Video packet loss synchronous compensation method and system based on deep learning

Abstract

The invention discloses a video packet loss synchronous compensation method and system based on deep learning, and the method comprises the steps: obtaining a to-be-compensated video stream, extracting the features of an original video frame, loading the features to a packet loss compensation model containing an LSTM prediction network and a GAN generator network, and enabling a time sequence prediction unit to comprise a long short-term memory assembly with attention gating; the LSTM prediction network sequentially performs inter-frame motion mapping, region enhancement mapping and region feature fusion on the original video frame features through a time sequence prediction unit to obtain a prediction feature sequence; and the GAN generator generates and decodes the prediction feature sequence to obtain a target replacement frame feature, and further obtains a synchronous compensation frame from the video stream to be compensated. According to the method, time sequence state adjustment is guided through attention gating, cooperation of time sequence prediction and detail generation is realized in combination with LSTM and GAN, time sequence coherence and detail accuracy of compensation frames are improved, and the problem of synchronous compensation in a video packet loss scene is effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for synchronously compensating for video packet loss based on deep learning. Background Art

[0002] During video transmission, network bandwidth fluctuations, transmission delays, and other factors can lead to packet loss in video streams. This can cause playback freezes and image distortion at the receiving end, severely impacting the user's viewing experience. Existing methods for compensating for packet loss often rely on frame replication or simple interpolation, which struggles to restore the temporal coherence and detailed features of lost frames. Some deep learning-based methods, while incorporating timing prediction networks, suffer from inaccurate timing state adjustments and insufficient regional feature fusion, resulting in poor synchronization between the compensated frames and the original video stream and a blurry image. Summary of the Invention

[0003] The purpose of the present invention is to provide a method and system for synchronous compensation of video packet loss based on deep learning.

[0004] In a first aspect, an embodiment of the present invention provides a method for synchronously compensating for video packet loss based on deep learning, comprising: Obtaining a video stream to be compensated, performing frame feature extraction on the video stream to be compensated to obtain original video frame features, and loading the original video frame features into a packet loss compensation model, wherein the packet loss compensation model includes an LSTM prediction network and a GAN generator network arranged in sequence, the LSTM prediction network and the GAN generator network each including a plurality of sequentially connected timing prediction units, the timing prediction unit including a long short-term memory component, the long short-term memory component being provided with an attention gate, the attention gate being used to guide adjustment of an internal timing state of the long short-term memory component or maintain the internal timing state constant; Based on the first time series prediction unit in the LSTM prediction network, inter-frame motion mapping is performed on the original video frame feature to obtain a first time series feature, regional enhancement mapping is performed on the first time series feature to obtain a second time series feature, regional feature fusion is performed on the second time series feature to obtain the output of the first time series prediction unit, and the output of the first time series prediction unit is loaded into the next time series prediction unit until a prediction feature sequence of the LSTM prediction network is obtained; Generate and decode the predicted feature sequence of the LSTM prediction network based on the GAN generator network to obtain target replacement frame features; Based on the target replacement frame feature, a frame is synchronously generated from the video stream to be compensated to obtain a synchronous compensation frame.

[0005] In a second aspect, an embodiment of the present invention provides a server system, including a server, wherein the server is used in the method described in the first aspect.

[0006] Compared with the existing technology, the beneficial effects provided by the present invention include: using a deep learning-based video packet loss synchronization compensation method and system disclosed in the present invention, by obtaining the video stream to be compensated and extracting the original video frame features, loading them into a packet loss compensation model containing an LSTM prediction network and a GAN generator network, wherein the timing prediction unit includes a long short-term memory component with attention gating; the LSTM prediction network sequentially performs inter-frame motion mapping, regional enhancement mapping, and regional feature fusion on the original video frame features through the timing prediction unit to obtain a predicted feature sequence; the GAN generator generates and decodes the predicted feature sequence to obtain the target replacement frame features, and then obtains a synchronized compensation frame from the video stream to be compensated. This method guides the timing state adjustment through attention gating, combines LSTM and GAN to achieve synergy between timing prediction and detail generation, improves the timing coherence and detail accuracy of the compensated frame, and effectively solves the synchronization compensation problem in video packet loss scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly describes the drawings required for use in the embodiments. It should be understood that the following drawings illustrate only certain embodiments of the present invention and should not be construed as limiting the scope of the present invention. Those skilled in the art can, without inventive effort, derive other relevant drawings from these drawings.

[0008] Figure 1 A schematic flow chart of the steps of a method for synchronously compensating for video packet loss based on deep learning provided by an embodiment of the present invention; Figure 2 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0009] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the embodiments of the present invention. It should be understood that the described embodiments are only a portion of the embodiments of the present invention, not all of them. Generally, the components of the embodiments of the present invention described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations.

[0010] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0011] In order to solve the technical problems in the above background technology, Figure 1This is a flow chart of a method for synchronously compensating for packet loss in video based on deep learning provided in an embodiment of the present disclosure. The method for synchronously compensating for packet loss in video based on deep learning is introduced in detail below.

[0012] Step S201: Obtain a video stream to be compensated, extract frame features from the video stream to be compensated, obtain original video frame features, and load the original video frame features into a packet loss compensation model, wherein the packet loss compensation model includes a sequentially arranged LSTM prediction network and a GAN generator network, wherein the LSTM prediction network and the GAN generator network each include a plurality of sequentially connected timing prediction units, wherein the timing prediction unit includes a long short-term memory component, wherein the long short-term memory component is provided with an attention gate, wherein the attention gate is used to guide the adjustment of the internal timing state of the long short-term memory component or maintain the internal timing state constant; Step S202: Based on the first time series prediction unit in the LSTM prediction network, inter-frame motion mapping is performed on the original video frame features to obtain a first time series feature, regional enhancement mapping is performed on the first time series feature to obtain a second time series feature, regional feature fusion is performed on the second time series feature to obtain the output of the first time series prediction unit, and the output of the first time series prediction unit is loaded into the next time series prediction unit until a prediction feature sequence of the LSTM prediction network is obtained; Step S203, generating and decoding the prediction feature sequence of the LSTM prediction network based on the GAN generator network to obtain target replacement frame features; Step S204 : synchronously generating frames from the to-be-compensated video stream based on the target replacement frame feature to obtain synchronous compensation frames.

[0013] In this embodiment of the present invention, for example, a real-time video conferencing system is used as the application scenario. The server, as the executor of video stream processing and compensation, receives a real-time video stream from a client (using TCP to transmit key frames and UDP to transmit non-key frames). When UDP packet loss is detected, resulting in the loss of non-key frames, the deep learning-based video packet loss synchronization compensation method of the present invention generates replacement frames to ensure the smoothness and synchronization accuracy of the receiving end video. The following describes each step of the method in detail, combining specific scenarios.

[0014] The server first receives the video stream sent by the client via the Real-time Transport Protocol (RTP). In this video stream, key frames (such as I frames, which contain complete image information) are transmitted via TCP to ensure reliability, while non-key frames (such as P frames and B frames, which are differential information based on the previous frame) are transmitted via UDP to reduce latency. The server has a built-in packet loss detection module that monitors in real time whether non-key frames transmitted via UDP are lost by parsing the sequence number field of the RTP packet. For example, in a video conferencing scenario, client A sends a video stream to client B. The server detects that the P frame (non-key frame) with RTP sequence number 1001 does not arrive within the preset time window (such as 50ms), and the sequence numbers of the adjacent frames are 1000 (P frame) and 1002 (B frame). It determines that the P frame is lost, triggering the packet loss compensation process.

[0015] The server identifies the valid video frames before and after the packet loss (i.e., P-frame 1000 and B-frame 1002) as the "video stream to be compensated" and extracts frame features from them. Specifically, the server uses a pre-trained convolutional neural network (such as the first 10 layers of ResNet-50) to extract features from the two frames. Each frame is first resized to 256×256 resolution. A convolutional layer (3×3 kernel, stride 1, padding 1) extracts low-level visual features (such as edges, textures, and color channel distribution). This is then reduced in dimension by a pooling layer (2×2 max pooling). The final output is a 64×64×128 feature map (containing spatial details and temporal motion trends), which is denoted as the "raw video frame features." For example, for a person speaking in a video conference, the raw video frame features would focus on capturing texture features of the person's face (such as the outlines of the eyes and mouth), motion vectors of the upper body (such as the direction of hand gestures), and static features of the background (such as the edges of conference room tables and chairs).

[0016] The server then loads the original video frame features into a pre-trained "packet loss compensation model." This model consists of a sequentially connected LSTM prediction network and a GAN generator network: the LSTM prediction network is used to predict the temporal feature sequence of the lost frame based on the features of the previous and next frames, and the GAN generator network is used to decode the predicted feature sequence into a replacement frame that can be directly displayed. The LSTM prediction network consists of three serially connected "temporal prediction units," and the GAN generator network consists of four serially connected temporal prediction units. Each temporal prediction unit contains a "long short-term memory component" and an "attention gate." The role of the attention gate is to dynamically adjust the internal temporal state of the long short-term memory component based on the video content. For example, in the facial area (high-information area), the temporal state is maintained stable to preserve details, and in the background area (low-information area), the temporal state is adjusted to reduce redundant calculations.

[0017] The core function of the LSTM prediction network is to output a "predicted feature sequence" for the lost frame based on the features of the original video frame (features of frames before and after packet loss) through a cascade of multiple time series prediction units. The following details the processing of the first time series prediction unit in the LSTM prediction network as an example. The processing logic of subsequent time series prediction units is consistent, with only the parameters and feature dimensions dynamically adjusting with network depth.

[0018] The server first uses the long short-term memory component of the first time series prediction unit to perform "inter-frame motion mapping" on the original video frame features to capture the motion correlation between the previous and next frames and output the "first time series features". The specific steps are as follows: Normalization: The server performs Z-score normalization on the original video frame features (64×64×128) using the formula The eigenvalues ​​are adjusted to a distribution with mean 0 and variance 1, where is the global mean of the feature map, is the global standard deviation. The purpose of normalization is to eliminate the interference of dimensional differences in different feature channels on subsequent motion mapping. For example, the texture feature value range (0-255) of the character's facial area and the brightness feature value range (0-100) of the background area need to be scaled uniformly. Motion mapping of the long short-term memory component: The normalized features are input into the long short-term memory component (LSTM unit), which contains an input gate, a forget gate, an output gate, and a cell state, and is used to learn the temporal dependencies between frames. In a video conferencing scenario, inter-frame motion is mainly reflected in the character's head rotation, hand gestures, etc. The LSTM unit filters irrelevant historical information (such as the brief appearance of a water cup in the previous frame) through the forget gate, retains key motion features (such as the displacement vector of the character's face) through the input gate, and updates the cell state. At this time, attention gating intervenes to adjust the internal temporal state of the long short-term memory component. The server dynamically calculates the “region attention confidence” according to the regional importance of the video content: for the current processing stage (the first processing cycle of the time series prediction unit), the server obtains the “initial region attention confidence” (e.g., the confidence of the facial region is 0.8, the background region is 0.2) and the “initial region attention confidence fluctuation” (e.g., the facial region fluctuation is 0.05, the background region is 0.15) of the previous time series stage (the feature processing stage of the 1000 frames before the packet loss), and calculates the initial region attention confidence by formula C. target =C init+α·△C calculates the "target region attention confidence" for the current stage (where α is an adjustment coefficient, dynamically adjusted based on the intensity of video motion; α=0.3 for rapid head turns and α=0.1 for static scenes). For example, when a person's head turns from left to right, the target attention confidence for the facial region is updated to 0.8+0.3×0.05=0.815, and for the background region to 0.2+0.3×0.15=0.245. This indicates that attention gating guides the LSTM component to prioritize maintaining the temporal state stability of the facial region while paying appropriate attention to changes in the background region. Temporal Dimension Integration: The LSTM component outputs a feature map (64×64×128) for a single temporal step. The server performs a temporal dimension integration operation (average pooling along the time axis) to fuse the motion features of the two frames before and after the packet loss into a "first temporal feature" (64×64×256 dimensions). This feature not only contains the spatial details of a single frame, but also integrates the motion trend between two frames (for example, the displacement of a person's face in the x-axis direction is 5 pixels / frame, and the displacement in the y-axis direction is 0 pixels / frame).

[0019] To further highlight the features of key regions in the video (such as faces and hands), the server performs "region enhancement mapping" on the first temporal features, outputting "second temporal features." The specific process is as follows: Spatial filtering and local relationship partitioning: The server first applies a 3×3 convolution filter (the convolution kernel weights are determined through pre-training, and the bias term is 0) to the first temporal features (64×64×256) in the spatial dimension based on the temporal dimension. This generates a "second feature map" (64×64×256) to smooth noise and enhance regional correlations. The second feature map is then normalized (LayerNorm, calculating the mean and variance along the channel dimension) and partitioned into "first local relationship features" and "second local relationship features" based on the spatial dimension. For example, a sliding window (16×16 pixels) is used to partition the 64×64 feature map into 16 local regions. The four regions containing the face are designated "first local relationship features" (high-priority regions), while the remaining 12 background regions are designated "second local relationship features" (low-priority regions). Local Region Feature Enhancement: The server normalizes the first local relationship feature (high-priority region) again and applies a 3×3 convolution filter to the local region based on the temporal dimension (the convolution kernel is optimized for facial features, such as enhancing edge response). This yields a "first feature map" (16×16×256, corresponding to the refined features of the facial region). The first feature map is then element-wise multiplied with the second local relationship feature (background region) to highlight the relationship between the face and background. This is then added to the second local relationship feature (preserving basic background information). Finally, the result is integrated through a 3×3 convolution filter to yield a "second temporal feature" (64×64×256). For example, in a hand-raising gesture scenario, the second temporal feature enhances the motion vector features of the hand region (such as wrist rotation angle) while mitigating interference from the background wall texture.

[0020] The server performs "regional feature fusion" on the second temporal features, dynamically adjusting the weights of different regions through a feature selection mechanism, and outputs the processing results of the current temporal prediction unit. The specific steps are as follows: Parallel spatial filtering and feature selection: The server first normalizes the second temporal features (BatchNorm, normalization along the batch dimension). Then, spatial filtering is performed through two parallel convolutional layers. The first convolutional layer uses a 3×3 convolution kernel (stride 1, padding 1) to output a "third feature map" (64×64×128, containing global semantic features); the second convolutional layer uses a 1×1 convolution kernel to output a "fourth feature map" (64×64×128, used to generate feature selection weights). The server applies a sigmoid function to the fourth feature map (output range 0-1) to obtain "feature selection weights"—for example, a weight of 0.85 for the face, 0.7 for the hand, and 0.2 for the background. This weighting prioritizes features from high-information regions. Weighted fusion output: The server performs element-wise multiplication of the third feature map with the feature selection weights (giving stronger representation to features in high-weighted regions), resulting in the output features of the current time series prediction unit (64×64×128). This output is then fed into the next time series prediction unit in the LSTM prediction network (three in series), repeating the aforementioned "inter-frame motion mapping → regional enhancement mapping → regional feature fusion" process. The final output is a "prediction feature sequence" (64×64×512 in dimension), which contains the temporal features and spatial details of the lost frame (P-frame number 1001).

[0021] The predicted feature sequence output by the LSTM prediction network is input into the GAN generator network, which converts the abstract feature sequence into directly displayable image features (i.e., "target replacement frame features") through a "generative decoding" process. The GAN generator network consists of four serially connected time series prediction units (the structure is consistent with that of the units in the LSTM prediction network) and transposed convolutional layers. Its core is to optimize the visual realism of the generated frames through adversarial training. The generator network generates replacement frame features, and the discriminator network (jointly trained during pre-training) distinguishes the generated features from the real frame features. The parameters of both are dynamically adjusted using a minimax loss function.

[0022] In a video conferencing scenario, the generator network first upsamples the predicted feature sequence (64×64×512): Four layers of transposed convolution (each with a 4×4 kernel, stride 2, and padding 1) gradually restore the feature map resolution from 64×64 to 128×128 to 256×256, while the number of channels increases from 512 to 256 to 128 to 3 (corresponding to the three RGB channels). For example, the transposed convolution layer learns the skin color distribution of the face in the real frame (such as the range of the Cr component in the YCrCb color space) to ensure natural facial colors in the generated replacement frame. It also captures the continuity of gestures (such as the intermediate gestures from "raising" to "lowering") to ensure that the generated frame has consistent motion trends with the previous and next frames.

[0023] During the decoding process, the generator network also introduces feedback from the "frame quality fluctuation detection unit": this unit calculates in real time the spatial domain detail deviation (such as edge clarity difference), temporal domain detail deviation (such as motion vector continuity error) and spectral envelope feature deviation (such as blurring caused by high-frequency component loss) between the generated features and the real frame features, and optimizes the generator parameters through backpropagation of the loss function (such as perceptual loss, GAN loss), and finally outputs the "target replacement frame feature" (256×256×3), that is, the image feature of the lost frame.

[0024] Based on the characteristics of the target replacement frame, the server extracts synchronization information (such as RTP timestamps and motion vector synchronization parameters) from the video stream to be compensated. It then aligns the timestamps and motion vectors of the generated replacement frame with those of the preceding and following frames, ultimately obtaining a "synchronized compensation frame." Specifically, the server reads the timestamps of the frame before the packet loss (frame 1000) (e.g., 1620000000ms) and the timestamp of the frame after the packet loss (frame 1002) (1620000033ms) in the video stream to be compensated. Based on the video frame rate (30fps, 33ms per frame interval), the server calculates the target timestamp of the lost frame (frame 1001) to be 1620000016.5ms. The server sets the timestamp of the generated replacement frame to this value, ensuring that the time interval error with the preceding and following frames is ≤15ms (meeting the synchronization accuracy requirement).

[0025] At the same time, the server uses the motion vector synchronization module to match the motion vector of the replacement frame with the preceding and following frames. For example, if the motion vector of the face in the preceding frame 1000 is (+5px, 0px) (moved 5 pixels to the right), and the motion vector in the following frame 1002 is (+10px, 0px), the motion vector of the replacement frame 1001 is calibrated to (+7.5px, 0px) to ensure a continuous motion trajectory. Finally, the server sends the video stream containing the synchronization compensation frame to the receiving client B via the RTP protocol. Client B's player decodes and plays the video in timestamp order, achieving smooth video playback.

[0026] In this embodiment, the server successfully generates replacement frames with high visual consistency and timing synchronization with the original frames in video conferencing scenarios with UDP packet loss through the entire process of "obtaining the video stream to be compensated → extracting original frame features → LSTM prediction network timing feature prediction → GAN generator network decoding → synchronous compensation frame output", effectively solving the problems of screen distortion and freeze caused by packet loss.

[0027] In an embodiment of the present invention, performing inter-frame motion mapping on the original video frame features to obtain the first temporal features may be implemented as follows: performing normalization processing on the original video frame features, and performing inter-frame motion mapping on the normalized original video frame features using the long short-term memory component; The output of the long short-term memory component is integrated in a time series dimension to obtain the first time series feature.

[0028] In an exemplary embodiment of the present invention, in a video conferencing scenario, after detecting the loss of P frame 1001 transmitted via UDP, the server retrieves P frame 1000 and B frame 1002 before and after the packet loss as the video stream to be compensated. Frame feature extraction is performed on these two frames to obtain 64×64×128 raw video frame features, which include facial texture, upper body motion vectors, and background static features. The server first performs Z-score normalization on the raw video frame features, adjusting the feature values ​​to a distribution with a mean of 0 and a variance of 1. This eliminates the dimensionality difference between the texture features of the facial region (value range 0-255) and the brightness features of the background region (value range 0-100). The normalized features are then input into the long-short-term memory component. This component uses a forget gate to filter out irrelevant historical information, such as the brief appearance of a water cup in the previous frame. The input gate retains key motion features, such as the facial displacement vector, and updates the cell state. During this process, attention gating dynamically adjusts internal temporal states based on regional importance. For example, when a person's head rotates, it prioritizes maintaining temporal state stability in the facial region while also paying appropriate attention to changes in the background. The server then performs average pooling along the time axis on the single temporal step feature map (64×64×128) output by the long short-term memory component, fusing the motion features of the two frames before and after the packet loss to obtain a first temporal feature with dimensions of 64×64×256. This feature integrates the spatial details of a single frame with the motion trends between the two frames, such as the displacement of the face in the x-axis direction of 5 pixels per frame.

[0029] In an embodiment of the present invention, the packet loss compensation model is set to perform cyclic processing through multiple timing stages, and the inter-frame motion mapping of the normalized original video frame features through the long short-term memory component can be implemented through the following examples.

[0030] For any current stage of the plurality of sequential stages, obtaining the initial region attention confidence and the initial region attention confidence fluctuation of the long short-term memory component in the previous sequential stage of the current stage, and determining the target region attention confidence of the long short-term memory component in the current stage according to the initial region attention confidence and the initial region attention confidence fluctuation; Quantifying the attention confidence of the target area to obtain the attention weight corresponding to the attention gate in the current stage; The internal temporal state output by the long short-term memory component in the current stage is determined according to the attention weight, and inter-frame motion mapping is performed on the standardized original video frame features through the long short-term memory component based on the internal temporal state.

[0031] In an embodiment of the present invention, for example, in a video conference scenario, user A is speaking and gesturing with their hands. The server detects the loss of P frame 2003 (corresponding to the middle gesture of A's raised hand gesture) transmitted via UDP, triggering the packet loss compensation process. The packet loss compensation model uses three time stages (T1, T2, and T3) to cyclically process the video frame features before and after packet loss (P frame 2002: A lowers his head to look at a document; B frame 2004: A raises his hand to his chest) to gradually refine the motion features. The following focuses on the current time stage T2 (which follows the preliminary motion analysis of stage T1 and transitions to the refined motion prediction of stage T3), detailing the inter-frame motion mapping process of the long-short-term memory component. The server first obtains the "initial region attention confidence" and "initial region attention confidence fluctuation" at the end of the previous time stage T1. During the T1 phase, feature analysis of P frame 2002 identified high-information regions: the face region (A's facial features, including expression and head-up movement) had an initial confidence of 0.85 (highest attention), the hand region (A's right hand, just lifted from the table) had an initial confidence of 0.6 (second highest attention), and the background region (the conference room tables, chairs, and whiteboard) had an initial confidence of 0.2 (lowest attention). The fluctuations in initial regional attention confidence reflect the magnitude of attention changes in each region: the face region, which is stable due to the head-up movement, has a fluctuation of 0.03 (small fluctuation); the hand region, which has just started the lift movement and has more intense movement, has a fluctuation of 0.12 (moderate fluctuation); and the background region, which has no significant change, has a fluctuation of 0.05 (slight fluctuation). The server dynamically adjusts the coefficient α based on the intensity of the video motion: for the face region, which has stable movement, α is 0.1 (to minimize the impact of fluctuations); for the hand region, which has intense movement, α is 0.4 (to enhance fluctuation response); and for the background region, which has minimal changes, α is 0.2 (to balance stability with update requirements). The target confidence for each region during the T2 phase is calculated using the formula: target region attention confidence C_target = initial confidence C_init + α × fluctuation ΔC: face region: 0.85 + 0.1 × 0.03 = 0.853 (slightly increased attention, maintaining stability); hand region: 0.6 + 0.4 × 0.12 = 0.648 (significantly increased attention, adapting to accelerated motion); background region: 0.2 + 0.2 × 0.05 = 0.21 (slightly increased attention, capturing slight background changes). The server quantifies the target region attention confidence and normalizes each region's target confidence to a "attention weight" ranging from 0 to 1 (with the sum of the weights being 1) using the Softmax function to clarify the LSTM component's attention priority for different regions.The specific calculation is as follows: The sum of target confidences for each region = 0.853 (face) + 0.648 (hand) + 0.21 (background) = 1.711; the attention weight for the face region is 0.853 / 1.711≈0.5 (50% of the total focus, primary attention); the attention weight for the hand region is 0.648 / 1.711≈0.38 (38% of the total focus, primary attention); and the attention weight for the background region is 0.21 / 1.711≈0.12 (12% of the total focus, secondary attention). The internal temporal state of the long-term short-term memory component consists of the cell state (long-term memory, storing historical motion features) and the hidden state (short-term output, reflecting current motion trends). The server dynamically adjusts these two states based on attention weights: For the face region (weight 0.5), due to its highest attention and stable movement, the cell state retains 80% of the historical information from stage T1 (e.g., A's facial contours and eye opening), updating only 20% of new features (e.g., the change in chin angle when looking up) to ensure continuity between expression and head posture. The hidden state focuses on outputting the vertical displacement vector of the face (displacement rate from lowering the head to raising it: 1.2 pixels / time-series step). For the hand region (weight 0.38), due to its intense movement and lower attention, the cell state updates 60% of the historical information from stage T1 (e.g., the initial position coordinates of the right hand), retaining 40% of the core information (palm orientation) to accommodate changes in hand lift momentum. The hidden state outputs the composite horizontal and vertical motion vector of the hand (a horizontal displacement of 2.5 pixels to the right / time-series step, and a vertical displacement of 3 pixels upward / time-series step, matching the trajectory of the hand raised to the chest). Background region (weight 0.12): Due to low attention, the cell state retains only 30% of the historical information from stage T1 (such as the edge texture of the whiteboard), rapidly updating 70% of redundant information (filtering out subtle fluctuations in desktop glare). The hidden state deemphasizes spatial details and outputs smooth static features. Based on this internal temporal state, the long short-term memory component performs inter-frame motion mapping on the normalized raw video frame features (including the face, hand, and background features of P-frame 2002). The cell state anchors the historical motion baseline, while the hidden state captures the current motion trend. The resulting output is a motion feature map (64×64×128) that fuses the region priorities, completing the inter-frame motion mapping in stage T2. This feature map serves as input for the region enhancement mapping in stage T3, further refining motion details in the lost frame.

[0032] In an embodiment of the present invention, after the attention confidence of the target area is quantified to obtain the attention weight corresponding to the attention gate in the current stage, the method further provides the following implementation methods.

[0033] Determining the initial region attention confidence corresponding to the current stage according to the attention weight corresponding to the current stage; According to the attention weight corresponding to the current stage and the target area attention confidence, the fluctuation amount of the initial area attention confidence corresponding to the current stage is determined.

[0034] In an exemplary embodiment of the present invention, in a video conferencing scenario, after the server completes the calculation of attention weights in stage T2 (0.5 for face, 0.38 for hands, and 0.12 for background), it then determines the initial regional attention confidence and fluctuation for that stage, which serves as the "previous temporal stage" of the next temporal stage, T3. This provides a basis for motion mapping in stage T3. The server converts the attention weights in stage T2 into initial confidences using a linear mapping: using the historically highest initial confidence of 0.85 corresponding to the maximum weight (0.5 for face) as a benchmark, a scaling factor of 1.7 is calculated (0.85 / 0.5 = 1.7). The initial confidence for each region is calculated as follows: 0.5 × 1.7 = 0.85 for the face region (maintaining core attention), 0.38 × 1.7 = 0.646 for the hand region (matching the hand lift momentum requirement), and 0.12 × 1.7 = 0.204 for the background region (maintaining basic static attention). These values ​​serve as the "initial regional attention confidence" for stage T3. The server calculates the fluctuation amount based on the difference between the target region attention confidence in stage T2 (face 0.853, hand 0.648, background 0.21) and the initial confidence in the current stage (face 0.85, hand 0.646, background 0.204), combined with the attention weight: Fluctuation = (target confidence - initial confidence) × attention weight. For the face region, (0.853-0.85) × 0.5 = 0.0015 (stable movement, minimal fluctuation); for the hand region (0.648-0.646) × 0.38 = 0.00076 (hand lift momentum stabilizes, fluctuation narrows); and for the background region (0.21-0.204) × 0.12 = 0.00072 (mostly static, minimal fluctuation). These values ​​will be used as the "initial region attention confidence fluctuation amount" in stage T3.

[0035] In an embodiment of the present invention, the packet loss compensation model further includes a frame quality fluctuation detection unit. Before the original video frame features are loaded into the packet loss compensation model, the following implementation is also provided.

[0036] Obtaining a packet loss video stream instance and a reference video stream instance corresponding to the packet loss video stream instance, and calling the packet loss compensation model to generate packet loss compensation for the packet loss video stream instance; The frame quality fluctuation detection unit is used to determine the frame detail fluctuation of the compensation frame generation result of the packet loss video stream instance, and a training error value is determined based on the deviation between the compensation frame generation result of the packet loss video stream instance and the reference video stream instance and the frame detail fluctuation. The packet loss compensation model is trained based on the training error value.

[0037] In an embodiment of the present invention, during the model training phase for a video conferencing scenario, the server optimizes the parameters of the packet loss compensation model using historical data to ensure that the compensated frames generated in actual applications are highly consistent with the real frames. The following describes the error calculation process before model training, using the example of a server processing high-dynamic-motion video data involving a person rapidly turning and making hand gestures. The server obtains a packet loss video stream example from the training dataset: this example is a video conference recording (1000 frames, 30 fps) simulating a 30% packet loss rate. The example includes a complete scene: a person speaking from the front (frames 1-200), rapidly turning to write on a whiteboard (frames 201-400, with intense movement), and then turning back to their original position and continuing to gesture (frames 401-1000). 30% of the non-key frames (P / B frames) transmitted via UDP are randomly lost (e.g., frames 250, 280, and 320), while key frames (I frames) are transmitted intact via TCP. The corresponding reference video stream example is a packet loss-free version of the same video (the original, real frames, without any information loss), which serves as the "gold standard" for the quality of the compensated frames. The server feeds the packet loss video stream instance into an untrained packet loss compensation model. Using an LSTM prediction network and a GAN generator network, the model generates a compensation frame for the lost non-key frames (e.g., frame 250: a character turning in a semi-sideways position, holding a pen in their right hand, preparing to write). This compensation frame is recorded as the "compensated frame generation result." The server then activates the frame quality fluctuation detection unit, which calculates the "frame detail fluctuation" of the compensated frame generation result along three dimensions: Spatial domain frame detail fluctuation: This unit uses the Sobel operator to extract the gradient map (edge ​​and texture details) between the compensated frame and the reference frame, and calculates the variance of the gradient values ​​(reflecting the stability of spatial details). For example, in frame 250, the gradient variance of the character's shoulder edge due to motion blur is 0.18 (large fluctuation), while the gradient variance of the background whiteboard edge is 0.05 (small fluctuation). The overall spatial domain fluctuation is taken as a regional weighted average of 0.12. Temporal domain frame detail fluctuation: This unit calculates the rate of change of the motion vector (using optical flow) between the compensated frame and its preceding and succeeding frames (frames 249 and 251). When the character turns, the waist motion vector suddenly changes from (-3 pixels / frame, leftward shift) in frame 249 to (+5 pixels / frame, rightward shift) in frame 250, with an absolute value of 8. The temporal domain fluctuation is 0.15 (fluctuations caused by discontinuous motion). To determine the fluctuation of the spectral envelope feature, Fourier transforms are performed on the compensated and reference frames to extract the power spectrum envelope of the high-frequency components (corresponding to detailed textures). The root mean square error (RMSE) of the envelope curve is calculated. The RMSE of the character's pen tip (high-frequency detail) in the compensated frame is 0.21, and the spectral envelope fluctuation is 0.17 (fluctuations caused by loss of high-frequency detail).The server calculates the "deviation" between the compensated frame generation result and the reference video stream instance: spatial domain detail deviation (the SSIM value of the compensated frame and the reference frame is 0.82, corresponding to a deviation of 0.18), temporal domain detail deviation (motion vector mean square error 2.3 pixels / frame), and spectral envelope feature deviation (high-frequency component energy is 15% lower than that of the reference frame). Combined with the fluctuation of frame details, the server calculates the training error according to a preset formula: The error of the first training link: using the spatial domain fluctuation of 0.12 as the normalization coefficient, the spatial domain deviation of 0.18 is weighted (0.18×0.12=0.0216); the temporal domain fluctuation of 0.15 is weighted on the temporal domain deviation of 2.3 (2.3×0.15=0.345), and the sum of the two is the first error of 0.3666; the spectral envelope fluctuation of 0.17 is weighted on the spectral deviation of 0.15 (0.15×0.17=0.0255) as the second error; superimposing the visual distortion error (PSNR value 28dB, corresponding to an error of 0.22) and the state transition frequency error (attention gate switching frequency 1.2 times / frame, exceeding the threshold 0.3 times / frame, error 0.09), the total first link error is 0.3666+0.0255+0.22+0.09=0.6921. The error of the second training link: directly use the spatial domain fluctuation of 0.12 to weight the deviation of 0.18 (0.18×0.12=0.0216), the temporal domain fluctuation of 0.15 to weight the deviation of 0.23 (0.23×0.15=0.0345), the third error is 0.0561; the spectral envelope fluctuation of 0.17 gives a weight of 0.0255 to the deviation of 0.15 as the fourth error; superimpose the same visual distortion error of 0.22 and the state transition frequency error of 0.09, the total second link error is 0.0561+0.0255+0.22+0.09=0.3916. The server takes a weighted average of the errors in the two steps (weight 0.6 for the first step and 0.4 for the second step), obtaining a total training error of 0.6921×0.6+0.3916×0.4≈0.5719. The server then optimizes the model parameters (such as the attention gate weights of the LSTM and the convolution kernel parameters of the GAN generator) through the backpropagation algorithm, reducing the error to below 0.2 (corresponding to a PSNR of ≥35dB between the compensated frame and the reference frame), completing this round of model training.

[0038] In an embodiment of the present invention, the training error value is determined based on the compensation frame generation result of the packet loss video stream instance, the deviation between the reference video stream instance and the frame detail fluctuation, and the packet loss compensation model is trained based on the training error value. This can be implemented through the following example.

[0039] Determining a training error value of a first training session based on a compensation frame generation result of the packet loss video stream instance, a deviation between the reference video stream instance, and the frame detail fluctuation, wherein the training error value of the first training session is used to regulate the stability of the frame detail fluctuation; Determining a training error value for a second training phase based on a deviation between a compensation frame generation result of the packet loss video stream instance and the reference video stream instance and the frame detail fluctuation, wherein the training error value for the second training phase is used to set a corresponding regional attention for the regional timing component based on regional credibility fluctuations of the regional timing component when determining the deviation between the compensation frame generation result of the packet loss video stream instance and the reference video stream instance; The packet loss compensation model is trained in sequence based on the training error value of the first training link and the training error value of the second training link.

[0040] In an embodiment of the present invention, for example, in a video conferencing scenario, the server performs model training on a high-dynamic packet loss video stream instance of "a person turning around to write on a whiteboard" (simulating a 30% packet loss rate, with the 250th non-key frame lost, corresponding to the person holding a pen half-sideways). At this point, the server has obtained key data through the frame quality fluctuation detection unit: the compensation frame generation result (the 250th frame generated by the model) deviates from the reference video stream instance (a real frame with no packet loss) (the spatial domain detail deviation is manifested as blurred shoulder edges, the temporal domain detail deviation is manifested as discontinuous waist motion vectors, and the spectral envelope feature deviation reflects the loss of high-frequency texture at the pen tip). At the same time, the frame detail fluctuation is detected (the spatial domain fluctuation is higher due to blurred shoulder edges, the temporal domain fluctuation is larger due to jumps in waist motion vectors, and the spectral envelope fluctuation is significant due to the loss of pen tip texture). The following details how the server determines the training error and optimizes the model in sequence through the first and second training steps. The core goal of the first training step is to suppress the dramatic fluctuations in the details of the compensation frame in the spatial, temporal, and spectral dimensions to ensure the stability of the generated frame. The server dynamically adjusts the deviation using a "fluctuation normalization weighting" strategy. When the fluctuation of a particular dimension increases (the more unstable the details), normalization is used to reduce the impact of the deviation in that dimension on the overall error, preventing the model from over-focusing on highly fluctuating noise information. Specifically, the server first processes the deviations in the spatial and temporal domains. In the spatial domain, due to the blurred shoulder edge, resulting in high spatial fluctuation, the server normalizes the deviations using the fluctuation (the greater the fluctuation, the lower the normalized deviation weight), weakening the interference of the blurred edge on the overall error and ensuring that the spatial texture of the generated frame (such as wrinkles in the character's clothing and whiteboard grid lines) remains stable. In the temporal domain, due to the jump in the waist motion vector, resulting in high temporal fluctuation, the server similarly processes the temporal domain deviation using fluctuation normalization to suppress sudden changes in the motion vector, making the transition from frame 249 to frame 251 smoother (for example, when the waist rotation angle changes from 15° to the left to 20° to the right, the angular deviation of the intermediate frames is controlled within 3°). Subsequently, the server accumulates the normalized spatial and temporal domain deviations and the corresponding fluctuations to obtain a comprehensive error reflecting spatial and temporal stability. Next, the server processes the spectral envelope feature deviation: due to the loss of high-frequency texture of the pen tip, the spectral envelope fluctuation is significant. The server normalizes the deviation based on the spectral envelope fluctuation to reduce the interference of high-frequency noise on details and ensure that the sharpness of the pen tip in the generated frame (such as the edge clarity of the contact point between the pen tip and the whiteboard) is consistent with the reference frame. Finally, the server superimposes the visual distortion error (to compensate for the difference between the overall visual effect of the frame and the reference frame, such as the color cast of the character's skin color and uneven background brightness) and the state transition frequency error (the instability of feature extraction caused by too frequent switching of attention gates, such as frequent jumps in the attention of the facial area and the hand area) to obtain the total error of the first training link.This error is used to guide the model to prioritize the stability of frame details. For example, it adjusts the cell state update rate of the long-short-term memory component in the LSTM prediction network to reduce feature fluctuations caused by intense motion. A smoothing filter layer is added to the GAN generator network to suppress sudden changes in high-frequency components. The core goal of the second training phase is to dynamically adjust the priority of attention to deviations in each region based on regional credibility fluctuations (variable fluctuations in different regions). For regions with high volatility (such as hands, pen tips, and other areas with movement or rich details), the deviation weight in the error is increased, guiding the model to prioritize optimizing details in these areas. For regions with low volatility (such as static areas like background walls and desktops), the deviation weight is reduced to reduce redundant calculations. Specifically, the server first processes spatial and temporal domain deviations. For the hand region (which experiences high temporal domain fluctuation due to changes in hand lift momentum), the server directly uses the temporal domain fluctuation in this region as a weight to amplify the contribution of the hand motion vector deviation to the error, prompting the model to focus on learning the hand's trajectory from the tabletop to the chest (e.g., wrist rotation angle and finger bending). For the shoulder edge (which experiences high spatial domain fluctuation), the server uses the spatial domain fluctuation as a weight to amplify the spatial detail deviation, pushing the model to optimize the clarity of the shoulder contour (e.g., the transition between the shoulder and the folds of clothing). The server then accumulates the weighted spatial and temporal domain deviations to produce a composite error reflecting the priority of the region. Next, the server processes the spectral envelope feature deviation. For the pen tip region (which experiences high spectral envelope fluctuation and high-frequency texture loss), the server uses the spectral envelope fluctuation as a weight to amplify the spectral deviation, guiding the model to enhance the high-frequency details of the pen tip (e.g., metallic reflections and ink marks). Finally, the server also superimposes the visual distortion error and state transition frequency error to obtain the total error of the second training phase. This error is used to guide the model in optimizing regional attention allocation. For example, adjustments are made to the regional attention confidence calculation logic of the attention gate to increase the attention weight for high-fluctuation areas such as the hand and pen tip. Furthermore, targeted regional feature fusion is implemented in the temporal prediction unit of the LSTM prediction network (for example, a more refined local relationship feature segmentation is used for the pen tip region). The server trains the model sequentially using a "stabilize fluctuations first, then optimize attention" strategy. First, based on the error backpropagation from the first training phase, model parameters are adjusted to stabilize frame detail fluctuations. After 50 rounds of iterative training, the spatial domain fluctuation of the compensation frame is reduced from "high" to "smooth" (shoulder edge blur is reduced, and texture consistency with the reference frame is improved). The temporal domain fluctuation is reduced from "large" to "smooth" (the jump amplitude of the waist motion vector is reduced, and the turning movement is smoother). The spectral envelope fluctuation is reduced from "significant" to "slight" (the high-frequency texture of the pen tip is restored, and its sharpness approaches that of the reference frame). At this point, the error of the first training phase is optimized to below the preset threshold, and frame detail stability meets the standard.Subsequently, the server continued backpropagation based on the error from the second training phase, optimizing regional attention allocation. After 30 rounds of iterative training, the model's attention weight for high-fluctuation areas (hands, pen tips) was significantly improved. The hand motion vector deviation decreased from "significant" to "minor" (the hand-lifting trajectory was essentially consistent with the reference frame), and the pen tip's high-frequency texture deviation decreased from "significant" to "negligible" (the pen tip details were highly consistent with the reference frame). At this point, the error in the second training phase was optimized below the preset threshold, and regional attention accuracy met the target. Finally, after two rounds of training, the packet loss compensation model generated compensated frames in an environment with a 30% packet loss rate, achieving significantly improved visual consistency with the reference frames. The video's freeze-free rate reached 95%, and synchronization accuracy was controlled within 15ms, meeting the smoothness and real-time requirements of video conferencing.

[0041] In an embodiment of the present invention, the frame detail fluctuation includes spatial domain frame detail fluctuation, temporal domain frame detail fluctuation and spectral envelope feature fluctuation. The training error value of the first training link is determined based on the deviation between the compensation frame generation result of the packet loss video stream instance and the reference video stream instance and the frame detail fluctuation. It can be implemented through the following example.

[0042] Determining a spatial domain detail deviation, a temporal domain detail deviation, and a spectrum envelope feature deviation between a compensation frame generation result of the packet loss video stream instance and the reference video stream instance; The spatial domain detail deviation is weighted according to a Gaussian distribution function with the spatial domain frame detail fluctuation as a normalization coefficient, the temporal domain detail deviation is weighted according to a Gaussian distribution function with the temporal domain frame detail fluctuation as a normalization coefficient, and a first error is determined according to the sum of the spatial domain frame detail fluctuation, the temporal domain frame detail fluctuation, the weighted spatial domain detail deviation, and the weighted temporal domain detail deviation; weighting the spectrum envelope feature deviation according to a Gaussian distribution function using the spectrum envelope feature fluctuation as a normalization coefficient, and determining a second error according to the sum of the spectrum envelope feature fluctuation and the weighted spectrum envelope feature deviation; The first error and the second error are weighted and accumulated to obtain a training error value of the first training link.

[0043] In an exemplary embodiment of the present invention, in a video conferencing scenario, the server performs error calculations for the first training phase for a high-dynamic packet loss video stream example (simulating a 30% packet loss rate, with the 250th non-keyframe lost, corresponding to a person half-sideways, holding a pen, and preparing to write) depicting a person turning to write. At this point, the server has obtained key data through the frame quality fluctuation detection unit: the detail deviation between the compensation frame generation result (the 250th frame generated by the model) and the reference video stream example (a real frame with no packet loss), as well as frame detail fluctuations in various dimensions (spatial domain fluctuations are higher due to shoulder edge blur, temporal domain fluctuations are larger due to waist motion vector jumps, and spectral envelope fluctuations are significant due to the loss of high-frequency texture at the pen tip). The server determines the training error values ​​for the first training phase in the following steps to control the stability of frame detail fluctuations. The server first uses a multi-dimensional comparison to identify detail deviations between the compensation frame generation result and the reference video stream example: Spatial domain detail deviation: This is manifested as the difference between the blurred shoulder edge of the person in the compensation frame and the sharp edge of the reference frame. In the reference frame, the edges of the clothing wrinkles (cotton and linen texture) created by the character's turn are clearly discernible on their left shoulder. In the compensated frame, this area exhibits a 1-2 pixel halo, blurring the edges, weakening the layering of the wrinkles. Furthermore, the grid lines (1-pixel-wide solid black lines) on the whiteboard background are partially disjointed in the compensated frame (e.g., the grid line break at row 3, column 5), creating a visual discrepancy with the continuous lines in the reference frame. Temporal detail deviations manifest as discontinuities in the motion vectors between the compensated frame and the preceding and following frames (frames 249 and 251). In the reference frame, the character's waist rotation smoothly transitions from 15° to the left in frame 249 to 20° to the right in frame 251. The angle in frame 250 should be 5° to the left, but in the compensated frame, the waist angle jumps directly to 10° to the right, creating a noticeable disconnect with the preceding and following frames, resulting in a jerky effect on the turn. Spectral envelope feature deviations reflect a loss of energy in high-frequency detail. In the reference frame, the pen tip (made of metal) held by the figure has a subtle reflective texture (rich in high-frequency components). After Fourier transform, the high-frequency band (>10kHz) accounts for 22% of the energy. In the compensation frame, the reflective texture of the pen tip is blurred, with the high-frequency band accounting for only 12% of the energy, a 10% difference from the reference frame, resulting in a lack of sharpness in the pen tip. The server uses the Gaussian distribution function to weight the deviations in the spatial and temporal domains. The core function uses the "frame detail fluctuation" of the corresponding dimension as a normalization coefficient to dynamically adjust the impact of the deviation on the error. When the fluctuation of a particular dimension is higher (the detail is more unstable), the Gaussian distribution function suppresses the weight of the deviation in that dimension through normalization, preventing the model from over-focusing on the highly fluctuating "noisy details" and instead prioritizing overall feature stability. Due to the blurred shoulder edge and broken grid lines, the spatial domain frame details have high fluctuations (poor detail stability).The server uses spatial domain fluctuation as a normalization coefficient and weights spatial detail deviations using a Gaussian distribution function. The higher the fluctuation, the stronger the "suppression effect" of the Gaussian distribution function, thereby reducing the interference of blurred edges and grid line discontinuities on the overall error. For example, for shoulder wrinkle blur deviation, the "effective weight" of the deviation is reduced after normalization (to prevent the model from over-adjusting the overall texture due to local blur), ensuring that the spatial structure of the generated frame (such as the figure's torso outline and the overall layout of the whiteboard) is consistent with the reference frame. For grid line break deviation, the normalized deviation weight is also reduced, so that the model prioritizes grid line continuity over individual breakpoints, avoiding distortion of entire lines caused by repairing breakpoints. Due to the jump in the waist motion vector, the temporal domain frame details have large fluctuations (poor motion continuity). The server uses temporal domain fluctuation as a normalization coefficient and weights temporal domain detail deviations using a Gaussian distribution function. The greater the fluctuation, the stronger the "smoothing effect" of the Gaussian distribution function, suppressing sudden changes in motion vectors. For example, to address jumps in waist angle, the normalized "effective weight" of the deviation is reduced (to prevent the model from overcompensating to correct for jumps, which could lead to new motion distortion). This allows for a more natural transition from frame 249 to frame 251 (for example, adjusting the waist rotation angular velocity from a sudden 30° / frame to a gradual 15° / frame), ensuring smooth timing. The server directly adds the weighted spatial and temporal detail deviations with the corresponding frame detail fluctuations (spatial and temporal) to produce the "first error" that reflects spatial and temporal stability. This error comprehensively reflects the overall deviation in spatial texture continuity (such as wrinkles and grid lines) and temporal motion smoothness (such as turn angles) of the compensated frame. For example, the first error is the sum of spatial domain fluctuation (basic fluctuations caused by edge blur) and weighted spatial deviation (effective interference from blurred edges), plus temporal domain fluctuation (basic fluctuations caused by motion jumps) and weighted temporal deviation (effective interference from jumps). This error decreases as the stability of spatial and temporal details improves. The server uses the same logic as in the spatial and temporal domains to process spectral envelope feature deviations: due to the loss of high-frequency texture from the pen tip, the spectral envelope frame details fluctuate significantly (poor high-frequency stability). The server uses the spectral envelope fluctuation as a normalization coefficient and weights the spectral envelope feature deviations using a Gaussian distribution function. The higher the fluctuation, the stronger the "filtering effect" of the Gaussian function on the deviations, preventing high-frequency noise (such as random noise from pen tip reflections) from interfering with detail. For example, in response to the deviation of missing high-frequency energy at the pen tip, the "effective weight" of the deviation is reduced after normalization (to avoid the model introducing unnecessary noise in order to enhance high-frequency energy), ensuring that the high-frequency details of the generated frame (such as the reflection of the pen tip and the sharpness of the edge of the whiteboard handwriting) are consistent with the reference frame (such as the reflection texture of the pen tip is corrected from "blurred light spot" to "clear strip reflection", and the proportion of high-frequency energy is increased from 12% to 18%).The server then adds the weighted spectral envelope feature deviations and the spectral envelope frame detail fluctuations (basic fluctuations caused by high-frequency loss) to obtain a "second error" reflecting the stability of high-frequency details. Based on the priority of the video content (spatial and temporal stability takes precedence over high-frequency details, as smooth movement is more important than pen tip reflection in a meeting), the server assigns a higher weight (e.g., 60%) to the first error (spatial + temporal stability) and a lower weight (e.g., 40%) to the second error (spectral envelope stability). The weighted sum of the two results in the "training error value for the first training phase." For example, the first error may appear as a "high error" due to spatial edge blur and temporal motion jumps, while the second error may appear as a "medium error" due to high-frequency detail loss. After weighted summation, the total error intuitively reflects the overall stability deficiencies of the current compensated frame. The server backpropagates this error to the packet loss compensation model, adjusts the cell state update threshold of the long-short-term memory component in the LSTM prediction network (reducing the frequency of state updates in areas of high fluctuation), and adds a smoothing filter kernel (3×3 mean filter) to the convolutional layer of the GAN generator network to suppress excessive fluctuations in spatial edges and sudden motion in the temporal domain. After 20 rounds of iterative training, the spatial edge blurring in the 250th frame is reduced (shoulder wrinkles are clearly visible), and the temporal motion vector is continuous (the waist rotation angle transitions smoothly). The error of the first training phase has been reduced to below the preset threshold, and the stability of frame detail fluctuation has met the standard.

[0044] In the embodiment of the present invention, the weighted accumulation of the first error and the second error to obtain the training error value of the first training link can be implemented through the following example.

[0045] Determining a visual distortion error based on a compensation frame generation result of the packet loss video stream instance and the reference video stream instance, and determining a state transition frequency error based on a deviation between a state transition frequency of the attention gate and a preset frequency threshold; The first error and the second error are weighted and accumulated to obtain the first region credibility fluctuation error, and the first region credibility fluctuation error, the visual distortion error and the state transition frequency error are weighted and accumulated to obtain the training error value of the first training link.

[0046] In an exemplary embodiment of the present invention, in a video conferencing scenario, the server calculates the training error value for the first training phase for a packet loss video stream instance of a person turning to write on a whiteboard (the 250th non-keyframe is lost, and the compensation frame corresponds to the person's semi-sideways pen-holding posture). This requires integrating regional credibility fluctuations, visual distortion, and the stability of attention gating state transitions. The specific process is as follows: The server first determines the visual distortion error through subjective visual quality assessment and objective parameter testing. The generated compensation frame differs from the reference video stream instance (a real frame without packet loss) in overall visual quality. The facial skin tone of the person in the reference frame is natural yellow (RGB mean values ​​of approximately 230, 190, 160). Due to color mapping deviations in the GAN generator, the compensated frame has a reddish skin tone (RGB mean values ​​of 240, 180, 150), with a slight color cast in the cheek area. The background whiteboard has uniform brightness in the reference frame (grayscale value of 240±5). However, due to light estimation deviations, the upper left corner of the compensated frame has a low brightness (grayscale value of 220±8), creating a brightness difference with the lower right corner. Combining color shift and brightness unevenness, the visual distortion error is characterized as "moderate distortion" (corresponding to a quantized value of 0.25, with 0 indicating no distortion and 1 indicating severe distortion). The server then detects state transition frequency error: Attention gating dynamically allocates attention across different regions (face, hand, and background), with a preset frequency threshold of "1 time / frame" (to avoid frequent switching that can lead to unstable feature extraction). During the compensation frame generation process, the attention gating experienced two state transitions within a single frame (from "face focus" to "hand focus" and then back to "background focus") due to shoulder edge blur and waist motion abruptness. This exceeded the threshold once, resulting in a state transition frequency deviation of "1 time / frame" (corresponding to a quantized value of 0.15, with 0 indicating no distortion and 1 indicating severe distortion). The server then weighted the first error (spatial and temporal stability error) and the second error (spectral envelope stability error) based on scenario priority. Because the smoothness of human movement (spatial and temporal stability) is more important than the high-frequency texture of the pen tip (spectral envelope) in a meeting scenario, the first error is weighted 60% and the second error 40%. The first error, due to shoulder edge blur and temporal motion jumps, manifests as a "high fluctuation error" (quantized value 0.6). The second error, due to the loss of high-frequency texture at the pen tip, manifests as a "medium fluctuation error" (quantized value 0.4). Weighted summation yields the following: the first region's credibility fluctuation error = 0.6 × 60% + 0.4 × 40% = 0.36 + 0.16 = 0.52 (reflecting the overall credibility deficit due to regional detail fluctuations).Based on the video quality priority (regional fluctuation stability > visual distortion > state transition frequency), the server assigned a 70% weight to the first region's credibility fluctuation error (0.52), a 20% weight to the visual distortion error (0.25), and a 10% weight to the state transition frequency error (0.15). The weighted sum of these three factors yielded the following training error: 0.52 × 70% + 0.25 × 20% + 0.15 × 10% = 0.364 + 0.05 + 0.015 = 0.429 (the quantized value is < 0.5, initially meeting the fluctuation stability requirement, but further optimization is required). This error value guided the server's backpropagation model adjustments: To address visual distortion, the color mapping layer parameters of the GAN generator were optimized (correcting the skin RGB mean to 230, 190, and 160); to address state transition frequency, the region switching sensitivity of the attention gate was reduced (relaxing the switching threshold from 1 to 1.2 times per frame). Ultimately, the overall stability and visual consistency of the compensated frames were significantly improved.

[0047] In an embodiment of the present invention, the frame detail fluctuation includes spatial domain frame detail fluctuation, temporal domain frame detail fluctuation and spectral envelope feature fluctuation. The training error value of the second training link is determined based on the deviation between the compensation frame generation result of the packet loss video stream instance and the reference video stream instance and the frame detail fluctuation. It can be implemented through the following example.

[0048] weighting the spatial domain detail deviation according to the spatial domain frame detail fluctuation, weighting the temporal domain detail deviation according to the temporal domain frame detail fluctuation, and determining a third error according to the sum of the weighted spatial domain detail deviation and the weighted temporal domain detail deviation; weighting the spectrum envelope feature deviation according to the spectrum envelope feature fluctuation, and determining the weighted spectrum envelope feature deviation as a fourth error; The third error and the fourth error are weighted and accumulated to obtain a training error value of the second training link.

[0049] In an exemplary embodiment of the present invention, in a video conferencing scenario, the server calculates the training error value for the second training phase for a packet loss video stream example of a person turning to write on a whiteboard (the 250th non-keyframe is lost, and the compensation frame corresponds to the person's semi-sideways pen-holding posture). The core goal is to dynamically adjust the deviation weight based on regional credibility fluctuations (highly fluctuating regions are prioritized for optimization), focusing on improving the quality of details in high-information areas such as the hand and pen tip. The specific process is as follows: The server focuses on detail deviations in highly fluctuating regions and directly weights the deviations based on "frame detail fluctuation." The higher the fluctuation (the more unstable the regional details), the larger the deviation weight, ensuring that the model prioritizes optimization in these key areas. Spatial domain detail deviation weighting: In the compensation frame, the area where the person is holding the pen in their right hand generates spatial domain detail deviation (blurred hand outline edges and lost knuckle texture) due to motion blur. While the knuckle protrusions in this area (e.g., the second knuckle of the index finger) are clearly visible in the reference frame, a 2-3 pixel blur band appears around the knuckle edge in the compensation frame, resulting in a "large deviation" (quantized value 0.3). Due to the intense hand movement, the spatial domain frame detail fluctuation is "high" (quantized to 0.6, with 0 representing stability and 1 representing extreme instability). After weighting, the spatial domain deviation = large deviation × high volatility = 0.3 × 0.6 = 0.18 (the weighting is increased, requiring key optimization). Regarding the weighting of temporal domain detail deviation, the hand motion vector in the compensation frame shows a jump compared to the preceding and following frames (frames 249 and 251), resulting in significant temporal domain detail deviation. In the reference frame, the hand moves from its "initial desktop position" (coordinates (120, 180)) in frame 249 to its "writing position on the chest" (coordinates (160, 150)) in frame 251. The motion vector should increase smoothly (20 pixels per frame). However, in the compensation frame, the hand position suddenly changes to (170, 140), differing by 50 pixels from the motion vector in frame 249, indicating a "significant deviation" (quantized to 0.4). Due to the intense hand movement, the temporal domain frame detail fluctuates significantly (quantized to 0.5). The weighted temporal domain deviation is calculated as: significant deviation × large fluctuation = 0.4 × 0.5 = 0.2 (the weight is increased, prioritizing motion continuity correction). The server adds the weighted spatial domain deviation (0.18) and temporal domain deviation (0.2), resulting in a "third error" of 0.18 + 0.2 = 0.38 (reflecting the combined optimization requirements for spatial detail and temporal motion in the hand region; higher error values ​​indicate higher optimization priority). The server directly weights the spectral envelope feature deviations of the high-frequency detail area (pen tip) using "spectral envelope feature fluctuation" to enhance optimization of high-frequency textures. The spectral envelope characteristic deviation of the pen tip (metal material) in the compensation frame is caused by the loss of high-frequency texture: in the reference frame, the reflective texture of the pen tip (0.5-pixel wide highlight bar) is clear, and after Fourier transform, the high-frequency band (>10kHz) energy accounts for 22%; in the compensation frame, the pen tip reflection is blurred and becomes a "diffuse light spot", and the high-frequency band energy accounts for only 12%, and the deviation is manifested as "medium deviation" (quantization value 0.35).Because the pen tip is the core area of ​​writing, the spectral envelope feature has "significant" fluctuations (quantized value 0.7). After weighting, the spectral envelope feature deviation = medium deviation × significant fluctuation = 0.35 × 0.7 = 0.245, resulting in a "fourth error" of 0.245 (reflecting the need to optimize the high-frequency details of the pen tip). Based on the priority of "hand movement (spatial + temporal)" and "pen tip details (spectral)" in the meeting scenario (hand movement smoothness directly affects information transmission and therefore has a higher weight), the server assigns a 70% weight to the third error and a 30% weight to the fourth error. The weighted sum of these two values ​​is: the training error value for the second training phase = third error × 70% + fourth error × 30% = 0.38 × 0.7 + 0.245 × 0.3 = 0.266 + 0.0735 = 0.3395 (error values ​​> 0.3 indicate that the hand and pen tip areas require key optimization). This error guided the server's backpropagation model adjustments. To address the third error (hand spatial and temporal deviation), the attention weight of the hand region in the LSTM prediction network was increased (from 0.38 to 0.5), enhancing the accuracy of hand motion vector prediction. To address the fourth error (pen tip spectrum deviation), the learning rate in the high-frequency convolutional layer of the GAN generator network was increased (from 0.001 to 0.003), enhancing the high-frequency components of the pen tip's reflective texture. After 25 rounds of training, the hand knuckle texture in the compensation frame was restored (the blur band was reduced to 0.5 pixels), motion vector jumps were reduced to within 15 pixels, and the proportion of high-frequency energy in the pen tip increased to 19%. The error in the second training phase was optimized to 0.28 (<0.3), significantly improving regional attention distribution and detail quality.

[0050] In the embodiment of the present invention, the weighted accumulation of the third error and the fourth error to obtain the training error value of the second training link can be implemented through the following example.

[0051] Determining the visual distortion error based on the compensation frame generation result of the packet loss video stream instance and the reference video stream instance, and determining the state transition frequency error based on the deviation between the state transition frequency of the attention gate and the preset frequency threshold; The third error and the fourth error are weighted and accumulated to obtain the second region credibility fluctuation error, and the second region credibility fluctuation error, the visual distortion error and the state transition frequency error are weighted and accumulated to obtain the training error value of the second training link.

[0052] In an embodiment of the present invention, for example, in a video conferencing scenario, the server calculates the training error value for the second training phase for a packet loss video stream instance of a person turning to write on a whiteboard (the 250th non-keyframe is lost, and the compensation frame corresponds to the person's semi-sideways pen-holding posture). The core of the training phase is to guide the model to prioritize the quality of details in high-information areas such as the hand and pen tip through a comprehensive evaluation of regional credibility fluctuations, visual distortion, and state transition frequency. The specific process is as follows: The server first evaluates the visual distortion error between the generated compensation frame and the reference video stream instance: After optimization in the first training phase, the overall visual distortion is reduced, but residual distortion still exists in high-fluctuation areas. The knuckles in the area where the person is holding the pen in his right hand are blurred (spatial domain deviation), resulting in a slight difference in the hand outline compared to the reference frame (the knuckles are clearly convex in the reference frame, while there is still a 1-pixel blurred band in the compensation frame). The pen tip reflective texture (spectral envelope deviation) improves from a "diffuse light spot" to a "striped reflective texture," but the length is 1 pixel shorter than in the reference frame. Taking into account the residual distortion of regional details, the visual distortion error is reduced to "slight distortion" (corresponding to a quantized value of 0.2, with 0 indicating no distortion and 1 indicating severe distortion). The server then checks the state transition frequency error: After the first stage of attention gating optimization, the state transition frequency has dropped from 2 times / frame to 1.5 times / frame (still exceeding the preset threshold of 1 time / frame), with a deviation of 0.5 times / frame (corresponding to a quantized value of 0.1, with 0 indicating no deviation and 1 indicating severe deviation). At this point, attention gating primarily switches between "hand attention" and "pen tip attention" (due to the high volatility of these two regions, dynamic weighting is required). Although the switching frequency has decreased, further stabilization is still required to avoid fluctuations in feature extraction. The server weights the third error (hand spatial + temporal stability error, 0.38) and the fourth error (pen tip spectral envelope stability error, 0.245) according to regional priority. Because hand movements (convey writing intention) are more critical than pen tip reflections (auxiliary details) in meeting scenarios, the third error is given a 70% weight and the fourth error is given a 30% weight. The second region's credibility fluctuation error = the third error × 70% + the fourth error × 30% = 0.38 × 0.7 + 0.245 × 0.3 = 0.266 + 0.0735 = 0.3395 (approximately 0.34, reflecting the urgency of optimizing details in high-fluctuation regions. The higher the error value, the more priority the model should place on that region). Based on the priority of "regional detail quality > overall visual effect > state transition stability," the server assigns a 70% weight to the second region's credibility fluctuation error (0.34), a 20% weight to the visual distortion error (0.2), and a 10% weight to the state transition frequency error (0.1). The weighted sum of these three factors yields the following training error for the second training phase: 0.34 × 70% + 0.2 × 20% + 0.1 × 10% = 0.238 + 0.04 + 0.01 = 0.288 (approximately 0.29, with an error value < 0.3, close to the optimization target).This error guided the server's backpropagation model adjustments: To address the second-region credibility fluctuation error (0.34), the weight of the temporal prediction unit in the LSTM prediction network for the hand region was increased (from 0.45 to 0.55), strengthening motion vector continuity learning. To address the visual distortion error (0.2), the color mapping layer parameters of the GAN generator were fine-tuned (correcting the RGB channel ratio of the pen tip reflection). To address the state transition frequency error (0.1), the region switching sensitivity threshold of the attention gate was lowered (from 1 to 1.2 times per frame). After 20 rounds of training, the hand knuckle blur bands in the compensation frame were eliminated, the pen tip reflection length was consistent with the reference frame, and the error in the second training phase was optimized to 0.25, achieving the required regional detail quality and attention distribution.

[0053] In the embodiment of the present invention, performing regional enhancement mapping on the first time series feature to obtain the second time series feature can be implemented through the following examples.

[0054] Dividing the first time series feature based on a spatial dimension to obtain a first local relationship feature and a second local relationship feature; performing normalization processing on the first local relationship features, and performing spatial filtering on the normalized first local relationship features in a local area dimension based on a time series dimension to obtain a first feature map; Determine a first multiplication result of the first feature map and the second local relationship feature, and obtain the second temporal feature based on the sum of the first multiplication result and the second local relationship feature.

[0055] In an embodiment of the present invention, for example, in a video conferencing scenario, the server has obtained the first temporal feature (a 64×64×256 feature map containing motion-related features of the character's face, hands, and background, focusing on capturing the spatial relationship between the motion vector of the character's right hand holding the pen and the shoulder outline when the character turns around) through inter-frame motion mapping in the compensation process for the 250th frame loss. The server needs to perform regional enhancement mapping on this feature to highlight the details of high-information areas such as the hands and face to obtain the second temporal feature. The specific process is as follows: The server divides the first temporal feature into regions based on the spatial dimension, focusing on differentiated enhancement of high-information areas and low-information areas: The first local relationship feature corresponds to the character's right-hand pen-holding area (a high-information area, containing key details such as finger joint movement and pen tip trajectory) and the facial area (including facial expression changes when raising the head). The server uses a sliding window (16×16 pixels) to extract feature subgraphs of these two regions from the first temporal feature. These subgraphs have dimensions of 16×16×256 and include hand motion vectors (2.5 pixels / frame horizontally to the right and 3 pixels / frame vertically upward) and facial vertical displacement (head-up rate of 1.2 pixels / temporal step). The second local relationship feature, corresponding to the background region (low-information areas, such as static features such as conference room tables and chairs and whiteboard edges), extracts feature subgraphs of the remaining region with dimensions of 48×48×256. These subgraphs include the basic background texture (such as the grid lines on the whiteboard and the wood grain of the tabletop). The server performs LayerNorm on the first local relationship feature (hand + face subgraphs), eliminating dimensional differences between feature channels. For example, this aligns the numerical ranges of hand motion vectors and facial texture features. Next, spatial filtering is performed on the local region dimension based on the temporal dimension. A 3×3 convolution kernel (with weights optimized for hand joint texture and enhanced edge response) is applied to the normalized sub-image. This filtering emphasizes the raised contours of the knuckles (e.g., the gradient change at the second joint of the index finger) and the trajectory of the pen tip (the linear vector from the tabletop to the chest). The resulting "first feature map" (16×16×256) contains detailed motion and texture features of the hand and face (e.g., contour deformation caused by hand muscle contraction, and changes in brow height when the face raises its head). The server uses a "multiplication-addition" operation to fuse features from high-information regions with features from low-information regions, enhancing details in key areas while preserving background information. The first multiplication result is the element-wise multiplication of the first feature map (fine-grained hand and face features) with the second local relational feature (background features). This emphasizes the correlation between high-information regions and the background (e.g., local background lighting changes caused by hand movement, and the relative position of the pen tip to the whiteboard grid lines). Additive fusion: Directly add the first multiplication result to the second local relationship feature, while retaining the basic background texture (such as the continuity of the whiteboard grid lines and the direction of the desktop wood grain) while injecting the refined features of the hands and face into the overall feature map.Finally, the server outputs the "second temporal feature" (64×64×256). In this feature map, the clarity of the knuckle contours in the area where the person holds the pen on his right hand is improved by 30% (the blur band is reduced from 2 pixels to 0.5 pixels), and the continuity error of the pen tip motion vector is reduced to 1.5 pixels / frame (meeting the compensation requirements for hand movement details in conference scenarios). At the same time, the static features of the background area (such as the straightness of the whiteboard edge) remain stable, laying the foundation for subsequent regional feature fusion.

[0056] In the embodiment of the present invention, the dividing of the first time series features based on the spatial dimension to obtain the first local relationship features and the second local relationship features can be implemented through the following examples.

[0057] Performing spatial filtering on the first time series feature in the spatial dimension based on the time series dimension to obtain a second feature map; The second feature map is normalized, and the normalized second feature map is divided based on a spatial dimension to obtain the first local relationship feature and the second local relationship feature.

[0058] In an embodiment of the present invention, for example, in a video conferencing scenario, the server has obtained a first temporal feature (a 64×64×256 feature map, including the hand motion vector of the person turning around and holding a pen, the facial head movement, and the associated features of the static background texture) through inter-frame motion mapping to compensate for the loss of the 250th non-keyframe. To highlight the details of high-information areas (hands and faces), the server needs to perform spatial division on the first temporal feature to obtain the first and second local relationship features. The specific process is as follows: The server first performs spatial filtering on the first temporal feature in the spatial dimension based on the temporal dimension to enhance regional correlation and suppress noise. The first temporal features are filtered using a 3×3 convolutional kernel (with a stride of 1, padding of 1, and weights optimized through pre-training, with a focus on enhancing continuity in motion regions). For the pen-holding area of ​​the character's right hand, the filtering improves the coherence of the hand motion vector (from the tabletop position in frame 249 to the chest position in frame 251), reducing the horizontal displacement error from 3 pixels per frame to 1.5 pixels per frame. For the facial area, the filtering also smooths the vertical displacement (brow bone height change) of the head-up movement, preventing facial contour distortion caused by timing jumps. For the background area, the filtering enhances the continuity of the whiteboard grid lines, repairs local breakpoints (such as the grid line in row 3 and column 5), and reduces noise (random pixel fluctuations) in the wood grain texture of the desktop by 30%. The final output is a "second feature map" (64×64×256). While preserving the original spatial structure, this map significantly improves the correlation between features within the region (for example, the linkage between hand movement and facial expression is more clearly defined). The server performs LayerNorm on the second feature map (calculating the mean and variance along the channel dimension and normalizing the feature values ​​of each channel to a distribution with a mean of 0 and a variance of 1), eliminating the dimensional differences in features in different regions. For example, the hand motion vector (value range -5 to 5 pixels) and background texture features (value range 0 to 255 grayscale values) are standardized to dimensionless values ​​of -1 to 1, ensuring that the feature weights are comparable during subsequent division. Subsequently, the server divides the standardized second feature map into regions based on the spatial dimension: a 16×16 pixel sliding window is used to extract feature sub-graphs of high-information areas as the "first local relationship features", specifically the right-hand pen-holding area of ​​the person (including knuckle movement and pen tip trajectory, corresponding to feature map coordinates (20~35,30~45)) and the facial area (including changes in brow bones and chin contours when raising the head, corresponding to coordinates (10~25,15~30)). The sub-graph dimension is 16×16×256, which includes refined features such as contour deformation caused by hand muscle contraction and tiny movements of facial expression muscles; the remaining area (background area, including static features such as conference room tables and chairs, whiteboard edges, and the part of the coordinates (0~63,0~63) except the above area) is used as the "second local relationship feature". The sub-graph dimension is 48×48×256, which includes the basic texture of the background (such as the direction of the whiteboard grid lines and the periodic distribution of the wood grain on the desktop).Through the above division, the server achieves feature separation between high-information areas and low-information areas, laying the foundation for subsequent targeted enhancement of high-information area details.

[0059] In the embodiment of the present invention, the obtaining of the second time series feature based on the sum of the first multiplication result and the second local relationship feature can be implemented through the following example.

[0060] Based on the time series dimension, spatial filtering is performed on the sum of the first multiplication result and the second local relationship feature in the spatial dimension to obtain the second time series feature.

[0061] In this embodiment of the present invention, for example, during the compensation process for frame 250 of a video conference, the server calculates the sum of the first multiplication result (the correlation feature between the refined hand / facial features and the background features) and the second local relationship feature (the background basic texture). This sum contains information about the fusion of the hand knuckle contour, the pen tip trajectory, and the background grid lines. The server applies a 3×3 convolution filter (with a stride of 1, padding of 1, and kernel weights optimized for regional transition smoothness) to this sum in the temporal dimension. This filter suppresses the abrupt transition between the hand edge and the background (for example, the edge of the right hand holding area is reduced from a 2-pixel blur band to a 0.5-pixel blur band), while preserving the orthogonal relationship between the pen tip trajectory vector and the background grid lines. The resulting second temporal feature (64×64×256) significantly enhances the details of high-information areas (hand and face).

[0062] In the embodiment of the present invention, the step of performing regional feature fusion on the second time series feature to obtain the output of the time series prediction unit located in the first position may be implemented through the following example.

[0063] performing normalization processing on the second time series features, and performing parallel spatial filtering on the second time series features after the normalization processing, to obtain a third feature map and a fourth feature map after regional feature fusion; Performing Sigmoid scaling on the fourth feature map to obtain a feature selection weight; The third feature map is weighted based on the feature selection weight to obtain the output of the time series prediction unit located in the first position.

[0064] In an embodiment of the present invention, for example, during the compensation process for the 250th frame of a video conference, the server has obtained the second temporal features (a 64×64×256 feature map containing detailed hand knuckle motion, facial contours, and background whiteboard texture, with a natural transition between high-information areas and the background). Regional feature fusion is then used to dynamically adjust the weights of different regions, outputting the feature results of the first temporal prediction unit to provide input for the next unit of the subsequent LSTM prediction network. The specific process is as follows: the server first performs BatchNorm on the second temporal features (normalizing along the batch dimension to adjust the feature values ​​to a distribution with mean 0 and variance 1), eliminating dimensional differences between features in different regions. For example, the gradient change of the hand knuckles (value range -2 to 2) and the texture intensity of the background whiteboard grid lines (value range 0 to 1) are normalized to dimensionless values ​​between -1 and 1, ensuring comparable feature weights during subsequent filtering. The server then performs parallel spatial filtering on the normalized second temporal features, extracting global semantic features and regional weighted features through two convolutional layers. The third feature map is filtered using a 3×3 convolution kernel (stride 1, padding 1, and weights optimized for the overall pose of the person, such as torso rotation angle and the spatial relationship between the hand and face). This outputs a global semantic feature map (64×64×128 pixels), which contains the overall outline of the person holding a pen in a semi-lateral position (e.g., waist rotation angle of 30°, right hand relative distance of 20 pixels to face), and the hierarchical relationship between the background and the person (depth difference between the foreground and background whiteboards). The fourth feature map is filtered using a 1×1 convolution kernel (no spatial expansion, focusing on channel-wise weight distribution), outputting a regional weighted feature map (64×64×128 pixels), which is used to generate feature selection weights (to emphasize the importance of high-information regions). The server applies sigmoid scaling to the fourth feature map, compressing the feature values ​​to the range of 0–1, generating the "feature selection weights" (64×64×128 pixels). Weights are dynamically assigned based on the amount of information contained in each region: Hand regions (knuckles, pen tip) are weighted 0.8-0.85 (high attention) because their motion vectors are crucial (affecting the transmission of writing intent); facial regions (eyebrow contours, brow bone outlines) are weighted 0.7-0.75 (medium-high attention) because their expression information is important (aiding communication and understanding); background regions (whiteboard grid lines, desk wood grain) are weighted 0.15-0.2 (low attention) because they are static and contain little information. The weight matrix clearly prioritizes regions, ensuring that features from high-information regions are prioritized during subsequent weighting.The server multiplies the feature selection weights element-wise with the third feature map (global semantic features), giving stronger representation to features in high-weighted regions. For example, hand region features (knuckle motion vectors and pen tip trajectory) weighted by 0.8 improve contour clarity by 40% (sharpening the raised edges of the knuckles from 1 pixel to 0.5 pixels). Facial region features (chin angle and brow bone height when looking up) weighted by 0.7 enhance the response of facial details (such as a slight raise of the mouth corner) by 30%. Background region features (whiteboard gridline continuity) weighted by 0.2 retain only basic textures to avoid redundant information interference. Finally, the server obtains the "output of the first time series prediction unit" (a 64×64×128 feature map). This map highlights key features of high-information regions such as the hands and face, while suppressing background noise. This provides accurate local-global fusion features for the next time series prediction unit (the second unit) in the LSTM prediction network, supporting the generation of subsequent prediction feature sequences.

[0065] The embodiment of the present invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned video packet loss synchronization compensation method based on deep learning. Figure 2 As shown, Figure 2 This is a block diagram of the structure of a computer device 100 provided in an embodiment of the present invention. Computer device 100 includes a memory 111, a processor 112, and a communication unit 113. To enable data transmission or exchange, memory 111, processor 112, and communication unit 113 are electrically connected to each other, directly or indirectly. For example, these components can be electrically connected via one or more communication buses or signal lines.

[0066] For illustrative purposes, the foregoing description has been made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Numerous modifications and variations are possible in light of the above teachings. These embodiments have been selected and described in order to best illustrate the principles of the present disclosure and its practical application, thereby enabling those skilled in the art to best utilize the present disclosure and to utilize various embodiments with various modifications as appropriate for the specific application contemplated.

Claims

1. A video packet loss synchronization compensation method based on deep learning, characterized in that: include: Obtaining a video stream to be compensated, performing frame feature extraction on the video stream to be compensated to obtain original video frame features, and loading the original video frame features into a packet loss compensation model, wherein the packet loss compensation model includes an LSTM prediction network and a GAN generator network arranged in sequence, the LSTM prediction network and the GAN generator network each including a plurality of sequentially connected timing prediction units, the timing prediction unit including a long short-term memory component, the long short-term memory component being provided with an attention gate, the attention gate being used to guide adjustment of an internal timing state of the long short-term memory component or maintain the internal timing state constant; Based on the first time series prediction unit in the LSTM prediction network, inter-frame motion mapping is performed on the original video frame feature to obtain a first time series feature, regional enhancement mapping is performed on the first time series feature to obtain a second time series feature, regional feature fusion is performed on the second time series feature to obtain the output of the first time series prediction unit, and the output of the first time series prediction unit is loaded into the next time series prediction unit until a prediction feature sequence of the LSTM prediction network is obtained; Generate and decode the predicted feature sequence of the LSTM prediction network based on the GAN generator network to obtain target replacement frame features; Based on the target replacement frame feature, a frame is synchronously generated from the video stream to be compensated to obtain a synchronous compensation frame.

2. The method according to claim 1, characterized in that The performing inter-frame motion mapping on the original video frame feature to obtain the first temporal feature includes: Standardizing the original video frame features, and performing inter-frame motion mapping on the standardized original video frame features using the long short-term memory component; The output of the long short-term memory component is integrated in a time series dimension to obtain the first time series feature.

3. The method according to claim 2, characterized in that The packet loss compensation model is configured to perform a cyclic process through multiple time-series stages, and the long short-term memory component is used to perform inter-frame motion mapping on the normalized original video frame features, including: For any current stage of the plurality of sequential stages, obtaining the initial region attention confidence and the initial region attention confidence fluctuation of the long short-term memory component in the previous sequential stage of the current stage, and determining the target region attention confidence of the long short-term memory component in the current stage according to the initial region attention confidence and the initial region attention confidence fluctuation; Quantifying the attention confidence of the target area to obtain the attention weight corresponding to the attention gate in the current stage; The internal temporal state output by the long short-term memory component in the current stage is determined according to the attention weight, and inter-frame motion mapping is performed on the standardized original video frame features through the long short-term memory component based on the internal temporal state.

4. The method according to claim 1, wherein The packet loss compensation model further includes a frame quality fluctuation detection unit. Before loading the original video frame features into the packet loss compensation model, the method further includes: Obtaining a packet loss video stream instance and a reference video stream instance corresponding to the packet loss video stream instance, and calling the packet loss compensation model to generate packet loss compensation for the packet loss video stream instance; The frame quality fluctuation detection unit is used to determine the frame detail fluctuation of the compensation frame generation result of the packet loss video stream instance, and a training error value is determined based on the deviation between the compensation frame generation result of the packet loss video stream instance and the reference video stream instance and the frame detail fluctuation. The packet loss compensation model is trained based on the training error value.

5. The method according to claim 4, characterized in that The step of determining a training error value according to a compensation frame generation result of the packet loss video stream instance, a deviation between the reference video stream instance, and the frame detail fluctuation, and training the packet loss compensation model based on the training error value includes: Determining a training error value of a first training session based on a compensation frame generation result of the packet loss video stream instance, a deviation between the reference video stream instance, and the frame detail fluctuation, wherein the training error value of the first training session is used to regulate the stability of the frame detail fluctuation; Determining a training error value for a second training phase based on a deviation between a compensation frame generation result of the packet loss video stream instance and the reference video stream instance and the frame detail fluctuation, wherein the training error value for the second training phase is used to set a corresponding regional attention for the regional timing component based on regional credibility fluctuations of the regional timing component when determining the deviation between the compensation frame generation result of the packet loss video stream instance and the reference video stream instance; The packet loss compensation model is trained in sequence based on the training error value of the first training link and the training error value of the second training link.

6. The method according to claim 5, characterized in that The frame detail fluctuation includes spatial domain frame detail fluctuation, temporal domain frame detail fluctuation, and spectrum envelope feature fluctuation. The determining of the training error value of the first training link based on the deviation between the compensation frame generation result of the packet loss video stream instance and the reference video stream instance and the frame detail fluctuation includes: Determining a spatial domain detail deviation, a temporal domain detail deviation, and a spectrum envelope feature deviation between a compensation frame generation result of the packet loss video stream instance and the reference video stream instance; The spatial domain detail deviation is weighted according to a Gaussian distribution function with the spatial domain frame detail fluctuation as a normalization coefficient, the temporal domain detail deviation is weighted according to a Gaussian distribution function with the temporal domain frame detail fluctuation as a normalization coefficient, and a first error is determined according to the sum of the spatial domain frame detail fluctuation, the temporal domain frame detail fluctuation, the weighted spatial domain detail deviation, and the weighted temporal domain detail deviation; weighting the spectrum envelope feature deviation according to a Gaussian distribution function using the spectrum envelope feature fluctuation as a normalization coefficient, and determining a second error according to the sum of the spectrum envelope feature fluctuation and the weighted spectrum envelope feature deviation; Determining a visual distortion error based on a compensation frame generation result of the packet loss video stream instance and the reference video stream instance, and determining a state transition frequency error based on a deviation between a state transition frequency of the attention gate and a preset frequency threshold; The first error and the second error are weighted and accumulated to obtain the first region credibility fluctuation error, and the first region credibility fluctuation error, the visual distortion error and the state transition frequency error are weighted and accumulated to obtain the training error value of the first training link.

7. The method according to claim 6, characterized in that The frame detail fluctuation includes spatial domain frame detail fluctuation, temporal domain frame detail fluctuation, and spectrum envelope feature fluctuation. The determining of the training error value of the second training link based on the deviation between the compensation frame generation result of the packet loss video stream instance and the reference video stream instance and the frame detail fluctuation includes: weighting the spatial domain detail deviation according to the spatial domain frame detail fluctuation, weighting the temporal domain detail deviation according to the temporal domain frame detail fluctuation, and determining a third error according to the sum of the weighted spatial domain detail deviation and the weighted temporal domain detail deviation; weighting the spectrum envelope feature deviation according to the spectrum envelope feature fluctuation, and determining the weighted spectrum envelope feature deviation as a fourth error; Determining a visual distortion error based on a compensation frame generation result of the packet loss video stream instance and the reference video stream instance, and determining a state transition frequency error based on a deviation between a state transition frequency of the attention gate and a preset frequency threshold; The third error and the fourth error are weighted and accumulated to obtain the second region credibility fluctuation error, and the second region credibility fluctuation error, the visual distortion error and the state transition frequency error are weighted and accumulated to obtain the training error value of the second training link.

8. The method according to claim 1, characterized in that The performing regional enhancement mapping on the first time series feature to obtain a second time series feature includes: Performing spatial filtering on the first time series feature in the spatial dimension based on the time series dimension to obtain a second feature map; performing normalization processing on the second feature map, and dividing the normalized second feature map based on a spatial dimension to obtain a first local relationship feature and a second local relationship feature; performing normalization processing on the first local relationship features, and performing spatial filtering on the normalized first local relationship features in a local area dimension based on a time series dimension to obtain a first feature map; Determine a first multiplication result of the first feature map and the second local relationship feature, and perform spatial filtering on the sum of the first multiplication result and the second local relationship feature in a spatial dimension based on a temporal dimension to obtain the second temporal feature.

9. The method according to claim 1, characterized in that The performing regional feature fusion on the second time series feature to obtain the output of the first time series prediction unit includes: performing normalization processing on the second time series features, and performing parallel spatial filtering on the second time series features after the normalization processing, to obtain a third feature map and a fourth feature map after regional feature fusion; Performing Sigmoid scaling on the fourth feature map to obtain a feature selection weight; The third feature map is weighted based on the feature selection weight to obtain the output of the time series prediction unit located in the first position.

10. A server system, characterized in that: The method comprises a server, wherein the server is configured to execute the method according to any one of claims 1 to 9.

Images