A video compression method and system based on LivePortrait-GAN video technology

Abstract

This invention proposes a video recording compression method and system based on LivePortrait-GAN video technology. The method includes: extracting a set of key point coordinates of the teacher's body and matching them with a preset teaching action semantic template library. If a match is successful, it is encoded as a template number and spatial offset; otherwise, it is retained as a residual key point sequence. The residual key point sequence is adjusted by combining synchronous speech rhythm features, structural masks, and spatial offsets. A temporally smooth driving vector is generated by fusing semantic anchor vectors, residual vectors, and gating parameters generated based on the ratio of residual amplitude to template offset amplitude. Using the identity features of the teacher's static reference image and the driving vector, the cropped and fixed-point LivePortrait-GAN generator is used to reconstruct the portrait image, and the final video frame is output through frame-level fusion. The key model in this invention can be efficiently deployed on domestically produced artificial intelligence acceleration chips (such as AX650N) to achieve end-to-end real-time encoding and rendering, meeting the requirements of domestically produced and controllable deployment.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence and multimedia communication technology, and particularly relates to a video compression method and system based on LivePortrait-GAN video technology. BACKGROUND

[0002] With the continuous promotion of education informatization, classroom video recording, remote teaching and after-school playback have gradually become the basic capabilities of schools and education platforms. The storage scale and playback experience of classroom videos directly affect the long-term operation cost of teaching resource construction, teaching supervision and personalized learning. However, the existing classroom video system still has obvious bottlenecks in compression efficiency, real-time performance, localization controllability and scene adaptation. Pixel domain coding represented by H.265 and AVS3 mainly relies on intra-frame transformation and inter-frame prediction to reduce redundancy. In the classroom scene with fixed background, high teacher body proportion and frequent changes in action details, the high-frequency changes in the character area will significantly reduce the prediction efficiency, making it difficult to further improve the compression rate, resulting in a continuous increase in the storage cost of long-term video. At the same time, the dependence of the core technology and ecology of the related coding system also makes it difficult to fully meet the deployment requirements of education data localization and end-side controllability. On the other hand, the development of generative technology makes it possible to replace pixel sequences with structured features and reconstruct pictures with a generative model. However, existing generative compression researches are mostly aimed at general videos or entertainment content, and often lack specific design for the classroom scene. On the one hand, under the condition of real-time at the end side, if the input of the generative model is still high-dimensional key points or unstable frame-by-frame features, it is easy to produce time sequence jitter, action mutation and other artifacts that affect viewing. On the other hand, in the classroom, the teacher's voice and teaching actions are highly synchronized, and the strength, pause and emphasis of the voice often correspond to the changes in gesture amplitude, expression tension and action rhythm. However, the traditional compression path does not take advantage of this synchronization to improve the continuity and stability of the action, resulting in insufficient restoration in the transition section or individualized detail action section that is not covered by the template. SUMMARY

[0003] The purpose of the present application is to design a video compression method and system based on LivePortrait-GAN video technology, which can utilize the classroom action structure rules and voice synchronization rules and perform real-time video compression and rendering.

[0004] To achieve the above purpose, in the first aspect of the present application, a video compression method based on LivePortrait-GAN video technology is provided, which comprises:

[0005] Collecting classroom video frames and extracting a set of teacher human key point coordinates;

[0006] The key point sequence of the continuous frame is matched with a preset teaching action semantic template library. If the matching is successful, the corresponding template number and the spatial offset relative to the template are coded. If the matching is not successful, the residual key point sequence is retained.

[0007] A voice signal synchronized with the classroom video frame is obtained, a voice rhythm feature is extracted, and the residual key point sequence is adjusted by combining the structure mask corresponding to the current template number and the spatial offset, to generate a corrected residual key point sequence.

[0008] The template number is mapped to a semantic embedding vector, and a semantic anchor vector is generated in combination with the spatial offset. The corrected residual key point sequence is projected into a residual vector, and the amplitude of the residual vector is calculated as a residual amplitude. The amplitude of the offset vector corresponding to the spatial offset is calculated as a template offset amplitude. A residual injection gating parameter is generated based on the ratio of the template offset amplitude to the residual amplitude. The semantic anchor vector, the residual vector, and the residual injection gating parameter are fused, and a driving vector at the current time is generated by introducing a driving vector at the previous time for time smoothing.

[0009] An identity feature is extracted from a teacher static reference image, and a LivePortrait-GAN generator is driven based on the driving vector to generate a teacher portrait picture.

[0010] The current generated portrait picture and the output picture at the previous time are frame-level fused to output the final classroom video frame.

[0011] Further, the teaching action semantic template library is constructed before system deployment, and each template is represented by a standardized key point time sequence table, which is used to describe the structural change characteristics of any teaching action in the time dimension, such as board writing, pointing, turning or moving.

[0012] Further, the key point coordinate set is output by a lightweight human key point detection model, covering parts of the head, upper limbs and torso that have teaching semantic significance.

[0013] Further, the structure mask is determined according to the current template number, and is used to limit the voice rhythm feature to act only on a local key point subset related to the current teaching action semantic.

[0014] Further, after the voice rhythm feature is divided into short windows, the short-time energy fluctuation and rhythm change trend are extracted by a one-dimensional convolution stacking network, and a fixed-length vector is generated by time pooling.

[0015] Further, the semantic embedding vector is learned and fixed in the system training phase jointly with the LivePortrait-GAN generator, and is directly obtained from the template number by table lookup in the runtime.

[0016] Further, the residual injection gating parameter is generated by saturation function mapping on the ratio of the template offset amplitude to the residual amplitude.

[0017] Further, the frame-level fusion adopts weighted average of the current generated portrait picture and the last time output picture, and the weight is determined by a fixed smoothing coefficient.

[0018] Further, the LivePortrait-GAN generator is deployed on an artificial intelligence acceleration chip with a neural network processing unit after channel pruning and INT8 quantization.

[0019] In the second aspect of the present application, a video compression system based on LivePortrait-GAN video technology is provided, and the system comprises:

[0020] The key point extraction and action template matching module is used for collecting classroom video frames, extracting a set of teacher human body key point coordinates from the video frames, matching the key point sequence of the continuous frames with a preset teaching action semantic template library, and outputting a template number and a spatial offset or a residual key point sequence.

[0021] The voice-guided residual adjustment module is used for extracting voice rhythm features from a synchronous voice signal, adjusting the residual key point sequence to generate a corrected residual key point sequence in combination with the voice rhythm features, the structure mask and the spatial offset.

[0022] The driving vector generation module is used for mapping the template number into a semantic embedding vector, generating a semantic anchor vector in combination with the spatial offset, projecting the corrected residual key point sequence into a residual vector, calculating the amplitude of the residual vector as a residual amplitude, calculating the amplitude of the offset vector corresponding to the spatial offset as a template offset amplitude, generating a residual injection gating parameter based on the ratio of the template offset amplitude to the residual amplitude, and fusing the semantic anchor vector, the residual vector and the residual injection gating parameter, and introducing a last time driving vector for time smoothing to generate a driving vector at the current time.

[0023] The video rendering and output module comprises a cropped and quantized LivePortrait-GAN generator, which is used for generating a teacher portrait picture based on the driving vector and the identity features of the teacher static reference image, and fusing the current generated picture and the last time output picture to output a final classroom video frame.

[0024] The beneficial technical effects of the present application are at least the following:

[0025] To solve the above problems, the present application provides a video compression method and system based on LivePortrait-GAN video technology. By encoding the high-dimensional action information of the teacher in the continuous video frames into a low-data semantic template number, a space offset and an adjusted residual parameter, the traditional pixel domain inter-frame prediction is replaced, thereby significantly reducing the video code rate and greatly reducing the storage and transmission cost of classroom videos. Secondly, by using the preset "teaching action semantic template library", the typical teaching behaviors such as board writing and pointing are structured and matched, ensuring that the structure and semantic information of the core teaching action are accurately retained and reconstructed with high quality during the compression process. At the same time, by introducing the synchronous voice rhythm feature to fine-tune the residual action, and combining the gate smoothing mechanism during the generation of the driving vector and the frame-level fusion during the output, the common time sequence jitter, action mutation and picture flicker problems in the generated video are effectively suppressed, making the reconstructed video more smooth and natural.

[0026] The key models in the entire method flow are designed to be lightweight, channel pruned and fixed-point, which can be efficiently deployed on domestic artificial intelligence acceleration chips (such as AX650N) to realize end-to-end real-time encoding and rendering, meeting the deployment requirements of domestic controllability. BRIEF DESCRIPTION OF DRAWINGS

[0027] The present application is further illustrated by the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present application. For those skilled in the art, other drawings can be obtained without creative labor on the basis of the following drawings.

[0028] Figure 1 A video compression method based on LivePortrait-GAN video technology of the present application is shown in the flowchart.

[0029] Figure 2 A video compression system framework based on LivePortrait-GAN video technology of the present application is shown in the flowchart. DETAILED DESCRIPTION

[0030] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference signs represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation on the present application.

[0031] It should be noted that the system requires initialization configuration before operation. After obtaining authorization from the teacher, a static reference image of the teacher (such as a frontal image of the teacher standing in front of the podium) is captured before the start of the class. This image is only used for a one-time extraction of the identity feature vector, which will remain unchanged throughout the entire course recording and reside in the chip cache to ensure the consistency of the generated portrait. This initialization process is independent of the real-time video stream compression process described below.

[0032] It should be noted that the LivePortrait-GAN mentioned in this invention is not a general standard model, but rather a generative adversarial network architecture customized for semantic-driven portrait video generation tasks. Its core function is to synthesize dynamic teacher video frames with natural expressions, head poses, and upper limb movements based on a static teacher portrait image (as an identity reference) and a structured driving signal (i.e., the driving vector described in this invention).

[0033] The generator part of LivePortrait-GAN mainly includes three cooperating sub-modules: an identity encoder, which extracts motion-independent identity features from a static reference image; a motion modulation module, which receives a driving vector and performs spatial adaptive deformation control on the identity features accordingly; and a rendering decoder, which reconstructs the modulated features into a high-fidelity RGB portrait image.

[0034] The discriminator is used to improve the realism of the generated images during the training phase, but only the generator is used during the inference and deployment phase.

[0035] To meet the low power consumption and low latency requirements of edge computing scenarios, this invention implements two lightweight processing steps on the original generator: channel pruning and INT8 fixed-point quantization. This enables it to run efficiently on AI acceleration chips with neural network processing units (NPUs) without relying on high-performance GPUs.

[0036] Therefore, in the context of this invention, "LivePortrait-GAN" specifically refers to the aforementioned generative model that has undergone instructional semantic adaptation and edge deployment optimization. Its input is a static image of the teacher and a driving vector, and its output is a reconstructed video frame of the teacher's portrait. It is a key component in realizing the technical path of "extreme compression - semantic transmission - edge reconstruction".

[0037] In one or more embodiments, such as Figure 1 As shown, a video recording compression method based on LivePortrait-GAN video technology is disclosed, the method comprising the following:

[0038] S1: Collect classroom video frames and extract the set of key point coordinates of the teacher's body; match the key point sequence of consecutive frames with the preset teaching action semantic template library. If the match is successful, it is encoded as the corresponding template number and the spatial offset relative to the template. If the match is unsuccessful, it is retained as the residual key point sequence.

[0039] Specifically, this step transforms repetitive and structured actions in teacher classroom videos into efficient template indexes and offset representations, thereby achieving significant data compression while preserving the semantics of instruction. This process does not rely on traditional pixel-level video compression methods; instead, it is based on an abstract representation of human action structure, using semantic template matching to structurally encode frequently occurring behavioral patterns in the teaching scenario. Because classroom teaching is characterized by a limited number of action types and a relatively stable behavioral rhythm, this step leverages this characteristic to perform targeted structural modeling of the action data.

[0040] Classroom video data is captured by a fixed-focus camera installed directly in front of the classroom, covering the area of ​​the podium and blackboard. The video resolution is fixed at 1920×1080, and the sampling frame rate is 30 frames per second. The video stream is input through the MIPI interface of the domestically produced AX650N chip, and basic image processing, including brightness stabilization, background suppression, and enhancement of the teacher's main subject area, is performed internally by the ISP module. This processing is only used to improve the stability of keypoint extraction and does not generate new data categories. Each frame is represented as a tensor in the system. The image data resides in the on-chip cache only during the current frame processing cycle for key point localization calculation. Once the key point coordinates are generated, the original image will no longer participate in subsequent processing or enter any long-term storage structure.

[0041] Obtaining a single frame image Subsequently, the system invokes a lightweight human keypoint detection model deployed in the AX650N's NPU to analyze the teacher's movements. This model employs a convolutional network structure, consisting of multiple layers of convolutional layers and non-linear activation units. The input is the current frame image, and the output is a set of explicit keypoint coordinates for the teacher's body. .in The keypoints are predefined in number and cover parts of the head, upper limbs, and torso that have clear semantic meaning in the teaching movements. The keypoint results only reflect the relative spatial structural relationships between the parts and do not contain any texture, appearance, or features that can be used for individual identification. The purpose of the design is to describe the movement form rather than to portray a specific individual.

[0042] To further reduce redundant representations of action data, a teaching action semantic template library is introduced. This template library was built before system deployment and is derived from the abstract summarization of common teaching actions performed by multiple teachers during lectures, such as basic behaviors like writing on the blackboard, pointing, turning around, and moving. Each template uses a standardized key point time series as a reference representation, with a fixed length, to describe the structural changes of that type of action over time. The templates themselves do not correspond to any specific individual teacher, but are abstractions of the common structure of teaching behaviors; therefore, they only serve as a matching reference during operation and do not change with usage.

[0043] In actual operation, the system employs a sliding time window approach, assembling keypoint sequences from consecutive frames into local segments and comparing them with reference sequences in the template library. During the matching process, to accommodate differences in the range of motion and habits among different teachers, a temporal alignment mechanism is introduced to normalize the timeline, and the sum of Euclidean distances between keypoints is used as the basis for evaluating the matching degree in the spatial dimension. For a given keypoint sequence and template... Match score The calculation method is as follows:

[0044] ;

[0045] in Indicates the current time. The location of a key point template Reference positions of corresponding key points in the middle. This represents the total number of key points. The above distance calculation is based on a normalized coordinate system to avoid the influence of different shooting distances or perspectives on the matching results. When the matching score is lower than a preset threshold, the system determines that the current action segment matches the template. It has structural consistency.

[0046] For a successfully matched action segment, the system no longer retains the complete keypoint sequence, but instead encodes the action as a template number. And the offset relative to the template reference position This offset is only used to describe subtle changes in the current action within the semantic framework of this template, such as local differences in hand trajectories during whiteboard writing, thereby significantly reducing the data size while ensuring accuracy in reconstruction.

[0047] For keypoint sequences that fail to match any template, the system treats them as residual motion data. This data is retained. It typically corresponds to atypical teacher actions or short-term transitional behaviors, and its proportion is relatively low; it will be processed later in conjunction with other information sources. This step does not perform further analysis or storage expansion of the residual key points; it is only passed as an intermediate result to the next processing stage. The output of this step consists of two parts: one part is the structured encoding result formed after successful action template matching. One part represents the type of teaching action the teacher takes at that moment and its offset state; the other part is the sequence of key points that do not match the template. The above outputs are all in structured and de-identified data format, do not contain the original image content, and do not support reverse reconstruction of the actual shooting scene. Their use is limited to subsequent action adjustment and generation control, and user privacy is further protected through this method with the knowledge of teachers and students.

[0048] S2: Acquire the speech signal synchronized with the classroom video frame, extract the speech rhythm features, and adjust the residual key point sequence by combining the structural mask corresponding to the current template number and the spatial offset to generate a corrected residual key point sequence.

[0049] This step processes the two types of results output from step one: one type is the structured results from template matching. It provides "which teaching action semantic mode is being used in the current time period" and "the spatial offset state relative to the template"; the other type is the residual keypoint sequence that was retained even though the template was not hit. These residual segments typically appear during short transitions between different teacher action templates, or in individualized detail segments not covered by the templates. In classroom settings, these residual segments often share a distinct characteristic: the amplitude of the movement is synchronized with the intensity of the speech (e.g., when emphasizing a concept, the speech energy increases, and the hand swing amplitude increases; when pausing to think, the speech energy decreases, and the movement tends to converge). However, relying solely on the keypoint sequence itself can easily lead to jitter in low-amplitude segments and underfitting by "falling behind" in high-amplitude segments. This step, within the semantic framework given in Step 1, uses speech rhythm to fine-tune the residual movements, ensuring a more stable temporal continuity for the residual segments during subsequent fusion and generation.

[0050] This step starts from the classroom characteristic of "how the speech rhythm signal lands on specific key points," limiting the effect of speech to a subset of key points related to the semantics of the current action, avoiding indiscriminate perturbation of key points throughout the body. In specific implementation, the system first... Read a structure mask vector from on-chip read-only memory. This mask is used to indicate which key points are allowed to be adjusted by speech rhythm under the current semantic action. The construction of the mask is related to the semantics of classroom actions, for example: when When representing the semantics of "blackboard writing", the mask gives high weight to key points related to the upper limbs and wrists; when When representing the semantics of "pointing," the mask assigns high weights to key points on the pointing side of the arm; when When representing the semantics of "turning / transition," the mask gives high weight to key points on the torso and shoulders. The purpose of this is to ensure that the speech only affects the "movement parts that should actually change with the tone at this moment," aligning with the physical laws of classroom movements.

[0051] Subsequently, the speech signal Perform short window analysis to obtain rhythmic features In engineering implementation, the DSP performs fixed-length framing on the speech segment and inputs it into a lightweight one-dimensional convolutional stacking network: the first layer is a one-dimensional convolution to extract short-term energy fluctuations, the second layer is a one-dimensional convolution to extract rhythmic change trends, followed by a time pooling layer to obtain a fixed-length vector. This vector does not contain reversible speech content information; it only depicts the relative changes in "intensity and rhythm." Meanwhile, The offset state output from step one is directly incorporated into the calculation of this step, so that the voice adjustment will not conflict with the template offset, but will be finely adjusted in a directional manner based on the "current spatial offset".

[0052] In obtaining , and Afterwards, the motion fine-tuning model Forward computation is performed on the NPU side, and the key point adjustments for the residual segment are output. . A two-layer fully connected structure is adopted, with nonlinear activation in the middle. Semantic mask constraints and temporal inertia terms are added to the output to suppress jitter and abrupt changes common in residual segments. To facilitate matrix operations, the adjustment amount... Spatial offset and structure mask This is represented as a flattened vector. The adjustment process is described as follows:

[0053] ;

[0054] in, Indicates time The key point adjustment amount serves as an additive correction to the residual key points; For the reason The semantic structure mask obtained from the index is used to restrict the adjustment to a subset of key points relevant to the current instructional action; This indicates that a mask constraint is applied on an element-by-element basis; It is the time inertia coefficient, which allows the adjustment amount to transition smoothly between adjacent time points; This is the adjustment amount from the previous moment, used to suppress instantaneous jumps in the residual segment; For voice signal The rhythm feature vector obtained by one-dimensional convolution and pooling on the DSP side; The template offset state output from step one is used to provide the current spatial position conditions; This indicates vector concatenation; and This is the parameter matrix of a two-layer fully connected system; It is a non-linear activation function. The calculation is performed in a normalized coordinate system to avoid uncontrolled adjustment due to changes in shooting distance.

[0055] get Then, apply it to the remaining key points output in step one. Generate corrected residual key points This update only occurs in the residual segments and does not rewrite the template-hitting segments, thus ensuring that the structured template expression from step one is not destroyed. The update relationship is as follows:

[0056] ;

[0057] in, This is the corrected set of residual keypoint locations, used to replace the original residual keypoints in subsequent fusion steps; These are the remaining key points output from step one; This is the adjustment amount calculated in this step. It can be intuitively understood for typical classroom scenarios: when a teacher explains a key point, their voice rhythm is stronger. Corresponding changes will occur. Under masking constraints, the adjustment of key points of the hand or upper limb related to the current semantic action is mainly amplified to make the amplitude of the residual action more consistent with the tone; when the teacher pauses to think or the vocal energy decreases, Under the influence of the time inertia term, it gradually converges, and the residual segment will not exhibit meaningless tremors.

[0058] In terms of data processing, this step does not introduce new data categories: the speech signal is only converted into a rhythm vector within a short window on the DSP. It participates in the current calculation, but the rhythm vector is not stored in long-term storage; the remaining key points are updated. It remains structured geometric data, independent of the original speech and original image, and does not carry reversible appearance or speech content.

[0059] This step outputs two results. The first is the corrected residual keypoint sequence. Used to replace step one The next step is to proceed with the fusion process; the second step is to check the template matching results. The result is used in this step to select the mask and provide offset conditions, but its value remains unchanged and continues to be passed on.

[0060] S3: Map the template number to a semantic embedding vector, and generate a semantic anchoring vector by combining the spatial offset; project the corrected residual keypoint sequence into a residual vector, and calculate the magnitude of the residual vector as the residual amplitude; calculate the magnitude of the offset vector corresponding to the spatial offset as the template offset amplitude; generate residual injection gating parameters based on the ratio of the template offset amplitude to the residual amplitude; fuse the semantic anchoring vector, the residual vector, and the residual injection gating parameters, and introduce the driving vector from the previous time step for temporal smoothing to generate the driving vector at the current time step;

[0061] Specifically, this step builds upon the results of the previous two steps, converging the two information links of classroom actions into a unified driving vector for LivePortrait-GAN within the same temporal coordinate system. The real-world engineering problem addressed in this step is that generative models cannot directly consume mixed data such as "discrete template numbers + high-dimensional keypoint sets" when running on the edge; otherwise, inconsistent driving scales, temporal jitter, and abrupt changes will be amplified, leading to shaky mouths / hands and flickering transition frames in the generated portrait. The characteristics of teacher actions in classroom scenarios are "long semantically stable segments, short transition segments, and limited action amplitude." Therefore, this step focuses on "semantic anchoring + residual injection + temporal gating" to... As a stable anchor point, As a local detail injection, and with gating parameters restricting residual injection to only function in transition segments where it is truly needed, a low-dimensional, stable, and end-execution-capable driving vector is formed. .

[0062] This step first constructs a "semantic anchoring representation" to ensure that long-term stable action segments in the classroom (such as continuous blackboard writing or continuous pointing) do not drift in the driving space. Specifically, each template is numbered. A set of semantic embedding vectors is pre-defined. This embedding is learned and solidified along with the generator during the system training phase, and is generated at runtime by... The results are obtained via table lookup, eliminating the need for online calculations. Since the number of classroom action templates is limited and fixed, the table lookup method has extremely low cost on the client side, while also avoiding jumps caused by directly inputting discrete numbers. (The rest of the text is missing.) Mapped to and through a fully connected layer Offset vectors of the same scale, and with By concatenating the vectors, we obtain the semantic anchoring vector. The parameters of this fully connected layer are fixed before deployment, and only one matrix multiplication is performed during the inference phase, adapting to edge operators.

[0063] Next, a "residual injection representation" is constructed to be used to... Local details are injected into the driving space in a controllable manner. Considering that most residual segments in classroom scenarios occur during action transitions and short-term emphasized actions, the details are directly... Full mapping introduces unnecessary perturbations. This step employs two-stage compression: first, [the compression process is performed on the data]... Perform a linear projection with a fixed dimension to obtain the residual vector. Then through gating parameters The residual injection ratio is determined. The gating parameters do not depend on new data but are adaptively generated from the "relative relationship between template offset magnitude and residual magnitude." Intuitively, this means that residual injection should be suppressed when the template offset is sufficient to explain the current action; and reinforced when the residual segment truly contains details not covered by the template. To this end, a set of directly calculable magnitude ratios are used at the endpoint to generate these parameters. and use it to Weighted.

[0064] To unify the aforementioned "semantic anchoring" and "residual injection" into the LivePortrait-GAN driver interface, this step constructs the driver vector. Furthermore, it incorporates a "temporal stability constraint specific to the classroom scenario": stable phases require gradual changes, while transition phases allow for faster changes but still need to avoid abrupt changes. This constraint is based on the driving vector from the previous time step. The first-order smooth implementation. Based on the above, the driving vector generation relationship is as follows:

[0065] ;

[0066] in, This is the generator driving vector at the current time step; This is the driving vector from the previous time step, used to provide temporal smoothing; This is a smoothing coefficient used to control the continuity of the stable segment; For the reason The constructed semantic anchor vector represents the main structural control of the current teaching action under the template semantics; For the reason The residual vector obtained by linear projection represents the local details not covered by the template; This is the residual injection gating parameter, representing the intensity of the residual to be injected.

[0067] Gating parameters The calculations use a "relative amplitude ratio" rather than an additional model to ensure end-to-end feasibility and align with the physical intuition of classroom movements. Specifically, firstly, the calculations are performed on... and The scalar representation of its variation amplitude is calculated, and then a saturated gating function is constructed, which tends to enhance when the residual amplitude is significantly greater than the template offset amplitude, and tends to suppress when the residual amplitude is small. The calculation method is as follows:

[0068] ;

[0069] in, For residual injection gating parameters; This is the set of remaining key points from the previous moment, used to calculate the magnitude of change in the remaining segment; This represents the template offset state at the previous moment, used to calculate the magnitude of change at the template level. To avoid stable terms with a denominator of zero, this gating method has direct significance for classroom scenarios: when the teacher maintains the same template action (e.g., continuous blackboard writing), and The changes were relatively small. Maintaining low values, the driving force is mainly semantic anchoring. Control; when the teacher makes a brief emphasis or a rapid transition using gestures, The magnitude of change increased. The automatic elevation allows for a stronger injection of the speech rhythm-corrected details from step two into the driving vector, enabling the generator to reproduce the intensity and rhythm of the emphasized actions.

[0070] In terms of engineering implementation, The construction is accomplished through "table lookup embedding + one layer of fully joined mapping". The construction is accomplished through a "one-layer fully connected projection". The calculations are performed using a small number of vector differences and norms, both of which are compatible with edge operators and cache access modes. The entire process does not introduce new data types, nor does it generate high-frequency, large-scale storage; it only maintains the on-chip cache. , and To support continuous computation. This step outputs a single result. This vector contains both action semantic anchoring information and residual action detail information in the same representation space, and its stability in the classroom scenario is ensured by gating and temporal smoothing constraints.

[0071] S4: Extract identity features using the teacher's static reference image, and generate a teacher portrait image using the cropped and fixed-point LivePortrait-GAN generator based on the driving vector; perform frame-level fusion of the currently generated portrait image with the output image of the previous moment, and output the final classroom video frame;

[0072] Specifically, the system deploys the LivePortrait-GAN generator model on the domestically produced AX650N chip as the core module for classroom video rendering, denoted as... Before system deployment, the model underwent cropping and localization, retaining only the generation path and containing no discrimination or training-related structures. The generator consists of multi-level upsampling convolutional modules, progressively mapping low-dimensional control features to high-resolution image features, ultimately outputting a single frame of the teacher's portrait. All model parameters are fixed during deployment and are not updated during runtime.

[0073] To ensure the generated portrait maintains consistency of identity throughout the class, the system acquires a static reference image of the teacher during the initialization phase. For example, a single frame image captured by a camera before the class begins, showing the teacher standing directly in front of the podium. This image is used only once to extract identity reference features, which are then converted into a fixed feature representation through an encoding path within the generator and subsequently stored in the on-chip cache. During subsequent rendering, these identity features remain unchanged over time, thus ensuring consistency regardless of the context. No matter how things change, the generated portrait always corresponds to the same teacher appearance.

[0074] At each time step The generation process follows the following relationship:

[0075] ;

[0076] in, Indicates the generator at time... The output image of the teacher. This represents the LivePortrait-GAN generator model deployed on the AX650N. This is the driving vector output from step three. Instead of directly describing pixel positions, it acts as a conditional signal to adjust the responses of features at various levels within the generator, ensuring that the generated results are consistent with classroom behavior in terms of head pose, facial expressions, and hand gestures.

[0077] Considering the actual viewing needs of classroom recordings, relying solely on frame-by-frame output from the generator may still result in slight flickering during action transitions or emphasis. Therefore, this step introduces a frame-level smoothing mechanism at the generation output layer to fuse the results of the current generated frame with those of the previous frame. This processing does not alter the structure of the generation model, nor does it backtrack to historical driving information; it only performs simple calculations at the output layer, expressed as follows:

[0078] ;

[0079] in, For the final output of classroom video frames, This refers to the frames that were already output in the previous time step. This is the result directly output by the current generator. This is a smoothing coefficient used to control the continuity of time. In actual classroom scenarios, when teachers stand for extended periods to explain or write on the blackboard, and The differences are minimal, and the fusion result introduces almost no delay; when the teacher suddenly turns around, raises their hand, or emphasizes a key point, the newly generated action transitions naturally within a few frames, avoiding abrupt jumps and thus better reflecting the real classroom viewing experience. At the engineering implementation level, Forward inference runs on the AX650N's NPU unit, while frame-level fusion computation runs on the chip's internal vector computation unit, ultimately yielding... The video is sent directly to the display interface or written to the local storage buffer via the video processing module, forming a continuous video stream. Throughout the process, there is no need to access or reconstruct the original classroom video, nor is it necessary to store the complete action history; real-time rendering can be completed solely based on the current and previous states.

[0080] In one or more embodiments, such as Figure 2 As shown, a video recording compression system based on LivePortrait-GAN video technology is disclosed, the system comprising:

[0081] The key point extraction and action template matching module is used to collect classroom video frames; extract the set of key point coordinates of the teacher's human body from the video frames; match the key point sequence of consecutive frames with the preset teaching action semantic template library, and output the template number and spatial offset or residual key point sequence.

[0082] The voice-guided residual adjustment module is used to extract speech rhythm features from the synchronous speech signal; and combine the speech rhythm features, structural mask and spatial offset to adjust the residual key point sequence to generate a corrected residual key point sequence.

[0083] The driving vector generation module is used to map the template number to a semantic embedding vector and generate a semantic anchoring vector by combining the spatial offset; project the corrected residual keypoint sequence into a residual vector and calculate the magnitude of the residual vector as the residual amplitude; calculate the magnitude of the offset vector corresponding to the spatial offset as the template offset amplitude; generate residual injection gating parameters based on the ratio of the template offset amplitude to the residual amplitude; fuse the semantic anchoring vector, the residual vector, and the residual injection gating parameters, and introduce the driving vector of the previous time step for temporal smoothing to generate the driving vector of the current time step.

[0084] The video rendering and output module includes a cropped and fixed-point LivePortrait-GAN generator, which generates a teacher portrait based on the identity features of the driving vector and the teacher's static reference image; it merges the currently generated image with the output image from the previous moment to output the final classroom video frame.

[0085] It is worth noting that the specific workflow of the video recording compression system based on LivePortrait-GAN video technology provided in this embodiment of the invention is the same as that of the video recording compression method based on LivePortrait-GAN video technology described in the above embodiment, and will not be repeated here.

[0086] This invention also provides a video recording compression device based on LivePortrait-GAN video technology, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the steps described in the above embodiment of a video recording compression method based on LivePortrait-GAN video technology, for example... Figure 1 The steps S1 to S4 described above; or, when the processor executes the computer program, it implements the functions of each module in the above system embodiments.

[0087] For example, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the LivePortrait-GAN video recording and compression device.

[0088] The video recording and compression device based on LivePortrait-GAN video technology can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. This device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the device may also include input / output devices, network access devices, and buses.

[0089] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASACs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the LivePortrait-GAN video recording and compression device, connecting all parts of the device via various interfaces and lines.

[0090] The memory can be used to store the computer program and / or modules. The processor implements various functions of the video recording and compression device based on LivePortrait-GAN video technology by running or executing the computer program and / or modules stored in the memory and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created according to the operation of the air conditioner controller, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD card), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0091] The module integrated into the video recording and compression device based on LivePortrait-GAN technology, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0092] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0093] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A video recording compression method based on LivePortrait-GAN video technology, characterized in that, The method includes: Collect classroom video frames and extract the coordinate set of key points on the teacher's body; The key point sequence of consecutive frames is matched with a preset teaching action semantic template library. If the match is successful, it is encoded as the corresponding template number and the spatial offset relative to the template. If the match is unsuccessful, it is retained as a residual key point sequence. Acquire the speech signal synchronized with the classroom video frame, extract the speech rhythm features, and combine the structural mask corresponding to the current template number and the spatial offset to adjust the residual key point sequence and generate a corrected residual key point sequence. The template number is mapped to a semantic embedding vector, and a semantic anchoring vector is generated by combining the spatial offset. The corrected residual keypoint sequence is projected into a residual vector, and the magnitude of the residual vector is calculated as the residual amplitude. The magnitude of the offset vector corresponding to the spatial offset is calculated as the template offset amplitude. A residual injection gating parameter is generated based on the ratio of the template offset amplitude to the residual amplitude. The semantic anchoring vector, the residual vector, and the residual injection gating parameter are fused, and the driving vector of the previous time step is introduced for temporal smoothing to generate the driving vector of the current time step. Identification features are extracted using static reference images of teachers, and the cropped and fixed-point LivePortrait-GAN generator is driven by the driving vector to generate a portrait of the teacher. The currently generated human image is merged with the output image from the previous moment at the frame level to output the final classroom video frame.

2. The video recording compression method based on LivePortrait-GAN video technology according to claim 1, characterized in that, The teaching action semantic template library was built before the system was deployed. Each template is represented by a standardized key point time series and is used to describe the structural change characteristics of any teaching action in the time dimension, such as writing on the blackboard, pointing, turning around or moving.

3. The video recording compression method based on LivePortrait-GAN video technology according to claim 1, characterized in that, The set of key point coordinates is output by a lightweight human key point detection model, covering parts of the head, upper limbs, and torso that have pedagogical semantic significance.

4. The video recording compression method based on LivePortrait-GAN video technology according to claim 1, characterized in that, The structure mask is determined based on the current template number and is used to limit the speech rhythm features to only apply to a subset of local key points that are semantically related to the current teaching action.

5. The video recording compression method based on LivePortrait-GAN video technology according to claim 1, characterized in that, The speech rhythm features are segmented into frames through a short window, and then extracted using a one-dimensional convolutional stacking network to extract short-term energy fluctuations and rhythmic change trends. Finally, a fixed-length vector is generated through time pooling.

6. The video recording compression method based on LivePortrait-GAN video technology according to claim 1, characterized in that, The semantic embedding vector is jointly learned and solidified with the LivePortrait-GAN generator during the system training phase, and can be directly obtained from the template number by looking up a table during runtime.

7. The video recording compression method based on LivePortrait-GAN video technology according to claim 1, characterized in that, The residual injection gating parameters are generated by performing a saturated function mapping on the ratio of the template offset amplitude to the residual amplitude.

8. The video recording compression method based on LivePortrait-GAN video technology according to claim 1, characterized in that, The frame-level fusion uses a weighted average of the currently generated portrait image and the output image from the previous moment, with the weights determined by a fixed smoothing coefficient.

9. The video recording compression method based on LivePortrait-GAN video technology according to claim 1, characterized in that, The LivePortrait-GAN generator, after channel pruning and INT8 localization, is deployed on an AI acceleration chip equipped with a neural network processing unit for execution.

10. A video recording compression system based on LivePortrait-GAN video technology, characterized in that, The system includes: The key point extraction and action template matching module is used to collect classroom video frames; extract the set of key point coordinates of the teacher's human body from the video frames; match the key point sequence of consecutive frames with the preset teaching action semantic template library, and output the template number and spatial offset or residual key point sequence. The voice-guided residual adjustment module is used to extract speech rhythm features from the synchronous speech signal; and combine the speech rhythm features, structural mask and spatial offset to adjust the residual key point sequence to generate a corrected residual key point sequence. The driving vector generation module is used to map the template number to a semantic embedding vector and generate a semantic anchoring vector by combining the spatial offset; project the corrected residual keypoint sequence into a residual vector and calculate the magnitude of the residual vector as the residual amplitude; calculate the magnitude of the offset vector corresponding to the spatial offset as the template offset amplitude; generate residual injection gating parameters based on the ratio of the template offset amplitude to the residual amplitude; fuse the semantic anchoring vector, the residual vector, and the residual injection gating parameters, and introduce the driving vector of the previous time step for temporal smoothing to generate the driving vector of the current time step. The video rendering and output module includes a cropped and fixed-point LivePortrait-GAN generator, which generates a teacher portrait based on the identity features of the driving vector and the teacher's static reference image; it merges the currently generated image with the output image from the previous moment to output the final classroom video frame.

Images