Video coding method, computer equipment, storage medium and computer program product

Abstract

The invention relates to a video coding method, computer equipment, a storage medium and a computer program product. The method comprises the following steps: identifying a target video frame in a video, and determining a source image and a to-be-coded image from the target video frame; determining motion difference features between the motion features of the source image and the motion features of the to-be-coded image, and extracting facial features of the source image; and obtaining an image coding result of the source image, adjusting the image coding result of the source image according to the action difference feature and the facial feature of the source image, and taking the adjusted image coding result of the source image as an image coding result of the to-be-coded image. By adopting the method, the facial features of the source image and the action difference between the source image and the to-be-coded image can be combined, the image coding result of the source image can be accurately deformed and adjusted, the image coding result capable of representing the three-dimensional space information of the to-be-coded image is generated, complex geometric and texture information is not needed, and the image coding efficiency is improved. And accurate and efficient video compression is realized.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video encoding method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Video compression can reduce the size of video files by encoding the video while keeping the visual quality of the video as unchanged as possible. Video compression can effectively solve the problem that video files usually take up a lot of storage space and bandwidth resources by removing or reducing redundant information in the video. With the popularization of remote work and online communication, users have put forward higher requirements for the efficiency and accuracy of video compression.

[0003] In traditional technologies, video compression of facial animation based on 3D models requires complex geometry and texture information, and consumes a lot of bandwidth; video compression based on optical flow estimation is not accurate enough when processing fast motion or occlusion; video compression based on audio-driven facial animation may not be able to handle complex non-verbal facial movements. Therefore, traditional technologies are not conducive to improving the accuracy of video compression when performing efficient video compression. Summary of the invention

[0004] Based on this, it is necessary to provide a video encoding method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of video compression when performing high-efficiency video compression in response to the above-mentioned technical problems.

[0005] In a first aspect, the present application provides a video encoding method, comprising:

[0006] Identify a target video frame in which a face appears in the video, and determine a source image and an image to be encoded from the target video frame; the face region of the source image and the face region of the image to be encoded satisfy a preset similarity condition;

[0007] Determining motion difference features between motion features of the source image and motion features of the image to be encoded, and extracting facial features of the source image;

[0008] An image coding result corresponding to the source image is obtained, the image coding result corresponding to the source image is adjusted according to the action difference feature and the facial feature of the source image, and the adjusted image coding result of the source image is used as the image coding result corresponding to the image to be encoded.

[0009] In one embodiment, adjusting the image encoding result corresponding to the source image according to the action difference feature and the facial feature of the source image includes:

[0010] splicing the facial features of the source image and the action difference features to obtain a splicing result;

[0011] The image encoding result corresponding to the source image is subjected to multi-scale transformation with the splicing result as a condition to obtain the adjusted image encoding result.

[0012] In one of the embodiments, determining the source image and the image to be encoded from the target video frame includes:

[0013] Dividing the video into at least one video segment according to differences between facial regions of the target video frames in the video;

[0014] For each of the video segments, the source image and the image to be encoded are screened out from each of the video segments according to the facial area in each of the video segments.

[0015] In one of the embodiments, dividing the video into at least one video segment according to the difference between the facial regions of each of the target video frames in the video includes:

[0016] Sort the facial regions of the target video frames according to the sequence of the target video frames in the video to obtain a facial region sequence;

[0017] Taking the first target video frame among the target video frames as the video frame to be analyzed, and determining the intersection-over-union ratio between the facial region of the video frame to be analyzed and the facial region of each target video frame after the video frame to be analyzed;

[0018] When the IoU is greater than a preset IoU threshold and the time difference between the target video frames corresponding to the facial region associated with the IoU is less than a preset time difference threshold, the facial region associated with the IoU is removed from the facial region sequence to obtain a new facial region sequence, and the video frame to be analyzed is used as a video segmentation node;

[0019] The video frame corresponding to the first facial region in the new facial region sequence is used as a new video frame to be analyzed, and the step of determining the intersection-and-union ratio between the facial region of the video frame to be analyzed and the facial regions of each target video frame after the video frame to be analyzed is returned to be executed until the new facial region sequence is empty, and at least one video segmentation node is obtained;

[0020] The video is divided into at least one video segment according to each of the video segmentation nodes.

[0021] In one of the embodiments, after the step of using the adjusted image encoding result as the image encoding result corresponding to the image to be encoded, the method further includes:

[0022] Obtaining a video encoding result at least according to the image encoding result corresponding to the source image and the image encoding result corresponding to the image to be encoded; and sending the video encoding result to an image decoding end;

[0023] The image decoding end is used to perform upsampling and mapping processing on the image encoding result corresponding to the image to be encoded according to the facial features and the action difference features in the video encoding result, so as to obtain a decoded image corresponding to the image to be encoded.

[0024] In a second aspect, the present application further provides a method for training a video processing model, wherein the video processing model includes a facial feature extraction model, an action feature extraction model, a feature deformation model, and an encoder, and the method includes:

[0025] Inputting a first sample image to an encoder to be trained, obtaining an image encoding result corresponding to the first sample image, and decoding the image encoding result corresponding to the first sample image to obtain an image decoding result corresponding to the first sample image; training the encoder to be trained according to a difference between the first sample image and the image decoding result corresponding to the first sample image to obtain a trained encoder; the trained encoder is used to obtain an image encoding result corresponding to an image;

[0026] Training a facial feature extraction model to be trained according to the first sample image; the trained facial feature extraction model is used to extract facial features of the image;

[0027] Acquire a second sample image different from the first sample image, respectively extract the motion features of the first sample image and the motion features of the second sample image through the motion feature extraction model to be trained, and determine the motion difference features between the motion features of the first sample image and the motion features of the second sample image, and train the motion feature extraction model to be trained according to the motion difference features to obtain a trained motion feature extraction model; the trained motion feature extraction model is used to obtain the motion features of the image;

[0028] adjusting the image coding result corresponding to the first sample image according to the action difference feature between the action feature of the first sample image and the action feature of the second sample image through the feature deformation model to be trained, obtaining the image coding result corresponding to the second sample image, obtaining the image coding reference corresponding to the second sample image, and training the feature deformation model to be trained according to the image coding reference and the image coding result corresponding to the second sample image to obtain the trained feature deformation model; the trained feature deformation model is used to adjust the image coding result of the image;

[0029] A trained video processing model is obtained based on the trained encoder, the trained facial feature extraction model, the trained motion feature extraction model and the trained feature deformation model; the trained video processing model is used to adjust the image encoding result corresponding to the image to obtain an image encoding result corresponding to another image different from the image.

[0030] In one embodiment, the video processing model further includes a discriminator, and the method further includes:

[0031] Obtaining a first expected value of the discriminator for the first sample image and a second expected value of an image decoding result corresponding to the first sample image, and determining an adversarial loss function value according to the first expected value and the second expected value;

[0032] Determining a reconstruction loss function value according to the second sample image and an image decoding result corresponding to the second sample image;

[0033] determining a perceptual loss function value according to an image feature of the second sample image and an image feature of an image decoding result corresponding to the second sample image;

[0034] determining a consistency loss function value according to an action difference feature between an action feature of the first sample image and an action feature of the second sample image and an action difference feature between an action feature of the first sample image and an action feature of an image decoding result corresponding to the second sample image;

[0035] The video processing model is trained according to the adversarial loss function value, the reconstruction loss function value, the perceptual loss function value and the consistency loss function value.

[0036] In a third aspect, the present application further provides a video encoding device, including:

[0037] A determination module, used to identify a target video frame in which a face appears in a video, and determine a source image and an image to be encoded from the target video frame; the face area of ​​the source image and the face area of ​​the image to be encoded satisfy a preset similarity condition;

[0038] An extraction module, used to determine the action difference characteristics between the action characteristics of the source image and the action characteristics of the image to be encoded, and to extract the facial features of the source image;

[0039] An adjustment module is used to obtain an image coding result corresponding to the source image, adjust the image coding result corresponding to the source image according to the action difference feature and the facial feature of the source image, and use the adjusted image coding result of the source image as the image coding result corresponding to the image to be encoded.

[0040] In a fourth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above method are implemented.

[0041] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above method when executed by a processor.

[0042] In a sixth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0043] The above-mentioned video encoding method, apparatus, computer equipment, computer-readable storage medium and computer program product, by identifying the target video frame in which the face appears in the video, determine the source image and the image to be encoded from the target video frame, and adjust the image encoding result of the source image based on the source image, combined with the facial features of the source image and the motion difference between the source image and the image to be encoded, to generate an image encoding result corresponding to the image to be encoded, so that when the facial features of the source image can guide the generated image to be close to the image of the source image, based on the image encoding result that can represent the three-dimensional spatial information of the source image, combined with the facial features of the source image and the motion difference between the source image and the image to be encoded, the image encoding result of the source image is accurately deformed and adjusted to generate an image encoding result that can represent the three-dimensional spatial information of the image to be encoded, without complex geometry and texture information, thereby achieving efficient compression of the video, and can also make the image encoding result include the motion features in the image, thereby improving the accuracy of video compression. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 A diagram of an application environment of a video encoding method in an embodiment;

[0046] Figure 2 is a schematic flow chart of a video encoding method in an embodiment;

[0047] Figure 3A schematic diagram of dividing a video segment in one embodiment;

[0048] Figure 4 A schematic diagram of a process of compressing a video in one embodiment;

[0049] Figure 5 A schematic diagram of a process of training an image coding model in one embodiment;

[0050] Figure 6 is a structural block diagram of a video encoding device in one embodiment;

[0051] Figure 7 The figure is a diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] The video encoding method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 identifies the target video frame where the face appears in the video, and determines the source image and the image to be encoded from the target video frame; the facial area of ​​the source image and the facial area of ​​the image to be encoded meet the preset similarity condition; the server 104 determines the action difference feature between the action feature of the source image and the action feature of the image to be encoded, and extracts the facial features of the source image; the server 104 obtains the image encoding result corresponding to the source image, adjusts the image encoding result corresponding to the source image according to the action difference feature and the facial features of the source image, and uses the adjusted image encoding result of the source image as the image encoding result corresponding to the image to be encoded. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart TVs, smart car-mounted devices, projection devices, etc. Portable wearable devices can be smart watches, head-mounted devices, etc. The head mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0054] In an exemplary embodiment, Figure 2 As shown, a video encoding method is provided, and the method is applied to a server as an example for description, including the following steps S202 to S206. Among them:

[0055] Step S202, identifying a target video frame in which a face appears in the video, and determining a source image and an image to be encoded from the target video frame; the face region of the source image and the face region of the image to be encoded satisfy a preset similarity condition.

[0056] The video may refer to a video to be compressed. The face may refer to a human face. The target video frame may refer to a video frame image containing a human face. The source image may refer to a video frame image containing a human face. In practical applications, the source image may be selected or randomly selected from the target video frame. The image to be encoded may refer to a video frame image in the target video frame that is different from the source image.

[0057] As an example, the server can obtain a video that needs to be compressed, and decompose the video into several video frames through an open source multimedia processing tool (such as ffmpeg), and perform face detection on each of the several video frames, and filter out the video frames where faces appear as target video frames. In actual applications, in order to save computing time, the server can start from the first video frame of the video, and perform face detection on the video frames at intervals of a preset number of frames, and use the video frames where faces appear as target video frames. The server can then sort the target video frames according to the frame order of the target video frames in the video, and divide the video into several video clips according to the facial detection results / face detection results of the target video frames. The facial detection results / face detection results of the target video frame in each video clip are matched (such as the same or similar). For each video clip, the server can randomly select two video frames from the video clip as the source image and the image to be encoded. In practical applications, when any (target) video frame in the video is used as the source image, the server can arbitrarily select a video frame from the video frames other than the any (target) video frame in the video as the image to be encoded, and the server can also arbitrarily select a video frame from the video frames other than the any (target) video frame in the video clip where the any (target) video frame is located as the image to be encoded.

[0058] Step S204, determining the motion difference features between the motion features of the source image and the motion features of the image to be encoded, and extracting the facial features of the source image.

[0059] The motion difference feature may refer to the difference between the motion feature representing the source image and the motion feature of the image to be encoded. In practical applications, the motion difference feature includes the difference between the motion feature encoding of the source image and the motion feature encoding of the image to be encoded.

[0060] Among them, the facial features of the source image may refer to the information characterizing the facial structure of the source image. In practical applications, the facial features of the source image may be used as a high-level abstract feature to compare the image to be encoded with the source image, thereby guiding the generated image to be closer to the image (or features) of the source image.

[0061] As an example, the server can use the action feature extraction model to respectively obtain the action features of the source image and the action features of the image to be encoded, and analyze the differences between the action features of the source image and the action features of the image to be encoded to generate action difference features between the source image and the image to be encoded. The server can also obtain the facial features of the source image through the facial feature extraction model, so as to subsequently generate the image encoding result of the image to be encoded based on the action difference features and the facial features of the source image.

[0062] Step S206, obtaining the image coding result corresponding to the source image, adjusting the image coding result corresponding to the source image according to the action difference feature and the facial feature of the source image, and using the adjusted image coding result of the source image as the image coding result corresponding to the image to be coded.

[0063] The image encoding result corresponding to the source image may refer to information used to characterize the three-dimensional spatial information of the source image. In practical applications, the image encoding result corresponding to the source image may include latent variables corresponding to the source image.

[0064] As an example, the server can use the image coding model to obtain the image coding result corresponding to the source image, and adjust the image coding result corresponding to the source image in combination with the action difference features and the facial features of the source image to achieve dynamic deformation. The adjusted image coding result can be used as the image coding result corresponding to the image to be encoded. The image coding process accurately decouples the character's action from the character's expression, and adjusts the image coding result corresponding to the source image according to the action difference features and the facial features of the source image, so that the image coding process focuses on facial features and action features, thereby ensuring the accuracy of facial expression capture in the image coding process.

[0065] In the above-mentioned video encoding method, by identifying the target video frame in which the face appears in the video, the source image and the image to be encoded are determined from the target video frame, and based on the source image, combined with the facial features of the source image and the motion difference between the source image and the image to be encoded, the image encoding result of the source image is adjusted to generate the image encoding result corresponding to the image to be encoded. It can be seen that in this embodiment, based on the image encoding result that can represent the three-dimensional spatial information of the source image, the image encoding result of the source image is accurately deformed and adjusted to guide other images to approach the image of the source image, thereby generating an image encoding result that can represent the three-dimensional spatial information of the image to be encoded. On the one hand, no complex geometry and texture information is required, thereby achieving efficient compression of the video. On the other hand, the image encoding result can also include the motion features in the image to be encoded, thereby improving the accuracy of video compression.

[0066] In an exemplary embodiment, the image coding result corresponding to the source image is adjusted according to the action difference features and the facial features of the source image, including: splicing the facial features and the action difference features of the source image to obtain a splicing result; performing a multi-scale transformation on the image coding result corresponding to the source image with the splicing result as a condition to obtain an adjusted image coding result.

[0067] The splicing result may refer to information obtained by splicing facial features and action difference features.

[0068] As an example, the server can use a feature splicing model (or a feature deformation model) to splice facial features and motion difference features, and use the splicing result as a condition to perform multi-scale transformation on the image encoding result corresponding to the source image, thereby optimizing the image quality of the encoded image. The feature splicing model can combine the splicing result with the image encoding result corresponding to the source image through a convolutional layer, a fully connected layer, or an attention mechanism, perform dynamic deformation, and generate an adjusted image encoding result. Specifically, the feature splicing model can first convert the facial features of the source image and the motion difference features between the source image and the image to be encoded into a form compatible with the image encoding result corresponding to the source image, and then use the converted facial features and motion difference features to determine the focus area in the source image, and use the attention mechanism to combine the converted facial features and motion difference features to determine the weight of the focus area. After that, the server can weight the data of the image encoding result corresponding to the source image in the focus area according to the weight to achieve dynamic deformation.

[0069] In this embodiment, the facial features and the motion difference features are spliced ​​to obtain a splicing result; the splicing result and the image coding result corresponding to the source image are fused. Since the image coding result corresponding to the source image can represent the high-level facial features and facial structure of the source image, the motion difference features are spliced ​​on the basis of the image coding result corresponding to the source image, and the image coding result corresponding to the source image is subjected to a multi-scale transformation using the splicing result. This enables the adjusted image coding result to include the high-level facial features and facial structure corresponding to the image to be encoded, thereby improving the accuracy of the adjusted image coding result (i.e., the image coding result corresponding to the image to be encoded), thereby improving the accuracy of the video compression.

[0070] In some embodiments, obtaining an image encoding result corresponding to a source image includes: obtaining image features of the source image; converting the image features of the source image into latent space features; and using the latent space features of the source image as the image encoding result corresponding to the source image.

[0071] The image feature of the source image may refer to information that characterizes the image content of the source image. In practical applications, the image feature of the source image may include deep features of the source image. The latent space feature of the source image may refer to information obtained after converting the image feature of the source image to a preset space. The preset space may refer to the latent space.

[0072] As an example, the server can use the convolutional layer and residual block of a deep residual network (such as ResNet18) for forward propagation to extract the image features of the source image, and map or convert the image features of the source image to a preset latent space to obtain the latent space features of the source image. The latent space features of the source image can be used as the image encoding result corresponding to the source image. At this time, the latent space features of the source image may not have actual physical meanings, but can be used as an abstract representation of facial features for model learning.

[0073] In this embodiment, by converting the image features of the source image into latent space features and generating an image coding result corresponding to the source image, it is possible to analyze the facial features and facial structure of the source image, gradually extract the deep features of the image, and obtain an image coding result as an abstract facial representation of the source image, thereby providing a data basis for subsequently generating an image coding result of the image to be encoded based on the image coding result of the source image, thereby improving the accuracy of video compression.

[0074] In some embodiments, extracting motion difference features between a source image and an image to be encoded includes: acquiring motion features of the source image and motion features of the image to be encoded; and analyzing the difference between the motion features of the source image and the motion features of the image to be encoded to obtain motion difference features.

[0075] Among them, the motion feature may refer to information characterizing the motion of a face or a person in an image. In practical applications, the motion of a face or a person can be decomposed into rotation, translation, and facial micro-motions, etc.

[0076] As an example, the server can use a motion feature extraction model (such as a deep residual network ResNet18) to obtain the motion features of the source image and the motion features of the image to be encoded, and analyze the differences between the motion features of the source image and the motion features of the image to be encoded at different motion decomposition angles to obtain motion difference features.

[0077] In this embodiment, by analyzing the differences in motion features between the source image and the image to be encoded, the motion difference features can be accurately obtained, providing a data basis for subsequently adjusting the image encoding results of the source image with the motion difference features and facial features, thereby improving the accuracy of the image encoding results of the image to be encoded and improving the accuracy of video compression.

[0078] In some embodiments, determining a source image and an image to be encoded from a target video frame includes: dividing the video into at least one video segment according to facial areas in each target video frame in the video and according to differences between facial areas; for each video segment, filtering out a source image and an image to be encoded from each video segment according to the facial areas in each video segment.

[0079] The facial region may be referred to as a facial detection result, which may refer to information obtained after performing face detection on each video frame in a video using a face detection algorithm. In practical applications, the facial detection result may include a face detection bounding box, which may describe or identify a face contour. The video clip may refer to a video obtained by dividing the video according to the difference between the facial detection results.

[0080] As an example, the server may use a face detection algorithm (Single Shot Scale-invariant Face Detector, S3FD) to perform face detection on each video frame in the target video frame, and extract a face detection bounding box as a face detection result. In a specific implementation, to save computing time, the server may start from the first frame of the video and obtain the face detection bounding box of the video frame at intervals of a preset number of frames. Then, the server may divide the video into at least one video segment based on the difference between the face detection results. In a specific implementation, the server may group video frames whose face detection results have a similarity greater than a first threshold into video segments, thereby dividing the video into several video segments. For each video segment, the server may filter out a source image and an image to be encoded from the video segment based on the face detection results of each video frame in the video segment. In a specific implementation, for each video segment, the server may use two video frames whose face detection results have a similarity greater than a second threshold as the source image and the image to be encoded, respectively.

[0081] In the present embodiment, based on the face detection results of each video frame in the video, the video is divided into a number of video segments, and based on the face detection results of the video frames in each video segment, the source image and the image to be encoded are screened out, and the face detection results of the video frames can be combined to segment the video based on the video frames with similar or identical face detection results, and the source image and the image to be encoded can be accurately determined from the video segments, so as to perform video compression on the video frames with similar or identical face detection results, thereby avoiding the source image and the image to be encoded corresponding to different objects, resulting in an erroneous image encoding result of the image to be encoded, thereby improving the accuracy of video compression.

[0082] In some embodiments, dividing the video into at least one video segment according to differences between facial regions of target video frames in the video includes:

[0083] According to the sequence of each target video frame in the video, the facial region of each target video frame is sorted to obtain a facial region sequence; the first target video frame in each target video frame is used as the video frame to be analyzed, and the intersection-and-union ratio between the facial region of the video frame to be analyzed and the facial regions of each target video frame after the video frame to be analyzed in the target video frame is determined; when the intersection-and-union ratio is greater than a preset intersection-and-union ratio threshold, and the time difference between the target video frames corresponding to the facial region associated with the intersection-and-union ratio is less than a preset time difference threshold, the facial region associated with the intersection-and-union ratio is removed from the facial region sequence to obtain a new facial region sequence, and the video frame to be analyzed is used as a video segmentation node;

[0084] The video frame corresponding to the first facial region in the new facial region sequence is used as a new video frame to be analyzed, and the step of determining the intersection-and-union ratio between the facial region of the video frame to be analyzed and the facial regions of each target video frame after the video frame to be analyzed is returned until the new facial region sequence is empty, and at least one video segmentation node is obtained; and the video is divided into at least one video segment according to each video segmentation node.

[0085] The IoU may refer to information about the degree of similarity between the face detection results of two video frames. In practical applications, the IoU may include the ratio between the intersection of the face detection bounding boxes of the two video frames and the union of the face detection bounding boxes of the two video frames. The preset IoU threshold may refer to information used to determine whether the degree of similarity between the face detection results of the video frames meets the preset similarity requirement. This condition may be called a content similarity condition.

[0086] The time difference may refer to the time length between two video frames. In practical applications, the time difference may include the frame number difference between video frames corresponding to the face detection results associated with the I / O information. The preset time difference threshold may refer to whether the time difference between video frames corresponding to the face detection results associated with the I / O information meets the preset time requirement. This condition may be called a close time difference condition.

[0087] The facial detection results to be removed may refer to facial detection results that need to be removed from the facial detection results of each video frame in the video. The filtered facial detection results may refer to a number of facial detection results obtained by removing the facial detection results to be removed from the facial detection results of each video frame in the video.

[0088] As an example, Figure 3As shown, a schematic diagram of dividing a video segment is provided, the server can sort the facial detection results of each video frame according to the frame order of the video frames in the video to obtain a facial detection result sequence, and then use the first facial detection result in the facial detection result sequence (such as facial detection result 1 of video frame 1) as a reference facial detection result (such as the facial detection result of the video frame to be analyzed). At this time, the reference facial detection result can be removed from the facial detection result sequence, and the server can traverse the facial detection results other than the reference facial detection result in the facial detection result sequence (such as facial detection result 2 of video frame 2-facial detection result n of video frame n), calculate the intersection-and-union ratio between the facial detection results other than the reference facial detection result in the facial detection result sequence and the reference facial detection result, if the intersection-and-union ratio is greater than a preset intersection-and-union ratio threshold (such as 0.5), and the time difference / distance between the video frames corresponding to the two facial detection results (including the reference facial detection result) associated with the intersection-and-union ratio is less than 0.5, At a preset time difference threshold (such as 5 frames), the server determines that two facial detection results associated with the intersection-and-union ratio overlap, and the server removes the facial detection result associated with the intersection-and-union ratio from the facial detection result sequence to obtain a new facial detection result sequence. At this time, the video frame to be analyzed can be used as the first video segmentation node, and then the server can use the video frame corresponding to the facial detection result after the facial detection result of the video frame to be analyzed in the new facial detection result sequence as the new video frame to be analyzed, determine the intersection-and-union ratio between the facial detection result of the video frame to be analyzed at this time and the facial detection result of the video frame after the video frame to be analyzed in the target video frame, and determine whether the facial detection result associated with the intersection-and-union ratio at this time needs to be removed from the facial detection result sequence, until the last video frame in the target video frame is used as the video frame to be analyzed (or the facial detection result sequence is empty). Each video frame to be analyzed in the process can be used as a video segmentation node, so at least one video segmentation node can be obtained. It can be understood that when all facial monitoring results except the reference facial detection result in the facial detection result sequence cannot simultaneously satisfy the conditions that the intersection-and-union ratio between them and the reference facial detection result is greater than a preset intersection-and-union ratio threshold (such as 0.5), and the time difference / distance between the video frames corresponding to the two facial detection results (including the reference facial detection result) associated with the intersection-and-union ratio is less than a preset time difference threshold (such as 5 frames) (that is, the content similarity condition and the time difference close condition), the server can remove the reference facial detection result at this time from the facial detection result sequence to obtain a new facial detection result sequence. At this time, the video frame corresponding to the reference facial detection result can be used as a video segmentation node.

[0089] For example, a video includes 30 target video frames. The intersection-and-union ratio between the face detection result 1 of the first video frame (i.e., video frame 1) and the face detection result 2 of the second video frame (i.e., video frame 2) in the video is greater than a preset intersection-and-union ratio threshold (e.g., 0.5), and since the time difference / distance between video frame 1 and video frame 2 is less than a preset time difference threshold (e.g., 5 frames), the server can determine that the face detection result 1 of video frame 1 and the face detection result 2 of video frame 2 need to be removed. Similarly, assuming that the face detection result 3 of video frame 3 needs to be removed, the server can use face detection result 1, face detection result 2, and face detection result 3 as face detection results to be removed, and use the first video frame (i.e., video frame 1) as the first video segmentation node. At this time, after the face detection result to be removed is removed from the sorting result corresponding to the face detection result of the video, the actual first face detection result in the new face detection result is face detection result 4 of video frame 4. Therefore, the second segmentation node of the video is video frame 4, and so on, until the last segmentation node of the video is video frame 30.

[0090] In a specific implementation, the server may also calculate the intersection-and-union ratio between each face detection result in the new face detection result, determine a new face detection result to be removed and remove it, and repeat the steps until the face detection result sequence is empty. In this case, several video segmentation nodes may be obtained in the process, and the server may divide the video into at least one video segment according to each video segmentation node. Specifically, the video frames between two adjacent video segmentation nodes may form a video segment.

[0091] When processing facial detection results, the position of the facial detection results on the timeline is fully considered when analyzing the differences between facial detection results in combination with the intersection-in-union ratio and the time difference threshold, so as to avoid using two video frames with large differences in image content / facial detection results as source images and target images for video compression, thereby reducing false detections. At the same time, the intersection-in-union ratio and the time difference threshold are used to divide video frames with similar or identical facial detection results into the same video segment. Therefore, when segmenting the video based on the facial detection results to obtain video segments, the position of the facial detection results on the timeline is fully considered, which helps to reduce false detections and repeated detections and improve the accuracy and stability of target detection.

[0092] In this embodiment, the facial detection results are filtered by the intersection-and-union ratio between the facial detection results, and the video is segmented based on the filtered facial detection results. This can fully consider the position of the facial detection results on the timeline of the video, reduce false detections and duplicate detections, obtain accurate video segments, and provide a data basis for subsequently obtaining target video frames based on each video segment.

[0093] In some embodiments, after the step of using the adjusted image coding result as the image coding result corresponding to the image to be encoded, the above method also includes: obtaining a video coding result based on at least the image coding result corresponding to the source image and the image coding result corresponding to the image to be encoded; sending the video coding result to the image decoding end; wherein the image decoding end is used to upsample and map the image coding result corresponding to the image to be encoded according to the facial features and motion difference features in the video coding result to obtain a decoded image corresponding to the image to be encoded.

[0094] The image decoding end may refer to a data processing end for decoding an image encoding result.

[0095] As an example, the image encoding result corresponding to the image to be encoded can be used as the compression result of the image to be encoded. Based on the source image in the video, the video frames except the source image in the video are used as the image to be encoded. The image encoding results of each image to be encoded are determined by the above-mentioned video encoding method. The video can be compressed to obtain the video compression result. It can be understood that the video compression result can include the image encoding result of the source image, the image encoding results of each image to be encoded, the facial features of the source image, and the action difference features between the source image and each image to be encoded. The server can form a video encoding result based on the image encoding result corresponding to the source image and the image encoding result corresponding to the image to be encoded, and then send the video encoding result to the image decoding end (in real time). When the video compression result (such as the video encoding result) needs to be decompressed (or decoded), the image decoding end can obtain the image encoding result of the source image, the image encoding results of each image to be encoded, the facial features of the source image, and the action difference features between the source image and each image to be encoded based on the video encoding result, and use the decoding model / decoder to decode the image encoding result corresponding to the image to be encoded. In a specific implementation, the image decoding end can use a transposed convolution (also called deconvolution or upsampling) layer to upsample the image encoding result corresponding to the image to be encoded according to facial features and motion difference features, gradually increase the spatial dimension of the feature map, and reduce the number of channels to obtain upsampling information, and map the upsampling information to a preset color space to obtain a decoded image corresponding to the image to be encoded. The decoded image is usually the same size as the image to be encoded.

[0096] In this embodiment, facial features and motion difference features are used to decode the image encoding result corresponding to the image to be encoded in the decoding stage. The facial features and motion difference features in the encoding stage can be combined with the image encoding result corresponding to the image to be encoded, the image detail information is retained, and an accurate decoded image is obtained, thereby improving the accuracy of video compression.

[0097] In one of the exemplary embodiments, a method for training a video processing model is provided, the video processing model including a facial feature extraction model, an action feature extraction model, a feature deformation model and an encoder, and the method is applied to a server as an example for explanation, including: inputting a first sample image into an encoder to be trained to obtain an image encoding result corresponding to the first sample image, and decoding the image encoding result corresponding to the first sample image to obtain an image decoding result corresponding to the first sample image; training the encoder to be trained according to the difference between the first sample image and the image decoding result corresponding to the first sample image to obtain a trained encoder; the trained encoder is used to obtain the image encoding result corresponding to the image; training the facial feature extraction model to be trained according to the first sample image; the trained facial feature extraction model is used to extract the facial features of the image; obtaining a second sample image different from the first sample image, extracting the action features of the first sample image and the action features of the second sample image respectively through the action feature extraction model to be trained, and determining the action features of the first sample image and the second sample image. The motion difference feature between the motion features of the sample images is used to train the motion feature extraction model to be trained according to the motion difference feature to obtain the trained motion feature extraction model; the trained motion feature extraction model is used to obtain the motion features of the image; the image coding result corresponding to the first sample image is adjusted according to the motion difference feature between the motion features of the first sample image and the motion features of the second sample image through the feature deformation model to be trained to obtain the image coding result corresponding to the second sample image, the image coding reference corresponding to the second sample image is obtained, and the feature deformation model to be trained is trained according to the image coding reference and the image coding result corresponding to the second sample image to obtain the trained feature deformation model; the trained feature deformation model is used to adjust the image coding result of the image; the trained video processing model is obtained according to the trained encoder, the trained facial feature extraction model, the trained motion feature extraction model and the trained feature deformation model; the trained video processing model is used to adjust the image coding result corresponding to the first sample image to obtain the image coding result corresponding to the second sample image.

[0098] Among them, the facial feature extraction model may refer to a model used to extract facial features of an image.

[0099] The motion feature extraction model may refer to a model used to extract motion features of an image.

[0100] The image coding reference corresponding to the second sample image may refer to pre-marked information used to characterize the image coding of the second sample image. In practical applications, the image coding reference corresponding to the second sample image may be pre-set known information.

[0101] As an example, the server may utilize the video processing model to implement encoding and decoding (such as compression and decompression) of the video. In order to optimize the model performance of the video processing model, the server may train the video processing model.

[0102] Specifically, the server can input the first sample image into the encoder to be trained to obtain the image encoding result corresponding to the first sample image, and input the image encoding result corresponding to the first sample image into the decoder to be trained to obtain the image decoding result corresponding to the first sample image, wherein the encoder can compress the input sample image (face image) into a point in a latent space, and the decoder can reconstruct the sample image from the point, and the server can train the encoder to be trained and the decoder to be trained based on the difference between the first sample image and the image decoding result corresponding to the first sample image, to obtain the trained encoder and the trained decoder.

[0103] After the encoder and decoder are trained, the server can obtain a second sample image different from the first sample image, and train the facial feature extraction model to be trained and the motion feature extraction model to be trained according to the first sample image and the second sample image to obtain the trained facial feature extraction model and the trained motion feature extraction model.

[0104] In practical applications, the facial feature extraction model can extract the facial features of the first sample image, and the server can combine the facial features of the first sample image and the facial feature labels of the first sample image to train the facial feature extraction model to be trained, and obtain the trained facial feature extraction model. The motion feature extraction model can extract the motion features of the first sample image and the motion features of the second sample image, and the video processing model analyzes the difference between the motion features of the first sample image and the motion features of the second sample image to obtain the motion difference features between the first sample image and the second sample image, and then the server can splice the facial features of the first sample image and the motion difference features between the first sample image and the second sample image through the feature deformation model to obtain the splicing result, and based on the splicing result as a condition, perform multi-scale transformation on the image encoding result of the first sample image after training The encoder obtains an adjusted image encoding result, and the adjusted image encoding result can be used as the image encoding result of the second sample image. The trained decoder can decode the image encoding result of the second sample image to obtain the image decoding result corresponding to the second sample image, and the server can train the motion feature extraction model to be trained according to the motion difference features between the motion features of the first sample image and the motion features of the second sample image to obtain a trained motion feature extraction model.

[0105] In practical applications, the server can adjust the image encoding result corresponding to the first sample image according to the action difference characteristics between the action characteristics of the first sample image and the action characteristics of the second sample image through the feature deformation model to be trained, and obtain the image encoding result corresponding to the second sample image. Then the server can obtain the image encoding reference corresponding to the second sample image, and train the feature deformation model to be trained according to the image encoding reference and the image encoding result corresponding to the second sample image to obtain the trained feature deformation model. Then the server can generate a trained video processing model according to the trained encoder, the trained decoder, the trained facial feature extraction model, the trained action feature extraction model and the trained feature deformation model.

[0106] In this embodiment, by training the encoder, decoder, facial feature extraction model and motion feature extraction model in stages respectively, it is possible to ensure that the encoder, decoder, facial feature extraction model and motion feature extraction model are fully trained, accelerate model convergence, and obtain an accurate video processing model, thereby improving the accuracy of the encoding results output by the video processing model.

[0107] In one of the exemplary embodiments, the video processing model also includes a discriminator, and the method also includes: obtaining a first expected value of the discriminator for the first sample image and a second expected value of the discriminator for the image decoding result corresponding to the first sample image, and determining an adversarial loss function value based on the first expected value and the second expected value; determining a reconstruction loss function value based on the second sample image and the image decoding result corresponding to the second sample image; determining a perceptual loss function value based on image features of the second sample image and image features of the image decoding result corresponding to the second sample image; determining a consistency loss function value based on action difference features between action features of the first sample image and the action features of the second sample image and action difference features between action features of the first sample image and the action features of the image decoding result corresponding to the second sample image; and training the video processing model based on the adversarial loss function value, the reconstruction loss function value, the perceptual loss function value and the consistency loss function value.

[0108] The first expected value may refer to data representing the probability that the first sample image is real data.

[0109] The second expected value may refer to data representing the probability that the image decoding result corresponding to the first sample image is true data.

[0110] As an example, the video processing model may also include a discriminator, and adversarial training may be performed on the encoder and the decoder based on the output result of the discriminator. Specifically, the encoder to be trained may output the image encoding result of the first sample image based on the first sample image, and the decoder to be trained may output the image decoding result of the first sample image based on the image encoding result of the first sample image. The discriminator may output a first expected value based on the first sample image, and the discriminator may output a second expected value based on the image decoding result of the first sample image. The server may respectively obtain the first expected value of the discriminator for the first sample image and the second expected value of the discriminator for the image decoding result corresponding to the first sample image, and calculate the adversarial loss function value based on the first expected value and the second expected value. When the adversarial loss function value is greater than a preset loss threshold, the server may continuously adjust the model parameters of the encoder to be trained and the decoder to be trained according to the adversarial loss function value until the adversarial loss function value is less than or equal to the preset threshold, at which time the server may obtain the trained encoder and the trained decoder.

[0111] It can be understood that, on the basis of training the video processing model based on the adversarial loss function value, the server can also determine the reconstruction loss function value according to the difference between the image decoding results corresponding to the second sample image and the second sample image. The server can also obtain the image features of the second sample image and the image features of the image decoding result corresponding to the second sample image through the image feature extraction model, and determine the perceptual loss function value according to the image features of the second sample image and the image features of the image decoding result corresponding to the second sample image. The server can also determine the consistency loss function value according to the action difference features between the action features of the first sample image and the action features of the second sample image and the action difference features between the action features of the first sample image and the action features of the image decoding result corresponding to the second sample image. Then the server can train the video processing model according to the adversarial loss function value, the reconstruction loss function value, the perceptual loss function value and the consistency loss function value to obtain the trained video processing model, and the trained video processing model can be used to adjust the image encoding result corresponding to the source image to obtain the image encoding result corresponding to the coded image.

[0112] In this embodiment, adversarial training of the encoder and decoder is performed in combination with the discriminator, which can improve the model robustness and generalization ability of the encoder and decoder, thereby improving the accuracy of the encoding and decoding results.

[0113] In some embodiments, Figure 4As shown, a flow chart of video compression is provided, the server can obtain the source image and the image to be encoded (such as the driving image) from the video, the server can input the source image into the image coding model, the facial feature extraction model and the motion feature extraction model, the image coding model (such as the variational auto-encoder, VAE) outputs the image coding result of the source image (a latent space representation), the facial feature extraction model outputs the facial features of the source image, the facial features can be one-dimensional vectors, the facial features can contain the key facial information of the source image, the motion feature extraction model outputs the motion features of the source image, the server can input the image to be encoded into the motion feature extraction model, and the motion feature extraction model outputs the motion features of the image to be encoded.

[0114] In practical applications, the action feature extraction model can decompose the action into rotation, translation and facial micro-movement, so the action feature can include three one-dimensional vectors. The server analyzes the difference between the action features of the source image and the action features of the image to be encoded, and obtains the action difference features (such as new three one-dimensional vectors). The server can input the facial features and action difference features of the source image into a feature stitching model (such as a U-Net network). The feature stitching model can stitch the facial features and action difference features of the source image, and fuse the information corresponding to the facial features and action difference features at multiple scales, and output the adjusted image encoding result. The adjusted image encoding result can be used as the latent space representation of the deformed image. At this time, the adjusted image encoding can be used as the image encoding result of the image to be encoded, and the image encoding result of the image to be encoded can be used as the compression result of the image to be encoded. The compression results of each image in the video are obtained in the above manner, and video compression can be achieved.

[0115] When it is necessary to decode / decompress the image encoding result of the image to be encoded, the server can combine the facial features of the source image and the motion difference features between the source image and the image to be encoded, and use the decoding model / decoder to decode the image encoding result corresponding to the image to be encoded, so as to obtain a decoded image corresponding to the image to be encoded. By decoding / decompressing the compression results of each image in the video one by one in the above manner, video decompression can be achieved, thereby efficiently compressing the portrait video and generating a realistic and accurate portrait video stream according to the portrait motion in real time.

[0116] In some embodiments, in order to optimize the performance of the image coding model, facial feature extraction model, motion feature extraction model, feature splicing model and decoding model / decoder, the server needs to train the image coding model, facial feature extraction model, motion feature extraction model, feature splicing model and decoding model / decoder. The server first trains the image coding model and decoding model / decoder.

[0117] like Figure 5 As shown, a flow chart of training an image coding model is provided, where a face image (such as a source image) can be represented as x, and the server uses a training data set containing a face image to train an image coding model (such as a variational autoencoder), and the face image is input into the image coding model. The image coding model generates a Gaussian distribution for the face image x, and the parameters of the Gaussian distribution may include μ (the mean calculated based on the face image) and σ (the standard deviation calculated based on the face image), and generates a reconstructed image based on the latent vector z obtained by sampling using a decoding model / decoder , the image coding model and decoding model / decoder are trained with the goal of reconstructing images as close to face images as possible, and the model parameters are optimized by minimizing the reconstruction error and KL divergence. The loss function of the image coding model can be expressed as:

[0118] .

[0119] After training, the image coding model can compress the input face image into a point in a latent space (i.e., the image coding result), and the decoder can reconstruct the face image from the point. The trained image coding model can be used as a fixed feature extractor to initialize the facial feature extraction model, action feature extraction model, and feature splicing model.

[0120] In some embodiments, after obtaining the trained image encoding model and the trained decoding model, the server can use the sample video to train the facial feature extraction model, the motion feature extraction model and the feature splicing model. Specifically, the server can perform data preprocessing on the sample video, divide the sample video into several video frames, select one frame from the several video frames as the starting frame, start from the starting frame, perform face detection on the video frame at intervals of a preset number of frames, obtain the face detection bounding box corresponding to the video frame, and then the server uses a non-maximum suppression algorithm to merge the face detection bounding boxes: sort all detected face detection bounding boxes according to the frame order of the face detection bounding boxes in the sample video, select the first face detection bounding box from the sorted bounding box list as the reference bounding box, remove it from the list, traverse the remaining face detection bounding boxes, and calculate the intersection-union ratio between the remaining face detection bounding boxes and the reference bounding box. If the IoU is greater than a preset threshold (e.g., 0.5), and the distance between the face detection bounding box corresponding to the IoU greater than the preset threshold and the reference bounding box on the time axis is less than a preset time threshold (e.g., 5 frames), the two bounding boxes are considered to overlap, and the currently traversed bounding box is removed from the list. The above steps are repeated until the bounding box list is empty. The server then divides the sample video into several video clips (Clips) based on the changes in the face detection bounding box on the time axis corresponding to the sample video. The face detection bounding boxes used in the same video clip can be the same.

[0121] For each video clip, the server can randomly select two pictures as the source image and the image to be encoded (or driving image). The source image and the image to be encoded can be used to train the facial feature extraction model, the motion feature extraction model and the feature splicing model. The facial feature extraction model, the motion feature extraction model and the feature splicing model can learn the decoupled motion and expression information.

[0122] In one of the exemplary embodiments, the image coding model (or source image feature coding module) can extract features that can represent three-dimensional spatial information from a 2D image. The input format of the image coding model is [3, 512, 512], where 3 represents the color channel (such as RGB) and 512 is the resolution of the image. The image coding model can perform 2D convolution and downsampling, feature matrix transformation, reshape operation and 3D convolution on the source image. Among them, 2D convolution and downsampling can extract local features of the image through multiple 2D convolution layers. The convolution layer can capture basic visual patterns such as edges and textures in the image. The downsampling (or pooling) layer is used to reduce the spatial dimensions (i.e., width and height) of the feature map while increasing the number of channels of the feature map, which helps the model capture higher-level visual information and reduces the amount of calculation; the feature matrix transformation can convert the image into a higher-dimensional feature matrix through a series of 2D convolutions and downsampling, such as [512, 32, 32], where 512 represents the number of channels of the feature map and 32x32 is the resolution of the feature map after downsampling; the reshaping operation can reshape the feature matrix of [512, 32,32] into [16, 32, 32, 32], this step is to convert 2D features into 3D features, where 32 can be seen as a feature dimension representing depth, in order to simulate the 3D spatial structure; using the 3D convolution layer, features can be further extracted to obtain image encoding results. The 3D convolution not only considers the width and height of the image, but also the depth (here it is the dimension added by the reshaping operation), which can help the model better understand and simulate the three-dimensional spatial information of the image.

[0123] The motion feature extraction model can decompose the motion of the source image into rotation, translation and facial micro-movement. The motion feature extraction model can input the image into the multi-output ResNet18, which can realize the encoding of the corresponding features, generate rotation motion features, translation motion features and facial micro-movement features, and then obtain the motion features of the image.

[0124] The feature splicing model can be used as an image deformation model, which aims to accurately deform the spatial features of the input image (such as the image encoding result of the source image) according to the conditional features. That is, it accepts a high-dimensional spatial feature tensor [16, 32,32, 32] as the basic image content and four

[256] -dimensional conditional vectors (such as the splicing result) as input. These conditional vectors can represent facial feature abstraction, rotation, translation and facial micro-motion respectively. The feature splicing model can include a series of network layers, such as convolutional layers, fully connected layers or attention mechanisms, which combine the conditional vectors with spatial features for dynamic deformation and generate a new [16, 32, 32, 32] feature tensor. The feature tensor encodes the high-level facial features and structural information of the target image. The feature tensor can be used as the image encoding result corresponding to the image to be encoded.

[0125] The decoding model / decoder can convert the high-level abstract information after feature encoding back into a visual image format, that is, it receives the high-dimensional feature representation from the feature splicing model (such as the image encoding result of the image to be encoded), such as a feature tensor of the shape of [16,32, 32, 32], and uses a transposed convolution (also called deconvolution or upsampling) layer to gradually increase the spatial dimension of the feature map while reducing the number of channels. It may include skip connections or residual connections, which combine the intermediate features of the encoding stage with the features of the decoding stage to retain more detailed information. During the decoding process, nonlinear activation functions such as ReLU or Leaky ReLU are applied to increase the nonlinear expression capability. The last layer is usually a fully connected layer or a convolution layer that maps the features to the RGB color space of the image. The size of the output image is usually the same as the input image.

[0126] It can be understood that the image coding model, the facial feature extraction model, the action feature extraction model, the feature splicing model and the decoding model / decoder can constitute a video compression model, and the loss function of the video compression model may include reconstruction loss (Reconstruction Loss), perceptual loss (Perceptual Loss), adversarial loss (AdversarialLoss) and consistency loss (Consistency Loss), wherein the reconstruction loss is used to ensure that the generated image Igenerated (such as the decoded image corresponding to the image to be encoded) is as close as possible to the target image Itarget (such as the image to be encoded), and the pixel-level L1 loss (mean absolute error) or L2 loss (mean square error) can be used. The calculation expression of the reconstruction loss Lrec can be expressed as follows:

[0127] .

[0128] Perceptual loss is used to ensure that the generated image is similar to the target image in terms of high-level features (such as texture, edges, etc.). The calculation expression of perceptual loss Lpreceptual can be expressed as:

[0129] .

[0130] Here, F(x) may refer to a pre-trained feature extraction network for extracting image features of an image.

[0131] Adversarial loss involves the adversarial training process of a video compression model (generator) and a discriminator. The video compression model attempts to generate fake images that look like real images based on the image encoding results of the image through the decoding model / decoder, while the task of the discriminator is to distinguish between real images and fake images. The calculation expression of the adversarial loss Ladv can be expressed as:

[0132] .

[0133] Where D is the discriminator, G is the video compression model, pdata(x) is the distribution of real data, and pz(z) is the distribution of random noise.

[0134] The consistency loss is used to ensure that the action difference between the source image and the image to be encoded is correctly encoded and applied. For example, if the only difference between the source image Isource and the image to be encoded Itarget is the opening and closing of the eyes, then the generated decoded image Igenerated should only differ in the eye part. The calculation expression of the consistency loss can be expressed as:

[0135] .

[0136] Among them, Esource,target() can represent the action coding between the source image and the image to be encoded, Esource, generated() can represent the action coding between the source image and the decoded image, I source Can refer to the source image, I target It can refer to the image to be encoded, I generated It may refer to the decoded image corresponding to the image to be encoded.

[0137] In this embodiment, through the deep learning framework, the image is mapped into a 4-dimensional vector to achieve advanced feature mapping. The model can adapt to different deformation modules more effectively, thereby enhancing the scalability and flexibility of the model. The facial feature extraction model and the action feature extraction model are respectively used to learn action and expression information, thereby achieving action and expression decoupling, and improving the model's processing capabilities for complex portrait videos. Multi-scale network structures such as U-Net are used to achieve fine transformation of the image latent space, improve the details and quality of the generated image, thereby vectorizing the image features, achieving accurate decoupling of action and expression, and optimizing the image quality through a multi-scale network. In addition, the pre-trained variational autoencoder (VAE) enhances the recognition ability of facial features, and the application of a comprehensive loss function ensures the authenticity and accuracy of the generated image, which not only improves the compression efficiency of the video, but also ensures real-time response capabilities under various network conditions, meeting the growing needs of remote work and online communication.

[0138] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0139] Based on the same inventive concept, the embodiment of the present application also provides a video encoding device for implementing the above-mentioned video encoding method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in the one or more video encoding device embodiments provided below can refer to the above-mentioned limitations on the video encoding method, and will not be repeated here.

[0140] In an exemplary embodiment, Figure 6 As shown, a video encoding device is provided, including: a determination module 602, an extraction module 604 and an adjustment module 606, wherein:

[0141] The determination module 602 is used to identify a target video frame in which a face appears in the video, and determine a source image and an image to be encoded from the target video frame; the face area of ​​the source image and the face area of ​​the image to be encoded meet a preset similarity condition.

[0142] The extraction module 604 is used to determine the motion difference features between the motion features of the source image and the motion features of the image to be encoded, and to extract the facial features of the source image.

[0143] The adjustment module 606 is used to obtain the image coding result corresponding to the source image, adjust the image coding result corresponding to the source image according to the action difference feature and the facial feature of the source image, and use the adjusted image coding result of the source image as the image coding result corresponding to the image to be encoded.

[0144] In one of the exemplary embodiments, the adjustment module 606 is further specifically used to splice the facial features of the source image and the action difference features to obtain a splicing result; and to perform a multi-scale transformation on the image encoding result corresponding to the source image based on the splicing result as a condition to obtain the adjusted image encoding result.

[0145] In one of the exemplary embodiments, the above-mentioned determination module 602 is specifically used to divide the video into at least one video segment according to the difference between the facial areas in each video frame in each of the target video frames in the video; for each of the video segments, the source image and the image to be encoded are filtered out from each of the video segments according to the facial detection result in each of the video segments.

[0146] In one of the exemplary embodiments, the determination module 602 is further used to sort the facial regions of the target video frames according to the sequence of the target video frames in the video to obtain a facial region sequence; take the first target video frame in the target video frames as the video frame to be analyzed, and determine the intersection-and-union ratio between the facial region of the video frame to be analyzed and the facial regions of the target video frames after the video frame to be analyzed; when the intersection-and-union ratio is greater than a preset intersection-and-union ratio threshold, and the time difference between the video frames corresponding to the facial region associated with the intersection-and-union ratio is less than a preset time difference threshold, remove the facial region associated with the intersection-and-union ratio from the facial region sequence to obtain a new facial region sequence, and take the video frame to be analyzed as a video segmentation node; take the video frame corresponding to the first facial region in the new facial region sequence as the new video frame to be analyzed, and return to execute the step of determining the intersection-and-union ratio between the facial region of the video frame to be analyzed and the facial regions of the target video frames after the video frame to be analyzed, until the new facial region sequence is empty, and at least one video segmentation node is obtained; and divide the video into at least one video segment according to each video segmentation node.

[0147] In one of the exemplary embodiments, the above-mentioned device also includes a sending module, which is specifically used to obtain a video encoding result based on at least the image encoding result corresponding to the source image and the image encoding result corresponding to the image to be encoded; and send the video encoding result to an image decoding end; wherein the image decoding end is used to upsample and map the image encoding result corresponding to the image to be encoded according to the facial features and the action difference features in the video encoding result to obtain a decoded image corresponding to the image to be encoded.

[0148] Each module in the above video encoding device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0149] In an exemplary embodiment, a training device for a video processing model is provided, wherein the video processing model includes a facial feature extraction model, an action feature extraction model, a feature deformation model and an encoder, including: a coding and decoding result acquisition module, a first training module, a second training module and a model generation module, wherein:

[0150] The encoding and decoding result acquisition module is used to input the first sample image into the encoder to be trained, obtain the image encoding result corresponding to the first sample image, and decode the image encoding result corresponding to the first sample image to obtain the image decoding result corresponding to the first sample image; according to the difference between the first sample image and the image decoding result corresponding to the first sample image, train the encoder to be trained to obtain the trained encoder; the trained encoder is used to obtain the image encoding result corresponding to the image.

[0151] The first training module is used to train the facial feature extraction model to be trained according to the first sample image; the trained facial feature extraction model is used to extract facial features of the image.

[0152] The second training module is used to obtain a second sample image different from the first sample image, and train a facial feature extraction model to be trained according to the first sample image; the trained facial feature extraction model is used to extract facial features of the image; the action features of the first sample image and the action features of the second sample image are respectively extracted by the action feature extraction model to be trained, and the action difference features between the action features of the first sample image and the action features of the second sample image are determined, and the action feature extraction model to be trained is trained according to the action difference features to obtain the trained action feature extraction model; the trained action feature extraction model is used to obtain the action features of the image.

[0153] The third training module is used to adjust the image coding result corresponding to the first sample image according to the motion difference characteristics between the motion characteristics of the first sample image and the motion characteristics of the second sample image through the feature deformation model to be trained, obtain the image coding result corresponding to the second sample image, obtain the image coding reference corresponding to the second sample image, and train the feature deformation model to be trained according to the image coding reference and the image coding result corresponding to the second sample image to obtain the trained feature deformation model; the trained feature deformation model is used to adjust the image coding result of the image.

[0154] A model generation module is used to obtain a trained video processing model based on the trained encoder, the trained facial feature extraction model, the trained motion feature extraction model and the trained feature deformation model; the trained video processing model is used to adjust the image encoding result corresponding to the first sample image to obtain the image encoding result corresponding to the second sample image.

[0155] In one of the exemplary embodiments, the video processing model also includes a discriminator, and the above-mentioned device includes a loss function module, which is specifically used to obtain a first expected value of the discriminator for the first sample image and a second expected value of the discriminator for the image decoding result corresponding to the first sample image, and determine an adversarial loss function value according to the first expected value and the second expected value; determine a reconstruction loss function value according to the second sample image and the image decoding result corresponding to the second sample image; determine a perceptual loss function value according to image features of the second sample image and image features of the image decoding result corresponding to the second sample image; determine a consistency loss function value according to action difference features between action features of the first sample image and action features of the second sample image and action difference features between action features of the first sample image and action features of the image decoding result corresponding to the second sample image; train the video processing model according to the adversarial loss function value, the reconstruction loss function value, the perceptual loss function value and the consistency loss function value.

[0156] Each module in the training device of the video processing model can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0157] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a video encoding method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0158] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0159] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0160] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0161] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0162] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0163] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0164] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0165] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video encoding method, characterized in that: The method comprises: Identify a target video frame in which a face appears in the video, and determine a source image and an image to be encoded from the target video frame; the face region of the source image and the face region of the image to be encoded satisfy a preset similarity condition; Determining motion difference features between motion features of the source image and motion features of the image to be encoded, and extracting facial features of the source image; An image coding result corresponding to the source image is obtained, the image coding result corresponding to the source image is adjusted according to the action difference feature and the facial feature of the source image, and the adjusted image coding result of the source image is used as the image coding result corresponding to the image to be encoded.

2. The method according to claim 1, characterized in that The adjusting the image encoding result corresponding to the source image according to the action difference feature and the facial feature of the source image comprises: splicing the facial features of the source image and the action difference features to obtain a splicing result; The image encoding result corresponding to the source image is subjected to multi-scale transformation with the splicing result as a condition to obtain the adjusted image encoding result.

3. The method according to claim 1, characterized in that: The step of determining a source image and an image to be encoded from the target video frame comprises: Dividing the video into at least one video segment according to differences between facial regions of the target video frames in the video; For each of the video segments, the source image and the image to be encoded are screened out from each of the video segments according to the facial area in each of the video segments.

4. The method according to claim 3, characterized in that The step of dividing the video into at least one video segment according to the difference between the facial regions of the target video frames in the video comprises: Sort the facial regions of the target video frames according to the sequence of the target video frames in the video to obtain a facial region sequence; Taking the first target video frame among the target video frames as the video frame to be analyzed, and determining the intersection-over-union ratio between the facial region of the video frame to be analyzed and the facial region of each target video frame after the video frame to be analyzed; When the IoU is greater than a preset IoU threshold and the time difference between the target video frames corresponding to the facial region associated with the IoU is less than a preset time difference threshold, the facial region associated with the IoU is removed from the facial region sequence to obtain a new facial region sequence, and the video frame to be analyzed is used as a video segmentation node; The video frame corresponding to the first facial region in the new facial region sequence is used as a new video frame to be analyzed, and the step of determining the intersection-and-union ratio between the facial region of the video frame to be analyzed and the facial regions of each target video frame after the video frame to be analyzed is returned to be executed until the new facial region sequence is empty, and at least one video segmentation node is obtained; The video is divided into at least one video segment according to each of the video segmentation nodes.

5. The method according to claim 1, characterized in that: After the step of using the adjusted image encoding result as the image encoding result corresponding to the image to be encoded, the method further includes: Obtaining a video encoding result at least according to the image encoding result corresponding to the source image and the image encoding result corresponding to the image to be encoded; and sending the video encoding result to an image decoding end; The image decoding end is used to perform upsampling and mapping processing on the image encoding result corresponding to the image to be encoded according to the facial features and action difference features in the video encoding result, so as to obtain a decoded image corresponding to the image to be encoded.

6. A video processing model training method, characterized in that: The video processing model includes a facial feature extraction model, an action feature extraction model, a feature deformation model and an encoder, and the method includes: Inputting a first sample image to an encoder to be trained, obtaining an image encoding result corresponding to the first sample image, and decoding the image encoding result corresponding to the first sample image to obtain an image decoding result corresponding to the first sample image; training the encoder to be trained according to a difference between the first sample image and the image decoding result corresponding to the first sample image to obtain a trained encoder; the trained encoder is used to obtain an image encoding result corresponding to an image; Training a facial feature extraction model to be trained according to the first sample image; the trained facial feature extraction model is used to extract facial features of the image; Acquire a second sample image different from the first sample image, respectively extract the motion features of the first sample image and the motion features of the second sample image through the motion feature extraction model to be trained, and determine the motion difference features between the motion features of the first sample image and the motion features of the second sample image, and train the motion feature extraction model to be trained according to the motion difference features to obtain a trained motion feature extraction model; the trained motion feature extraction model is used to obtain the motion features of the image; adjusting the image coding result corresponding to the first sample image according to the action difference feature between the action feature of the first sample image and the action feature of the second sample image through the feature deformation model to be trained, obtaining the image coding result corresponding to the second sample image, obtaining the image coding reference corresponding to the second sample image, and training the feature deformation model to be trained according to the image coding reference and the image coding result corresponding to the second sample image to obtain the trained feature deformation model; the trained feature deformation model is used to adjust the image coding result of the image; A trained video processing model is obtained according to the trained encoder, the trained facial feature extraction model, the trained motion feature extraction model and the trained feature deformation model.

7. The method according to claim 6, characterized in that The video processing model also includes a discriminator, and the method further includes: Obtaining a first expected value of the discriminator for the first sample image and a second expected value of an image decoding result corresponding to the first sample image, and determining an adversarial loss function value according to the first expected value and the second expected value; Determining a reconstruction loss function value according to the second sample image and an image decoding result corresponding to the second sample image; determining a perceptual loss function value according to an image feature of the second sample image and an image feature of an image decoding result corresponding to the second sample image; determining a consistency loss function value according to an action difference feature between an action feature of the first sample image and an action feature of the second sample image and an action difference feature between an action feature of the first sample image and an action feature of an image decoding result corresponding to the second sample image; The video processing model is trained according to the adversarial loss function value, the reconstruction loss function value, the perceptual loss function value and the consistency loss function value.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images