Video encoding method, apparatus and storage medium
Patent Information
- Application Number
- HK62022064902
- Authority / Receiving Office
- HK · HK
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-09-30
- Filing Date
- 2022-11-30
- Publication Date
- 2026-08-28
AI Technical Summary
Existing video coding technologies based on deep neural networks suffer from insufficient compressibility, poor accuracy, and unnecessary information loss in video conferencing. They are particularly ineffective in handling occlusion and large movements, resulting in insufficient fidelity in face reconstruction.
By detecting facial landmark features in video data, a neural network is used to encode the set of facial landmark features, and an adversarial generative network is used to reconstruct the features of the extended facial region. Combined with downsampling and upsampling techniques, efficient encoding of video data is achieved.
It improves the compressibility and accuracy of video encoding, ensures the fidelity of face reconstruction, adapts to occlusion and large movements, and enhances the quality of video conferencing.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[0001] Cross-references to related applications
[0002] This application claims priority to U.S. Provisional Application 63 / 134,522, filed January 6, 2021, and U.S. Application 17 / 490,103, filed September 30, 2021, the entire contents of which are incorporated herein by reference. Technical Field
[0003] This disclosure relates to video coding methods, apparatus, and storage media that can restore real details from real low-quality (LQ) faces to high-quality (HW) faces based on landmark features. Background Technology
[0004] The International Organization for Standardization (ISO), the International Electrotechnical Commission (IEC), and the Institute of Electrical and Electronics Engineers (IEEE) are actively seeking AI-based video coding technologies, with a particular focus on deep neural network (DNN) technologies. Various ad hoc groups have been established to study neural network compression (NNR), video coding for machines (VCM), and neural network-based video coding (NNVC). The China Alliance for New Generation Artificial Intelligence Industry Technology Innovation (AITISA) and the Digital Audio and Video Coding Technology Standards Working Group (AVS) have also established corresponding expert groups to study the standardization of similar technologies.
[0005] Video conferencing has recently become increasingly important, often requiring low-bandwidth transmission to support joint meetings of multiple end users. Compared to typical video compression tasks, videos in conferencing scenarios primarily feature similar content: one or more speakers who are the main subject of the video and occupy a significant portion of the overall scene. The unconstrained background can be arbitrarily complex, indoor or outdoor, but this is less critical. Recently, Nvidia's Maxine video conferencing platform proposed an AI-based framework based on face replay technology. 2D or 3D facial landmarks (e.g., any one or more of nose, chin, eyes, proportions, position, wrinkles, ears, geometry, etc.) are extracted from a DNN to capture pose and emotional information of the human face. These features, along with high-quality features computed at low frequency to capture the shape and texture of the face, are sent to the decoder side, where a high-quality face is reconstructed by transmitting shape and texture based on pose and expression information from each reconstructed frame. This framework significantly reduces transmission bit consumption because, for most frames, only landmark features related to pose and expression are transmitted instead of raw pixels. However, a redefined framework cannot guarantee fidelity to the original face appearance and can lead to dramatic artifacts in many cases. For example, the framework is often very sensitive to occlusion, large movements, etc., and cannot be robustly used in practical video conferencing products.
[0006] Therefore, there are also technical shortcomings, including a lack of compressibility, accuracy, and unnecessary discarding of other information related to neural networks. Summary of the Invention
[0007] According to an exemplary embodiment, a method and apparatus are included, the method and apparatus comprising a memory and one or more processors, the memory configured to store computer program code, and the one or more processors configured to access the computer program code and operate according to the instructions of the computer program code. The computer program code includes: acquisition code configured to cause the at least one processor to acquire video data; detection code configured to cause the at least one processor to detect at least one face from at least one frame of the video data; determination code configured to cause the at least one processor to determine a set of face landmarks of the at least one face from the at least one frame of the video data; and encoding code configured to cause the at least one processor to encode the video data, at least partially, by a neural network based on the determined set of face landmarks.
[0008] According to an exemplary embodiment, the video data includes an encoded bitstream of the video data, and determining the face landmark set includes upsampling at least one downsampled sequence, the at least one downsampled sequence being obtained by decompressing the encoded bitstream.
[0009] According to an exemplary embodiment, the computer program code further includes determining code configured to cause the at least one processor to determine an extended face area (EFA), the extended face area including a boundary region extending from a region of the at least one face detected in the at least one frame of the video data; and to determine an EFA feature set from the EFA; and further encoding code configured to cause the at least one processor to encode the video data at least in part by the neural network based on the determined set of face landmark features.
[0010] According to an exemplary embodiment, determining the EFA and determining the EFA feature set includes upsampling the at least one downsampled sequence, which is obtained by decompressing the encoded bitstream.
[0011] According to an exemplary embodiment, determining the EFA and determining the EFA feature set further includes: reconstructing the EFA features by a generative adversarial network, wherein each of the EFA features corresponds to one of the face landmark features in the face landmark feature set.
[0012] According to an exemplary embodiment, encoding the video data by the neural network based on the determined set of face landmarks, at least in part, further includes: encoding the video data by the neural network based on the determined set of face landmark features by aggregating the set of face landmarks, the reconstructed EFA features, and an upsampled sequence obtained by upsampling the at least one downsampled sequence.
[0013] According to an exemplary embodiment, the at least one face from the at least one frame of the video data is determined to be the largest face among a plurality of faces in the at least one frame of the video data.
[0014] According to an exemplary embodiment, the determining code is further configured to cause the processor to: determine, for each of the plurality of faces in the at least one frame of the video data, a plurality of face landmark feature sets other than the face landmark feature set of the at least one face, from the at least one frame of the video data; and the encoding code is further configured to cause the processor to: encode the video data at least in part by the neural network based on the determined face landmark set and the determined plurality of face landmark feature sets. Attached Figure Description
[0015] Other features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, wherein:
[0016] Figure 1 This is a simplified illustration of a schematic diagram according to an embodiment.
[0017] Figure 2 This is a simplified illustration of a schematic diagram according to an embodiment.
[0018] Figure 3 This is a simplified illustration of a schematic diagram according to an embodiment.
[0019] Figure 4 This is a simplified illustration of a schematic diagram according to an embodiment.
[0020] Figure 5 This is a simplified illustration of a diagram according to an embodiment.
[0021] Figure 6 This is a simplified illustration of a diagram according to an embodiment.
[0022] Figure 7 This is a simplified illustration of a diagram according to an embodiment.
[0023] Figure 8 This is a simplified illustration of a diagram according to an embodiment.
[0024] Figure 9A This is a simplified illustration of a diagram according to an embodiment.
[0025] Figure 9B This is a simplified illustration of a diagram according to an embodiment.
[0026] Figure 10 This is a simplified illustration of a flowchart according to an embodiment.
[0027] Figure 11 This is a simplified illustration of a flowchart according to an embodiment.
[0028] Figure 12 This is a simplified illustration of a block diagram according to an embodiment.
[0029] Figure 13This is a simplified illustration of a block diagram according to an embodiment.
[0030] Figure 14 This is a simplified illustration of a schematic diagram according to an embodiment. Detailed Implementation
[0031] The features discussed below can be used individually or in any combination in any order. Furthermore, embodiments can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable storage medium.
[0032] Figure 1 A simplified block diagram of a communication system 100 according to an embodiment of the present disclosure is shown. The communication system 100 may include at least two terminals 102 and 103 interconnected via a network 105. For unidirectional data transmission, the first terminal 103 may encode video data locally for transmission to the other terminal 102 via the network 105. The second terminal 102 may receive the encoded video data from the other terminal from the network 105, decode the encoded data, and display the recovered video data. Unidirectional data transmission is common in media service applications and the like.
[0033] Figure 1 A second pair of terminals 101 and 104 is shown for supporting bidirectional transmission of encoded video, for example, during video conferencing. For bidirectional data transmission, each terminal 101 and 104 can encode video data acquired at a local location for transmission to the other terminal via network 105. Each terminal 101 and 104 can also receive encoded video data sent by the other terminal, decode the encoded data, and display the recovered video data on a local display device.
[0034] exist Figure 1 In this disclosure, terminals 101, 102, 103, and 104 may be represented as servers, personal computers, and smartphones, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 105 refers to any number of networks transmitting encoded video data between terminals 101, 102, 103, and 104, including, for example, wired and / or wireless communication networks. Communication network 105 may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet. For the purposes of this application, unless explained below, the architecture and topology of network 105 may be irrelevant to the operation of this disclosure.
[0035] As an example of the application of public topics Figure 2 The placement of a video encoder and decoder in a streaming environment is illustrated. The disclosed subject matter is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.
[0036] The streaming system may include an acquisition subsystem 203, which may include a video source 201, such as a digital camera, to create, for example, an uncompressed video sample stream 213. This sample stream 213 may be emphasized as having a high data volume compared to an encoded video stream and may be processed by an encoder 202 coupled to the camera 201. The encoder 202 may include hardware, software, or a combination thereof to allow implementation or execution of aspects of the disclosed subject matter as described in more detail below. An encoded video stream 204 may be stored on a streaming server 205 for future use; this encoded video stream 204 may be emphasized as having a lower data volume compared to the sample stream. One or more streaming clients 212 and 207 may access the streaming server 205 to retrieve copies 208 and 206 of the encoded video stream 204. Client 212 may include video decoder 211, which decodes an input copy of the encoded video stream 208 and creates an output video sample stream 210 that can be displayed on display 209 or other presentation device (not depicted). In some streaming systems, video streams 204, 206, and 208 may be encoded according to certain video encoding / compression standards. Examples of these standards have been mentioned above and are described further here.
[0037] Figure 3 This can be a functional block diagram of the video decoder 300 according to an embodiment of the present invention.
[0038] Receiver 302 can receive one or more codec video sequences to be decoded by decoder 300. In the same or another embodiment, one encoded video sequence is received at a time, with each encoded video sequence being decoded independently of the others. Encoded video sequences can be received from channel 301, which can be a hardware / software link to a storage device storing the encoded video data. Receiver 302 can receive encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams, which can be forwarded to their respective user entities (not depicted). Receiver 302 can separate the encoded video sequences from other data. To prevent network jitter, buffer memory 303 can be coupled between receiver 302 and entropy decoder / resolver 304 (hereinafter referred to as the "resolver"). Buffer 303 may not be necessary or can be made smaller when receiver 302 receives data from a store / forward device with sufficient bandwidth and controllability or from a synchronous network. For use on business packet networks such as the Internet, a buffer 303 may also be required, which may be relatively large and advantageously have an adaptive size.
[0039] Video decoder 300 may include parser 304 to reconstruct symbols 313 from an encoded video sequence. These symbols may include information for managing the operation of decoder 300, and potential information for controlling presentation devices such as display 312, which is not integral to the decoder but may be coupled to it. Control information for one or more presentation devices may be fragments of parameter sets (not depicted) of Supplemental Enhancement Information (SEI messages) or Video Usability Information (VUI). Parser 304 may perform parsing / entropy decoding on the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow various principles known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. Parser 304 may extract a subset of parameters from the encoded video sequence for at least one subset of pixels in a subgroup for use in the video decoder, based on at least one parameter corresponding to a group. Subgroups can include Group of Pictures (GOP), pictures, tiles, slices, macroblocks, Coding Units (CU), blocks, Transform Units (TU), Prediction Units (PU), and so on. The parser can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.
[0040] The parser 304 can perform entropy decoding / parsing operations on the video sequence received from the buffer 303 to create symbols 313. The parser 304 can receive encoded data and selectively decode specific symbols 313. Furthermore, the parser 304 can determine whether to provide specific symbols 313 to the motion compensation prediction unit 306, the scaler / inverse transform unit 305, the intra-frame prediction unit 307, or the loop filter 311.
[0041] Depending on the type of encoded video frames or a subset of encoded video frames (e.g., inter-frame and intra-frame frames, inter-frame and intra-frame blocks) and other factors, the reconstruction of symbol 313 may involve multiple different units. Which units are involved and how they are involved can be controlled by the subgroup control information parsed by the parser 304 from the encoded video sequence. For brevity, the flow of such subgroup control information between the parser 304 and the various units described below is not described.
[0042] In addition to the functional blocks already mentioned, the decoder 300 can be conceptually subdivided into several functional units as described below. In practical embodiments operating under commercial constraints, many of these units interact closely with each other and can be at least partially integrated with each other. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide into the functional units described below.
[0043] The first unit is the scaler / inverse transform unit 305. The scaler / inverse transform unit 305 receives quantization transform coefficients and control information for one or more symbols 313 from the parser 304, including the transform mode used, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit 305 can output a block containing sample values, which can be input into the aggregator 310.
[0044] The output samples of the scaler / inverse transform unit 305 may belong to intra-coded blocks, i.e., blocks that do not use predictive information from previously reconstructed images but can use predictive information from previously reconstructed portions of the current image. This predictive information may be provided by the intra-prediction unit 307. In some cases, the intra-prediction unit 307 uses surrounding reconstructed information extracted from the (partially reconstructed) current image 309 to generate blocks of the same size and shape as the block being reconstructed. In some cases, the aggregator 310 adds the predictive information already generated by the intra-prediction unit 307 to the output sample information provided by the scaler / inverse transform unit 305 based on each sample.
[0045] In other cases, the output samples of the scaler / inverse transform unit 305 may belong to inter-frame coding and latent motion compensation blocks. In this case, the motion compensation prediction unit 306 can access the reference image memory 308 to obtain samples for prediction. After motion compensation is performed on the extracted samples according to the symbols 313 belonging to the block, these samples can be added by the aggregator 310 to the output of the scaler / inverse transform unit (referred to as residual samples or residual signals in this case) to generate output sample information. The motion compensation unit's retrieval of predicted samples from addresses in the reference memory can be controlled by motion vectors, and these motion vectors are available to the motion compensation unit in the form of symbols 313, which may have, for example, X, Y, and reference image components. Motion compensation may also include interpolation of sample values extracted from the reference image memory, motion vector prediction mechanisms, etc., when using subsample precise motion vectors.
[0046] The output samples of aggregator 310 can undergo various loop filtering techniques in loop filter unit 311. Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video bitstream and used as symbols 313 from parser 304 for loop filter unit 311. However, video compression techniques may also respond to metadata acquired during decoding of a previous (in decoding order) portion of an encoded picture or encoded video sequence, and to previously reconstructed and loop-filtered sample values.
[0047] The output of the loop filter unit 311 can be a sample stream, which can be output to the rendering device 312 and stored in the reference image memory 557 for subsequent inter-frame prediction.
[0048] Once a certain encoded image is fully reconstructed, it can be used as a reference image for future predictions. For example, once the encoded image corresponding to the current image has been fully reconstructed and the encoded image (via, for example, parser 304) is identified as a reference image, the current reference image 309 can become part of the reference image buffer 308, and new current image memory can be reallocated before the reconstruction of subsequent encoded images begins.
[0049] The video decoder 300 can perform decoding operations according to a predetermined video compression technique, such as that specified in the ITU-T H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technique or standard in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the configuration file recorded in the video compression technique or standard. Specifically, the configuration file may select certain tools from all available tools in the video compression technique or standard as the only tools available under that configuration file. For compliance, the complexity of the encoded video sequence is also required to be within the limits defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (measured in megasamples per second, for example), maximum reference image size, etc. In some cases, the limitations set by the hierarchy can be further limited by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffer, which is represented by signals in the encoded video sequence.
[0050] In one embodiment, receiver 302 may receive supplemental (redundant) data along with the encoded video. This supplemental data may be included as part of one or more encoded video sequences. The supplemental data may be used by video decoder 300 to properly decode the data and / or more accurately reconstruct the original video data. The supplemental data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.
[0051] Figure 4 This may be a functional block diagram of a video encoder 400 according to an embodiment of the present disclosure.
[0052] Encoder 400 can receive video samples from video source 401 (not part of the encoder), which can capture one or more video images that will be encoded by encoder 400.
[0053] Video source 401 can provide a source video sequence in the form of a digital video sample stream to be encoded by video encoder 303. This digital video sample stream can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCb, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, video source 401 can be a storage device storing previously prepared video. In a video conferencing system, video source 401 can be a storage device storing previously prepared video. In a video conferencing system, video source 401 can be a camera capturing local image information as a video sequence. Video data can be provided as multiple individual pictures, which are given motion when viewed sequentially. The pictures themselves can be constructed as spatial pixel arrays, where each pixel can include one or more samples, depending on the sampling structure, color space, etc., used. Those skilled in the art can easily understand the relationship between pixels and samples. The following focuses on describing samples.
[0054] According to an embodiment, encoder 400 can encode and compress images of a source video sequence into an encoded video sequence 410 in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of controller 402. In some embodiments, the controller controls and is functionally coupled to other functional units described below. For simplicity, the coupling is not depicted in the figures. Parameters set by the controller may include rate control related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. Other functions of controller 402 will be readily identified by those skilled in the art, as these functions relate to a video encoder 400 optimized for a particular system design.
[0055] Some video encoders operate in a manner readily recognizable to those skilled in the art as an "encoding loop." In a simplified description, an encoding loop may include an encoding portion of encoder 402 ("source encoder") (responsible for creating symbols based on the input image to be encoded and one or more reference images), and a (local) decoder 406 embedded in encoder 400, which reconstructs the symbols to create sample data in a manner similar to how a (remote) decoder creates sample data (because in video compression techniques, any compression between the symbols and the encoded video stream is lossless). The reconstructed sample stream is input to reference image memory 405. Since decoding the symbol stream produces bit-precise results independent of the decoder's location (local or remote), the contents of the reference image buffer are also bit-precisely corresponding between the local and remote encoders. In other words, the reference image samples "seen" by the encoder's prediction portion are exactly the same sample values that the decoder will "see" during prediction. This fundamental principle of reference image synchronization (and the drift that occurs, for example, due to channel errors that prevent the maintenance of synchronization) is well known to those skilled in the art.
[0056] The operation of the "local" decoder 406 can be combined with the above. Figure 3 The operation is the same as that of the "remote" decoder 300 described in detail. However, please refer to another brief reference. Figure 4 When symbols are available and the entropy encoder 408 and the parser 304 are able to encode / decode the symbols into an encoded video sequence without loss, the entropy decoding portion of the decoder 300, including the channel 301, receiver 302, buffer 303 and parser 304, may not be fully implemented in the local decoder 406.
[0057] It can then be observed that any decoder technique other than parsing / entropy decoding, which exists in the decoder, must also exist in the corresponding encoder in essentially the same functional form. The description of encoder techniques can be simplified because encoder techniques are inverses of the fully described decoder techniques. More detailed descriptions are only required in certain areas, and are provided below.
[0058] As part of the operation, the source encoder 403 may perform motion-compensated predictive coding. Referring to one or more previously encoded frames in the video sequence designated as "reference frames," this motion-compensated predictive coding predictively encodes the input frame. In this manner, the encoding engine 407 encodes the differences between pixel blocks of the input frame and pixel blocks of one or more reference frames, which may be selected as one or more predictive references for the input frame.
[0059] The local video decoder 406 can decode encoded video data of a frame that can be designated as a reference frame, based on symbols created by the source encoder 403. The operation of the encoding engine 407 can advantageously be a lossy process. When the encoded video data can be decoded by the video decoder (… Figure 4 When decoded at (not shown), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder 406 replicates the decoding process, which can be performed by the video decoder on the reference frame, and allows the reconstructed reference frame to be stored in the reference image cache 405. In this way, the encoder 400 can locally store a copy of the reconstructed reference frame that shares common content (no transmission errors) with the reconstructed reference frame to be acquired by the remote video decoder.
[0060] Predictor 404 can perform a prediction search on encoding engine 407. That is, for a new frame to be encoded, predictor 404 can search in reference image memory 405 for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new image. Predictor 404 can operate pixel-by-pixel based on sample blocks to find suitable prediction references. In some cases, as determined by the search results obtained by predictor 404, the input image may have prediction references obtained from multiple reference images stored in reference image memory 405.
[0061] The controller 402 can manage the encoding operations of the video encoder 403, including, for example, setting parameters and subgroup parameters for encoding video data.
[0062] The outputs of all the above-mentioned functional units can be entropy encoded in the entropy encoder 408. The entropy encoder (645) performs lossless compression on the symbols generated by the various functional units according to techniques known to those skilled in the art, such as Huffman coding, variable length coding, and arithmetic coding, thereby converting the symbols into an encoded video sequence.
[0063] Transmitter 409 may buffer one or more encoded video sequences created by entropy encoder 408 in preparation for transmission via communication channel 411, which may be a hardware / software link to a storage device that will store the encoded video data. Transmitter 409 may combine encoded video data from video encoder 403 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).
[0064] Controller 402 manages the operation of encoder 400. During encoding, controller 405 can assign a specific encoded image type to each encoded image, but this may affect the encoding techniques applicable to the corresponding images. For example, images can typically be assigned to any of the following image types:
[0065] An intra-frame picture (I-picture) is a picture that can be encoded and decoded without using any other pictures in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will understand variations of I-pictures and their corresponding applications and characteristics.
[0066] A predictive picture (P-picture) can be a picture that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and reference index to predict sample values for each block.
[0067] A bidirectional predictive picture (B-picture) can be a picture that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most two motion vectors and a reference index to predict sample values for each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata to reconstruct a single block.
[0068] Source images are typically spatially subdivided into multiple sample blocks (e.g., 4×4, 8×8, 4×8, or 16×16 sample blocks), and encoded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, which are determined by the coding assignment of the corresponding images applied to the block. For example, a block of an I image can be non-predictively coded, or the block can be predictively coded (spatial prediction or intra-frame prediction) with reference to already coded blocks of the same image. A pixel block of a P image can be predictively coded with reference to a previously coded reference image via spatial prediction or temporal prediction. A block of a B image can be predictively coded with reference to one or two previously coded reference images via spatial prediction or temporal prediction.
[0069] The video encoder 400 can perform encoding operations according to predetermined video coding techniques or standards, such as those specified in ITU-T H.265 Recommendation. In operation, the video encoder 400 can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.
[0070] In this embodiment, transmitter 409 may transmit additional data while transmitting encoded video. Source encoder 403 may include such data as part of the encoded video sequence. Additional data may include temporal / spatial / SNR enhancement layers, redundant images and slices, other forms of redundant data, SEI messages, VUI parameter set fragments, etc.
[0071] Figure 5 This illustrates the intra-frame prediction modes used in High Efficiency Video Coding (HEVC) and the Joint Exploration Model (JEM). To capture arbitrary edge orientations presented in natural video, the number of directional intra-frame modes is expanded from the 33 used in HEVC to 65. Additional directional modes in JEM, built upon HEVC, are shown below. Figure 5 As shown by the dashed arrows, the planar mode and DC mode remain the same. These denser directional intra-prediction modes are applicable to all block sizes and for both luma and chroma intra-prediction. Figure 5 As shown, the directional intra-prediction modes associated with odd-numbered intra-prediction mode indices, indicated by dashed arrows, are called odd-numbered intra-prediction modes. The directional intra-prediction modes associated with even-numbered intra-prediction mode indices, indicated by solid arrows, are called even-numbered intra-prediction modes. In this paper, as... Figure 5 The intra-frame prediction mode indicated by the solid or dashed arrow is also called angular mode.
[0072] In JEM, a total of 67 intra-prediction modes are used for lumen intra-prediction. To encode intra-modes, a most probable mode (MPM) list of size 6 is constructed based on the intra-modes of neighboring blocks. If an intra-mode is not in the MPM list, a flag is signaled to indicate whether the intra-mode belongs to the selected mode. In JEM-3.0, there are 16 selected modes, which are uniformly selected according to the four corner modes. In JVET-D0114 and JVET-G0060, 16 auxiliary MPMs are derived to replace the uniformly selected modes.
[0073] Figure 6 N reference tiers for intra-frame orientation mode are shown. These include block unit 611, segment A 601, segment B 602, segment C 603, segment D 604, segment E 605, segment F 606, first reference layer 610, second reference layer 609, third reference layer 608, and fourth reference layer 607.
[0074] In HEVC, JEM, and other standards such as H.264 / AVC, the reference samples used to predict the current block are limited to the nearest reference line (row or column). In multi-reference-line intra-prediction methods, for intra-direction modes, the number of candidate reference lines (rows or columns) increases from 1 (i.e., nearest) to N, where N is an integer greater than or equal to 1. Taking a 4x4 prediction unit (PU) as an example... Figure 6 This illustrates the concept of a multi-line intra-frame orientation prediction method. The intra-frame orientation pattern can arbitrarily select one of N reference layers to generate the predictor. In other words, the predictor P(x, y) is generated based on one of the reference samples S1, S2, ..., SN. A signal flag indicates which reference layer was selected for the intra-frame orientation pattern. If N is set to 1, the intra-frame orientation prediction method is the same as the traditional method in JEM 2.0. Figure 6 In this diagram, reference lines 610, 609, 608, and 607 consist of six segments 601, 602, 603, 604, 605, and 606, as well as a top-left reference sample. In this paper, the reference layer is also referred to as a reference line. The coordinates of the top-left pixel within the current block cell are (0, 0), and the coordinates of the top-left pixel in the first reference line are (-1, -1).
[0075] In JEM, for the luma component, neighboring samples generated for intra-prediction samples are filtered before generation processing. This filtering is controlled by the given intra-prediction mode and transform block size. If the intra-prediction mode is DC or the transform block size is equal to 4x4, neighboring samples are not filtered. If the distance between the given intra-prediction mode and the vertical mode (or horizontal mode) is greater than a predefined threshold, filtering is enabled. For neighboring sample filtering, a [1, 2, 1] filter and a bilinear filter are used.
[0076] The Position Dependent Intra Prediction Combination (PDPC) method is an intra prediction method that calls a combination of unfiltered boundary reference samples and HEVC-style intra predictions with filtered boundary reference samples. The calculation of each predicted sample pred[x][y] located at (x, y) is as follows:
[0077] pred[x][y]=(wL*R -1,y +wT*R x,-1 +wTL*R -1,-1 +(64-wL-wT-wTL)*pred[x][y]+32>>6 (Equation 2-1)
[0078] Where R x,-1 R -1,y R represents the unfiltered reference samples located at the top and left of the current sample (x, y), respectively. -1,-1 This represents the unfiltered reference sample located at the top left corner of the current block. The weights are calculated as follows:
[0079] wT=32>>((y<<1)>>shift) (Equation 2-2)
[0080] wL=32>>((x<<1)>>shift) (Equation 2-3)
[0081] wTL=-(wL>>4)-(wT>>4) (Equation 2-4)
[0082] shift = (log2(width) + log2(height) + 2) >> 2 (Equation 2-5)
[0083] Figure 7 Figure 700 illustrates the DC mode PDPC weights (wL, wT, wTL) for the (0, 0) and (1, 0) positions within a 4x4 block. If PDPC is applied to DC, planar, horizontal, and vertical intra-frame modes, no additional boundary filters, such as HEVC DC mode boundary filters or horizontal / vertical mode edge filters, are required. Figure 7 A reference sample R is shown for the PDPC applied to the upper right diagonal mode. x,-1 R -1,y and R -1,-1 Definition: The predicted sample pred(x', y') is located at (x', y') within the prediction block. Reference sample R x,-1 The coordinate x is given by the following formula: x = x' + y' + 1, with reference sample R. -1,y The coordinates y are similarly given by the following formula: y = x' + y' + 1.
[0084] Figure 8 Figure 800 illustrates Local Illumination Compensation (LIC), which performs illumination changes based on a linear model using a scaling factor a and an offset b. LIC can be adaptively enabled or disabled for coding units (CUs) coded in inter-frame modes.
[0085] When LIC is applied to CU, parameters a and b are derived using the least squares error method, utilizing the neighboring samples of the current CU and the reference samples corresponding to those neighboring samples. More specifically, as follows... Figure 8 As shown, subsampling (2:1 subsampling) of the CU is used to sample neighboring samples and corresponding samples in the reference image (identified by motion information of the current CU or subCU). IC parameters are derived and applied separately for each prediction direction.
[0086] When encoding a CU in merge mode, the LIC flag is copied from the adjacent block in a manner similar to motion information copying in merge mode; otherwise, for the CU, the LIC flag is signaled to indicate whether LIC applies.
[0087] Figure 9A The intra-prediction mode 900 used in HEVC is shown. HEVC has a total of 35 intra-prediction modes, of which mode 10 is a horizontal mode, mode 26 is a vertical mode, and modes 2, 18, and 34 are diagonal modes. The intra-prediction mode is notified by signals from three most probable modes (MPMs) and 32 remaining modes.
[0088] Figure 9BThe example shown in the Universal Video Coding (VVC) implementation has a total of 87 intra-prediction modes, where mode 18 is a horizontal mode, mode 50 is a vertical mode, and modes 2, 34, and 66 are diagonal modes. Modes -1 to -10 and modes 67 to 76 are called Wide-Angle Intra Prediction (WAIP) modes.
[0089] Based on the following PDPC expression, a predicted sample pred(x, y) located at position (x, y) is predicted using a linear combination of the intra-frame prediction mode (DC, plane, angle) and the reference sample:
[0090] pred(x,y)=(wL×R-1,y+wT×Rx,-1–wTL×R-1,-1+(64–wL–wT+wTL)×pred(x,y)+32)>>6
[0091] Where Rx,-1 and R-1,y represent the reference samples located at the top and left of the current sample (x, y), respectively, and R-1,-1 represents the reference sample located at the top left corner of the current block.
[0092] For DC mode, the weights are calculated as follows for blocks with width and height dimensions:
[0093] wT=32>>((y<<1)>>nScale), wL=32>>((x<<1)>>nScale), wTL=(wL>>4)+(wT>>4)
[0094] Where nScale = (log2(width) – 2 + log2(height) – 2 + 2) >> 2, wT represents the weight factor of the reference sample located on the upper reference line with the same horizontal coordinate, wL represents the weight factor of the reference sample located on the left reference line with the same vertical coordinate, wTL represents the weight factor of the reference sample at the upper left of the current block, and nScale specifies the rate at which the weight factor decreases along the axis (wL decreases from left to right or wT decreases from top to bottom), i.e., the weight factor decrease rate. In the current design, nScale is the same along the x-axis (from left to right) and the y-axis (from top to bottom). 32 represents the initial weight factor of adjacent samples. The initial weight factor is also the top (left or upper left) weight assigned to the upper left sample in the current CB, and the weight factor of adjacent samples in PDPC processing should be equal to or less than this initial weight factor.
[0095] For the planar mode wTL = 0, for the horizontal mode wTL = wT, and for the vertical mode wTL = wL, the PDPC weights can only be calculated through addition and shifting. The value of pred(x, y) can be further calculated using Equation 2-1.
[0096] The methods presented herein can be used individually or in any combination in any order. Furthermore, each of the methods (or embodiments), encoders, and decoders can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable storage medium. According to embodiments, the term "block" can be interpreted as a prediction block, coding block, or coding unit, i.e., a CU.
[0097] Figure 10 An exemplary flowchart 1000 is shown, and further reference will be made to... Figure 12 To describe Figure 10 , Figure 12 Workflow 1200 of an exemplary framework according to an exemplary embodiment is shown. Workflow 1200 includes modules such as a face detection and face landmark extraction module 122, a spatial-temporally (ST) downsampling module 123, a landmark feature compression and transmission module 126, an EFA feature compression and transmission module 127, a face detail reconstruction module 130, an EFA reconstruction module 131, a video compression and transmission module 135, an ST upsampling module 137, and a fusion module 139, and workflow 1200 also includes various data 121, 124, 125, 128, 129, 132, 133, 134, 136, 138, and 140.
[0098] At S101, given the input video sequence X = x1, x2, ..., such as data 121, the face detection and face landmark extraction module 122 first extracts data from each video frame x at S102. i One or more valid faces are identified. In one embodiment, only the most salient (e.g., largest) face is detected; in another embodiment, all faces in the frame that meet the condition (e.g., have a sufficiently large size exceeding a threshold) are detected. In S103, for x... i Given the j-th face in the dataset, determine the set of face landmarks and calculate the face landmark features f accordingly. l,i,j Set, facial landmark features f l,i,j The set will be used by the decoder to restore x i The j-th face in the dataset. At S103, the face landmark features F of all faces are... l,i =f l,i,1 ,f l,i,2... are placed together as data 124, and the landmark feature compression and transmission module 126 encodes and transmits data 124. At S105, simultaneously, for x... i The j-th face in the dataset can be computed by expanding the boundary region of the originally detected face (with a rectangular, elliptical, or fine-grained segmentation boundary) to include additional hair, body parts, or even the background. The EFA feature f can be computed accordingly at S106 and S107. b,i,j The set of EFA features will be used by the decoder to reconstruct x. i The EFA of the j-th face in the dataset. At S107, all EFA features F of all faces are processed. b,i =f b,i,1 ,f b,i,2 ... are placed together as data 125, and the EFA compression and transmission module 127 encodes and transmits data 125.
[0099] According to an exemplary embodiment, the face detection and face landmark extraction module 122 can use any face detector to locate each video frame x. i The face region in the image can be localized, for example, by treating faces as a special object category or by other DNN architectures specifically designed for face localization, using any object detection DNN. The face detection and face landmark extraction module 122 can also use any face landmark detector to localize a predetermined set of face landmarks for each detected face (e.g., landmarks around the left / right eye, nose, mouth, etc.). In some embodiments, a multi-task DNN can be used to simultaneously localize faces and associated landmarks. Face landmark features f l,i,j The intermediate latent representation can be computed by a face landmark detector, which is directly used to locate the landmark of the j-th face. An additional DNN can be applied to further process the intermediate latent representation and compute the face landmark features f. l,i,j For example, we can aggregate information from feature maps corresponding to various landmarks around a facial portion (e.g., the right eye) into joint features about that facial portion. Similarity, EFA features f b,i,j This can be an intermediate latent representation computed by a face detector corresponding to the j-th face. An additional DNN can also be used to compute f based on this intermediate latent representation. b,i,j For example, by emphasizing background regions other than the actual face region. Various exemplary embodiments are not limited to the methods or DNN architectures for face detectors, face landmark detectors, face landmark feature extractors, or EFA feature extractors.
[0100] According to an exemplary embodiment, the landmark feature compression and transmission module 126 can use various methods to effectively compress face landmark features. In a preferred embodiment, a codebook-based mechanism is used, and a codebook can be generated for each face portion (e.g., the right eye). Then, for a specific face portion of a specific face (e.g., the right eye of the current face in the current frame), its face landmark features can be represented by a weighted combination of codewords in the codebook. In this case, the codebook is stored on the decoder side, and it may only be necessary to transmit the weight coefficients of the codewords to the decoder side to recover the face landmark features. Similarly, the EFA feature compression and transmission module 127 can also use various methods to compress EFA features. In a preferred embodiment, an EFA codebook is also used, wherein a specific EFA feature is represented by a weighted combination of EFA codewords, and it may subsequently only be necessary to transmit the weight coefficients of the codewords to recover the EFA feature.
[0101] On the other hand, the input video sequence X = x1, x2, ..., i.e., data 121, is ST-downsampled by ST downsampling module 123 to X′ = x′1, x′2, ... Compared to X, X′ can be downsampled spatially, temporally, or both. When X is downsampled spatially, each x... i and x′ i They all have the same timestamp, and, for example, through traditional or DNN-based interpolation, from x with reduced resolution. i To calculate x′ i When X is downsampled over time, each x′ i x corresponding to different timestamps ki , where k is the downsampling frequency (one frame is sampled from every k frames of X to generate X′). When X is downsampled both spatially and temporally, for example by conventional or DNN-based interpolation, from x at different timestamps with reduced resolution... ki Calculate each x′ i Then, the downsampled sequence X′=x′1,x′2,…, i.e., data 134, can be considered as an LQ version of the original HQ input X=x1,x2,…. X′=x′1,x′2,… can then be encoded and transmitted by the video compression and transmission module 135. According to an exemplary embodiment, the video compression and transmission module 135 can use any video compression framework, such as HEVC, VVC, NNVC, or end-to-end video coding.
[0102] On the decoder side, as referenced Figure 11 Flowchart 1100 and Figure 12 As described in each module, firstly, the encoded bitstream received at S111 is decompressed at S112 to obtain the decoded downsampled sequence. That is, data 136, the decoded EFA feature. That is, data 129, and the decoded facial landmark features. That is, data 128. Each decoded frame. The frame x′ corresponding to the downsampled data i Each decoded EFA feature Corresponding to EFA feature F b,i Each decoded landmark feature Corresponding to the landmark feature F l,i At S113, the decoded downsampled sequence The upsampling sequence is generated using the ST upsampling module 137. That is, data 138. Corresponding to the encoder size, this ST upsampling module performs spatial sampling, temporal sampling, or spatial and temporal sampling as the inverse operation of the downsampling process in the ST downsampling module 123. When spatial downsampling is used on the encoder side, spatial upsampling is used on the decoder side, where, for example, each... Upsampling is performed at the same timestamp. and Will have with x i Same resolution. When using temporal downsampling on the encoder side, temporal upsampling is used on the decoder side, where each... All And, for example, by using traditional motion interpolation or DNN-based frame synthesis methods, based on and calculate and An additional (k-1) frames between them. When spatial and temporal downsampling is used simultaneously on the encoder side, spatial and temporal upsampling is used on the decoder side, where spatial upsampling is performed by using conventional interpolation or DNN-based super-resolution methods to obtain the data from the encoder. Calculate each And by using traditional motion interpolation or DNN-based frame synthesis methods, based on and Further generation and Additional frames between.
[0103] At S114, the decoded EFA feature The EFA reconstruction module 131 is used to calculate the sequence of the reconstructed EFA. That is, data 133, each Contains an EFA set, For frames EFA of the j-th face in the dataset. Decoded landmark features. That is, data 128 is processed by the facial detail reconstruction module 130 to calculate the sequence of restored facial details. That is, data 132. Each It contains a set of facial detail representations. Corresponding to a frame The j-th face in the image. In a preferred embodiment, the EFA reconstruction module 131 is a DNN composed of a stack of remaining blocks and convolutional layers. The face detail reconstruction module 130 is a Conditional Generative Adversarial Network (GAN) conditioned on landmark features corresponding to different face parts. To compute the face detail for timestamp i... EFA reconstruction module 131 can use only the decoded EFA features of that timestamp. Alternatively, use several adjacent timestamps for EFA. (n, m are any positive integers). Similarly, to compute the result for timestamp i... The face detail reconstruction module 130 can use only the decoded landmark features of the timestamp. Alternatively, use several adjacent timestamps for EFA. Subsequently, at S115, the restored facial details are integrated using the fusion module 139. The rebuilt EFA and upsampling sequence They are aggregated together to generate the final reconstructed video sequence. That is, data 140. The fusion module can be a small DNN, where, in order to generate at timestamp i The fusion module can use only those from the same timestamp. and Alternatively, use timestamps from several adjacent timestamps. and Exemplary embodiments may exclude any limitations on the DNN architecture of the face detail reconstruction module 130, the EFA reconstruction module 131, and / or the fusion module 139.
[0104] The purpose of using EFA features is to improve the reconstruction quality of extended facial regions (e.g., hair, body parts, etc.). In some embodiments, EFA-related processing may be optional, depending on the trade-off between reconstruction quality and computational and transmission costs. Therefore, in Figure 12 In this context, this optional processing is marked with dashed lines, such as the dashed lines marking the spaces between components 125, 127, 129, 131, and 133.
[0105] Furthermore, according to an exemplary embodiment, multiple components exist in the proposed framework that needs to be trained, and reference will be made to... Figure 13Describing this training, Figure 13 A workflow 1300 for an exemplary training process according to an exemplary embodiment is shown. The workflow 1300 includes modules such as a face detection and face landmark extraction module 223, an ST downsampling module 236, a landmark feature noise modeling module 226, an EFA feature noise modeling module 227, a face detail reconstruction module 230, an EFA reconstruction module 231, a video noise modeling module 235, an ST upsampling module 237, a fusion module, an adversarial loss calculation module 241, a reconstruction loss calculation module 242, and a perceptual loss calculation module 243, and the workflow 1300 also includes various data 221, 224, 225, 229, 228, 232, 233, 236, 238, and 240.
[0106] According to an exemplary embodiment, the proposed framework, which needs to be trained before deployment, contains multiple components, including a face detector, a face landmark detector, a face landmark feature extractor, and an EFA feature extractor, an EFA reconstruction module 131, and a face detail reconstruction module 130 in the face detection and face landmark extraction module 122. Optionally, if a learning-based downsampling or upsampling method is used, the ST downsampling module 123 and the ST upsampling module 137 also need to be pre-trained. In a preferred embodiment, all these components use a DNN-based method, and the weight parameters of these DNNs need to be trained. In other embodiments, some of these components may use conventional learning-based methods, such as a conventional face landmark detector, and the corresponding model parameters need to be trained. Each learning-based component, DNN-based component, or conventional component needs to be pre-trained individually first, and then these components are jointly tuned through the training process described in this disclosure.
[0107] For example, Figure 13 A preferred embodiment of the training process, the overall workflow 1300, is provided. For training, we use a video noise modeling module 235 instead of the actual video compression and transmission module 135. This is because actual video compression includes, for example, non-differentiable quantization. The video noise modeling module 235 adds random noise to the downsampled sequence X′=x′1,x′2,… to generate the decoded downsampled sequence during the training process. In the final testing phase, the true data distribution of the decoded downsampled sequence is simulated. Therefore, the noise model used by the video noise modeling module 235 typically depends on the actual video compression method used in practice. Similarly, we replace the EFA feature compression and transmission module 127 with the EFA feature noise modeling module 227, which adds noise to F during the training phase. b,1 ,F b,2 ... to generate decoded EFA features This simulates the actual data distribution of decoded EFA features in practice. Furthermore, the landmark feature compression and transmission module 126 is replaced with a landmark feature noise modeling module 226. During the training phase, the landmark feature noise modeling module 226 adds noise to the F... l,1 ,F l,2 ... to generate decoded landmark features In practice, the true distribution of decoded landmark features is simulated. An exemplary embodiment calculates the following loss function for training.
[0108] Several types of losses are calculated during the training process to learn the learnable components. Distortion loss can be calculated in the reconstruction loss calculation module 242. To measure the difference between the original training sequence and the reconstructed training sequence, for example in It can be x i and The mean absolute error (MAE) or structural similarity (SSIM) index is used to measure the difference between the two regions. Importance weight maps can also be used to emphasize distortions in reconstructed face regions or different parts of face regions. Furthermore, perceptual loss can be calculated in the perceptual loss calculation module. For example Among them, the feature extraction DNN (e.g., the Visual Geometry Group (VGG) backbone network) is based on x i and Computational feature representation. Based on x i and The difference in the calculated feature representations (e.g., MSE) is used as the perceptual loss. The adversarial loss can be calculated by the adversarial loss calculation module 241. For example To measure the reconstructed input The degree to which it looks natural. This is achieved by comparing real x or reconstructed x. The data is fed into a discriminator (which is typically a classification DNN similar to ResNet) to classify it as either natural or reconstructed, and the classification error (e.g., cross-entropy loss) can be used as the basis for classification. Distortion loss Perceived loss and combat losses Weighted combinations can be used to form joint losses Its gradient can be calculated to update the model parameters via backpropagation:
[0109]
[0110] α and β are hyperparameters that balance the importance of different loss terms.
[0111] It is important to note that different components can be updated at different times using different update frequencies. In some cases, some components are updated periodically or frequently only after deployment, when new training data becomes available. In other cases, only a subset of model parameters are updated after deployment. This disclosure does not impose any limitations on the optimization methods, the frequency of model updates, or the percentage of model parameters to be updated.
[0112] Thus, an exemplary embodiment of either workflow 1200 or 1300 represents a new framework for video compression and transmission in video conferencing, based on face reconstruction that improves coding efficiency by transmitting LQ frames and facial features, a flexible and general framework for spatial, temporal, or space-time downsampled frames, a flexible and general framework for different DNN architectures, and a flexible and general framework for accommodating multiple faces with arbitrary backgrounds.
[0113] The embodiments further illustrate a video conferencing framework based on face restoration (or face phantomization), which restores realistic details from real low-quality (LQ) faces to high-quality (HQ) faces. Instead of relying on error-prone shape and texture transfer in face replay methods, we restore the details of HQ faces based on LQ faces and face landmark features. The exemplary framework disclosed herein guarantees robust quality of the restored faces, which is critical for real-world products. For example, only downsampled frames and face features can be transmitted to reduce transmission costs, and HQ frames can be restored on the decoder side based on downsampled frames and face features.
[0114] The above-described technologies can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable storage media, or implemented through one or more specifically configured hardware processors. For example, Figure 14 A computer system 1400 suitable for implementing certain embodiments of the disclosed subject matter is shown.
[0115] Computer software can be coded using any suitable machine code or computer language. Any suitable machine code or computer language can be assembled, compiled, linked, or similarly processed to create code containing instructions that can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or through decoding, microcode, etc.
[0116] The instructions can be executed on various types of computers or their components, including personal computers, tablets, servers, smartphones, gaming devices, and Internet of Things (IoT) devices.
[0117] Figure 14The components of the computer system 1400 shown are exemplary in nature and are not intended to impose any limitation on the scope of use or functionality of the computer software implementing the embodiments of this disclosure. The configuration of the components should also not be construed as having any dependency or requirement relating to any one or a combination of components shown in the exemplary embodiments of the computer system 1400.
[0118] Computer system 1400 may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, movement of a data glove), audio input (e.g., speech, clapping), visual input (e.g., gestures), and olfactory input (not depicted). Human-machine interface devices may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images acquired from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video), etc.
[0119] The input human-machine interface device may be one or more of the following (only one of each is shown): keyboard 1401, mouse 1402, touchpad 1403, touch screen 1410, joystick 1405, microphone 1406, scanner 1408, camera 1407.
[0120] Computer system 1400 may include certain human-machine interface (HMI) output devices. Such HMI output devices may stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such HMI output devices may include tactile output devices (e.g., haptic feedback of touchscreen 1410 or joystick 1405, but may also be haptic feedback devices that are not input devices), audio output devices (e.g., speakers 1409, headphones (not depicted)), and visual output devices (e.g., screen 1410 including CRT screens, LCD screens, plasma screens, OLED screens, each screen may or may not have touchscreen input functionality, each screen may or may not have haptic feedback functionality, some of which are capable of outputting two-dimensional or more three-dimensional visual outputs through devices such as stereoscopic image output, virtual reality glasses (not depicted), holographic displays and smoke boxes (not depicted), and printers (not depicted).
[0121] Computer system 1400 may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 1420 with media such as CD / DVD 1411, finger drives 1422, removable hard disk drives or solid-state drives 1423, conventional magnetic media such as magnetic tapes and floppy disks (not depicted), devices based on dedicated ROM / ASIC / PLD such as security dongles (not depicted), etc.
[0122] Those skilled in the art should also understand that the term "computer-readable storage medium" as used in connection with the subject matter of this disclosure does not cover transmission media, carrier waves, or other transient signals.
[0123] Computer system 1400 may also include an interface 1499 to one or more communication networks 1498. Network 1498 may be, for example, a wireless network, a wired network, or an optical network. Network 1498 may further be a local area network, a wide area network, a metropolitan area network, a vehicle and industrial network, a real-time network, a latency-tolerant network, etc. Examples of networks include, for example, local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., cable or wireless wide area digital television networks including cable television, satellite television, and terrestrial broadcast television, vehicle and industrial television including CANbus, etc. Some networks 1498 typically require external network interface adapters (e.g., USB ports of computer system 1400) to connect to certain general-purpose data ports or peripheral buses (1450 and 1451); as described below, other network interfaces are typically integrated into the core of computer system 1400 by connecting to the system bus (e.g., an Ethernet interface connected to a PC computer system or a cellular network interface connected to a smartphone computer system). Computer system 1400 can communicate with other entities using any of these networks 1498. Such communication can be one-way receiving (e.g., broadcast television), one-way transmitting (e.g., CANbus connected to certain CANbus devices), or bidirectional, such as connecting to other computer systems using a local area network (LAN) or wide area network (WAN). As mentioned above, certain protocols and protocol stacks can be used on each of those networks and network interfaces.
[0124] The aforementioned human-machine interface device, human-machine accessible storage device, and network interface can be attached to the kernel 1440 of the computer system 1400.
[0125] Core 1440 may include one or more central processing units (CPUs) 1441, graphics processing units (GPUs) 1442, dedicated programmable processing units in the form of field-programmable gate areas (FPGAs) 1417, hardware accelerators 1444 for certain tasks, etc. These devices, as well as read-only memory (ROM) 1445, random access memory (RAM) 1446, and internal mass storage 1447 such as internal non-user-accessible hard disk drives (HDDs), SSDs, etc., can be connected via system bus 1448. In some computer systems, system bus 1448 can be accessed via one or more physical connectors to allow for expansion with additional CPUs, GPUs, etc. Peripheral devices can be directly connected to the core's system bus 1448 or connected to the core's system bus 1448 via peripheral bus 1451. Peripheral bus architectures include PCI, USB, etc.
[0126] The CPU 1441, GPU 1442, FPGA 1443, and accelerator 1444 can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in read-only memory 1445 or random access memory 1446. Transient data can also be stored in random access memory 1446, while permanent data can be stored, for example, in internal mass storage 1447. Fast storage and retrieval of any memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs 1441, 1442, mass storage 1447, read-only memory 1445, random access memory 1446, etc.
[0127] Computer-readable storage media may have computer code thereon that performs various computer-implemented operations. The media and computer code may be media and computer code specifically designed and constructed for the purposes of this disclosure, or the media and computer code may be of a type known and available to those skilled in the art of computer software.
[0128] As a non-limiting example, a computer system having architecture 1400, particularly kernel 1440, can provide functionality by having one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) execute software contained in one or more tangible computer-readable storage media. Such computer-readable storage media can be media associated with user-accessible mass storage as described above, as well as some non-transitory memory of kernel 1440, such as kernel-internal mass memory 1447 or ROM 1445. Software implementing various embodiments of this disclosure can be stored in such devices and executed by kernel 1440. Depending on specific needs, the computer-readable storage media may include one or more storage devices or chips. The software can cause kernel 1440, particularly the processors therein (including CPUs, GPUs, FPGAs, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM 1446 and modifying such data structures according to software-defined processes. Additionally or alternatively, the computer system may provide functionality through hard-wired or otherwise embodied logic in circuitry (e.g., accelerator 1444), which may replace or operate with the software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable storage media may include circuitry (e.g., integrated circuits (ICs)) storing software for execution, circuitry embodying logic for execution, or both. This disclosure includes any suitable combination of hardware and software.
[0129] Although several exemplary embodiments have been described in this disclosure, modifications, substitutions, and various equivalent alternatives that fall within the scope of this disclosure exist. Therefore, it should be understood that those skilled in the art will be able to design numerous systems and methods that, while not expressly shown or described herein, embody the principles of this disclosure and thus fall within its spirit and scope.
Claims
1. A method for video encoding, executed by at least one processor, characterized in that, The method includes: Acquire video data, the video data including the encoded bitstream of the video data; Detect at least one face from at least one frame of the video data; Determining a set of facial landmark features of at least one face from at least one frame of the video data, wherein determining the set of facial landmark features includes upsampling at least one downsampled sequence, the at least one downsampled sequence being obtained by decompressing the encoded bitstream; Determine an extended face region, the extended face region comprising a boundary region extending from the region of the at least one face detected in the at least one frame of the video data, the extended face region comprising hair, body parts, or background; An extended face region feature set is determined from the extended face region. This extended face region feature set is used by the decoder to recover the extended face region. The extended face region and the extended face region feature set are determined by the following method: Extended face region features are reconstructed using an adversarial generative network, where each of the extended face region features corresponds to one of the face landmark features in the face landmark feature set; and The video data is encoded at least in part by a neural network based on a determined set of facial landmark features, and the extended facial region features are encoded and transmitted.
2. The method according to claim 1, characterized in that, The encoding of the video data by at least part of the neural network based on the determined set of facial landmark features includes: The video data is encoded at least in part by the neural network based on the determined set of facial landmark features by aggregating the set of facial landmark features, the extended facial region features, and the upsampled sequence obtained by upsampling the at least one downsampled sequence.
3. The method according to claim 2, characterized in that, The at least one face from the at least one frame of the video data is determined to be the largest face among a plurality of faces in the at least one frame of the video data.
4. The method according to claim 2, characterized in that, The method further includes: For each of a plurality of faces in at least one frame of the video data, determine from the at least one frame of the video data a plurality of face landmark feature sets other than the face landmark feature set of the at least one face; and The video data is encoded at least in part by the neural network based on the determined set of facial landmark features and the determined plurality of facial landmark feature sets.
5. The method according to any one of claims 1-4, characterized in that, The neural network includes a deep neural network.
6. An apparatus for video encoding, characterized in that, The device includes: At least one memory configured to store computer program code; At least one processor is configured to access the computer program code to implement the method of any one of claims 1-5.
7. A non-transitory computer-readable storage medium, characterized in that, It contains a program that causes a computer to perform the method of any one of claims 1-5.
8. An apparatus for video encoding, characterized in that, The device includes: The acquisition module is configured to acquire video data, wherein the video data includes the encoded bitstream of the video data; The detection module is configured to detect at least one face from at least one frame of the video data; A determination module is configured to determine a set of facial landmark features of at least one face from at least one frame of the video data, wherein determining the set of facial landmark features includes upsampling at least one downsampled sequence, the at least one downsampled sequence being obtained by decompressing the encoded bitstream; determining an extended face region, the extended face region including a boundary region extending from the region of the at least one face detected in the at least one frame of the video data, the extended face region including hair, body parts, or background; determining an extended face region feature set from the extended face region, the extended face region feature set being used by a decoder to recover the extended face region, the extended face region and the extended face region feature set being determined by: reconstructing the extended face region features through a generative adversarial network, each of the extended face region features corresponding to one of the facial landmark features in the set of facial landmark features; and The encoding module is configured to encode the video data, at least in part, by a neural network based on a determined set of facial landmark features, and to encode and transmit the extended facial region features.
