Video coding method, apparatus, computer readable medium and computer device

Abstract

This application provides a video encoding / decoding method, apparatus, computer-readable medium, and computer device. The video encoding method includes: acquiring a video content frame and corresponding auxiliary information, the auxiliary information describing the visual features of the video content frame; extracting first color channel data of the video content frame in a specified color space; using the first color channel data of the video content frame as the first color channel data of an auxiliary information frame associated with the video content frame, and using the auxiliary information as the second color channel data of the auxiliary information frame to generate an auxiliary information frame; encoding the video content frame and the auxiliary information frame to generate a video stream, and adding an association identifier to the video stream for associating the video content frame with the auxiliary information frame. The technical solution of this application can achieve reliable and synchronous transmission of auxiliary information with low bitrate overhead, which is beneficial to improving video encoding / decoding performance.

Description

Technical Field

[0001] This application relates to the fields of computer and communication technology, and more specifically, to a video encoding / decoding method, apparatus, computer-readable medium, and computer device. Background Technology

[0002] With the widespread adoption of artificial intelligence (AI) visual algorithms, videos, in addition to carrying the visual content, also simultaneously carry analysis results aligned with the timeline of the video, such as person segmentation masks, semantic region annotations, or alpha channels. This information is added to the video stream as auxiliary information. However, current technologies either increase the width or height of the original image by stitching it horizontally or vertically to carry this auxiliary information, or they require introducing additional transmission links. Stitching increases resolution, leading to higher bandwidth costs; while introducing additional transmission links increases both bitrate overhead and decoding complexity. Therefore, achieving reliable transmission of auxiliary information while minimizing bitrate overhead is a pressing technical challenge. Summary of the Invention

[0003] The embodiments of this application provide a video encoding / decoding method, apparatus, computer-readable medium, and computer device, which can achieve reliability and synchronization of auxiliary information transmission with low bit rate overhead, thereby improving video encoding / decoding performance.

[0004] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part by practice of this application.

[0005] According to one aspect of the embodiments of this application, a video encoding method is provided, comprising: acquiring a video content frame and auxiliary information corresponding to the video content frame, the auxiliary information being used to describe the visual features of the video content frame; extracting first color channel data of the video content frame in a specified color space; using the first color channel data of the video content frame as the first color channel data of an auxiliary information frame associated with the video content frame, and using the auxiliary information as the second color channel data of the auxiliary information frame to generate the auxiliary information frame; encoding the video content frame and the auxiliary information frame to generate a video stream, and adding an association identifier for associating the video content frame with the auxiliary information frame to the video stream.

[0006] According to one aspect of the embodiments of this application, a video decoding method is provided, comprising: parsing a video bitstream to obtain a video content frame and an auxiliary information frame associated with the video content frame, wherein the first color channel data of the auxiliary information frame is the first color channel data of the video content frame, and the second color channel data of the auxiliary information frame contains auxiliary information corresponding to the video content frame, the auxiliary information being used to describe the visual features of the video content frame; and rendering the video content frame according to the auxiliary information contained in the auxiliary information frame.

[0007] According to one aspect of the embodiments of this application, a video encoding apparatus is provided, comprising: an acquisition unit configured to acquire a video content frame and auxiliary information corresponding to the video content frame, the auxiliary information being used to describe the visual features of the video content frame; an extraction unit configured to extract first color channel data of the video content frame in a specified color space; a generation unit configured to use the first color channel data of the video content frame as first color channel data of an auxiliary information frame associated with the video content frame, and use the auxiliary information as second color channel data of the auxiliary information frame to generate the auxiliary information frame; and an encoding unit configured to encode the video content frame and the auxiliary information frame to generate a video stream, and add an association identifier for associating the video content frame with the auxiliary information frame to the video stream.

[0008] In some embodiments of this application, based on the foregoing scheme, the encoding unit is configured to: encode the video content frame and the auxiliary information frame using each candidate encoding parameter in the candidate encoding parameter set, and decode the encoding result to obtain the reconstruction result of the auxiliary information frame; calculate the error between the first color channel data of the video content frame and the first color channel data in the reconstruction result, as the reconstruction error between the video content frame and the auxiliary information frame; select a target encoding parameter from the candidate encoding parameter set according to the reconstruction error between the video content frame and the auxiliary information frame, and use the target encoding parameter to encode the video content frame and the auxiliary information frame.

[0009] In some embodiments of this application, based on the foregoing scheme, selecting a target coding parameter from the candidate coding parameter set according to the reconstruction error between the video content frame and the auxiliary information frame includes: calculating the total reconstruction error corresponding to the video to be encoded based on the reconstruction error between the video content frame and the auxiliary information frame and the number of video content frames contained in the video to be encoded; and selecting the candidate coding parameter with the smallest total reconstruction error from the candidate coding parameter set as the target coding parameter based on the total reconstruction error corresponding to the video to be encoded.

[0010] In some embodiments of this application, based on the aforementioned scheme, if the candidate coding parameter set contains multiple candidate coding parameters with the smallest total reconstruction error, then the candidate coding parameter with the smallest code rate is selected from the multiple candidate coding parameters with the smallest total reconstruction error as the target coding parameter.

[0011] In some embodiments of this application, based on the foregoing scheme, the total reconstruction error corresponding to the video to be encoded is calculated according to the reconstruction error between the video content frame and the auxiliary information frame, and the number of video content frames contained in the video to be encoded. This includes: calculating a statistical value of the reconstruction error according to the number of video content frames contained in the video to be encoded, and the reconstruction error between each video content frame and the associated auxiliary information frame, and using the calculated statistical value as the total reconstruction error corresponding to the video to be encoded.

[0012] In some embodiments of this application, based on the foregoing scheme, the generation unit is further configured to: perform hash calculation on at least one of the following information: the identification information of the video content frame, some or all of the data contained in the video content frame, the identification information of the auxiliary information frame, and some or all of the data contained in the auxiliary information frame; and use the calculated hash value as an association identifier for associating the video content frame with the auxiliary information frame.

[0013] In some embodiments of this application, based on the foregoing scheme, the designated color space includes the YUV color space, and the first color channel data includes luminance channel data;

[0014] Specifically, generating the auxiliary information frame by using the first color channel data of the video content frame as the first color channel data of the auxiliary information frame corresponding to the video content frame, and using the auxiliary information as the second color channel data of the auxiliary information frame, includes: using the luminance channel data of the video content frame as the luminance channel data of the auxiliary information frame, and using the auxiliary information as the chroma channel data of the auxiliary information frame, to generate the auxiliary information frame.

[0015] In some embodiments of this application, based on the foregoing scheme, the designated color space includes the YUV color space, and the first color channel data includes chroma channel data; Specifically, generating the auxiliary information frame by using the first color channel data of the video content frame as the first color channel data of the auxiliary information frame corresponding to the video content frame, and using the auxiliary information as the second color channel data of the auxiliary information frame, includes: using the chroma channel data of the video content frame as the chroma channel data of the auxiliary information frame, and using the auxiliary information as the luminance channel data of the auxiliary information frame, to generate the auxiliary information frame.

[0016] In some embodiments of this application, based on the foregoing scheme, an association identifier for associating the video content frame with the auxiliary information frame is added to the video bitstream, including: generating supplementary enhancement information, adding the association identifier to the supplementary enhancement information, and adding the supplementary enhancement information to the video bitstream.

[0017] According to one aspect of the embodiments of this application, a video decoding apparatus is provided, comprising: a parsing unit configured to parse a video bitstream to obtain a video content frame and an auxiliary information frame associated with the video content frame, wherein the first color channel data of the auxiliary information frame is the first color channel data of the video content frame, and the second color channel data of the auxiliary information frame contains auxiliary information corresponding to the video content frame, the auxiliary information being used to describe the visual features of the video content frame; and a processing unit configured to render the video content frame according to the auxiliary information contained in the auxiliary information frame.

[0018] In some embodiments of this application, based on the foregoing scheme, the parsing unit is configured to: parse the video stream to obtain multiple frames and the association identifier of each frame in the multiple frames; determine the associated video content frame and auxiliary information frame according to the association identifier of each frame obtained by parsing; wherein the associated video content frame and auxiliary information frame have matching association identifiers.

[0019] In some embodiments of this application, based on the aforementioned scheme, determining associated video content frames and auxiliary information frames according to the association identifiers of each frame obtained through parsing includes: determining two frames with the same association identifier based on the association identifiers of each frame obtained through parsing; and identifying the video content frames and auxiliary information frames in the two frames based on the information of the two frames with the same association identifier.

[0020] In some embodiments of this application, based on the foregoing scheme, the video content frame and auxiliary information frame in the two frames with the same association identifier are identified according to the information of the two frames, including at least one of the following methods: taking the frame with the smallest display timestamp among the two frames with the same association identifier as the video content frame, and taking the frame with the largest display timestamp among the two frames with the same association identifier as the auxiliary information frame; taking the frame with an odd number of sequence number among the two frames with the same association identifier as the video content frame, and taking the frame with an even number of sequence number among the two frames with the same association identifier as the auxiliary information frame.

[0021] In some embodiments of this application, based on the aforementioned scheme, the video content frame and auxiliary information frame in the two frames with the same association identifier are identified according to the information of the two frames with the same association identifier, including: obtaining the display timestamps of the two frames with the same association identifier; if the difference between the display timestamps of the two frames with the same association identifier is within a set range, then the video content frame and auxiliary information frame in the two frames are identified.

[0022] In some embodiments of this application, based on the foregoing scheme, the parsing unit is further configured to: parse the association identifier of the video content frame and the association identifier of the auxiliary information frame from the video bitstream; the processing unit is configured to: if the association identifier of the video content frame and the association identifier of the auxiliary information frame match, then render the video content frame according to the auxiliary information contained in the auxiliary information frame.

[0023] In some embodiments of this application, based on the foregoing scheme, the processing unit is further configured to: if the association identifier of the video content frame does not match the association identifier of the auxiliary information frame, then discard the auxiliary information frame.

[0024] In some embodiments of this application, based on the foregoing scheme, the processing unit is further configured to: after rendering the video content frame, use the display timestamp of the video content frame and the audio frame to perform audio-visual synchronization processing.

[0025] According to one aspect of the embodiments of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the video encoding method or video decoding method as described in the above embodiments.

[0026] According to one aspect of the embodiments of this application, a computer device is provided, including: one or more processors; and a storage device for storing one or more computer programs, which, when executed by the one or more processors, cause the computer device to implement the video encoding method or video decoding method as described in the above embodiments.

[0027] According to one aspect of the embodiments of this application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads from the computer-readable storage medium and executes the computer program, causing the computer device to perform the video encoding or video decoding methods provided in the various alternative embodiments described above.

[0028] In some embodiments of this application, the video encoding end can extract the first color channel data of the video content frame in a specified color space, then use the first color channel data of the video content frame as the first color channel data of the auxiliary information frame associated with the video content frame, and use the auxiliary information corresponding to the video content frame as the second color channel data of the auxiliary information frame to generate the auxiliary information frame. The video content frame and the auxiliary information frame are then encoded, and an association identifier for associating the video content frame and the auxiliary information frame is added to the generated video stream. It is evident that the technical solution of this application, by multiplexing the first color channel data of the video content frame as the first color channel data of the auxiliary information frame, leverages the strong spatial correlation between the video content and the auxiliary information, significantly reducing the encoding residual of the auxiliary information frame and thus effectively improving encoding compression efficiency. By embedding the auxiliary information into the second color channel of the auxiliary information frame and encoding it together with the video content frame, the reliability and synchronization of the auxiliary information transmission can be guaranteed. Furthermore, the association identifier added to the video stream enables the decoding end to accurately identify and extract the auxiliary information, providing a reliable data foundation for subsequent processing.

[0029] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0030] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown; Figure 2 This diagram illustrates the placement of the video encoding and decoding devices in a streaming system. Figure 3 A basic flowchart of a video encoder is shown; Figure 4 A flowchart illustrating the overall transmission process of a video file according to an embodiment of this application is shown; Figure 5 A flowchart of a video encoding method according to an embodiment of this application is shown; Figure 6 A flowchart of a video decoding method according to an embodiment of this application is shown; Figure 7 A processing flowchart of a video encoding end according to an embodiment of this application is shown; Figure 8 A flowchart illustrating the selection of optimal encoding parameters according to an embodiment of this application is shown; Figure 9 A processing flowchart of a video encoding end according to an embodiment of this application is shown; Figure 10 A processing flowchart of a video decoding end according to an embodiment of this application is shown; Figure 11 A block diagram of a video encoding apparatus according to an embodiment of this application is shown; Figure 12 A block diagram of a video decoding apparatus according to an embodiment of this application is shown; Figure 13 A schematic diagram of the structure of a computer system suitable for implementing the computer device of the present application is shown. Detailed Implementation

[0031] Exemplary embodiments will now be described in a more comprehensive manner with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in various forms and should not be construed as limited to these examples; rather, these embodiments are provided so that this application will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art.

[0032] Furthermore, the features, structures, or characteristics described in this application can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to provide a full understanding of the embodiments of this application. However, those skilled in the art will recognize that when implementing the technical solutions of this application, not all the detailed features in the embodiments may be used, one or more specific details may be omitted, or other methods, elements, devices, steps, etc., may be employed.

[0033] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0034] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0035] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0036] It should be noted that "multiple" in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0037] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown.

[0038] like Figure 1 As shown, system architecture 100 may include a first device 110 and a second device 120 interconnected via network 150. Figure 1 In one embodiment, the first device 110 and the second device 120 can perform one-way data transmission via the network 150.

[0039] As an example, the first device 110 may encode video data (e.g., a video image stream captured by the first device 110) for transmission over network 150 to the second device 120. The encoded video data is transmitted in the form of one or more encoded video streams. The second device 120 may receive the encoded video data from network 150, decode the encoded video data to recover the video data, and display video images based on the recovered video data.

[0040] Optionally, the first device 110 and the second device 120 can also perform bidirectional data transmission via the network 150. For example, in addition to the process described above where the first device 110 encodes video data and transmits it to the second device 120 via the network 150, the second device 120 can also encode video data and transmit it to the first device 110 via the network 150. Optionally, the first device 110 and the second device 120 can also selectively communicate directly (e.g., via...). Figure 1The dashed line between the first device 110 and the second device 120 shown in the diagram indicates that they can communicate directly without going through the network 150. For example, the first device 110 and the second device 120 can communicate directly through short-range wireless communication technologies such as Bluetooth, Wi-Fi, and Near Field Communication (NFC).

[0041] In one embodiment of this application, system architecture 100 may include a third device 130 and a fourth device 140 performing bidirectional transmission of encoded video data, such as during a video conference. For bidirectional data transmission, each of the third device 130 and the fourth device 140 may encode video data (e.g., a video image stream captured by the third device 130 or the fourth device 140) for transmission over network 150 to the other of the third device 130 and the fourth device 140. Each of the third device 130 and the fourth device 140 may also receive encoded video data transmitted by the other of the third device 130 and the fourth device 140, and may decode the encoded video data to recover the video data, and may display the video images on an accessible display device based on the recovered video data.

[0042] Optionally, the third device 130 and the fourth device 140 may also selectively communicate directly (e.g., via...). Figure 1 The dashed line between the third device 130 and the fourth device 140 shown in the diagram indicates that they can communicate directly without going through the network 150, for example, the third device 130 and the fourth device 140 can communicate directly through short-range wireless communication technologies such as Bluetooth, Wi-Fi, and NFC.

[0043] exist Figure 1 In the embodiments shown, the first device 110, the second device 120, the third device 130 and the fourth device 140 may be servers or terminal devices, but the principles disclosed in this application are not limited to these.

[0044] It should be noted that servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal devices can be smartphones, tablets, laptops, desktop computers, smart speakers, smart voice interaction devices, smartwatches, smart home appliances, in-vehicle terminals, aircraft, etc., but are not limited to these.

[0045] Figure 1The network 150 shown represents any number of networks, including, for example, wired and / or wireless communication networks, that transmit encoded video data between the first device 110, the second device 120, the third device 130, and the fourth device 140. The communication network 150 may exchange data in circuit-switched and / or packet-switched channels. This network may include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet. For the purposes of this application, unless explained below, the architecture and topology of network 150 may be irrelevant to the operation of the disclosure herein.

[0046] In one embodiment of this application, Figure 2 The illustration shows the placement of video encoding and decoding devices in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital television (TV), and storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0047] The streaming system may include an acquisition subsystem 213, which may include a video source 201 such as a digital camera, which creates an uncompressed video image stream 202. In an embodiment, the video image stream 202 includes samples captured by a digital camera. The video image stream 202 is depicted as a thick line to emphasize the high data volume of the video image stream compared to encoded video data 204 (or encoded video bitstream 204). The video image stream 202 may be processed by a computer device 220, which includes a video encoding device 203 coupled to the video source 201. The video encoding device 203 may include hardware, software, or a combination of hardware and software to implement or enforce aspects of the disclosed subject matter as described in more detail below. The encoded video data 204 (or encoded video bitstream 204) is depicted as a thin line to emphasize the lower data volume of the encoded video data 204 (or encoded video bitstream 204), which may be stored on a streaming server 205 for future use. One or more streaming client subsystems, such as Figure 2 Client subsystems 206 and 208 can access streaming server 205 to retrieve copies 207 and 209 of encoded video data 204. Client subsystem 206 may include, for example, a video decoding device 210 in computer device 230. Video decoding device 210 decodes the incoming copy 207 of the encoded video data and produces an output video picture stream 211 that can be displayed on display 212 (e.g., a screen) or another presentation device. In some streaming systems, the encoded video data 204, video data 207, and video data 209 (e.g., video stream) may be encoded according to certain video encoding / compression standards.

[0048] It should be noted that computer devices 220 and 230 may include other components not shown in the figures. For example, computer device 220 may include a video decoding device, and computer device 230 may also include a video encoding device. Optionally, computer devices 220 and 230 may be terminal devices or servers.

[0049] In one embodiment of this application, taking High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), and the Chinese national video coding standard AVS as examples, after an input video frame image is obtained, the video frame image is divided into several non-overlapping processing units according to a block size. Each processing unit performs a similar compression operation. This processing unit is called a Coding Tree Unit (CTU) or Largest Coding Unit (LCU). The CTU can be further subdivided into one or more basic Coding Units (CUs), where the CU is the most basic element in the coding process.

[0050] In another embodiment, this processing unit can also be called a tile, which is a rectangular area of ​​a multimedia data frame that can be independently decoded and encoded. In the Alliance for Open Media Video 1 (AV1) standard, the tile can be further subdivided into one or more superblocks. The superblock is the starting point for block partitioning and can be further divided into multiple subblocks. Each superblock is the most basic element in a coding process. Optionally, a superblock can contain several blocks.

[0051] The above method of dividing video frame images can be called block partition structure. The following introduces some concepts in the encoding process: Predictive coding includes intra-frame prediction and inter-frame prediction. The original video signal is predicted from a selected reconstructed video signal to obtain a residual video signal. The encoder needs to decide which predictive coding mode to choose for the current coding unit (or coding block) and inform the decoder. Intra-frame prediction refers to the predicted signal coming from a region within the same image that has already been encoded and reconstructed; inter-frame prediction refers to the predicted signal coming from another encoded image (called a reference image) that is different from the current image.

[0052] Transform and Quantization: After the residual video signal undergoes transformation operations such as Discrete Fourier Transform (DFT) and Discrete Cosine Transform (DCT), the signal is transformed into the transform domain, and these are called transform coefficients. The transform coefficients are then subjected to lossy quantization, losing some information to make the quantized signal more suitable for compression. In some video coding standards, there may be more than one transform method to choose from. Therefore, the encoder needs to select one of the transform methods for the current coding unit (or coding block) and inform the decoder. The fineness of quantization is usually determined by the quantization parameter (QP). A larger QP value means that coefficients with a wider range of values ​​will be quantized into the same output, which usually results in greater distortion and a lower bit rate. Conversely, a smaller QP value means that coefficients with a smaller range of values ​​will be quantized into the same output, which usually results in less distortion and a higher bit rate.

[0053] Entropy coding, or statistical coding, involves statistically compressing the quantized transform-domain signal based on the frequency of each value, ultimately outputting a binary (0 or 1) compressed bitstream. Simultaneously, other information generated during encoding, such as the selected coding mode and motion vector data, also requires entropy coding to reduce the bit rate. Statistical coding is a lossless coding method that effectively reduces the bit rate required to represent the same signal. Common statistical coding methods include Variable Length Coding (VLC) and Content-Adaptive Binary Arithmetic Coding (CABAC).

[0054] Context-Based Binary Arithmetic Coding (CABAC) primarily involves three steps: binarization, context modeling, and binary arithmetic coding. After binarizing the input syntax elements, the binary data can be encoded using either a regular coding mode or a bypass coding mode. The bypass coding mode eliminates the need to assign a specific probability model to each binary bit; the input binary bit bin value is directly encoded using a simple bypass encoder, thus accelerating the overall encoding and decoding speed. Generally, different syntax elements are not completely independent, and even identical syntax elements possess a certain degree of memory. Therefore, according to conditional entropy theory, using other encoded syntax elements for conditional coding can further improve coding performance compared to independent coding or memoryless coding. This encoded symbol information used as conditions is called the context. In the regular coding mode, the binary bits of the syntax elements sequentially enter the context modeler. The encoder assigns an appropriate probability model to each input binary bit based on the values ​​of previously encoded syntax elements or binary bits; this process is called context modeling. The context model corresponding to a grammatical element can be located using the context index increment (ctxIdxInc) and the context index start (ctxIdxStart). After the bin value and the assigned probability model are fed into the binary arithmetic encoder for encoding, the context model needs to be updated based on the bin value, which is the adaptive process in encoding.

[0055] Loop Filtering: The transformed and quantized signal undergoes inverse quantization, inverse transform, and prediction compensation to obtain a reconstructed image. Due to the effects of quantization, the reconstructed image differs from the original image in some aspects, resulting in distortion. Therefore, filtering operations can be performed on the reconstructed image, such as deblocking filters (DB), sample adaptive offset (SAO), or adaptive loop filters (ALF), to effectively reduce the distortion caused by quantization. Since these filtered reconstructed images will serve as a reference for subsequent coded images to predict future image signals, the aforementioned filtering operations are also called loop filtering, i.e., filtering operations within the coding loop.

[0056] In one embodiment of this application, Figure 3 A basic flowchart of a video encoder is shown, illustrating the process using intra-frame prediction as an example. The original image signal... With predicted image signal Perform the difference operation to obtain the residual signal. , the residual signal After transformation and quantization, quantization coefficients are obtained. These coefficients are then used to obtain the encoded bitstream through entropy encoding, and to obtain the reconstructed residual signal through inverse quantization and inverse transform. Predicting image signals With reconstructed residual signal Image signals generated by superposition Image signal On one hand, the signal is input to the intra-frame mode decision module and the intra-frame prediction module for intra-frame prediction processing; on the other hand, the reconstructed image signal is output through loop filtering. Reconstructing image signals It can be used as a reference image for the next frame for motion estimation and motion compensation prediction. Then, based on the results of the motion compensation prediction... Intra-frame prediction results Obtain the predicted image signal for the next frame. And continue repeating the above process until the encoding is complete.

[0057] Based on the above encoding process, at the decoding end, for each coding unit (or coding block), after acquiring the compressed bitstream (i.e., bitstream), entropy decoding is performed to obtain various mode information and quantization coefficients. Then, the quantization coefficients undergo inverse quantization and inverse transform processing to obtain the residual signal. On the other hand, based on the known coding mode information, the prediction signal corresponding to the coding unit (or coding block) can be obtained. Then, the residual signal and the prediction signal are added to obtain the reconstructed signal. The reconstructed signal then undergoes loop filtering and other operations to generate the final output signal. In this series of encoding processes, the coding framework mainly relies on the Rate Distortion Optimization (RDO) criterion to evaluate and select the optimal coding parameters and results.

[0058] In simple terms, the overall transmission process of a video file is as follows: Figure 4 As shown, a video file is obtained through video capture, and then after video encoding and video file encapsulation, the video file is transmitted to the receiver. After receiving the video file, the receiver decapsulates the video file, performs video decoding, and generates video frames through rendering to present the final video stream.

[0059] With the continuous development of video technology, auxiliary information is introduced into the video encoding process to improve encoding performance or achieve specific functions (such as video enhancement processing). For example, auxiliary information can be embedded into the video bitstream in the form of Supplemental Enhancement Information (SEI). However, when the auxiliary information reaches the same resolution as the image, using SEI to carry it will significantly increase the data volume of the video bitstream. Related technologies also include methods to increase the width or height of the original image by stitching it horizontally or vertically to carry auxiliary information. However, stitching increases resolution, which in turn increases bandwidth costs. Introducing additional transmission links to transmit auxiliary information not only increases bitstream overhead but also increases the complexity of the decoding process.

[0060] Based on the aforementioned technical problems, this application provides a novel video encoding and decoding scheme. The scheme uses the first color channel data of the video content frame as the first color channel data of the auxiliary information frame associated with the video content frame, and uses the auxiliary information corresponding to the video content frame as the second color channel data of the auxiliary information frame to generate the auxiliary information frame. This leverages the strong spatial correlation between the video content and the auxiliary information, significantly reducing the encoding residual of the auxiliary information frame and effectively improving encoding compression efficiency. Furthermore, embedding the auxiliary information into the second color channel of the auxiliary information frame and encoding it together with the video content frame ensures the reliability and synchronization of the auxiliary information transmission.

[0061] The implementation details of the technical solutions in the embodiments of this application are described in detail below: Figure 5 A flowchart of a video encoding method according to an embodiment of this application is shown. This video encoding method can be executed by a computer device, such as the terminal device in the foregoing embodiments or the server in the foregoing embodiments. (Refer to...) Figure 5 As shown, this video encoding method includes at least S510 to S540, which are detailed below: In S510, video content frames and corresponding auxiliary information are acquired. This auxiliary information is used to describe the visual features of the video content frames.

[0062] It should be noted that since video image data is typically processed frame by frame, video content frames primarily contain the raw video image data. Auxiliary information mainly describes the visual features of the video content frames, such as data strongly correlated with pixel-level or region-level visual features. Specifically, visual features such as object outlines, edges, and textures in video content frames can be represented in the auxiliary information as corresponding values ​​(such as depth abrupt changes, category boundaries, etc.).

[0063] As an example, if a video content frame is used to describe a human face, the corresponding auxiliary information could be a depth map of that face (such as features representing a high nose bridge and deep eye sockets). Similarly, if a video content frame is used to describe a street, the corresponding auxiliary information could be a semantic segmentation map (such as segmenting the street surface as black, the cars on the street as red, etc.).

[0064] Optionally, auxiliary information may include: a depth map for describing distance information, a semantic map for describing object categories, an optical flow field for describing motion vectors, and alpha channel information for defining which parts of the image are transparent.

[0065] Optionally, the video content frames and the corresponding auxiliary information can be strictly aligned in terms of display timestamp and spatial resolution. That is, the auxiliary information at a certain moment needs to correspond precisely to the video content frame at the same moment, and the length and width pixels of the two should be as consistent as possible.

[0066] In some optional embodiments, the acquired video content can be analyzed in advance before video encoding begins to determine the visual features corresponding to the video content, and then the visual features can be used as auxiliary information to establish a one-to-one correspondence between the video content and the spatial structure.

[0067] In S520, the first color channel data of the video content frame in the specified color space is extracted.

[0068] It should be noted that the specified color space can be any existing or future color space, such as RGB color space, YUV color space, YCbCr color space, etc.

[0069] As an example, let's consider a YUV color space. If the video content frame is an RGB image, then because luminance and color information are mixed in the R, G, and B color channels in the RGB color space, and the correlation between these three color channels is high, direct data processing results in significant data redundancy. Therefore, the video content frame can be converted from the RGB color space to the YUV color space to separate the video content frame into luminance and chrominance signals. The first color channel data can be luminance (Y) channel data, or it can be chrominance (such as U and V channels, or even just the U channel or the V channel) channel data.

[0070] In S530, the first color channel data of the video content frame is used as the first color channel data of the auxiliary information frame associated with the video content frame, and the auxiliary information is used as the second color channel data of the auxiliary information frame to generate the auxiliary information frame.

[0071] It should be noted that the S530 is primarily designed to construct a new auxiliary information frame based on the video content frame. The first color channel data of this auxiliary information frame can be directly reused from the first color channel data of the video content frame, while the second color channel data can be used to store auxiliary information (such as depth maps, semantic segmentation maps, optical flow data, alpha channel information, etc.). In this way, the auxiliary information frame maintains consistency with the video content frame in its first color channel, but stores auxiliary data in its second color channel. Due to the high content correlation between the auxiliary information frame and the video content frame, when the encoder uses the video content frame as a reference frame to encode and compress the auxiliary information frame, the calculated residual data is smaller, thus achieving a higher compression ratio.

[0072] In some optional embodiments, taking the YUV color space as an example, the first color channel data can be luminance channel data. Then, the luminance channel data of the video content frame can be used as the luminance channel data of the auxiliary information frame, and the auxiliary information can be used as the chroma channel data of the auxiliary information frame to generate the auxiliary information frame. Optionally, the auxiliary information can be used as the U channel data of the auxiliary information frame, or the auxiliary information can be used as the V channel data of the auxiliary information frame, or the auxiliary information can be used as both the U and V channels of the auxiliary information frame.

[0073] In some optional embodiments, taking the YUV color space as an example, the first color channel data can be chroma channel data. In this case, the chroma channel data of the video content frame can be used as the chroma channel data of the auxiliary information frame, and the auxiliary information can be used as the luminance channel data of the auxiliary information frame to generate the auxiliary information frame. Optionally, the U channel data of the video content frame can be used as the U channel data of the auxiliary information frame, or the V channel data of the video content frame can be used as the V channel data of the auxiliary information frame, or both the U and V channel data of the video content frame can be used as the U and V channel data of the auxiliary information frame.

[0074] Continue to refer to Figure 5 As shown, in S540, the video content frames and auxiliary information frames are encoded to generate a video stream, and an association identifier for associating the video content frames and auxiliary information frames is added to the generated video stream.

[0075] In some optional embodiments, when encoding video content frames and auxiliary information frames, a more suitable target encoding parameter can be selected from the candidate encoding parameter set for encoding processing, so as to improve the encoding efficiency of video content frames and auxiliary information frames, while ensuring that the auxiliary information is not over-compressed and thus becomes unusable at the decoding end.

[0076] Optionally, the video content frame and auxiliary information frame can be encoded using various candidate coding parameters from the candidate coding parameter set, and the encoding result can be decoded to obtain the reconstruction result of the auxiliary information frame. Then, the error between the first color channel data of the video content frame and the first color channel data in the reconstruction result is calculated as the reconstruction error between the video content frame and the auxiliary information frame. Furthermore, based on the reconstruction error between the video content frame and the auxiliary information frame, a target coding parameter is selected from the candidate coding parameter set, and the target coding parameter is used to encode the video content frame and the auxiliary information frame. The technical solution of this embodiment, by introducing a coding parameter optimization mechanism based on reconstruction error, can achieve fine-grained control over the joint encoding of the video content frame and the auxiliary information frame. This can improve coding efficiency while avoiding the problem of the auxiliary information frame being over-compressed, resulting in the decoding end being unable to parse the encoded information from it. It ensures that the encoder can select the optimal coding parameters from the candidate parameter set that can both guarantee the subjective quality of the video and maintain the structural integrity of the auxiliary information frame. This can improve the reconstruction fidelity of the auxiliary information without significantly increasing the bitrate, providing a high-quality data foundation for subsequent video applications based on auxiliary information (such as free-viewpoint video, intelligent analysis, etc.).

[0077] Optionally, in other embodiments of this application, the error between the reconstruction result of the first color channel data of the video content frame and the reconstruction result of the first color channel data of the auxiliary information frame can also be calculated as the reconstruction error between the video content frame and the auxiliary information frame.

[0078] In some optional embodiments, when selecting a target coding parameter from the candidate coding parameter set based on the reconstruction error between the video content frames and the auxiliary information frames, the total reconstruction error corresponding to the video to be encoded can be calculated based on the reconstruction error between the video content frames and the auxiliary information frames, as well as the number of video content frames contained in the video to be encoded. Then, based on the total reconstruction error corresponding to the video to be encoded, the candidate coding parameter with the smallest total reconstruction error is selected from the candidate coding parameter set as the target coding parameter. The technical solution of this embodiment can select the coding parameter with the smallest total reconstruction error, which can not only improve compression efficiency, but also effectively guarantee the reconstruction quality of the video content frames and auxiliary information frames.

[0079] Optionally, if the candidate coding parameter set contains multiple candidate coding parameters with the smallest total reconstruction error, the candidate coding parameter with the lowest bitrate can be selected from these multiple candidate coding parameters with the smallest total reconstruction error as the target coding parameter. The technical solution of this embodiment, by selecting the lowest bitrate parameter from multiple candidate coding parameters with the smallest total reconstruction error, can effectively reduce the bandwidth usage of the final bitstream, thereby improving the transmission efficiency of streaming media data.

[0080] Optionally, when calculating the total reconstruction error of the video to be encoded based on the reconstruction error between video content frames and auxiliary information frames, and the number of video content frames contained in the video to be encoded, a statistical value of the reconstruction error can be calculated based on the number of video content frames contained in the video to be encoded, and the reconstruction error between each video content frame and its associated auxiliary information frame. This statistical value is then used as the total reconstruction error of the video to be encoded. Optionally, the statistical value of the reconstruction error can be the average, sum, or variance of the reconstruction errors between each video content frame and its associated auxiliary information frame.

[0081] In some optional embodiments, a hash calculation can be performed on at least one of the following information: the identification information of the video content frame, some or all of the data contained in the video content frame, the identification information of the auxiliary information frame, and some or all of the data contained in the auxiliary information frame; then the calculated hash value is used as an association identifier for associating the video content frame with the auxiliary information frame. Optionally, the identification information of the video content frame may be one or more of the following: Picture Order Count (POC), Presentation Time Stamp (PTS), Temporal ID (TID), or other information that serves an identification function. Similarly, the identification information of the auxiliary information frame may be one or more of the following: POC, PTS, TID, or other information that serves an identification function. Optionally, some or all of the data contained in the video content frame may be data of one or more color channels in the video content frame; some or all of the data contained in the auxiliary information frame may be data of one or more color channels in the auxiliary information frame.

[0082] In some optional embodiments, an association identifier for associating video content frames with auxiliary information frames is added to the generated video bitstream. This can be done by generating supplemental enhancement information (SEI), then adding the association identifier to the SEI, and finally adding the SEI to the video bitstream. In this embodiment, since the SEI is a container defined in the video coding standard for carrying non-core encoded data, the technical solution of this embodiment does not require modification of the core syntax structure of video coding. When parsing the bitstream, the decoder can directly identify and extract the association identifier in the SEI according to the standard protocol. This ensures the standardized transmission of the association identifier in the bitstream and avoids compatibility issues introduced by custom syntax. This allows the solution to be adapted to various decoding devices that support standard video coding protocols, significantly improving the versatility of the solution.

[0083] In some optional embodiments, before encoding the video content frames and auxiliary information frames, the video content frames and auxiliary information frames can be arranged in sequence to obtain a sequence of frames to be encoded, and then the sequence of frames to be encoded is encoded. Optionally, the video content frames can be arranged before their associated auxiliary information frames, so that multiple pairs of frames can be formed for the video to be encoded, each pair of frames containing the video content frames and their associated auxiliary information frames.

[0084] Figure 5 This description focuses on the technical solutions of the embodiments of this application from the perspective of the video encoding end. The following is a combination of... Figure 6 The technical solutions of the embodiments of this application are described from the perspective of the video decoding end.

[0085] Figure 6 A flowchart of a video decoding method according to an embodiment of this application is shown. This video decoding method can be executed by a computer device, such as the terminal device in the foregoing embodiments or the server in the foregoing embodiments. (Refer to...) Figure 6 As shown, this video decoding method includes at least S610 to S620, which are described in detail below: In S610, the video stream is parsed to obtain video content frames and auxiliary information frames associated with the video content frames. The first color channel data of the auxiliary information frame is the first color channel data of the video content frame, and the second color channel data of the auxiliary information frame contains auxiliary information corresponding to the video content frame. This auxiliary information is used to describe the visual features of the video content frame.

[0086] It should be noted that the relevant descriptions of video content frames and auxiliary information frames can be found in the technical solutions of the foregoing embodiments, and will not be repeated here.

[0087] In some optional embodiments, the process of parsing the video stream to obtain video content frames and associated auxiliary information frames can involve parsing the video stream to obtain multiple frames and association identifiers for each frame. Since the associated video content frames and auxiliary information frames have matching association identifiers, the associated video content frames and auxiliary information frames can be determined based on the parsed association identifiers. Optionally, the association identifiers for each frame can be parsed from the SEI (Search Engine Information). That is, the SEI contains the association identifiers for each frame, and the method for generating these association identifiers can refer to the technical solutions of the foregoing embodiments.

[0088] Optionally, the matching association identifier can refer to the same association identifier, that is, the associated video content frame and the auxiliary information frame have the same association identifier. In other embodiments of this application, the associated video content frame and the auxiliary information frame may also have different association identifiers, but their association identifiers have a defined relationship, such as adding the identifier information of the associated auxiliary information frame (such as POC, PTS, etc.) to the association identifier of the video content frame to associate the video content frame with the auxiliary information frame.

[0089] In some optional embodiments, when determining the associated video content frames and auxiliary information frames based on the association identifiers of each frame obtained from parsing, two frames with the same association identifier can be determined based on the association identifiers of each frame obtained from parsing, and then the video content frames and auxiliary information frames in the two frames can be identified based on the information of the two frames with the same association identifier.

[0090] For example, the frame with the smallest display timestamp among two frames with the same association identifier can be used as the video content frame, and the frame with the largest display timestamp among two frames with the same association identifier can be used as the auxiliary information frame. This is because the video content frame is generally displayed before its associated auxiliary information frame, so the frame with the smallest display timestamp can be used as the video content frame.

[0091] For example, among two frames with the same association identifier, the frame with an odd-numbered sequence number can be considered the video content frame, and the frame with an even-numbered sequence number can be considered the auxiliary information frame. This is because video content frames and auxiliary information frames appear in pairs, and the video content frame precedes the auxiliary information frame. Therefore, the frame with an odd-numbered sequence number among two frames with the same association identifier can be considered the video content frame, and the frame with an even-numbered sequence number among two frames with the same association identifier can be considered the auxiliary information frame.

[0092] In some optional embodiments, when identifying the video content frame and auxiliary information frame in two frames based on information from two frames with the same association identifier, the display timestamps of the two frames with the same association identifier can be obtained first. If the difference between the display timestamps of the two frames with the same association identifier is within a set range, then the video content frame and auxiliary information frame in the two frames can be further identified. This embodiment's technical solution improves the accuracy of determining the association relationship between video content frames and auxiliary information frames by judging whether the difference between the display timestamps of two frames with the same association identifier is within a set range. This is because there may be some abnormal reasons (such as randomness in the calculation of the association identifier) ​​that cause video content frames and auxiliary information frames that are not related to each other to have the same association identifier. However, the display timestamps of related video content frames and auxiliary information frames are generally close to each other. Therefore, judging whether the difference between the display timestamps of two frames with the same association identifier is within a set range can determine whether there is a real association relationship between the two frames.

[0093] In some optional embodiments, the association identifiers of video content frames and auxiliary information frames can be parsed from the video stream. Then, based on the compatibility between the association identifiers of video content frames and auxiliary information frames, it can be determined whether there is an association between the video content frames and auxiliary information frames. If there is an association, the video content frames can be rendered based on the auxiliary information contained in the auxiliary information frames.

[0094] For example, video streams typically arrange frames to be decoded in the order of video content frames first, followed by their associated auxiliary information frames. After parsing a pair of frames (the video content frame first and the auxiliary information frame second), their association identifiers can be used to determine if a relationship exists between the frames. If a relationship exists, the video content frame can be rendered based on the auxiliary information contained in the auxiliary information frame. If no relationship exists (i.e., the association identifier of the video content frame and the association identifier of the auxiliary information frame do not match), the auxiliary information frame can be discarded, meaning it is not used to render the video content frame.

[0095] Continue to refer to Figure 6 As shown, in S620, the video content frame is rendered based on the auxiliary information contained in the auxiliary information frame.

[0096] Optionally, the process of rendering video content frames based on auxiliary information contained in auxiliary information frames mainly involves first extracting auxiliary information from the auxiliary information, such as extracting auxiliary information from the second color channel of the auxiliary information frame, and then rendering video content frames based on the visual features represented by the auxiliary information.

[0097] For example, auxiliary information can be depth information. The renderer can then calculate the blur radius based on this depth information to achieve a blurred background (farr in depth) and a sharp foreground (nearer in depth), thereby generating a video with a cinematic background blur effect. Alternatively, auxiliary information can be semantic segmentation information. The renderer can then identify the sky region in the video content frame based on this semantic segmentation information, thereby increasing the brightness or saturation of that region. Simultaneously, it can also identify facial regions in the video content frame and perform corresponding beautification processing to achieve intelligent image enhancement.

[0098] In some optional embodiments, after rendering the video content frames, the display timestamps of the video content frames are used to perform audio-visual synchronization processing with the audio frames. Since the video stream contains auxiliary information frames and video content frames, and this embodiment uses the display timestamps of the video content frames to perform audio-visual synchronization processing with the audio frames, it ensures strict alignment of the final video image and audio signal in playback timing after auxiliary information enhancement processing. This guarantees both the real-time presentation of high-quality rendered images and maintains millisecond-level synchronization accuracy with the audio stream, thereby improving visual quality while ensuring overall coordination of audio and video playback and a immersive user experience.

[0099] In summary, the technical solution of the above embodiments of this application generates auxiliary information frames by using the first color channel data of the video content frame as the first color channel data of the auxiliary information frame associated with the video content frame, and using the auxiliary information corresponding to the video content frame as the second color channel data of the auxiliary information frame. This can take advantage of the strong correlation between the video content and the auxiliary information in the spatial structure, significantly reducing the coding residual of the auxiliary information frame, thereby effectively improving the coding compression efficiency. Furthermore, embedding the auxiliary information into the second color channel of the auxiliary information frame and encoding it together with the video content frame can ensure the reliability and synchronization of the auxiliary information transmission.

[0100] The implementation details of the technical solutions in the embodiments of this application will be described in detail below with reference to specific examples: In one example of this application, it is assumed that the first video to be encoded is... A video content frame can be represented as: , No. The auxiliary information frame corresponding to the video content frame can be represented as: So, if Figure 7 As shown, the processing at the video encoding end may include the following steps: S701: For the input original video, extract the auxiliary information (such as transparency mask information), and split it into video content frames (i.e., the original video frames) and auxiliary information frames. Then arrange them in the order of video content frames first and auxiliary information frames last, as shown below:

[0101] Optionally, auxiliary information frames The width and height can be limited to half of the original image to reduce bitrate, and auxiliary information frames can be limited. Storing data from only two channels can be represented as In other embodiments of this application, auxiliary information frames It can also store data from one channel.

[0102] S702, based on the video's metadata, processes the video content frames (i.e., the original video frames). Perform color space conversion, that is, convert to the YUV color space. This process can be represented as:

[0103] S703 uses the Y channel data of the video content frame, along with the two channels of auxiliary information (i.e., The two frames are merged to generate a new auxiliary information frame. The Y-channel data in the new auxiliary information frame is the same as the Y-channel data in the video content frame. UV channel data in the new auxiliary information frame .

[0104] S704, followed by encoding of video content frames and auxiliary information frames.

[0105] Optionally, a sequence of frames to be encoded can be formed based on video content frames and generated auxiliary information frames, for example, it can be represented as:

[0106] Then, a synchronization identifier (also called an association identifier) ​​can be inserted for each pair of frames (i.e., each pair of video content frames and its associated auxiliary information frames) to identify the association relationship between each pair of frames. Optionally, a hash algorithm can be used to perform hash operations on the video content frames and / or auxiliary information frames to calculate the synchronization identifier, denoted as . .

[0107] Optionally, after inserting the synchronization flag, the sequence of frames to be encoded can be represented as:

[0108] It should be noted that: synchronization identifier During the encoding process, SEI (Sequence of Entities) can be inserted into the video stream. The resulting sequence of frames can then be encoded.

[0109] In some optional embodiments, although the auxiliary information frames in this application can maintain the same Y-channel data as the video content frames, different encoding parameters may still cause the auxiliary information to be over-compressed, making it unusable during playback. Therefore, appropriate encoding parameters can be set to ensure that the auxiliary information is usable after decoding, while keeping the encoding cost to a minimum. Specifically, refer to... Figure 8 As shown, the candidate encoding parameter set can be traversed. That is, for each encoding parameter Perform the following procedure: Use encoding parameters Each pair of frames in the video to be encoded (i.e., each pair of video content frames and its associated auxiliary information frames) is encoded, and the encoded output is decoded to obtain the reconstruction result of each pair of frames. Then, the luminance component reconstruction error of each pair of frames can be calculated. This refers to comparing the error between the original Y-channel data of the video content frame and the Y-channel data reconstructed from the decoded auxiliary frame. Alternatively, in addition to calculating the absolute value of the difference as shown in the example above, other calculation methods such as MSE (Mean Squared Error) can be used, and the comparison can also be performed block by block by dividing the frame.

[0110] The luminance component reconstruction error for each pair of frames is calculated. After that, it can be recorded as ,in Indicates encoding parameters The corresponding bitrate.

[0111] For including For the video to be encoded, the total error is It can be expressed using average error or other calculation methods (such as summation, variance, etc.). For example, it can be expressed using average error as follows: .

[0112] The total error was calculated. Then, comparisons can be made from the candidate encoding parameter set. The encoding parameter with the smallest total error is selected. If there are multiple encoding parameters with the smallest total error, the encoding parameter with the lowest bitrate is selected from these multiple encoding parameters. Then, the selected optimal encoding parameter is used for encoding to obtain the encoded video bitstream with auxiliary information.

[0113] It should be noted that the encoding parameters in the embodiments of this application may include, but are not limited to: quantization parameters, chroma QP offset, group of pictures (GOP) structure and reference frame configuration, loop filter strength, etc.

[0114] In some alternative embodiments, candidate coding parameter sets can be exhaustively enumerated. The optimal coding parameters described above are selected. However, when there are many candidate coding parameters in the candidate parameter set, the computational load is extremely large and the speed is slow. Therefore, optimization methods can also be used for search selection. For example, a grid search or coarse-grained search combined with local refinement can be used for search. For example, try several coding parameters at intervals (coarse-grained search) to find the approximate optimal region, and then perform dense testing within this small range (refinement method).

[0115] Alternatively, gradient descent, Bayesian optimization, genetic algorithms, or parameter prediction methods based on historical frame statistics can also be used. For example, by utilizing the temporal correlation of the video to find the optimal coding parameters of the previous frame, the optimal coding parameters of the next frame are likely to be near this optimal coding parameter.

[0116] It should be noted that: Figure 7 The description uses the Y-channel data of a video content frame to generate an auxiliary information frame as an example. In other embodiments of this application, the UV-channel data of the video content can also be reused to generate auxiliary information frames. For example, refer to... Figure 9 As shown, for the input original video, auxiliary information (such as transparency masking information) is extracted to obtain video content frames (i.e., the original video frames) and auxiliary information frames. Then, the UV channel data of the video content frames are used as the UV channel data of the auxiliary information frames, and the auxiliary information in the auxiliary information frames is used as the Y channel data to generate frame N+1 to be encoded, while the video content frames are used as frame N to be encoded. That is, the video content frames are arranged before the auxiliary information frames. Simultaneously, a synchronization identifier can be generated according to the aforementioned embodiment to obtain the sequence of frames to be encoded. Then, the sequence of frames to be encoded can be encoded to generate a video stream with auxiliary information. Optionally, a similar method to that in the aforementioned embodiment can be used to select the optimal encoding parameters to encode the sequence of frames to be encoded.

[0117] In some optional embodiments, the processing flow at the decoding end can be referred to Figure 10 As shown, after obtaining the video stream with auxiliary information, the video stream can be decoded to obtain video content frames (i.e., original video frames) and auxiliary information frames. Simultaneously, the corresponding synchronization identifier can be parsed from the SEI. For example, as a case study, the parsed frames and their corresponding synchronization identifiers can be represented as: .

[0118] Then, the synchronization identifiers can be compared to match the video content frames with the auxiliary information frames, i.e., if... That means These are video content frames. This is the video content frame. Associated auxiliary information frames are used because video content frames are usually arranged before auxiliary information frames, or they can be distinguished by their PTS (Pulse Time Signature) (since video content frames generally have a smaller PTS, while auxiliary information frames generally have a larger PTS). The next step of rendering can then be performed using the auxiliary information frames. The auxiliary information contained in the video content frames Perform rendering and compositing; if This indicates that the video content frames With auxiliary information frames There is no correlation between them, i.e., they do not match, so the auxiliary information frames can be discarded. Only video content frames are used. The included content is rendered and composited.

[0119] In some optional embodiments, after rendering the video content frames, the PTS of the video content frames can be used to perform audio-visual synchronization processing with the audio frames to ensure strict alignment of the final video picture and audio signal in the playback sequence after auxiliary information enhancement processing.

[0120] The technical solutions of the above embodiments of this application, by using independent auxiliary information frames combined with encoding optimization, can carry auxiliary information at a lower additional bit rate while maintaining compatibility with standard encoding and decoding processes. Limited by the YUV420 sampling format, the chroma resolution of the auxiliary information can be 1 / 4 of the original resolution. For applications requiring pixel-level precise alignment, the encoding format can be switched to YUV444. For most non-precisely aligned applications (such as HDR gain maps, depth maps, coarse-grained masks, etc.), the technical solutions of the embodiments of this application can carry sufficiently precise auxiliary information with a low bit rate overhead.

[0121] The following describes an apparatus embodiment of this application, which can be used to execute the video encoding method described above in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the video encoding method described above in this application.

[0122] Figure 11 A block diagram of a video encoding apparatus according to an embodiment of the present application is shown. The video encoding apparatus can be applied to a computer device, such as the terminal device in the foregoing embodiments or the server in the foregoing embodiments.

[0123] Reference Figure 11 As shown, a video encoding apparatus 1100 according to an embodiment of this application includes: an acquisition unit 1102, an extraction unit 1104, a generation unit 1106, and an encoding unit 1108.

[0124] The acquisition unit 1102 is configured to acquire a video content frame and the corresponding auxiliary information, wherein the auxiliary information is used to describe the visual features of the video content frame; the extraction unit 1104 is configured to extract the first color channel data of the video content frame in a specified color space; the generation unit 1106 is configured to use the first color channel data of the video content frame as the first color channel data of the auxiliary information frame associated with the video content frame, and use the auxiliary information as the second color channel data of the auxiliary information frame to generate the auxiliary information frame; the encoding unit 1108 is configured to encode the video content frame and the auxiliary information frame to generate a video stream, and add an association identifier for associating the video content frame with the auxiliary information frame in the video stream.

[0125] In some embodiments of this application, based on the foregoing scheme, the encoding unit 1108 is configured to: encode the video content frame and the auxiliary information frame using each candidate encoding parameter in the candidate encoding parameter set, and decode the encoding result to obtain the reconstruction result of the auxiliary information frame; calculate the reconstruction error between the first color channel data of the video content frame and the first color channel data in the reconstruction result, as the error between the video content frame and the auxiliary information frame; select a target encoding parameter from the candidate encoding parameter set according to the reconstruction error between the video content frame and the auxiliary information frame, and use the target encoding parameter to encode the video content frame and the auxiliary information frame.

[0126] In some embodiments of this application, based on the foregoing scheme, selecting a target coding parameter from the candidate coding parameter set according to the reconstruction error between the video content frame and the auxiliary information frame includes: calculating the total reconstruction error corresponding to the video to be encoded based on the reconstruction error between the video content frame and the auxiliary information frame and the number of video content frames contained in the video to be encoded; and selecting the candidate coding parameter with the smallest total reconstruction error from the candidate coding parameter set as the target coding parameter based on the total reconstruction error corresponding to the video to be encoded.

[0127] In some embodiments of this application, based on the aforementioned scheme, if the candidate coding parameter set contains multiple candidate coding parameters with the smallest total reconstruction error, then the candidate coding parameter with the smallest code rate is selected from the multiple candidate coding parameters with the smallest total reconstruction error as the target coding parameter.

[0128] In some embodiments of this application, based on the foregoing scheme, the total reconstruction error corresponding to the video to be encoded is calculated according to the reconstruction error between the video content frame and the auxiliary information frame, and the number of video content frames contained in the video to be encoded. This includes: calculating a statistical value of the reconstruction error according to the number of video content frames contained in the video to be encoded, and the reconstruction error between each video content frame and the associated auxiliary information frame, and using the calculated statistical value as the total reconstruction error corresponding to the video to be encoded.

[0129] In some embodiments of this application, based on the foregoing scheme, the generation unit 1106 is further configured to: perform hash calculation on at least one of the following information: the identification information of the video content frame, some or all of the data contained in the video content frame, the identification information of the auxiliary information frame, and some or all of the data contained in the auxiliary information frame; and use the calculated hash value as an association identifier for associating the video content frame with the auxiliary information frame.

[0130] In some embodiments of this application, based on the foregoing scheme, the designated color space includes the YUV color space, and the first color channel data includes luminance channel data; Specifically, generating the auxiliary information frame by using the first color channel data of the video content frame as the first color channel data of the auxiliary information frame corresponding to the video content frame, and using the auxiliary information as the second color channel data of the auxiliary information frame, includes: using the luminance channel data of the video content frame as the luminance channel data of the auxiliary information frame, and using the auxiliary information as the chroma channel data of the auxiliary information frame, to generate the auxiliary information frame.

[0131] In some embodiments of this application, based on the foregoing scheme, the designated color space includes the YUV color space, and the first color channel data includes chroma channel data; Specifically, generating the auxiliary information frame by using the first color channel data of the video content frame as the first color channel data of the auxiliary information frame corresponding to the video content frame, and using the auxiliary information as the second color channel data of the auxiliary information frame, includes: using the chroma channel data of the video content frame as the chroma channel data of the auxiliary information frame, and using the auxiliary information as the luminance channel data of the auxiliary information frame, to generate the auxiliary information frame.

[0132] In some embodiments of this application, based on the foregoing scheme, an association identifier for associating the video content frame with the auxiliary information frame is added to the video bitstream, including: generating supplementary enhancement information, adding the association identifier to the supplementary enhancement information, and adding the supplementary enhancement information to the video bitstream.

[0133] Figure 12 A block diagram of a video decoding apparatus according to an embodiment of the present application is shown. The video decoding apparatus can be applied to a computer device, such as the terminal device in the foregoing embodiments or the server in the foregoing embodiments.

[0134] Reference Figure 12 As shown, a video decoding apparatus 1200 according to an embodiment of this application includes: a parsing unit 1202 and a processing unit 1204.

[0135] The parsing unit 1202 is configured to parse the video stream to obtain a video content frame and an auxiliary information frame associated with the video content frame. The first color channel data of the auxiliary information frame is the first color channel data of the video content frame, and the second color channel data of the auxiliary information frame contains auxiliary information corresponding to the video content frame. The auxiliary information is used to describe the visual features of the video content frame. The processing unit 1204 is configured to render the video content frame according to the auxiliary information contained in the auxiliary information frame.

[0136] In some embodiments of this application, based on the foregoing scheme, the parsing unit 1202 is configured to: parse the video stream to obtain multiple frames and the association identifier of each frame in the multiple frames; determine the associated video content frame and auxiliary information frame according to the association identifier of each frame obtained by parsing; wherein the associated video content frame and auxiliary information frame have matching association identifiers.

[0137] In some embodiments of this application, based on the aforementioned scheme, determining associated video content frames and auxiliary information frames according to the association identifiers of each frame obtained through parsing includes: determining two frames with the same association identifier based on the association identifiers of each frame obtained through parsing; and identifying the video content frames and auxiliary information frames in the two frames based on the information of the two frames with the same association identifier.

[0138] In some embodiments of this application, based on the foregoing scheme, the video content frame and auxiliary information frame in the two frames with the same association identifier are identified according to the information of the two frames, including at least one of the following methods: taking the frame with the smallest display timestamp among the two frames with the same association identifier as the video content frame, and taking the frame with the largest display timestamp among the two frames with the same association identifier as the auxiliary information frame; taking the frame with an odd number of sequence number among the two frames with the same association identifier as the video content frame, and taking the frame with an even number of sequence number among the two frames with the same association identifier as the auxiliary information frame.

[0139] In some embodiments of this application, based on the aforementioned scheme, the video content frame and auxiliary information frame in the two frames with the same association identifier are identified according to the information of the two frames with the same association identifier, including: obtaining the display timestamps of the two frames with the same association identifier; if the difference between the display timestamps of the two frames with the same association identifier is within a set range, then the video content frame and auxiliary information frame in the two frames are identified.

[0140] In some embodiments of this application, based on the foregoing scheme, the parsing unit 1202 is further configured to: parse the association identifier of the video content frame and the association identifier of the auxiliary information frame from the video bitstream; the processing unit 1204 is configured to: if the association identifier of the video content frame and the association identifier of the auxiliary information frame are compatible, then render the video content frame according to the auxiliary information contained in the auxiliary information frame.

[0141] In some embodiments of this application, based on the foregoing scheme, the processing unit 1204 is further configured to: discard the auxiliary information frame if the association identifier of the video content frame does not match the association identifier of the auxiliary information frame.

[0142] In some embodiments of this application, based on the foregoing scheme, the processing unit 1204 is further configured to: after rendering the video content frame, use the display timestamp of the video content frame and the audio frame for audio-visual synchronization processing.

[0143] Figure 13 A schematic diagram of a computer system suitable for implementing the computer device of the present application is shown. The computer device may be the device used to perform video encoding method or video decoding method in the foregoing embodiments.

[0144] It should be noted that, Figure 13 The computer system 1300 of the computer device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0145] like Figure 13 As shown, the computer system 1300 may include a Central Processing Unit (CPU) 1301, which can perform various appropriate actions and processes based on a program stored in Read-Only Memory (ROM) 1302 or a program loaded from storage portion 1308 into Random Access Memory (RAM) 1303, such as performing the methods described in the above embodiments. Various programs and data required for system operation are also stored in RAM 1303. The CPU 1301, ROM 1302, and RAM 1303 are interconnected via bus 1304. An input / output (I / O) interface 1305 is also connected to bus 1304.

[0146] The following components can be connected to I / O interface 1305: an input section 1306 including a keyboard, mouse, etc.; an output section 1307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1308 including a hard disk, etc.; and a communication section 1309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to I / O interface 1305 as needed. Removable media 1311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1310 as needed so that computer programs read from them can be installed into storage section 1308 as needed.

[0147] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1309, and / or installed from removable medium 1311. When the computer program is executed by central processing unit (CPU) 1301, it performs various functions defined in the system of this application.

[0148] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a computer program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0149] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and a computer program.

[0150] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0151] In another aspect, this application also provides a computer-readable medium, which may be included in the computer device described in the above embodiments; or it may exist independently and not assembled into the computer device. The computer-readable medium carries one or more computer programs, which, when executed by the computer device, cause the computer device to perform the methods described in the above embodiments.

[0152] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0153] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computer device to execute the method according to the embodiments of this application. For example, the computer device can execute... Figure 5 The video encoding method or execution shown Figure 6 The video decoding method shown.

[0154] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0155] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method of video coding, the method comprising: include: Obtain video content frames and corresponding auxiliary information, wherein the auxiliary information is used to describe the visual features of the video content frames; Extract the first color channel data of the video content frame in the specified color space; The first color channel data of the video content frame is used as the first color channel data of the auxiliary information frame associated with the video content frame, and the auxiliary information is used as the second color channel data of the auxiliary information frame to generate the auxiliary information frame. The video content frames and the auxiliary information frames are encoded to generate a video stream, and an association identifier for associating the video content frames and the auxiliary information frames is added to the video stream.

2. The video coding method of claim 1, wherein, Encoding the video content frames and the auxiliary information frames includes: The video content frame and the auxiliary information frame are encoded using each candidate encoding parameter in the candidate encoding parameter set, and the encoding result is decoded to obtain the reconstruction result of the auxiliary information frame. The error between the first color channel data of the video content frame and the first color channel data in the reconstruction result is calculated as the reconstruction error between the video content frame and the auxiliary information frame. Based on the reconstruction error between the video content frame and the auxiliary information frame, a target coding parameter is selected from the candidate coding parameter set, and the target coding parameter is used to encode the video content frame and the auxiliary information frame.

3. The video coding method of claim 2, wherein, Based on the reconstruction error between the video content frame and the auxiliary information frame, a target coding parameter is selected from the candidate coding parameter set, including: The total reconstruction error of the video to be encoded is calculated based on the reconstruction error between the video content frame and the auxiliary information frame, and the number of video content frames contained in the video to be encoded. Based on the total reconstruction error corresponding to the video to be encoded, the candidate coding parameter with the smallest total reconstruction error is selected from the set of candidate coding parameters and used as the target coding parameter.

4. The video coding method of claim 3, wherein, The video encoding method further includes: If the candidate coding parameter set contains multiple candidate coding parameters with the smallest total reconstruction error, then the candidate coding parameter with the smallest code rate is selected from the multiple candidate coding parameters with the smallest total reconstruction error as the target coding parameter.

5. The video coding method of claim 3, wherein, Based on the reconstruction error between the video content frames and the auxiliary information frames, and the number of video content frames contained in the video to be encoded, the total reconstruction error corresponding to the video to be encoded is calculated, including: Based on the number of video content frames contained in the video to be encoded and the reconstruction error between each video content frame and the associated auxiliary information frame, a statistical value of the reconstruction error is calculated, and the calculated statistical value is used as the total reconstruction error corresponding to the video to be encoded.

6. The video coding method of claim 1, wherein, The video encoding method further includes: Hash calculation is performed on at least one of the following: the identification information of the video content frame, some or all of the data contained in the video content frame, the identification information of the auxiliary information frame, and some or all of the data contained in the auxiliary information frame. The calculated hash value is used as an association identifier to associate the video content frame with the auxiliary information frame.

7. The video coding method of claim 1, wherein, The specified color space includes the YUV color space, and the first color channel data includes luminance channel data. Specifically, the auxiliary information frame is generated by using the first color channel data of the video content frame as the first color channel data of the corresponding auxiliary information frame, and using the auxiliary information as the second color channel data of the auxiliary information frame. The luminance channel data of the video content frame is used as the luminance channel data of the auxiliary information frame, and the auxiliary information is used as the chroma channel data of the auxiliary information frame to generate the auxiliary information frame.

8. The video coding method of claim 1, wherein, The specified color space includes the YUV color space, and the first color channel data includes chroma channel data. Specifically, the auxiliary information frame is generated by using the first color channel data of the video content frame as the first color channel data of the corresponding auxiliary information frame, and using the auxiliary information as the second color channel data of the auxiliary information frame. The chroma channel data of the video content frame is used as the chroma channel data of the auxiliary information frame, and the auxiliary information is used as the luminance channel data of the auxiliary information frame to generate the auxiliary information frame.

9. The video coding method of any of claims 1-8, wherein, Add an association identifier to the video bitstream for associating the video content frames with the auxiliary information frames, including: Generate supplementary enhancement information, and add the associated identifier to the supplementary enhancement information; The supplementary enhancement information is added to the video stream.

10. A method of video decoding, comprising: include: The video stream is parsed to obtain video content frames and auxiliary information frames associated with the video content frames. The first color channel data of the auxiliary information frames is the first color channel data of the video content frames, and the second color channel data of the auxiliary information frames contains auxiliary information corresponding to the video content frames. The auxiliary information is used to describe the visual features of the video content frames. The video content frame is rendered based on the auxiliary information contained in the auxiliary information frame.

11. The video decoding method of claim 10, wherein, The video stream is parsed to obtain video content frames and associated auxiliary information frames, including: The video stream is parsed to obtain multiple frames and the association identifier of each frame in the multiple frames; Based on the association identifiers of each frame obtained from the parsing, the associated video content frames and auxiliary information frames are determined; wherein, the associated video content frames and auxiliary information frames have matching association identifiers.

12. The video decoding method according to claim 11, characterized in that, Based on the association identifiers of each frame obtained from the parsing, the associated video content frames and auxiliary information frames are determined, including: Based on the association identifiers of each frame obtained from the parsing, determine two frames with the same association identifier; Based on the information of the two frames with the same association identifier, the video content frame and auxiliary information frame in the two frames are identified.

13. The video decoding method according to claim 12, characterized in that, Based on the information of two frames with the same association identifier, identify the video content frame and auxiliary information frame in the two frames, including at least one of the following methods: The frame with the smallest timestamp among the two frames with the same association identifier is taken as the video content frame, and the frame with the largest timestamp among the two frames with the same association identifier is taken as the auxiliary information frame. The frame with an odd-numbered sequence number among the two frames with the same association identifier is designated as the video content frame, and the frame with an even-numbered sequence number among the two frames with the same association identifier is designated as the auxiliary information frame.

14. The video decoding method according to claim 12, characterized in that, Based on the information of the two frames with the same association identifier, the video content frame and auxiliary information frame in the two frames are identified, including: Obtain the display timestamps of the two frames with the same association identifier; If the difference between the display timestamps of two frames with the same association identifier is within a set range, then the video content frame and auxiliary information frame in the two frames are identified.

15. The video decoding method according to claim 10, characterized in that, The video decoding method further includes: parsing the association identifiers of the video content frames and the association identifiers of the auxiliary information frames from the video bitstream; Rendering the video content frame according to the auxiliary information contained in the auxiliary information frame includes: if the association identifier of the video content frame matches the association identifier of the auxiliary information frame, then the video content frame is rendered according to the auxiliary information contained in the auxiliary information frame.

16. The video decoding method according to any one of claims 10 to 15, characterized in that, The video decoding method further includes: After rendering the video content frames, the display timestamps of the video content frames are used to perform audio-visual synchronization processing with the audio frames.

17. A video encoding device, characterized in that, include: The acquisition unit is configured to acquire video content frames and auxiliary information corresponding to the video content frames, wherein the auxiliary information is used to describe the visual features of the video content frames; The extraction unit is configured to extract the first color channel data of the video content frame in a specified color space; The generation unit is configured to use the first color channel data of the video content frame as the first color channel data of the auxiliary information frame associated with the video content frame, and use the auxiliary information as the second color channel data of the auxiliary information frame to generate the auxiliary information frame; The encoding unit is configured to encode the video content frame and the auxiliary information frame to generate a video stream, and to add an association identifier to the video stream for associating the video content frame and the auxiliary information frame.

18. A video decoding device, characterized in that, include: The parsing unit is configured to parse the video bitstream to obtain a video content frame and an auxiliary information frame associated with the video content frame. The first color channel data of the auxiliary information frame is the first color channel data of the video content frame, and the second color channel data of the auxiliary information frame contains auxiliary information corresponding to the video content frame. The auxiliary information is used to describe the visual features of the video content frame. The processing unit is configured to render the video content frame based on the auxiliary information contained in the auxiliary information frame.

19. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 16.

20. A computer device, characterized in that, include: One or more processors; A memory for storing one or more computer programs that, when executed by one or more processors, cause the computer device to perform the method of any one of claims 1 to 16.

Images