Video coding processing method, video decoding processing method, video coding processing device, video decoding processing device and related equipment

Abstract

The invention discloses a video coding processing method and device, a video decoding processing method and device and related equipment, and belongs to the technical field of video coding and decoding, and the video coding processing method comprises the steps that a coding end obtains a first residual image and a first reconstructed image which is reconstructed based on the first residual image, the first residual image is a residual image generated after inverse quantization processing and inverse transformation processing in a coding process; the coding end determines first input data of a neural network model based on the first residual image and the first reconstructed image; and the coding end inputs the first input data into the neural network model to generate a target reconstructed image.

Description

Technical Field

[0001] This application belongs to the field of video encoding and decoding technology, specifically relating to a video encoding processing method, a video decoding processing method, an apparatus, and related equipment. Background Technology

[0002] With the development of encoding and decoding technologies, video encoding and decoding techniques have been widely applied. However, filtering and quantization during the encoding process still result in significant quality loss in reconstructed images in areas with complex textures. Traditional methods use deep learning to recover more texture details, but at lower bitrates, the reconstructed image loses a large number of features, making it difficult to further extract and recover high-quality images. Therefore, existing technologies suffer from poor image quality during video encoding and decoding. Summary of the Invention

[0003] This application provides a video encoding processing method, a video decoding processing method, an apparatus, and related equipment, which can solve the problem of poor image quality during video encoding and decoding.

[0004] Firstly, a video encoding processing method is provided, including:

[0005] The encoding end acquires a first residual image and a first reconstructed image based on the first residual image. The first residual image is a residual image generated after inverse quantization and inverse transform processing during the encoding process.

[0006] The encoding end determines the first input data of the neural network model based on the first residual image and the first reconstructed image;

[0007] The encoding end inputs the first input data into the neural network model to generate the target reconstructed image.

[0008] Secondly, a video decoding processing method is provided, including:

[0009] The decoding end acquires a second residual image and a second reconstructed image based on the second residual image. The second residual image is a residual image generated after inverse quantization and inverse transform processing during the decoding process of the target bitstream.

[0010] The decoding end determines the second input data of the neural network model based on the second residual image and the second reconstructed image;

[0011] The decoding end inputs the second input data into the neural network model to generate the target decoded image.

[0012] Thirdly, a video encoding processing apparatus is provided, comprising:

[0013] The first acquisition module is used to acquire a first residual image and a first reconstructed image based on the first residual image, wherein the first residual image is a residual image generated after inverse quantization and inverse transform processing during the encoding process;

[0014] The first processing module is used to determine the first input data of the neural network model based on the first residual image and the first reconstructed image; and to input the first input data into the neural network model to generate the target reconstructed image.

[0015] Fourthly, a video decoding processing apparatus is provided, comprising:

[0016] The second acquisition module is used to acquire the second residual image and the second reconstructed image based on the second residual image. The second residual image is a residual image generated after inverse quantization and inverse transform processing during the decoding process of the target bitstream.

[0017] The second processing module is used to determine the second input data of the neural network model based on the second residual image and the second reconstructed image; and to input the second input data into the neural network model to generate the target decoded image.

[0018] Fifthly, a video encoding processing apparatus is provided, the apparatus being configured to perform the steps of the method described in the first aspect.

[0019] In a sixth aspect, a video decoding processing apparatus is provided, the apparatus being configured to perform the steps of the method described in the second aspect.

[0020] In a seventh aspect, an electronic device is provided, the terminal including a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the method as described in the first aspect, or implementing the steps of the method as described in the second aspect.

[0021] Eighthly, an electronic device is provided, including a processor and a communication interface, wherein,

[0022] When the electronic device is an encoding end, the processor is used to acquire a second residual image and a second reconstructed image based on the second residual image. The second residual image is a residual image generated after inverse quantization and inverse transform processing during the decoding process of the target bitstream. The processor determines the second input data of the neural network model based on the second residual image and the second reconstructed image. The processor inputs the second input data into the neural network model to generate the target decoded image.

[0023] When the electronic device is a decoding end, the processor is used to acquire a second residual image and a second reconstructed image based on the second residual image. The second residual image is a residual image generated after inverse quantization and inverse transform processing during the decoding process of the target bitstream.

[0024] The second input data of the neural network model is determined based on the second residual image and the second reconstructed image; the second input data is input into the neural network model to generate the target decoded image.

[0025] A ninth aspect provides an electronic device comprising: a memory configured to store video data, and processing circuitry configured to implement the steps of the method described in the first aspect, or the steps of the method described in the second aspect.

[0026] In a tenth aspect, a readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect, or implement the steps of the method described in the second aspect.

[0027] Eleventhly, an encoding / decoding system is provided, comprising: an encoding end device and a decoding end device, wherein the encoding end device can be used to perform the steps of the method described in the first aspect, and the decoding end device can be used to perform the steps of the method described in the second aspect.

[0028] In a twelfth aspect, a chip is provided, the chip including a processor and a communication interface coupled to the processor, the processor being configured to run a program or instructions to implement the steps of the method described in the first aspect, or to implement the steps of the method described in the second aspect.

[0029] In a thirteenth aspect, a computer program / program product is provided, which is stored in a storage medium and is executed by at least one processor to implement the steps of the method as described in the first aspect, or to implement the steps of the method as described in the second aspect.

[0030] In this embodiment, a first residual image and a first reconstructed image based on the first residual image are obtained through the encoding end. The first residual image is a residual image generated after inverse quantization and inverse transform processing during the encoding process. The encoding end determines the first input data of the neural network model based on the first residual image and the first reconstructed image. The first input data is then input into the neural network model to generate the target reconstructed image. Thus, since the residual image can effectively represent the difficulty of image compression, adding input data generated based on the residual image to the input of the neural network model can improve the quality of the reconstructed image, thereby improving the efficiency of video encoding. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the structure of the encoding / decoding system provided in the embodiments of this application;

[0032] Figure 2 This is a schematic diagram of the encoder provided in an embodiment of this application;

[0033] Figure 3 This is a schematic diagram of the decoder provided in an embodiment of this application;

[0034] Figure 4 This is a schematic flowchart of the video encoding processing method provided in the embodiments of this application;

[0035] Figure 5 This is a general framework diagram of the video encoding processing method provided in the embodiments of this application;

[0036] Figure 6 This is a schematic diagram of the neural network model in the video encoding processing method provided in this application embodiment;

[0037] Figure 7 This is a flowchart illustrating the video decoding processing method provided in an embodiment of this application;

[0038] Figure 8 This is a schematic diagram of the structure of the video encoding processing apparatus provided in the embodiments of this application;

[0039] Figure 9 This is a schematic diagram of the structure of the video decoding processing device provided in the embodiments of this application;

[0040] Figure 10 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0041] Figure 11 This is a schematic diagram of the terminal structure provided in the embodiments of this application. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0043] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, the first object can be one or more. Furthermore, "or" in this application indicates at least one of the connected objects. For example, the scope of protection for "A or B" covers at least three scenarios: Scenario 1: including A but not B; Scenario 2: including B but not A; Scenario 3: including both A and B. In addition, the terms "A and / or B," "at least one of A and B," and "at least one of A or B" also cover at least the above three scenarios. The character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0044] Figure 1 This is a schematic diagram of the encoding / decoding system 10 provided in an embodiment of this application. The technical solution of this application embodiment relates to encoding and decoding (CODEC) video data (including encoding or decoding). The video data includes original unencoded video, encoded video, decoded (e.g., reconstructed) video, or syntax elements, etc.

[0045] like Figure 1 As shown, the encoding / decoding system 10 includes a source device 100, which provides encoded video data to be decoded and displayed by the destination device 110. Specifically, the source device 100 provides video data to the destination device 110 via a communication medium 120. The source device 100 and the destination device 110 may include any one or more of the following: desktop computer, laptop computer, tablet computer, set-top box, mobile phone, wearable device (e.g., smartwatch or wearable camera), television, camera, display device, in-vehicle device, virtual reality (VR) device, augmented reality (AR) device, mixed reality (MR) device, digital media player, video game console, video conferencing equipment, video streaming equipment, broadcast receiver equipment, broadcast transmitter equipment, spacecraft, aircraft, robot, satellite, etc.

[0046] exist Figure 1 In this example, source device 100 includes a data source 101, a memory 102, an encoder 200, and an output interface 104. Destination device 110 includes an input interface 111, a decoder 300, a memory 113, and a display device 114. Source device 100 represents an example of a video encoding device, while destination device 110 represents an example of a video decoding device. In other examples, source device 100 and destination device 110 may not include... Figure 1 Some components, or may include Figure 1 Other components besides the source device 100. For example, the source device 100 can receive video data from an external data source (such as an external camera). Similarly, the destination device 110 can interface with an external display device, without including an integrated display device. Furthermore, the memory 102 and memory 113 can be external memories.

[0047] Although Figure 1 Source device 100 and destination device 110 are illustrated as separate devices, but in some examples, they may also be integrated into a single device. In such embodiments, the same hardware or software, or separate hardware or software, or any combination thereof, may be used to implement the functionality corresponding to source device 100 and destination device 110.

[0048] In some examples, source device 100 and destination device 110 can perform unidirectional or bidirectional video transmission. If it is bidirectional video transmission, source device 100 and destination device 110 can operate in a substantially symmetrical manner, that is, each of source device 100 and destination device 110 includes an encoder and a decoder.

[0049] Data source 101 represents the source of video data (i.e., raw, unencoded video data) and provides encoder 200 with a series of images containing video data, which encoder 200 encodes. Data source 101 of source device 100 may include video acquisition devices (such as video cameras), video archives containing previously acquired raw video, or video feed interfaces for receiving video from video content providers. Alternatively, data source 101 may generate computer graphics-based data as source video, or combine live video, archived video, and computer-generated video. In these cases, encoder 200 encodes the acquired, pre-acquired, or computer-generated video data. Encoder 200 may rearrange the images from the received order (sometimes referred to as the "display order") according to the encoded order. Encoder 200 may generate a bitstream including the encoded video data. Source device 100 may then output the encoded video data to communication medium 120 via output interface 104 for reception or retrieval, for example, by input interface 111 of destination device 110.

[0050] The memory 102 of the source device 100 and the memory 113 of the destination device 110 represent general-purpose memory. In some examples, memory 102 may store raw video data from data source 101, and memory 113 may store decoded video data from decoder 300. Additionally or alternatively, memories 102 and 113 may respectively store software instructions executable by, for example, encoder 200 and decoder 300. Although memories 102 and 113 are shown separately from encoder 200 and decoder 300 in this example, it should be understood that encoder 200 and decoder 300 may also include internal memory for functionally similar or equivalent purposes. If encoder 200 and decoder 300 are deployed on the same hardware device, memories 102 and 113 may be the same memory. Furthermore, memories 102 and 113 may store, for example, encoded video data output from encoder 200 and input to decoder 300. In some examples, portions of memories 102 and 113 may be allocated as one or more video buffers, for example, to store raw, decoded, or encoded video data.

[0051] In some examples, source device 100 can output encoded data from output interface 104 to memory 113. Similarly, destination device 110 can access encoded data from memory 113 via input interface 111. Memory 113 or memory 102 can include any of a variety of distributed or locally accessed data storage media, such as hard drives, Blu-ray discs, digital versatile discs (DVDs), compact disc read-only memory (CD-ROMs), flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

[0052] Output interface 104 may include any type of medium or device capable of transmitting encoded video data from source device 100 to destination device 110. For example, output interface 104 may include a transmitter or transceiver, such as an antenna, configured to transmit encoded video data directly from source device 100 to destination device 110 in real time. The encoded video data may be modulated according to the communication standards of a wireless communication protocol and transmitted to destination device 110.

[0053] Communication medium 120 may include transient media, such as wireless broadcasting or wired network transmission. For example, communication medium 120 may include radio frequency (RF) spectrum or one or more physical transmission lines (e.g., cables). Communication medium 120 may form part of a packet-based network (such as a local area network, a wide area network, or a global network such as the Internet). Communication medium 120 may also take the form of a storage medium (e.g., a non-transitory storage medium), such as a hard disk, flash drive, compact disc, digital video disc, Blu-ray disc, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

[0054] In some implementations, the communication medium 120 may include a router, switch, base station, or any other device that can be used to facilitate communication from source device 100 to destination device 110. For example, a server (not shown) may receive encoded video from source device 100 and provide the encoded video data to destination device 110, for example, via network transmission. The server may include (e.g., a web server for a website), a server configured to provide file transfer protocol services (such as File Transfer Protocol (FTP) or File Delivery Over Unidirectional Transport (FLUTE) protocol), a content delivery network (CDN) device, a Hypertext Transfer Protocol (HTTP) server, a Multimedia Broadcast Multicast Services (MBMS) or Evolved Multimedia Broadcast Multicast Service (eMBMS) server, or a Network-attached storage (NAS) device, etc. The server can implement one or more HTTP streaming protocols, such as MPEG Media Transport (MMT), Dynamic Adaptive Streaming over HTTP (DASH), HTTP Live Streaming (HLS), or Real Time Streaming Protocol (RTSP).

[0055] Destination device 110 can access encoded video data from a server, for example, via a wireless channel (e.g., Wi-Fi connection) or a wired connection (e.g., digital subscriber line (DSL), cable modem, etc.) for accessing encoded video data stored on the server.

[0056] Output interface 104 and input interface 111 can represent a wireless transmitter / receiver, a modem, a wired networking component (e.g., an Ethernet card), a wireless communication component operating according to the IEEE 802.11 or IEEE 802.15 standard (e.g., ZigBee™), Bluetooth standard, or other physical components. In an example where output interface 104 and input interface 111 include wireless components, output interface 104 and input interface 111 can be configured to transmit data, such as encoded video data, via Wi-Fi, Ethernet, or cellular networks (such as 4G, LTE (Long Term Evolution), Advanced LTE, 5G, 6G, etc.).

[0057] The technology provided in this application can be applied to support video encoding and decoding in one or more multimedia applications such as video conferencing, over-the-air television broadcasting, cable television transmission, satellite television transmission, internet streaming video transmission, digital video encoded onto a data storage medium, decoding of digital video stored on a data storage medium, or other applications.

[0058] The input interface 111 of the destination device 110 receives an encoded video bitstream from the communication medium 120. The encoded video bitstream may include syntax elements and encoded data units (e.g., sequences, image groups, images, slices, blocks, etc.), where the syntax elements are used to decode the encoded data units to obtain decoded video data. The display device 114 displays the decoded video data to the user. The display device 114 may include a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or other types of display devices.

[0059] The encoder 200 and decoder 300 can be implemented as one or more of various processing circuits, which may include microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, or any combination thereof. When the technology is implemented wholly or partially in software, the device may store instructions for the software in a suitable non-transitory computer-readable storage medium and use one or more processors to execute the instructions in hardware to perform the technology provided in the embodiments of this application.

[0060] The encoder 200 and decoder 300 can process based on the following video codec standards: H.263, H.264, H.265 (also known as High Efficiency Video Coding, HEVC), H.266 (also known as Versatile Video Coding, VVC), Moving Picture Experts Group 2 (MPEG-2), MPEG-4, VP8, VP9, ​​Alliance for Open Media Video 1 (AV1), Audio Video Coding Standard 1 (AVS1), AVS2, AVS3, or next-generation video standard protocols. This application does not specifically limit the implementation of these protocols.

[0061] Typically, encoder 200 and decoder 300 can perform block-based encoding and decoding of images. The term "block" generally refers to a structure that includes data to be processed (e.g., encoded, decoded, or otherwise used during encoding or decoding). For example, a block can include a two-dimensional matrix of samples of luminance or chrominance data. For example, encoder 200 and decoder 300 can encode and decode video data represented in YUV format.

[0062] See Figure 2 The figure is a schematic diagram of the encoder 200 provided in an embodiment of this application. The encoder 200 can be... Figure 1 Encoder 200 in. In Figure 2 In the example, encoder 200 includes memory 201, encoding parameter determination unit 210, residual generation unit 202, transform processing unit 203, quantization unit 204, inverse quantization unit 205, inverse transform processing unit 206, reconstruction unit 207, filter unit 208, decoded picture buffer (DPB) 209, and entropy encoding unit 220.

[0063] Memory 201 can store video data to be encoded; for example, encoder 200 can retrieve the data from the encoder. Figure 1 The data source 101 shown receives and stores video data. In some examples, the memory 201 may be on the same chip as other components of the encoder 200 (e.g., ...). Figure 2 (As shown), they can also be independent of the chip in which these components are located.

[0064] The coding parameter determination unit 210 includes a mode selection unit 211, an inter-frame prediction unit 212, and an intra-frame prediction unit 213. The inter-frame prediction unit 212 is used to obtain a first prediction block for the current block using an inter-frame prediction mode. The intra-frame prediction unit 213 is used to obtain a second prediction block for the current block using an intra-frame prediction mode. The mode selection unit 211 is used to obtain a target prediction block based on the first and second prediction blocks and determine the final prediction mode. Furthermore, the coding parameter determination unit 210 may also include other functional units, such as functional units for determining the partitioning method of coding units (CUs), functional units for determining the transformation type of the residual data of the CUs, or functional units for determining the quantization parameters of the residual data of the CUs.

[0065] For ease of description and understanding, in the embodiments of this application, the CU to be processed in the current image is referred to as the current CU, and the image block to be processed in the current CU is referred to as the current block or the image block to be processed. For example, in encoding, it refers to the block currently being encoded; in decoding, it refers to the block currently being decoded.

[0066] Inter-frame prediction unit 212 may include a motion estimation unit and a motion compensation unit. For inter-frame prediction of the current block, the motion estimation unit may perform a motion search to identify one or more matching reference blocks in one or more reference pictures (e.g., one or more previously encoded / decoded pictures stored in DPB 209).

[0067] The motion estimation unit can generate one or more motion vectors (MVs) representing the position of a reference block in a reference image relative to the position of the current block in the current image. The motion compensation unit can then use interpolation to obtain a predicted value with the precision indicated by the motion vectors.

[0068] The encoding parameter determination unit 210 can provide the target prediction block to the residual generation unit 202. The residual generation unit 202 receives the raw uncoded video data of the current block from the memory 201 and calculates the residual between the current block and the target prediction block to obtain the residual block. In some examples, the function of the residual generation unit 202 can be implemented using one or more subtractor circuits that perform binary subtraction.

[0069] As an example, the encoding parameter determination unit 210 can provide the entropy encoding unit 220 with syntax elements representing encoding parameters for encoding. The encoding parameters include one or more of the following: the partitioning method of the CU, the final prediction mode, the transformation type of the residual data of the CU, or the quantization parameters of the residual data of the CU.

[0070] The transformation processing unit 203 transforms the residual block output by the residual generation unit 202 to obtain a transform coefficient block. This transformation may include Discrete Cosine Transform (DCT), integer transformation, direction transformation, or Karhunen-Loeve transformation, etc. In some examples, the encoder 200 may not include the transformation processing unit 203.

[0071] Quantization unit 204 can quantize the transform coefficients in the transform coefficient block according to the quantization parameter (QP) value associated with the current block to generate a quantized transform coefficient block.

[0072] The inverse quantization unit 205 and the inverse transform processing unit 206 can perform inverse quantization and inverse transform on the transform coefficient block, respectively, to obtain the reconstructed residual block. The reconstruction unit 207 can generate a reconstructed block corresponding to the current block based on the reconstructed residual block and the target prediction block generated by the coding parameter determination unit 210.

[0073] Filter unit 208 can perform one or more filter operations on the reconstructed block. For example, filter unit 208 can be a deblocking filter (DBF), an adaptive loop filter (ALF), a sample adaptive offset (SAO) filter, etc. In some examples, encoder 200 may not include filter unit 208.

[0074] Encoder 200 stores the reconstructed image obtained from the reconstructed blocks in DPB 209. For example, in an example where the operation of filter unit 208 is not required, reconstruction unit 207 can store the reconstructed blocks in DPB 209. In an example where the operation of filter unit 208 is required, filter unit 208 can store the filtered reconstructed blocks in DPB 209. Inter-frame prediction unit 212 retrieves the reconstructed image from DPB 209 to perform inter-frame prediction on blocks of subsequent images to be encoded. In some examples, DPB 209 can be replaced with other types of memory.

[0075] Entropy coding unit 220 can entropy code the syntax elements of other components in encoder 200 to output encoded video data. For example, entropy coding unit 220 can entropy code the quantized transform coefficient block from quantization unit 204. As another example, entropy coding unit 220 can entropy code the syntax elements (e.g., motion information for inter-frame prediction or intra-frame mode information for intra-frame prediction) from coding parameter determination unit 210.

[0076] Understandable Figure 2 The composition of the encoder 200 described is merely illustrative and does not constitute a limitation on the embodiments of this application.

[0077] Figure 3 This is a schematic diagram of the structure of the decoder 300 provided in the embodiments of this application. The decoder 300 can be... Figure 1 The decoder 300 is described. Figure 3 In the example, the decoder 300 includes a coded picture buffer (CPB) 301, an entropy decoding unit 302, a prediction processing unit 310, an inverse quantization unit 303, an inverse transform processing unit 304, a reconstruction unit 305, a filter unit 306, and a DPB 307.

[0078] The entropy decoding unit 302 can receive encoded video data from the CPB 301 and perform entropy decoding on the video data to obtain syntax elements. The syntax elements indicate encoding parameters, including one or more of the following: CU partitioning method, final prediction mode, transformation type of CU residual data, or quantization parameters of CU residual data.

[0079] When the syntax element includes the final prediction mode, the prediction processing unit 310 obtains the final prediction mode. If the final prediction mode is an inter-frame prediction mode, the prediction block of the current CU can be obtained through the inter-frame prediction unit 311 of the prediction processing unit 310; if the final prediction mode is an intra-frame prediction mode, the prediction block of the current CU can be obtained through the intra-frame prediction unit 312 of the prediction processing unit 310. In some examples, the prediction processing unit 310 may also include a unit for performing prediction functions according to other prediction modes.

[0080] CPB 301 can be obtained from, for example Figure 1 The communication medium 120 shown acquires and stores encoded video data. DPB 307 is used to store decoded images. Optionally, CPB 301 and DPB 307 can be replaced with other types of memory; this application does not impose specific limitations. In some examples, CPB 301 can be on the same chip as other components of the decoder 300 (as shown), or it can be on a separate chip from those components.

[0081] Decoder 300 can perform reconstruction operations on each block individually. Entropy decoding unit 302 can entropy decode the syntax elements and transform information (e.g., QP or transform mode indication) of the quantized transform coefficients to obtain the quantized transform coefficients. Dequantization unit 303 dequantizes the quantized transform coefficients to obtain a transform coefficient block including the transform coefficients. Inverse transform processing unit 304 performs an inverse transform on the transform coefficient block to generate a residual block corresponding to the current block; this inverse transform is the reverse operation of the above transform.

[0082] Reconstruction unit 305 can reconstruct the current block based on the prediction block and the residual block. For example, reconstruction unit 305 can add samples from the residual block to the corresponding samples from the prediction block to reconstruct the current block.

[0083] Filter unit 306 can perform one or more filter operations on the reconstructed block. For example, the type of filter unit 306 can be referenced to the type of filter unit 208, and will not be described again here. In some examples, the operations of filter unit 306 can be skipped.

[0084] Decoder 300 can store the reconstructed image obtained from the reconstructed blocks in DPB 307. For example, in an example where filter unit 306 is not operated, reconstruction unit 305 can store the reconstructed blocks in DPB 307. In an example where filter unit 306 is operated, filter unit 306 can store the filtered reconstructed blocks in DPB 307. Decoder 300 can output the decoded image (e.g., decoded video) from DPB 307 for use with a display device (such as...). Figure 1 The subsequent presentation of the display device 114).

[0085] Applications of Neural Networks in Video Encoding and Decoding: Inspired by the tremendous success of neural network technology in computer vision and image processing tasks, many neural network-based methods have been introduced into the field of video encoding and decoding. Some technologies replace certain modules in traditional video standards, while others are end-to-end video encoding and decoding technologies.

[0086] In loop filtering and super-resolution, in addition to reconstructed pixels, some techniques incorporate video encoding / decoding information as additional input to guide the network. This information includes quantization parameters, frame type, predicted pixels, and boundary strength. The loss function is typically L1 (mean absolute error) or L2 (mean squared error).

[0087] Optionally, quantization can effectively reduce the range of values ​​for the transform coefficients, thereby achieving better compression. However, it inevitably introduces distortion, which is the root cause of distortion in video coding. The quantization parameter reflects the compression of spatial details; the smaller the quantization parameter, the finer the quantization, the higher the image quality, and the lower the compression ratio. Conversely, increasing the quantization parameter leads to more detail loss, greater image quality degradation, and a higher compression ratio.

[0088] Optionally, the loss function measures the quality of the model's predictions, thus guiding the next stage of training in the correct direction and enabling backpropagation to function. The quality of the loss function design directly affects the model's final convergence. The formulas for two commonly used loss functions in video encoding / decoding are as follows:

[0089] Mean absolute error:

[0090] Mean square error:

[0091] Among them, y i f(x) represents the original pixel values ​​of the video image sample points. i ) represents the pixel value of the sample point after the video image has passed through the neural network, and n is the number of sample points in the video image.

[0092] Currently, applications based on neural networks have made significant progress in improving the quality of decoded images. However, filtering and quantization during the encoding process still result in substantial quality loss in reconstructed images in areas with complex textures. Traditional methods use deep learning to recover more texture details, but at lower bitrates, the reconstructed image loses a large number of features, making it difficult to further extract and recover high-quality images. Therefore, it is necessary to utilize intermediate information from the decoding process to guide the neural network for image quality recovery. To this end, the video encoding and decoding processing method of this application is proposed.

[0093] The video encoding and decoding methods provided in this application are described below with reference to the accompanying drawings. The video encoding method provided in this application can be executed by an encoding end, for example... Figure 1 or Figure 2 The encoder 200 is shown. The video decoding processing method provided in this embodiment can be executed by the decoding end, for example... Figure 1 or Figure 3 The decoder 300 is described above. The encoding and decoding ends can be implemented by software, hardware, or a combination thereof. When implemented by hardware, the encoding end can be referred to as an encoding device or a video encoding device, and the decoding end can be referred to as a decoding device or a video decoding device.

[0094] Reference Figure 4This application provides a video encoding processing method, such as... Figure 4 As shown, the video encoding processing method includes:

[0095] Step 401: The encoding end acquires a first residual image and a first reconstructed image based on the first residual image. The first residual image is a residual image generated after inverse quantization and inverse transform processing during the encoding process.

[0096] Step 402: The encoding end determines the first input data of the neural network model based on the first residual image and the first reconstructed image;

[0097] Step 403: The encoding end inputs the first input data into the neural network model to generate the target reconstructed image.

[0098] In this embodiment of the application, the encoding end can perform encoding processing on the image to be encoded, such as... Figure 2 As shown, during the encoding process, the inverse transform processing unit can output the first residual image, and the reconstruction unit can output the first reconstructed image.

[0099] Optionally, the first input data can be data obtained after processing the first residual image and the first reconstructed image accordingly. For example, the first residual image and the first reconstructed image can be processed by extracting channel data and performing corresponding preprocessing to obtain data that can be recognized by the neural network model.

[0100] Optionally, after inputting the first input data into the neural network model, the target reconstructed image can be obtained based on the model's output. This reconstructed image can then be used in subsequent video encoding processes. Since residual images can effectively represent the ease or difficulty of image compression, incorporating input data generated based on residual images into the input of the neural network model can improve the quality of the reconstructed image, thereby increasing the efficiency of video encoding.

[0101] In this embodiment, a first residual image and a first reconstructed image based on the first residual image are obtained through the encoding end. The first residual image is a residual image generated after inverse quantization and inverse transform processing during the encoding process. The encoding end determines the first input data of the neural network model based on the first residual image and the first reconstructed image. The first input data is then input into the neural network model to generate the target reconstructed image. Thus, since the residual image can effectively represent the difficulty of image compression, adding input data generated based on the residual image to the input of the neural network model can improve the quality of the reconstructed image, thereby improving the efficiency of video encoding.

[0102] Optionally, in some embodiments, the encoding end determines the first input data of the neural network model based on the first residual image and the first reconstructed image, including:

[0103] The preprocessing includes normalizing the channel data, and the first input data includes the first channel data.

[0104] In this embodiment, information can first be extracted from the first residual image and the first reconstructed image to obtain the channel data of the first residual image. Then, preprocessing can be performed on each channel data to obtain the first channel data corresponding to the first residual image and the second channel data corresponding to the first reconstructed image.

[0105] It should be understood that by normalizing the channel data and encoding parameter information, the first input data that the neural network model can recognize can be obtained.

[0106] Optionally, in some embodiments, the process of determining the first input data of the neural network model based on the first residual image and the first reconstructed image by the encoding end further includes processing of the first reconstructed image, for example, further including:

[0107] The encoding end preprocesses the channel data of the first reconstructed image to obtain the second channel data;

[0108] The first input data also includes the second channel data.

[0109] It should be noted that, in some embodiments, the encoding parameter information can also be used as the input of the neural network model. For example, the encoding end performs normalization processing on the encoding parameter information to obtain the target encoding parameter information, and the aforementioned first input data also includes the target encoding parameter information.

[0110] Optionally, in some embodiments, the first residual image includes at least one of the following: Y channel data, U channel data, and V channel data. Alternatively, in some embodiments, the first residual image may include RGB data.

[0111] Optionally, the Y channel data can be understood as luminance (or luma) data. The U channel data mentioned above can be understood as one component of chroma data (e.g., chroma or chrominance), and the V channel data mentioned above can be understood as another component of chroma data.

[0112] Optionally, in some embodiments, the encoding parameter information includes quantization parameter (QP) values.

[0113] Of course, in some embodiments, the encoding parameter information may also include other information, such as frame type, etc., which are not further limited here.

[0114] In some embodiments, the normalization is performed as follows:

[0115]

[0116] Where [i,j] represents pixel coordinates, Rec LR_Y Rec represents the Y channel of the reconstructed image after normalization. Y Rec represents the Y channel of the reconstructed image. LR_U Rec represents the U channel of the reconstructed image after normalization. U Rec represents the U channel of the reconstructed image. LR_V Rec represents the V channel of the reconstructed image after normalization. V Represents the V channel of the reconstructed image; Resi LR_Y Resi represents the Y channel of the normalized residual image. Y Represents the Y channel of the residual image, Resi LR_U Represents the U channel of the normalized residual image, Resi U Represents the U channel of the residual image, Resi LR_V Represents the V channel of the normalized residual image, Resi V This represents the V channel of the residual image.

[0117] Optionally, QP norm The normalized QP, in some embodiments, can be based on QP. norm To fill the matrix, to construct the QP mapping, and the size of the QP mapping is related to Rec. LR_Y Similarly, in this case, the QP mapping can be used as the input to the neural network model.

[0118] Optionally, in some embodiments, the preprocessing further includes upsampling the normalized U-channel data and V-channel data;

[0119] The image resolution corresponding to the upsampled U-channel and V-channel data is the same as the image resolution corresponding to the Y-channel data.

[0120] In this embodiment, by upsampling the normalized U-channel and V-channel data, the image resolution corresponding to the U-channel and V-channel data is made the same as the image resolution corresponding to the Y-channel data, thereby reducing the difficulty of channel data fusion for the neural network model.

[0121] It should be noted that, in the embodiments of this application, residual images can be used to improve coding performance during the super-resolution process or image enhancement. To better understand this application, the following explanation uses the use of residual images to improve coding performance during the super-resolution process as an example.

[0122] Optionally, in some embodiments, the first input data includes first identification information, which indicates at least one of the following:

[0123] Input data associated with the residual image;

[0124] Input data associated with the reconstructed image.

[0125] In this embodiment of the application, at least one of the following can be indicated by whether or not the first identification information is carried, or by different values ​​of the first identification information: input data associated with the residual image; input data associated with the reconstructed image. For example, the first identification information is carried only for the input data associated with the residual image, or only for the input data associated with the reconstructed image.

[0126] It should be understood that by using the first identifier information as auxiliary information, the residual image and the reconstructed image can be distinguished. This helps guide the neural network model to extract more accurate features from the residual image and the reconstructed image respectively, thereby outputting a higher quality reconstructed image.

[0127] Optionally, in some embodiments, the first identification information includes a first identification value, which is used to indicate that the first channel data corresponding to the first residual image is associated with the residual image, and the first input data includes the first channel data and the first identification value.

[0128] In this embodiment, the first identifier value can be 0 or 1. The first identifier value and the first channel data can be understood as the input data corresponding to the residual image.

[0129] Optionally, in some embodiments, the first identification information includes a second identification value, which is used to indicate that the second channel data corresponding to the first reconstructed image is associated with the reconstructed image, and the first input data includes the second channel data and the second identification value.

[0130] In this embodiment, the second identifier value can be 1 or 0, and the first identifier value is different from the second identifier value. The second identifier value and the second channel data can be understood as the input data corresponding to the reconstructed image.

[0131] Optionally, in some embodiments, the target bitstream generated based on the encoding process carries target indication information, which is used to indicate at least one of the following:

[0132] Whether to use a neural network model to process the reconstructed image generated during the decoding process;

[0133] Whether to input the residual image into a neural network model to assist in processing the reconstructed image generated during the decoding process.

[0134] In this embodiment, whether or not a target indication information is carried in the target bitstream indicates whether a neural network model is used to process the reconstructed image generated during the decoding process, and whether the residual image is input into the neural network model to assist in processing the reconstructed image generated during the decoding process. For example, if the target bitstream carries target indication information, it indicates at least one of the following: using a neural network model to process the reconstructed image generated during the decoding process or not using a neural network model to process the reconstructed image generated during the decoding process; inputting the residual image into the neural network model to assist in processing the reconstructed image generated during the decoding process or not inputting the residual image into the neural network model to assist in processing the reconstructed image generated during the decoding process. Different values ​​of the target indication information can indicate different states. For example, when the target indication information is 1, it indicates at least one of the following: using a neural network model to process the reconstructed image generated during the decoding process; inputting the residual image into the neural network model to assist in processing the reconstructed image generated during the decoding process; when the target indication information is 0, it indicates at least one of the following: not using a neural network model to process the reconstructed image generated during the decoding process; not inputting the residual image into the neural network model to assist in processing the reconstructed image generated during the decoding process.

[0135] Optionally, the encoding end may determine whether to carry target indication information, or determine the specific value of the target indication information, based on at least one of the following:

[0136] Whether to use a neural network model to process the reconstructed image generated during the encoding process at the encoding end;

[0137] Whether the residual image is input into the neural network model at the encoding end to assist in processing the reconstructed image generated during the encoding process;

[0138] Is the encoded video image a super-resolution image?

[0139] Does the encoded video image need image enhancement?

[0140] For example, in some embodiments, when the video image encoded by the encoder is a super-resolution image, the target indication information can be used to instruct the decoder to use a neural network model to process the reconstructed image generated during the decoding process, or the residual image can be input into the neural network model to assist in processing the reconstructed image generated during the decoding process.

[0141] It should be noted that processing reconstructed images using neural network models can be understood as: using neural network models to extract features from reconstructed images, or using neural network models to regenerate new reconstructed images from reconstructed images.

[0142] Specifically, such as Figure 5 As shown, Figure 5 This is a general framework diagram of video encoding processing methods.

[0143] The first step in the preprocessing is to perform the following steps: reconstruct the image from the low-resolution image (Rec LR ) and low-resolution residual images (Resi LR Extract the Y-channel, U-channel, and V-channel data and normalize them respectively. Normalize QP. Upsample the U-channel and V-channel data of the normalized low-resolution reconstructed image and the U-channel and V-channel data of the low-resolution residual image respectively. The resolution of the upsampled data is the same as that of the Y-channel image.

[0144] Then, feature extraction is performed using a neural network model. In this model, shallow feature extraction can be performed using the same parameters, followed by deep feature extraction using the residual module, where the Y-channel data and UV components (i.e., U-channel and V-channel data) are processed separately for feature extraction.

[0145] Finally, denormalization is performed to obtain a high-quality image, i.e., Rec. HR .

[0146] Optionally, the structure of the neural network model is as follows: Figure 6 As shown. Here, Upsample indicates upsampling. "Conv kxk" represents convolution with kernel size k, and RB is the residual block connection. The input to the neural network model consists of five parts: QP during the encoding process, the Y component (i.e., Y channel data) of the reconstructed image and the residual image, and their respective UV components (i.e., U channel data and V channel data; the UV component of the residual image is denoted as Resi). LR_UV The UV components of the reconstructed image are represented as Rec LR_UV In this example, the upsampling layer is a pixel shuffling layer; alternatively, the upsampling block can use a deconvolution with a stride equal to the upscaling ratio.

[0147] It's important to note that the goal of training a neural network model is to find the optimal parameter values, including weights and biases. First, the training dataset is compressed using a codec to generate reconstructed frames with distortion. Then, these reconstructed frames (i.e., low-resolution and compressed frames), residual frames, and quantization parameters are fed into the neural network model, and the loss is calculated using the model's output and ground truth frames (or the original frames). Commonly used loss functions include Sum of Absolute Difference (SAD) and Mean Absolute Error (MSE). Next, the gradient of each parameter is calculated using backpropagation. These gradients are then used to update the parameter values. This process is repeated until the convergence criterion is met. After training, the optimized parameters are saved for use in the inference phase.

[0148] Reference Figure 7 This application also provides a video decoding processing method, such as... Figure 7 As shown, the video decoding processing method includes:

[0149] Step 701: The decoding end acquires the second residual image and the second reconstructed image based on the second residual image. The second residual image is the residual image generated after inverse quantization and inverse transform processing during the decoding process of the target bitstream.

[0150] Step 702: The decoding end determines the second input data of the neural network model based on the second residual image and the second reconstructed image;

[0151] Step 703: The decoding end inputs the second input data into the neural network model to generate the target decoded image.

[0152] In this embodiment of the application, the decoding end can perform decoding processing on the received video bitstream, such as... Figure 3 As shown, during the decoding process, the inverse transform processing unit can output a second residual image, and the reconstruction unit can output a second reconstructed image. Optionally, the aforementioned encoding parameter information is carried in the video bitstream.

[0153] It should be understood that, in the embodiments of this application, since the residual image can effectively represent the ease or difficulty of image compression, the input data generated based on the residual image is added to the input of the neural network model, thereby improving the quality of the reconstructed image.

[0154] Optionally, the decoding end determines the second input data of the neural network model based on the second residual image and the second reconstructed image, including:

[0155] The decoding end preprocesses the channel data of the second residual image to obtain the third channel data;

[0156] The preprocessing includes normalizing the channel data, and the second input data includes the third channel data.

[0157] Optionally, the second residual image includes at least one of the following: Y channel data, U channel data, and V channel data.

[0158] Optionally, the preprocessing further includes upsampling the normalized U-channel and V-channel data;

[0159] The image resolution corresponding to the upsampled U-channel and V-channel data is the same as the image resolution corresponding to the Y-channel data.

[0160] Optionally, the second input data includes second identification information, which indicates at least one of the following:

[0161] Input data associated with the residual image;

[0162] Input data associated with the reconstructed image.

[0163] Optionally, the second identification information includes a third identification value, which is used to indicate that the third channel data corresponding to the second residual image is associated with the residual image, and the second input data includes the third channel data and the third identification value.

[0164] Optionally, the second identification information includes a fourth identification value, which is used to indicate that the fourth channel data corresponding to the second reconstructed image is associated with the reconstructed image, and the second input data includes the fourth channel data and the fourth identification value.

[0165] Optionally, the decoding end determines the second input data of the neural network model based on the second residual image and the second reconstructed image, including:

[0166] Under the condition that the preset conditions are met, the decoding end determines the second input data of the neural network model based on the second residual image and the second reconstructed image;

[0167] The preset conditions include at least one of the following:

[0168] Based on the target bitstream, a neural network model is used to process the reconstructed image generated during the decoding process;

[0169] Based on the target bitstream, the residual image is input into a neural network model to assist in processing the reconstructed image generated during the decoding process.

[0170] Optionally, the target bitstream carries target indication information, which is used to indicate at least one of the following:

[0171] Whether to use a neural network model to process the reconstructed image generated during the decoding process;

[0172] Whether to input the residual image into a neural network model to assist in processing the reconstructed image generated during the decoding process.

[0173] The video encoding processing method provided in this application can be executed by a video encoding processing device. As an example, the device can be an electronic device or a component within an electronic device, such as a chip or circuit. This application uses a video encoding processing device executing the video encoding processing method as an example to illustrate the video encoding processing device provided in this application.

[0174] Reference Figure 8 This application also provides a video encoding processing apparatus, such as... Figure 8 As shown, the video encoding processing apparatus 800 includes:

[0175] The first acquisition module 801 is used to acquire a first residual image and a first reconstructed image based on the first residual image, wherein the first residual image is a residual image generated after inverse quantization and inverse transformation processing during the encoding process;

[0176] The first processing module 802 is used to determine the first input data of the neural network model based on the first residual image and the first reconstructed image; and input the first input data into the neural network model to generate the target reconstructed image.

[0177] Optionally, the first processing module 802 is specifically used to: preprocess the channel data of the first residual image to obtain the first channel data;

[0178] The preprocessing includes normalizing the channel data, and the first input data includes the first channel data.

[0179] Optionally, the first residual image includes at least one of the following: Y channel data, U channel data, and V channel data.

[0180] Optionally, the preprocessing further includes upsampling the normalized U-channel and V-channel data;

[0181] The image resolution corresponding to the upsampled U-channel and V-channel data is the same as the image resolution corresponding to the Y-channel data.

[0182] Optionally, the first input data includes first identification information, which indicates at least one of the following:

[0183] Input data associated with the residual image;

[0184] Input data associated with the reconstructed image.

[0185] Optionally, the first identification information includes a first identification value, which is used to indicate that the first channel data corresponding to the first residual image is associated with the residual image, and the first input data includes the first channel data and the first identification value.

[0186] Optionally, the first identification information includes a second identification value, which is used to indicate that the second channel data corresponding to the first reconstructed image is associated with the reconstructed image, and the first input data includes the second channel data and the second identification value.

[0187] Optionally, the target bitstream generated based on the encoding process carries target indication information, which is used to indicate at least one of the following:

[0188] Whether to use a neural network model to process the reconstructed image generated during the decoding process;

[0189] Whether to input the residual image into a neural network model to assist in processing the reconstructed image generated during the decoding process.

[0190] The frequency coding processing device provided in this application embodiment can achieve... Figure 4 The various processes implemented in the method embodiments achieve the same technical effect, and will not be described again here to avoid repetition.

[0191] The video decoding processing method provided in this application can be executed by a video decoding processing device. As an example, the device can be an electronic device or a component within an electronic device, such as a chip or circuit. This application uses a video decoding processing device executing the video decoding processing method as an example to illustrate the video decoding processing device provided in this application.

[0192] Reference Figure 9 This application also provides a video decoding processing apparatus, such as... Figure 9 As shown, the video decoding processing device 900 includes:

[0193] The second acquisition module 901 is used to acquire a second residual image and a second reconstructed image based on the second residual image. The second residual image is a residual image generated after inverse quantization and inverse transformation processing during the decoding process of the target bitstream.

[0194] The second processing module 902 is used to determine the second input data of the neural network model based on the second residual image and the second reconstructed image; and input the second input data into the neural network model to generate the target decoded image.

[0195] Optionally, the second processing module 902 is specifically used for:

[0196] The channel data of the second residual image is preprocessed to obtain the third channel data;

[0197] The preprocessing includes normalizing the channel data, and the second input data includes the third channel data.

[0198] Optionally, the second residual image includes at least one of the following: Y channel data, U channel data, and V channel data.

[0199] Optionally, the preprocessing further includes upsampling the normalized U-channel and V-channel data;

[0200] The image resolution corresponding to the upsampled U-channel and V-channel data is the same as the image resolution corresponding to the Y-channel data.

[0201] Optionally, the second input data includes second identification information, which indicates at least one of the following:

[0202] Input data associated with the residual image;

[0203] Input data associated with the reconstructed image.

[0204] Optionally, the second identification information includes a third identification value, which is used to indicate that the third channel data corresponding to the second residual image is associated with the residual image, and the second input data includes the third channel data and the third identification value.

[0205] Optionally, the second identification information includes a fourth identification value, which is used to indicate that the fourth channel data corresponding to the second reconstructed image is associated with the reconstructed image, and the second input data includes the fourth channel data and the fourth identification value.

[0206] Optionally, the second processing module 902 is specifically used to: determine the second input data of the neural network model based on the second residual image and the second reconstructed image when the preset conditions are met;

[0207] The preset conditions include at least one of the following:

[0208] The neural network model is determined based on the target bitstream;

[0209] Based on the target bitstream, the residual image is input into the neural network model.

[0210] Optionally, the target bitstream carries target indication information, which is used to indicate at least one of the following:

[0211] Whether to use a neural network model to process the reconstructed image generated during the decoding process;

[0212] Whether to input the residual image into a neural network model to assist in processing the reconstructed image generated during the decoding process.

[0213] The frequency decoding processing device provided in this application embodiment can achieve... Figure 7 The various processes implemented in the method embodiments achieve the same technical effect, and will not be described again here to avoid repetition.

[0214] like Figure 10 As shown, this application embodiment also provides an electronic device 1000, including a processor 1001 and a memory 1002. The memory 1002 stores a program or instructions that can run on the processor 1001. For example, when the electronic device 1000 is an encoding device, the program or instructions executed by the processor 1001 implement the various steps of the above-described video encoding processing method embodiment and achieve the same technical effect. When the electronic device 1000 is a decoding device, the program or instructions executed by the processor 1001 implement the various steps of the above-described video decoding processing method embodiment and achieve the same technical effect. To avoid repetition, this will not be described again here. Optionally, the memory 1002 may be... Figure 1 The processor 1001 can implement the memory 102 or memory 113 in the illustrated embodiment. Figure 1-3 The functions of the encoder 200 or decoder 300 in the illustrated embodiment.

[0215] This application also provides an electronic device, including: a memory configured to store video data; and a processing circuit configured to implement the steps of the video encoding processing method or video decoding processing method embodiments described above. Optionally, the memory may be... Figure 1 The processing circuitry of memory 102 or memory 113 in the illustrated embodiment can implement... Figure 1-3 The functions of the encoder 200 or decoder 300 in the illustrated embodiment.

[0216] This application embodiment also provides an electronic device, including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement, for example... Figure 4 or Figure 7The steps in the method embodiment shown are illustrated. This device embodiment corresponds to the above method embodiment, and all implementation processes and methods of the above method embodiments can be applied to this terminal embodiment and achieve the same technical effect.

[0217] The processor or processing circuit in this application embodiment may include general-purpose processors, special-purpose processors, etc., such as central processing units (CPUs), microprocessors, digital signal processors (DSPs), artificial intelligence (AI) processors, graphics processing units (GPUs), application-specific integrated circuits (ASICs), network processors (NPs), field-programmable gate arrays (FPGAs), or other programmable logic devices, gate circuits, transistors, discrete hardware components, etc. The communication interface in this application embodiment may include transceivers, pins, circuits, buses, etc.

[0218] The aforementioned electronic devices can be terminals or other devices besides terminals, such as servers, network attached storage (NAS), etc.

[0219] The terminal can be a mobile phone, tablet computer, laptop computer, notebook computer, personal digital assistant (PDA), handheld computer, netbook, ultra-mobile personal computer (UMPC), mobile internet device (MID), augmented reality (AR), virtual reality (VR) device, mixed reality (MR) device, robot, wearable device, flight vehicle, vehicle user equipment (VUE), shipboard equipment, pedestrian user equipment (PUE), smart home (home devices with wireless communication capabilities, such as refrigerators, televisions, washing machines, or furniture), game console, personal computer (PC), ATM or self-service machine, etc. Wearable devices include: smartwatches, smart bracelets, smart earphones, smart glasses, smart jewelry (smart bracelets, smart chains, smart rings, smart necklaces, smart anklets, smart anklets, etc.), smart wristbands, smart clothing, etc. Among these, in-vehicle devices can also be referred to as in-vehicle terminals, in-vehicle controllers, in-vehicle modules, in-vehicle components, in-vehicle chips, or in-vehicle units, etc. It should be noted that the embodiments in this application do not limit the specific type of terminal.

[0220] A server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server. A cloud server can provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), or cloud computing services based on big data and artificial intelligence platforms.

[0221] For example, the aforementioned electronic devices may include, but are not limited to, those described above. Figure 1 The type of source device 100 or destination device 110 shown.

[0222] Taking electronic devices as terminals as an example, Figure 11 A schematic diagram of the hardware structure of a terminal to implement an embodiment of this application.

[0223] The terminal 1100 includes, but is not limited to, at least some of the following components: radio frequency unit 1101, network module 1102, audio output unit 1103, input unit 1104, sensor 1105, display unit 1106, user input unit 1107, interface unit 1108, memory 1109, and processor 1110.

[0224] Those skilled in the art will understand that the terminal 1100 may also include a power supply (such as a battery) for powering various components. The power supply can be logically connected to the processor 1110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. 11 The terminal structure shown does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0225] It should be understood that, in this embodiment, the input unit 1104 may include a graphics processor 11041 and a microphone 11042. The graphics processor 11041 processes image data of still images or videos obtained by an image acquisition device (such as a camera) in video acquisition mode or image acquisition mode, or it may process the obtained point cloud data. The display unit 1106 may include a display panel 11061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 1107 includes at least one of a touch panel 11071 and other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 may include a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, joysticks, etc., which will not be described in detail here.

[0226] In this embodiment, after receiving downlink data from the network-side device, the radio frequency unit 1101 can transmit it to the processor 1110 for processing; in addition, the radio frequency unit 1101 can send uplink data to the network-side device. Typically, the radio frequency unit 1101 includes, but is not limited to, antennas, amplifiers, transceivers, couplers, low-noise amplifiers, duplexers, etc.

[0227] The memory 1109 can be used to store software programs or instructions, as well as various data. The memory 1109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1109 may include volatile memory or non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1109 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0228] Processor 1110 may include one or more processing units; optionally, processor 1110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 1110.

[0229] When the terminal is an encoding end, the processor 1110 is configured to: acquire a first residual image and a first reconstructed image based on the first residual image, wherein the first residual image is a residual image generated after inverse quantization and inverse transform processing during the encoding process; determine the first input data of the neural network model based on the first residual image and the first reconstructed image; and input the first input data into the neural network model to generate the target reconstructed image.

[0230] When the terminal is a decoding end, the processor 1110 is configured to: acquire a second residual image and a second reconstructed image based on the second residual image, wherein the second residual image is a residual image generated after inverse quantization and inverse transform processing during the decoding process of the target bitstream; determine the second input data of the neural network model based on the second residual image and the second reconstructed image; and input the second input data into the neural network model to generate the target decoded image.

[0231] It is understood that the implementation process of each implementation method mentioned in this embodiment can refer to the relevant descriptions of the above-mentioned video encoding processing method and video decoding processing method embodiments, and achieve the same or corresponding technical effects. To avoid repetition, it will not be described again here.

[0232] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described video encoding or video decoding methods and achieve the same technical effect. To avoid repetition, these will not be described again here.

[0233] The processor mentioned above is the processor in the terminal described in the above embodiments. The readable storage medium includes computer-readable storage media, such as ROM, RAM, magnetic disk, or optical disk. In some examples, the readable storage medium may be a non-transient readable storage medium.

[0234] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described video encoding processing method or video decoding processing method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0235] It should be understood that the chips mentioned in the embodiments of this application may include system-on-a-chip (also known as system chip, chip system, or system-on-a-chip) or discrete display chips, etc.

[0236] This application also provides a computer program / program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described video encoding or video decoding methods, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0237] This application also provides a video encoding and decoding system, including: an encoding end device and a decoding end device. The encoding end device can be used to perform the steps of the video encoding processing method described above, and the decoding end device can be used to perform the steps of the video decoding processing method described above.

[0238] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0239] From the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of computer software products plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disk, optical disk, etc.) and includes several instructions to cause the terminal or network-side device to execute the methods described in the various embodiments of this application.

[0240] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other implementations under the guidance of this application without departing from the spirit and scope of the claims. All of these implementations are within the protection scope of this application.

Claims

1. A method of video encoding processing, characterized by, The method comprises: an encoding end obtains a first residual image and a first reconstructed image reconstructed based on the first residual image, the first residual image being a residual image generated based on a dequantization process and a de-transformation process in an encoding process; the encoding end determines first input data of a neural network model based on the first residual image and the first reconstructed image; the encoding end inputs the first input data into the neural network model to generate a target reconstructed image.

2. The method of claim 1, wherein, The encoding end determines first input data of a neural network model based on the first residual image and the first reconstructed image, which comprises: the encoding end pre-processes channel data of the first residual image to obtain first channel data; wherein the pre-processing comprises normalizing the channel data, and the first input data comprises the first channel data.

3. The method of claim 2, wherein, The first residual image comprises at least one of Y channel data, U channel data and V channel data.

4. The method of claim 3, wherein, The pre-processing further comprises up-sampling the normalized U channel data and V channel data; wherein the image resolution corresponding to the up-sampled U channel data and V channel data is the same as the image resolution corresponding to the Y channel data.

5. The method according to any one of claims 1 to 4, characterized in that, The first input data comprises first identification information, the first identification information being used to indicate at least one of: input data associated with a residual image; input data associated with a reconstructed image.

6. The method of claim 5, wherein, The first identification information comprises a first identification value, the first identification value being used to indicate that first channel data corresponding to the first residual image is associated with a residual image, and the first input data comprises the first channel data and the first identification value.

7. The method of claim 5, wherein, The first identification information comprises a second identification value, the second identification value being used to indicate that second channel data corresponding to the first reconstructed image is associated with a reconstructed image, and the first input data comprises the second channel data and the second identification value.

8. The method according to any one of claims 1 to 7, characterized in that, Target indication information is carried in a target code stream generated based on the encoding process, the target indication information being used to indicate at least one of: whether to use a neural network model to process a reconstructed image generated in a decoding process; whether to input a residual image into a neural network model to assist in processing a reconstructed image generated in a decoding process.

9. A method of video decoding processing, the method comprising: The method comprises: a decoding end obtains a second residual image and a second reconstructed image reconstructed based on the second residual image, the second residual image being a residual image generated based on a dequantization process and a de-transformation process in a decoding process of a target code stream; the decoding end determines second input data of a neural network model based on the second residual image and the second reconstructed image; the decoding end inputs the second input data into the neural network model to generate a target decoded image.

10. The method of claim 9, wherein, The decoding end determines second input data of a neural network model based on the second residual image and the second reconstructed image, which comprises: the decoding end pre-processes channel data of the second residual image to obtain third channel data; wherein the pre-processing comprises normalizing the channel data, and the second input data comprises the third channel data.

11. The method of claim 10, wherein, The second residual image includes at least one of Y channel data, U channel data, and V channel data.

12. The method of claim 11, wherein, The preprocessing further includes up-sampling processing on the normalized U channel data and V channel data. The image resolution corresponding to the up-sampled U channel data and V channel data is the same as the image resolution corresponding to the Y channel data.

13. The method according to any one of claims 9 to 12, characterized in that, The second input data includes second identification information, and the second identification information is used to indicate at least one of the following: Input data associated with a residual image; Input data associated with a reconstructed image.

14. The method of claim 13, wherein, The second identification information includes a third identification value, and the third identification value is used to indicate that third channel data corresponding to the second residual image is associated with a residual image, and the second input data includes the third channel data and the third identification value.

15. The method of claim 13, wherein, The second identification information includes a fourth identification value, and the fourth identification value is used to indicate that fourth channel data corresponding to the second reconstructed image is associated with a reconstructed image, and the second input data includes the fourth channel data and the fourth identification value.

16. The method according to any one of claims 9 to 15, characterized in that, The second input data of the neural network model determined by the decoding end based on the second residual image and the second reconstructed image includes: In the case of satisfying a preset condition, the decoding end determines the second input data of the neural network model based on the second residual image and the second reconstructed image; The preset condition includes at least one of the following: Based on the target code stream, it is determined that the reconstructed image generated by the decoding process is processed by using the neural network model; Based on the target code stream, it is determined that the residual image is input into the neural network model to assist in processing the reconstructed image generated by the decoding process.

17. The method according to any one of claims 9 to 16, characterized in that, The target indication information carried in the target code stream is used to indicate at least one of the following: Whether to use the neural network model to process the reconstructed image generated by the decoding process; Whether to input the residual image into the neural network model to assist in processing the reconstructed image generated by the decoding process.

18. A video encoding processing apparatus, characterized by comprising: It includes: A first acquisition module is configured to acquire a first residual image and a first reconstructed image reconstructed based on the first residual image, the first residual image being a residual image generated based on a reverse quantization processing and a reverse transformation processing in an encoding process; A first processing module is configured to determine first input data of a neural network model based on the first residual image and the first reconstructed image, and input the first input data into the neural network model to generate a target reconstructed image.

19. The apparatus of claim 18, wherein, The first processing module is specifically configured to: Preprocess channel data of the first residual image to obtain first channel data; The preprocessing includes normalizing the channel data.

20. A video decoding processing apparatus, comprising: It includes: A second acquisition module is configured to acquire a second residual image and a second reconstructed image reconstructed based on the second residual image, the second residual image being a residual image generated based on a reverse quantization processing and a reverse transformation processing in a decoding process of a target code stream; A second processing module is configured to determine second input data of a neural network model based on the second residual image and the second reconstructed image; Input the second input data into the neural network model to generate a target decoded image.

21. The apparatus of claim 20, wherein, The second processing module is specifically configured to: perform preprocessing on the channel data of the second residual image to obtain third channel data; wherein the preprocessing comprises normalizing the channel data, and the second input data comprises the third channel data.

22. An electronic device, comprising: The apparatus comprises a processor and a memory, the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the video encoding processing method according to any one of claims 1 to 8, or implement the steps of the video decoding processing method according to claims 9 to 17.

23. A readable storage medium characterized by, The readable storage medium stores programs or instructions, and the programs or instructions are executed by the processor to implement the steps of the video encoding processing method according to any one of claims 1 to 8, or implement the steps of the video decoding processing method according to claims 9 to 17.

24. A chip, characterized by The chip comprises a processor and a communication interface, the communication interface is coupled with the processor, and the processor is configured to execute programs or instructions to implement the steps of the video encoding processing method according to any one of claims 1 to 8, or implement the steps of the video decoding processing method according to claims 9 to 17.

Images