Coding method and apparatus

Abstract

This application provides a decoding method and apparatus. The decoding method includes: decoding a bitstream to obtain a first feature map, the resolution of which is less than the resolution of the original image; reconstructing a second feature map using a first neural network to obtain a reconstructed image, wherein both the second feature map and the reconstructed image have a target resolution, which is less than the resolution of the original image, wherein the second feature map is the first feature map, or the second feature map is a feature map obtained by processing the first feature map using a second neural network. This application can improve the efficiency of acquiring reconstructed images, thereby increasing the speed at which digital video applications display thumbnails of the original images.

Description

Technical Field

[0001] This application relates to the field of data compression technology, and in particular to an encoding / decoding method and apparatus. Background Technology

[0002] Video encoding and decoding are widely used in digital video applications, such as video transmission over broadcast digital television, the Internet, photo albums and mobile networks, real-time conversational applications such as video chat and video conferencing, digital variadic discs (DVD) and Blu-ray discs, video content capture and editing systems, and security applications for portable cameras.

[0003] Video data is typically large, which can pose challenges when sending or transmitting it over networks with limited bandwidth. Therefore, video data is usually compressed before transmission over modern telecommunications networks. Video size can also be an issue when storing video on storage devices due to potentially limited memory resources. Video compression devices typically use software and / or hardware at the source side to encode video data before transmission or storage, thereby reducing the amount of data required to represent digital video. The compressed video data is then received at the destination by video decompression equipment. Given limited network resources and the growing demand for higher video quality, there is a need for improved compression and decompression techniques that can increase compression ratios with minimal impact on image quality.

[0004] In some operations of digital video applications, thumbnails of the original images need to be displayed to show a large number of images on the display interface. Related technologies first decode and reconstruct the received original image stream to obtain the original image. Then, the resolution of the original image is reduced to obtain a thumbnail. However, the above method of obtaining thumbnails of the original images is complex and time-consuming, thus affecting the speed at which digital video applications display thumbnails of the original images. Summary of the Invention

[0005] This application provides an encoding / decoding method and apparatus to improve the efficiency of acquiring and reconstructing images, thereby increasing the speed at which digital video applications display thumbnails of the original images.

[0006] In a first aspect, this application provides a decoding method, the method comprising: decoding a bitstream to obtain a first feature map, wherein the resolution of the first feature map is less than the resolution of the original image; reconstructing a second feature map according to a first neural network to obtain a reconstructed image, wherein the resolution of the second feature map and the reconstructed image are both target resolutions, wherein the target resolution is less than the resolution of the original image, wherein the second feature map is the first feature map, or the second feature map is a feature map obtained by processing the first feature map through a second neural network.

[0007] In a convolutional neural network, a feature map is the three-dimensional data output by layers such as convolutional layers, activation layers, pooling layers, and batch normalization layers. Its three dimensions are called width, height, and channel, respectively. A feature map can be M×W×H, indicating that the feature map is a two-dimensional feature map with a resolution of W×H, including M channels, where W represents width and H represents height.

[0008] The resolution of the reconstructed image output by the first neural network is the same as the resolution of the input second feature map, both being the target resolution. When the second feature map is the first feature map, the target resolution is equal to the resolution of the first feature map. When the second feature map is a feature map obtained by processing the first feature map through the second neural network, the target resolution is greater than the resolution of the first feature map.

[0009] The second neural network is used to process the first feature map to obtain the second feature map. This processing may include convolution operations and / or deconvolution operations, and further, the processing may also include normalization operations.

[0010] In this decoding method, the decoder decodes the bitstream to obtain a first feature map. Then, it reconstructs the second feature map using a first neural network to obtain a reconstructed image. Both the second feature map and the reconstructed image have a target resolution, which is smaller than the resolution of the original image. The second feature map includes the first feature map, or it is a feature map obtained by processing the feature map of the original image using a second neural network. Since the resolution of the reconstructed image is smaller than that of the original image, there is no need to reconstruct the original image first; instead, the reconstructed image is obtained directly, improving the efficiency of obtaining the reconstructed image and thus increasing the speed at which digital video applications display thumbnails of the original image.

[0011] When there is only one first neural network, in one example, there is no need to apply a second neural network, and the first neural network can exist independently. In this case, the second feature map is the first feature map, and the first neural network can only output a reconstructed image at one resolution, which is the resolution of the first feature map.

[0012] In another example, the first neural network can be connected to the input of the second neural network and / or to one or more output nodes. In this case, the second feature map is either the first feature map or a feature map obtained by processing the first feature map through the second neural network. The first neural network can output reconstructed images at one or more resolutions.

[0013] When there are multiple first neural networks, the input and / or each output node of the second neural network are connected to the first neural network. In this case, the second feature map is either the first feature map or a feature map obtained by processing the first feature map through the second neural network. The first neural network can output reconstructed images at various resolutions.

[0014] In one possible implementation, when the second feature map is a feature map obtained by processing the first feature map through a second neural network, the second neural network includes one or more output nodes and multiple convolutional layers. The output nodes are located between the output of the first convolutional layer and the input of the last convolutional layer, and the output nodes are connected to the input of the first neural network, the method further includes: inputting the first feature map into the second neural network to obtain the second feature map output by a target output node of the second neural network, wherein the target output node belongs to the one or more output nodes; and reconstructing the second feature map according to the first neural network to obtain a reconstructed image, which includes: inputting the second feature map output by the target output node into the first neural network to obtain the reconstructed image output by the first neural network.

[0015] The second neural network is an existing neural network used in encoding and decoding technologies to generate reconstructed images at the original resolution. Output nodes are set at different positions in the second neural network, and reconstructed images with various target resolutions can be generated using these output nodes and the existing second neural network. In this process, the second neural network processes the first feature map, and each output node shares layers in the second neural network during the generation of reconstructed images at different target resolutions. This reduces the size of the neural network used to generate the reconstructed images (including the first and second neural networks), reduces the storage space occupied by the neural network in the decoder, and thus reduces the operating overhead and complexity of the neural network in the decoder.

[0016] In one possible implementation, when there is only one first neural network, the second feature map output by the target output node is directly input into that single network to obtain the reconstructed image output by that single neural network. When there are multiple first neural networks, inputting the second feature map output by the target output node into the first neural network to obtain the reconstructed image output by the first neural network includes: inputting the second feature map into the first neural network connected to the target output node to obtain the reconstructed image output by the first neural network connected to the target output node.

[0017] In one possible implementation, when there are multiple target resolutions and multiple target output nodes, the step of inputting the first feature map into the second neural network to obtain the second feature map output by the target output node of the second neural network includes: inputting the first feature map into the second neural network to obtain multiple second feature maps of multiple resolutions output by multiple target output nodes of the second neural network; the step of inputting the second feature map into the first neural network connected to the target output node to obtain the reconstructed image output by the first neural network connected to the target output node includes: inputting the second feature map output by each target output node into the first neural network connected to the target output node to obtain reconstructed images of multiple resolutions output by the first neural networks respectively connected to the multiple target output nodes.

[0018] Each output node can output the second feature map in parallel or sequentially. Parallel output improves the efficiency of acquiring the reconstructed image. Sequential output allows the second feature maps from each node to be shared. This reduces the computational load of the second neural network, thereby decreasing its overall computational cost.

[0019] In one possible implementation, when the second neural network includes multiple output nodes, each output node corresponding to an output resolution, the method further includes: determining the target resolution; and determining the output node whose output resolution is the target resolution as the target output node.

[0020] The target resolution can be determined by the decoder or externally, such as by the external application layer or encoder. External application layers can refer to the aforementioned digital video applications, such as video players, photo albums, and web pages.

[0021] For example, the target resolution may be determined according to the thumbnail resolution. For example, among the candidate resolutions that can be output by the first neural network, the resolution closest to the thumbnail resolution is determined as the target resolution. In another example, the target resolution may be determined according to the complexity of the reconstructed image acquisition process for each candidate resolution and / or the power consumption constraint.

[0022] In a possible implementation, the first neural network includes at least one convolutional layer, and a convolution stride of the at least one convolutional layer is 1.

[0023] In a possible implementation, the bitstream corresponds to two-dimensional feature maps of M1 channels, and the decoding the bitstream to obtain a first feature map includes: decoding the bitstream corresponding to the two-dimensional feature maps of M2 channels among the M1 channels to obtain the first feature map, where M2<M1, and the first feature map includes the two-dimensional feature maps of the M2 channels.

[0024] The decoder may decode the bitstream corresponding to the two-dimensional feature maps of the first M2 channels among the M1 channels. In this way, after the bitstream corresponding to the two-dimensional feature maps of the M2 channels is decoded, the subsequent reconstruction process can be executed, without executing the subsequent reconstruction process after the entire bitstream is completely decoded, which improves the efficiency of obtaining the third feature map, thereby improving the efficiency of acquiring the reconstructed image.

[0025] In a possible implementation, the method further includes: performing upsampling processing on the reconstructed image to obtain a first image, where the resolution of the first image is the same as the resolution of the original image.

[0026] For example, the upsampling processing may include bilinear interpolation upsampling, or directly filling pixel values of adjacent positions for upsampling, or implementing upsampling through a deconvolution operation of a convolutional layer with a stride greater than 1.

[0027] It should be noted that if reconstruction is performed on the feature map obtained by decoding the entire bitstream to generate a reconstructed image of the original resolution, the resolution of the image of the original resolution is relatively large, the generation process takes a long time, which affects the speed at which a digital video application displays the image of the original resolution, and causes stuttering when a user browses the image of the original resolution. In the embodiments of this application, only part of the bitstream in the bitstream is decoded during the process of obtaining the reconstructed image, so that the data volume of the reconstructed image is small, and the efficiency of obtaining the reconstructed image is high. Further, when the first image is obtained based on the reconstructed image, the efficiency of obtaining the first image is improved, thereby reducing the time consumption of the process of obtaining the first image, and improving the speed at which the digital video application displays the image of the original resolution.

[0028] In conjunction with the above implementation, when the bitstream is the bitstream of the initial feature map, the initial feature map is obtained by feature extraction from the original image. The method further includes: decoding the bitstream corresponding to the two-dimensional feature maps of the channels other than the M2 channels in the M1 channels to obtain a third feature map, the third feature map including the two-dimensional feature maps of the M1-M2 channels; processing the first feature map and the third feature map according to the second neural network to obtain a second image, the resolution of the second image being the same as the resolution of the original image.

[0029] The second image is generated by reconstructing the first and third feature maps. The second image has a larger data volume than the first image, and its image quality is higher. Since the reconstruction process of the second image is time-consuming, a low-quality first image is generated quickly for display first, while a high-quality second image is reconstructed simultaneously. This ensures that digital video applications do not experience stuttering when displaying images at their original resolution, thus improving the display effect of images at their original resolution.

[0030] In one possible implementation, the second feature map includes a two-dimensional feature map with multiple channels. Before reconstructing the second feature map according to the first neural network to obtain a reconstructed image, the method further includes: performing channel reduction processing on the second feature map; the reconstructing the second feature map according to the first neural network to obtain a reconstructed image includes: reconstructing the channel-reduced second feature map according to the first neural network to obtain the reconstructed image.

[0031] In one possible implementation, the first feature map includes a two-dimensional feature map with multiple channels, and the method further includes: performing channel reduction processing on the first feature map; the second feature map is the first feature map after the channel reduction processing, or the second feature map is a feature map obtained by processing the first feature map after the channel reduction processing through the second neural network.

[0032] Channel reduction processing on the second feature map and / or the first feature map can reduce the complexity of the subsequent reconstruction process and improve the efficiency of the reconstruction process.

[0033] Secondly, this application provides a decoding method, comprising: decoding a bitstream to obtain a first feature map; reconstructing a second feature map of a first resolution based on a first neural network to obtain a reconstructed image of a second resolution, wherein the second resolution is different from the first resolution and is smaller than the resolution of the original image, and the second feature map includes the first feature map, or the second feature map is a feature map obtained by processing the first feature map through a second neural network.

[0034] The resolution of the reconstructed image output by the first neural network differs from the resolution of the input second feature map. Since the second neural network can output a fixed number of resolution values, the resolution value output by the first neural network connected to one or more output nodes of the second neural network is also fixed. In this method, the first neural network also has the ability to change the resolution of the input feature map. This allows for the achievement of various numerical resolutions through first neural networks with different structures, reducing the overhead and complexity of the neural network used to generate the reconstructed image in the decoder while improving the flexibility of generating the reconstructed image.

[0035] Thirdly, this application provides an encoding method, the method comprising: extracting features from an original image to obtain an initial feature map, the initial feature map comprising two-dimensional feature maps of multiple channels, the resolution of the initial feature map being less than the resolution of the original image; encoding the feature map to be encoded to obtain a bitstream, the feature map to be encoded being the initial feature map, or a two-dimensional feature map comprising some channels of the initial feature map.

[0036] When the feature map to be encoded includes two-dimensional feature maps of some channels in the initial feature map, the amount of data in the subsequent encoded bitstream can be reduced, thereby reducing the communication overhead of the bitstream transmission to the decoder.

[0037] Fourthly, this application provides a decoding device, the decoding device comprising: a processing module for decoding a bitstream to obtain a first feature map, wherein the resolution of the first feature map is less than the resolution of the original image; and a reconstruction module for reconstructing a second feature map according to a first neural network to obtain a reconstructed image, wherein the resolution of the second feature map and the reconstructed image are both target resolutions, wherein the target resolution is less than the resolution of the original image, wherein the second feature map is the first feature map, or the second feature map is a feature map obtained by processing the first feature map through a second neural network.

[0038] In a possible implementation, when the second feature map is a feature map obtained by processing the first feature map through the second neural network, the second neural network includes one or more output nodes and a plurality of convolution layers, the output nodes are located between the output end of the first convolution layer and the input end of the last convolution layer, the output nodes are connected to the input end of the first neural network, the reconstruction module is further configured to input the first feature map into the second neural network, and obtain the second feature map output by a target output node of the second neural network, where the target output node belongs to the one or more output nodes; the reconstruction module is specifically configured to input the second feature map output by the target output node into the first neural network, and obtain the reconstructed image output by the first neural network.

[0039] In a possible implementation, when the number of the first neural networks is multiple, the reconstruction module is specifically configured to input the second feature map into the first neural network connected to the target output node, and obtain the reconstructed image output by the first neural network connected to the target output node.

[0040] In a possible implementation, when the number of target resolutions is multiple and the number of target output nodes is multiple, the reconstruction module is specifically configured to input the first feature map into the second neural network, and obtain second feature maps with multiple resolutions output by the multiple target output nodes of the second neural network; and input the second feature map output by each of the target output nodes into the first neural network connected to the target output node, to obtain reconstructed images with multiple resolutions output by the first neural networks respectively connected to the multiple target output nodes.

[0041] In a possible implementation, when the second neural network includes a plurality of output nodes, each of the output nodes corresponds to one output resolution, the reconstruction module is further configured to determine the target resolution; and determine an output node whose output resolution is the target resolution as the target output node.

[0042] In a possible implementation, the first neural network includes at least one convolution layer, and the convolution stride of the at least one convolution layer is 1.

[0043] In a possible implementation, the code stream corresponds to two-dimensional feature maps of M1 channels, and the processing module is specifically configured to decode the code stream corresponding to the two-dimensional feature maps of M2 channels among the M1 channels to obtain the first feature map, where M2<M1, and the first feature map includes the two-dimensional feature maps of the M2 channels.

[0044] In one possible implementation, the reconstruction module is further configured to upsample the reconstructed image to obtain a first image, the resolution of which is the same as that of the original image.

[0045] In one possible implementation, the bitstream is a bitstream of an initial feature map, which is obtained by feature extraction from the original image. The processing module is further configured to decode the bitstream corresponding to the two-dimensional feature maps of the channels other than the M2 channels in the M1 channels to obtain a third feature map, which includes the two-dimensional feature maps of the M1-M2 channels. The reconstruction module is further configured to process the first feature map and the third feature map according to the second neural network to obtain a second image, the resolution of which is the same as the resolution of the original image.

[0046] In one possible implementation, the second feature map includes a two-dimensional feature map with multiple channels, and the reconstruction module is further configured to perform channel reduction processing on the second feature map; specifically, the reconstruction module is configured to reconstruct the channel-reduced second feature map based on the first neural network to obtain the reconstructed image.

[0047] In one possible implementation, the first feature map includes a two-dimensional feature map with multiple channels, and the reconstruction module is further configured to perform channel reduction processing on the first feature map; the second feature map is the first feature map after the channel reduction processing, or the second feature map is a feature map obtained by processing the first feature map after the channel reduction processing through the second neural network.

[0048] Fifthly, this application provides an encoding device, the encoding device comprising: a processing module for extracting features from an original image to obtain an initial feature map, the initial feature map including a two-dimensional feature map of multiple channels, the resolution of the initial feature map being smaller than the resolution of the original image; and an encoding module for encoding the feature map to be encoded to obtain a bitstream, the feature map to be encoded being the initial feature map, or a two-dimensional feature map including some channels of the initial feature map.

[0049] In a sixth aspect, this application provides an electronic device comprising: one or more processors; a memory for storing one or more computer programs or instructions; wherein, when the one or more computer programs or instructions are executed by the one or more processors, the one or more processors perform the method as described in any one of the first to third aspects.

[0050] In a seventh aspect, this application provides an electronic device, comprising: a processor for performing the method as described in any one of the first to third aspects.

[0051] Eighthly, this application provides a computer-readable storage medium, characterized in that it includes a computer program or instructions, which, when executed on a computer, cause the computer to perform the method described in any one of the first to third aspects. Attached Figure Description

[0052] Figure 1 An exemplary block diagram of a decoding system 10 provided in an embodiment of this application;

[0053] Figure 2 A network diagram for video encoding and decoding based on deep learning is provided as an embodiment of this application;

[0054] Figure 3 A schematic diagram of an end-to-end video encoding and decoding network structure based on deep learning is provided for an embodiment of this application;

[0055] Figure 4 A flowchart illustrating process 400 of an encoding / decoding method provided in an embodiment of this application;

[0056] Figure 5 This application provides a schematic diagram of the structure of a first neural network and a second neural network according to embodiments of the present application.

[0057] Figure 6 This application provides a schematic diagram of the structure of another first neural network and a second neural network in its embodiments.

[0058] Figure 7 A schematic diagram of a channel reduction process provided in an embodiment of this application;

[0059] Figure 8 A flowchart illustrating another encoding / decoding method 500 provided in an embodiment of this application;

[0060] Figure 9 A flowchart illustrating process 600 of another encoding / decoding method provided in an embodiment of this application;

[0061] Figure 10 A flowchart illustrating process 700 of another encoding / decoding method provided in an embodiment of this application;

[0062] Figure 11 A schematic diagram of a neural network structure provided in an embodiment of this application;

[0063] Figure 12 This is a schematic diagram of another neural network structure provided in an embodiment of this application;

[0064] Figure 13A schematic diagram illustrating another channel reduction process provided in this application embodiment;

[0065] Figure 14 This application provides a schematic diagram of an encoding / decoding process.

[0066] Figure 15 A schematic diagram of the network structure of a feature extraction module provided in an embodiment of this application;

[0067] Figure 16 This is a schematic diagram of the network structure of a reconstruction module provided in an embodiment of this application;

[0068] Figure 17 A flowchart illustrating a decoding end provided in an embodiment of this application;

[0069] Figure 18 This is a schematic diagram of a reconstructed network provided in an embodiment of this application;

[0070] Figure 19 This is a schematic diagram of another reconstructed network structure provided in an embodiment of this application;

[0071] Figure 20 A schematic diagram of a channel reduction process provided in an embodiment of this application;

[0072] Figure 21 A schematic diagram illustrating another channel reduction process provided in this application embodiment;

[0073] Figure 22 A schematic diagram illustrating another channel reduction process provided in an embodiment of this application;

[0074] Figure 23 A block diagram of a decoding device provided in an embodiment of this application;

[0075] Figure 24 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0076] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0077] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0078] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, rather than to describe a specific order of objects. For example, "first range" and "second range," etc., are used to distinguish different ranges, rather than to describe a specific order of ranges.

[0079] In the embodiments of this application, the words "in one example," "examplely," or "for example" are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design described as "in one example," "examplely," or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "in one example," "examplely," or "for example" is intended to present the relevant concepts in a specific manner.

[0080] In the description of the embodiments in this application, unless otherwise stated, "at least one" means one or more, and "multiple" means two or more. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0081] This application provides a video compression / decompression technology based on artificial intelligence (AI), and more particularly a video compression / decompression technology based on neural networks. Specifically, it provides an encoding / decoding technology, which may include entropy encoding / decoding technology.

[0082] Video encoding and decoding consist of two parts: video encoding and video decoding. Video encoding is performed on the source side (or commonly referred to as the encoder side) and typically involves processing (e.g., compressing) the raw image to reduce the amount of data required to represent it (thus enabling more efficient storage and / or transmission). Video decoding is performed on the destination side (or commonly referred to as the decoder side) and typically involves performing the inverse processing relative to the encoder side to reconstruct the original image. The term "encoding and decoding" in the embodiments should be understood as either "encoding" or "decoding" of the video.

[0083] Video encoding and decoding generally refers to the processing of image sequences that form videos or video sequences. In the field of video encoding and decoding, the terms "picture," "frame," or "image" can be used synonymously.

[0084] Figure 1This is an exemplary block diagram of a decoding system provided for embodiments of this application, such as a video decoding system 10 (or simply decoding system 10) that can utilize the technology of this application. The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) in the video decoding system 10 represent devices, etc., that can be used to perform various technologies according to the various examples described in this application.

[0085] like Figure 1 As shown, the decoding system 10 includes a source device 12, which provides encoded image data 21, such as encoded images, to a destination device 14 for decoding the encoded image data 21.

[0086] The source device 12 includes an encoder 20, and optionally may include an image source 16, a preprocessor (or preprocessing unit) 18 such as an image preprocessor, and a communication interface (or communication unit) 22.

[0087] Image source 16 may include or may be any type of image capture device for capturing real-world images, and / or any type of image generation device, such as a computer graphics processor for generating computer animation images or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage device storing any of the images described above.

[0088] To distinguish the processing performed by the preprocessor (or preprocessing unit) 18, the image (or image data) 17 may also be referred to as the raw image (or raw image data) 17.

[0089] The preprocessor 18 receives the raw image data 17 and preprocesses it to obtain a preprocessed image (or preprocessed image data) 19. For example, the preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It is understood that the preprocessing unit 18 may be an optional component.

[0090] The video encoder (or encoder) 20 is used to receive preprocessed image data 19 and provide encoded image data 21.

[0091] The communication interface 22 in the source device 12 can be used to receive encoded image data 21 and send the encoded image data 21 (or other arbitrarily processed version) to another device such as the destination device 14 or any other device via the communication channel 13 for storage or direct reconstruction.

[0092] Source device 12 may also include a memory ( Figure 1 (Not shown), the memory can be used to store at least one of the following data: raw image data 17, preprocessed image (or preprocessed image data) 19, and encoded image data 21.

[0093] The target device 14 includes a decoder 30, and optionally may include a communication interface (or communication unit) 28, a post-processor (or post-processing unit) 32 and a display device 34.

[0094] The communication interface 28 in the destination device 14 is used to receive encoded image data 21 (or other processed versions) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is an encoded image data storage device, and to provide the encoded image data 21 to the decoder 30.

[0095] Communication interfaces 22 and 28 can be used to send or receive encoded image data (or encoded data 21) through a direct communication link between source device 12 and destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination thereof.

[0096] For example, the communication interface 22 can be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission encoding or processing, so as to transmit it on a communication link or communication network.

[0097] Communication interface 28 corresponds to communication interface 22. For example, it can be used to receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain encoded image data 21.

[0098] Both communication interface 22 and communication interface 28 can be configured as follows: Figure 1 The arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14 indicates a one-way or two-way communication interface, which can be used to send and receive messages, establish connections, acknowledge and exchange any other information related to the communication link and / or data transmission, such as encoded image data transmission, etc.

[0099] Video decoder (or decoder) 30 is used to receive encoded image data 21 and provide decoded image data (or decoded image data) 31 (hereinafter referred to as...). Figure 3 (and so on, for further description).

[0100] The post-processor 32 is used to post-process the decoded image data 31 (also known as the reconstructed image data) to obtain post-processed image data 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color adjustment, trimming or resampling, or any other processing to generate the decoded image data 31 for display by the display device 34, etc.

[0101] Display device 34 is used to receive post-processed image data 33 to display the image to a user or viewer. Display device 34 can be or includes any type of display for representing the reconstructed image, such as an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen.

[0102] The destination device 14 may also include a memory ( Figure 1 (Not shown), the memory can be used to store at least one of the following data: encoded image data 21, decoded image data 31, and post-processed image data 33.

[0103] The decoding system 10 also includes a training engine 25, which is used to train the encoder 20 to process the input image or image region or image block to obtain a feature map of the input image or image region or image block, and to obtain an estimated probability distribution of the feature map and encode the feature map according to the estimated probability distribution.

[0104] The training engine 25 is also used to train the decoder 30 to obtain the estimated probability distribution of the bitstream, decode the bitstream according to the estimated probability distribution to obtain a feature map, and reconstruct the feature map to obtain a reconstructed image.

[0105] although Figure 1 The source device 12 and destination device 14 are shown as independent devices, but device embodiments may also include both source device 12 and destination device 14, or the functions of both source device 12 and destination device 14, that is, simultaneously including source device 12 or its corresponding functions and destination device 14 or its corresponding functions. In these embodiments, source device 12 or its corresponding functions and destination device 14 or its corresponding functions may be implemented using the same hardware and / or software, or by separate hardware and / or software, or any combination thereof.

[0106] According to the description, Figure 1 The presence and (accurate) division of different units or functions in the source device 12 and / or destination device 14 shown may vary depending on the actual device and application, which is obvious to those skilled in the art.

[0107] In recent years, applying deep learning to the field of video codecs has gradually become a trend. Deep learning refers to learning at multiple levels of abstraction through machine learning algorithms. Video codecs based on deep learning can also be called AI video codecs or neural network-based video codecs. Since the embodiments of this application involve the application of neural networks, for ease of understanding, some terms or nouns used in the embodiments of this application will be explained below, and these terms or nouns are also part of the invention content.

[0108] (1) Neural network (NN)

[0109] Neural networks are machine learning models. A neural network can be composed of neural units, which can be computational units that take xs and an intercept of 1 as input. The output of such a computational unit can be:

[0110]

[0111] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0112] (2) Deep Neural Networks

[0113] Deep neural networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three layers based on their position: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is therefore quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary: the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as... It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0114] (3) Convolutional Neural Network (CNN)

[0115] Convolutional Neural Networks (CNNs) are a deep learning architecture and a typical method in image processing and analysis. A CNN contains at least convolutional layers and may also include activation layers, pooling layers, batch normalization layers (BN), fully connected layers (FC), and other functional modules. Activation layers can be rectified linear units (ReLU), parametric rectified linear units (PReLU), etc. Typical CNNs include LeNet, AlexNet, the visual geometry group network (VGGNet), the deep residual network (ResNet), YOLO (You Only Look Once), Faster R-CNN (Region with CNN feature), Mask R-CNN, and ASLFeat.

[0116] A basic convolutional neural network can be composed of a backbone network and a head network, such as AlexNet in object recognition; while some complex CNNs, such as Faster RCNN with a feature pyramid structure in the field of object detection, are composed of several parts: a backbone network, a neck network, and a head network.

[0117] The backbone network is the first part of a convolutional neural network. Its function is to extract feature maps at multiple scales from the input image. It typically consists of convolutional layers, pooling layers, activation layers, etc., but does not contain fully connected layers. Generally, the feature maps output by layers closer to the input image in the backbone network have a higher resolution (width and height) but fewer channels. Typical backbone networks include VGG-16, ResNet-50, and ResNet-101. The backbone network can be further divided into two parts: the front part of the backbone network and the core part of the backbone network. The front part of the backbone network, that is, the layers closest to the input, is also called the stem. It usually includes a few convolutional layers, and may also include other types of layers such as pooling layers. It performs preliminary processing of the input signal, reducing the spatial resolution and increasing the number of channels. For example, in ResNet-50, the input side is a structure composed of convolutional layers with 7×7 kernels and max pooling layers. Apart from the front part of the backbone network, the rest of the backbone network consists of the core part. The core part of the backbone network usually contains a large number of convolutional layers and some network sub-modules with the same or similar structures connected in series, such as the residual block structure (Resblock) in ResNet.

[0118] The neck network is the middle part of a convolutional neural network. Its function is to further integrate and process the feature maps generated by the backbone network to obtain new feature maps. A common neck network is the Feature Pyramid Network (FPN).

[0119] The head network is the final part of a convolutional neural network, and its function is to process feature maps to obtain the prediction results output by the neural network. Common head networks include fully connected layers, a normalized exponential function (Softmax) module, etc.

[0120] A bottleneck structure is a multi-layer network structure in which the input data first passes through one or more neural network layers to obtain intermediate data, and the intermediate data then passes through one or more neural network layers to obtain the output data. The amount of intermediate data (i.e., the product of width, height and number of channels) is less than the amount of input data and output data.

[0121] A feature map is a three-dimensional data output from layers such as convolutional layers, activation layers, pooling layers, and batch normalization layers in a convolutional neural network. Its three dimensions are called width, height, and channel. A feature map can be M × W × H, indicating that it is a two-dimensional feature map with a resolution of W × H, including M channels, where W represents width and H represents height. For example, when the original image is in RGB format, R represents red, G represents green, and B represents blue, and the feature map can include three channels: R, G, and B. When the original image is in YUV format (e.g., YUV444 format), Y represents luminance, U represents chrominance, V represents hue, and V represents saturation, and the feature map can include three channels: Y, U, and V.

[0122] In a convolutional neural network, a convolutional layer refers to a layer of neurons that performs convolutional processing on the input signal. A convolutional layer can contain multiple convolution operators, also known as kernels. In image processing, these operators act as filters, extracting specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, multiple weight matrices of the same size (rows × columns) are used instead of a single weight matrix. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. This dimension can be understood as being determined by the "multiple" factors mentioned above. Different weight matrices can be used to extract different features from the image. For example, one weight matrix can be used to extract edge information, another to extract specific colors, and yet another to blur unwanted noise. These multiple weight matrices have the same size (rows × columns), and the feature maps extracted by these weight matrices also have the same size. These extracted feature maps are then merged to form the output of the convolution operation. The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by these trained weight values ​​can be used to extract information from the input image, enabling the convolutional neural network to make correct predictions. When a convolutional neural network has multiple convolutional layers, the initial convolutional layers often extract more general features, which can also be called low-level features. As the depth of the convolutional neural network increases, the features extracted by later convolutional layers become increasingly complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem being solved.

[0123] Because it's often necessary to reduce the number of training parameters, pooling layers are frequently introduced periodically after convolutional layers. This can be a single convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. Average pooling calculates the average value of pixel values ​​within a specific range as the result of average pooling. Max pooling takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after pooling can be smaller than the size of the input image of the pooling layer. Each pixel in the output image represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0124] After processing by convolutional / pooling layers, a convolutional neural network (CNN) is still insufficient to output the required information. As mentioned earlier, convolutional / pooling layers only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the CNN needs to utilize neural network layers to generate one or a set of required class numbers of output. Therefore, the neural network can include multiple hidden layers (e.g., activation layers, batch normalization (BN) layers, and / or fully connected (FC) layers). The parameters contained in these hidden layers can be pre-trained based on training data relevant to the specific task type, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0125] Optionally, after the multiple hidden layers in the neural network, there is also an output layer of the entire convolutional neural network. This output layer has a loss function similar to the classification cross-entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network is completed, the backpropagation will begin to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network through the output layer and the ideal result.

[0126] Neural networks require training to determine the parameters of each layer. During training, forward loss calculation combined with backward gradient propagation is used to update the trainable parameters in the neural network. Through multiple parameter updates, the parameters of each layer eventually converge to a relatively optimal analytical accuracy. After training is complete, the parameters of each layer are usually fixed, and the input signal is passed through the neural network to obtain the result. This actual process of using a neural network is called "inference".

[0127] (4) Recurrent Neural Network

[0128] Recurrent neural networks (RNNs) are used to process sequential data. In traditional neural network models, the layers from the input layer to the hidden layer and then to the output layer are fully connected, but the nodes within each layer are unconnected. While this type of neural network has solved many difficult problems, it remains inadequate for many others. For example, predicting the next word in a sentence generally requires using the preceding words because words in a sentence are not independent. RNNs are called recurrent neural networks because the current output of a sequence is related to the outputs of previous sequences. Specifically, the network memorizes previous information and applies it to the calculation of the current output; that is, nodes within the same hidden layer are no longer unconnected but connected, and the input to a hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous time step. Theoretically, RNNs can process sequential data of any length. Training an RNN is similar to training a traditional CNN or DNN. This algorithm also uses the backpropagation algorithm, but with one key difference: when an RNN is expanded, its parameters, such as W, are shared; however, this is not the case with traditional neural networks as illustrated above. Furthermore, in gradient descent, the output at each step depends not only on the network at the current step but also on the states of the network in previous steps. This learning algorithm is called Backpropagation Through Time (BPTT).

[0129] (5) Loss Function

[0130] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0131] (6) Backpropagation algorithm

[0132] Convolutional neural networks can employ backpropagation (BP) to correct the parameters of the initial super-resolution model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters of the initial super-resolution model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the super-resolution model, such as the weight matrix.

[0133] (7) Generative Adversarial Networks

[0134] Generative adversarial networks (GANs) are a type of deep learning model. This model comprises at least two modules: a generative model and a discriminative model. These two modules learn from each other through a game-like interaction, resulting in better outputs. Both the generative and discriminative models can be neural networks, specifically deep neural networks or convolutional neural networks. The basic principle of GANs is as follows: Taking an image-generating GAN as an example, suppose there are two networks, G (Generator) and D (Discriminator). G is a network that generates images by receiving random noise z and using this noise, denoted as G(z). D is a discriminative network used to determine whether an image is "real." Its input parameter is x, representing an image, and its output D(x) represents the probability that x is a real image. A value of 1 indicates that the image is 100% real, while a value of 0 indicates that the image is impossible to be real. During the training of this generative adversarial network (GAN), the goal of the generative network G is to generate realistic images to deceive the discriminator network D, while the goal of the discriminator network D is to distinguish the images generated by G from real images as much as possible. Thus, G and D constitute a dynamic "game," which is the "adversarial" aspect of the GAN. Ideally, the game will result in G generating images G(z) that are sufficiently realistic, while D struggles to determine whether the images generated by G are real or not, i.e., D(G(z)) = 0.5. This yields a superior generative model G that can be used to generate images.

[0135] Figure 2 This application provides a schematic diagram of a deep learning-based video encoding and decoding network. Figure 2 Let's take entropy encoding and decoding as an example. This network includes a feature extraction module, a feature quantization module, an entropy encoding module, an entropy decoding module, a feature inverse quantization module, and a feature decoding (or image reconstruction) module.

[0136] At the encoding end, the original image is input to the feature extraction module. This module uses stacked convolutional layers and a non-linear mapping activation function to output the extracted feature map of the original image. The feature quantization module quantizes the floating-point feature values ​​in the feature map to obtain the quantized feature map. The quantized feature map is then entropy encoded to obtain the bitstream.

[0137] At the decoding end, the entropy decoding module parses the bitstream to obtain a quantized feature map. The feature dequantization module dequantizes the integer feature values ​​in the quantized feature map to obtain a dequantized feature map. After reconstruction by the feature decoding module, the dequantized feature map yields the reconstructed image.

[0138] Alternatively, this network may not include feature quantization and feature dequantization modules. In this case, the network can directly perform a series of processes on the feature map, which is a floating-point number. Alternatively, the network can perform integer-based processing so that the feature values ​​in the feature map output by the feature extraction module are all integers.

[0139] Figure 3 This application provides a schematic diagram of an end-to-end video encoding and decoding network structure based on deep learning, as shown in the embodiments of this application. Figure 3 Let's take entropy encoding / decoding as an example. This neural network includes a feature extraction module g. a Quantization module Q, edge information extraction module h a Entropy encoding module, entropy decoding module, probability estimation module h s and the rebuild module g s Entropy encoding can be done using auto-encoders (AE), and entropy decoding can be done using auto-decoders (AD).

[0140] Among them, g a It includes four convolutional layers interleaved with each other and three normalization layers, the normalization layers of which may include GDN (generalized divisive normalization) layers. a The four convolutional layers all have 5×5 kernels and a stride of 2. The first to third convolutional layers have N output channels, and the last convolutional layer has M output channels. In this embodiment, the stride is used to control the resolution of the image or feature map input to the convolutional layer. When the stride is 1, the convolutional layer keeps the resolution of the input image or feature map unchanged; when the stride is greater than 1, the convolutional layer uses the stride as a sampling factor to upsample or downsample the input image or feature map. a Each convolutional layer is used to downsample the resolution of the input image or feature map by a factor of 2. Furthermore, the number of output channels controls the number of channels in the output image or feature map of the convolutional layer, g. a The feature map of the output original image has M channels.

[0141] h a It includes three convolutional layers, two activation layers, and one abs layer, interleaved and cascaded. a The first convolutional layer has a 3×3 kernel, a stride of 1, and N output channels.a The second and third convolutional layers both have 5×5 kernels, a stride of 2, and N output channels. s It includes three convolutional layers and three activation layers interleaved and cascaded. s The first and second convolutional layers both have 5×5 kernels, a stride of 2, and N output channels. s The third convolutional layer has a 3×3 kernel, a stride of 1, and M output channels. s It includes four convolutional layers interleaved with each other and three inverse normalization layers. The inverse normalization layers may include IGDN layers. a The kernels of the four convolutional layers are all 5 × 5, and the stride is 2. The number of output channels of the first to third convolutional layers is N, and the number of output channels of the last convolutional layer is 3.

[0142] At the encoding end, the original image x is input into g. a g a The original image's feature map y is output. On one hand, this feature map y is input into Q, and Q outputs a quantized feature map, which is then input into the entropy encoding module. On the other hand, the feature map y is input into h. a h a Output edge information z. Input edge information z into Q, and Q outputs quantized edge information. The quantized edge information is then processed by an entropy encoding module to obtain the bitstream of the edge information, and then by an entropy decoding module to obtain the decoded edge information. Input the decoded edge information into h. s h s The output feature map contains the probability distribution of each feature element [x][y][i], and this probability distribution is input into the entropy coding module. The entropy coding module performs entropy coding on each input feature element based on its probability distribution to obtain the super-prior bitstream.

[0143] Here, the edge information z is a type of feature information, represented as a three-dimensional feature map, which contains fewer feature elements than the feature map y. Both the feature map y and the quantized feature map have M channels and a resolution of W×H. M and g... a The last convolutional layer in the image has the same number of output channels, and W and H are the same as the width and height of the original image, as well as g. a The stride is related to the stride of each convolutional layer. For example... Figure 3 As shown, g a The original image was downsampled four times at a magnification of 2. Assuming the resolution of feature map y or the quantized feature map is W×H, then the resolution of the original image is 2. 4 W×2 4 H.

[0144] At the decoding end, the entropy decoding module parses the bitstream of the side information to obtain the side information, and inputs the side information into h. s h s Output the probability distribution of each feature element [x][y][i] in the symbol to be decoded. Input the probability distribution of each feature element [x][y][i] into the entropy decoding module. The entropy decoding module performs entropy decoding on each feature element according to its probability distribution to obtain the decoded feature map. Input the decoded feature map into g. s g s Output the reconstructed image.

[0145] In addition, some variational autoencoders (VAEs) utilize previously encoded or decoded feature elements around the current feature element in their probability estimation modules to more accurately estimate the probability distribution of the current feature element.

[0146] It should be noted that, Figure 2 and Figure 3 The network structure shown is for illustrative purposes only. This application does not limit the modules included in the network or the structure of the modules.

[0147] In some operations of digital video applications, thumbnails of the original images need to be displayed to show a large number of images on the screen. Related technologies first decode and reconstruct the received original image stream to obtain the original image. Then, the resolution of the original image is reduced to obtain a thumbnail. That is, the process of obtaining thumbnails always requires reconstructing the original resolution image first. The complexity of reconstructing the original resolution image is high, resulting in a long time consumption for obtaining thumbnails of the original images, thus affecting the speed at which digital video applications display thumbnails of the original images. For example, in photo album applications, which need to display thumbnails of multiple images simultaneously, the display interface cannot refresh in time, thus affecting the display effect of the photo album application.

[0148] This application provides an encoding / decoding method. Please refer to the following embodiments. Figure 4 , Figure 4 This is a flowchart illustrating process 400 of an encoding / decoding method provided in an embodiment of this application. Process 400 can be executed by an electronic device (including an encoder and a decoder). Specifically, process 400 can be executed by the electronic device by calling a neural network model. Process 400 is described as a series of operations; it should be understood that process 400 can be executed in various orders and / or occur simultaneously, and is not limited to... Figure 4 The execution order is shown. Process 400 may include the following processes:

[0149] 401. The encoder extracts features from the original image to obtain an initial feature map. The resolution of the initial feature map is smaller than that of the original image.

[0150] The encoder can input the original image into the feature extraction module of a deep learning-based network. The feature extraction module outputs the extracted initial feature map by stacking multiple layers of convolution and combining non-linear mapping activation functions.

[0151] Optionally, the feature extraction module can refer to the aforementioned Figure 2 and Figure 3 .by Figure 3 For example, this feature extraction module may include four cascaded convolutional layers and three normalization layers. Each convolutional layer has a kernel size of 5x5 and a stride of 2, meaning each convolutional layer downsamples the resolution of the input original image or feature map by a factor of 2 (both width and height are downsampled by a factor of 2). The first three convolutional layers have N output channels, and the last convolutional layer has M output channels.

[0152] Assuming the original image has a resolution of W×H, after inputting the original image into the feature extraction module, the first convolutional layer downsamples the original image resolution by a factor of 2, outputting a feature map with N channels and a resolution of W / 2×H / 2. The second convolutional layer downsamples the feature map output by the first convolutional layer by a factor of 2, outputting a feature map with N channels and a resolution of W / 4×H / 4. The third convolutional layer downsamples the feature map output by the second convolutional layer by a factor of 2, outputting a feature map with N channels and a resolution of W / 8×H / 8. The final convolutional layer downsamples the feature map output by the third convolutional layer by a factor of 2, outputting an initial feature map with M channels and a resolution of W / 16×H / 16. In other words, the initial feature map is a two-dimensional feature map with M channels and a resolution of W / 16×H / 16.

[0153] It should be noted that the aforementioned Figure 3 The feature extraction module shown is merely illustrative. This application does not specifically limit the structure of the feature extraction module, the connection relationships between layers, the number of convolutional layers, or the stride, kernel, and number of output channels of any convolutional layer. Furthermore, at least one of the following parameters can be the same for any two convolutional layers: stride, kernel, and number of output channels. For example, the kernel of any convolutional layer can be 3×3, 7×7, or 9×9, the stride can be 1, 1.5, 3, 3.2, 4, or 5, and the number of output channels can be 1, 2, 5, or 8. The feature extraction module may not include a normalization layer, or it may include at least one activation layer and / or pooling layer.

[0154] 402. The encoder encodes the first feature map to obtain a bitstream. The first feature map is the initial feature map, or the first feature map includes a two-dimensional feature map of some channels in the initial feature map.

[0155] The initial channel map includes two-dimensional feature maps of multiple channels. When the first feature map includes two-dimensional feature maps of some channels in the initial feature map, the encoder does not need to encode the two-dimensional feature maps of all channels in the initial feature map. This reduces the amount of data in the subsequently encoded bitstream, thereby reducing the communication overhead of transmitting the bitstream to the decoder.

[0156] For example, assuming the initial feature map includes two-dimensional feature maps of M channels, the first feature map may include two-dimensional feature maps of M1 channels, where M1 ≤ M. When M1 < M, the first feature map may include two-dimensional feature maps of any M1 channels out of the M channels, or it may include two-dimensional feature maps of the first M1 channels or the last M1 channels out of the M channels. This application does not limit this.

[0157] The first feature map includes multiple feature elements, and the encoder encodes each feature element to obtain the bitstream. Optionally, the first feature map can be encoded based on a probability distribution model, or it can be encoded using a hyper-prior entropy encoding method.

[0158] In one implementation, the first feature map is encoded based on a probability distribution model. First, a model is built based on the probability distribution model, then the context information of the feature elements to be encoded is obtained. The probability distribution of the feature elements to be encoded is obtained based on the context information, and then the feature elements to be encoded are encoded according to the probability distribution. After performing the aforementioned process on each feature element in the first feature map, a bitstream is obtained. The context information may include already encoded feature elements adjacent to the feature element to be encoded in the first feature map.

[0159] The probability distribution models include at least one of the following: Gaussian single model (GSM), asymmetric Gaussian model, Gaussian mixture model (GMM), and Laplace distribution model.

[0160] The encoder can invoke a neural network model to encode the first feature map based on a probability distribution model. For example, contextual information can be input into a probability estimation network, which outputs model parameters for the feature elements to be encoded. These model parameters are then input into the probability distribution model, which outputs the probability distribution of the feature elements to be encoded. The probability estimation network can include deep learning-based neural networks, such as RNNs and CNNs.

[0161] In the second implementation, a super-prior entropy coding method is used to encode the first feature map. First, edge information is extracted from the first feature map. This extracted edge information is then quantized, and entropy coding and decoding are performed sequentially on the quantized edge information to obtain the decoded edge information. The probability distribution of the feature elements to be encoded is obtained based on the decoded edge information, and then entropy coding is performed on the feature elements to be encoded based on this probability distribution. After performing the aforementioned process on each feature element in the first feature map, a bitstream is obtained.

[0162] The encoder can call a neural network model to encode the first feature map using a super-prior entropy encoding method. Optionally, the first feature map can be input into a side information extraction module, which outputs side information. The side information is then input into a probability estimation network, which outputs the model parameters of the feature elements to be encoded. The side information extraction module and the probability estimation network can be referenced from the previous section. Figure 3 The corresponding descriptions are not repeated here in the embodiments of this application.

[0163] Before encoding the first feature map, it can be quantized to obtain a quantized first feature map. Then, the quantized first feature map is encoded to obtain the bitstream. As described above... Figure 2 or Figure 3 As shown, the first feature map can be quantized using the quantization module.

[0164] For example, the quantization process includes: quantizing each feature element (or feature value) in the first feature map, and converting the floating-point feature element to an integer feature element. Optionally, the floating-point feature element can be rounded to obtain an integer feature element, or the floating-point feature element can be truncated to obtain an integer feature element, or an integer feature element can be obtained according to a preset quantization step size. The embodiments of this application do not limit the quantization process.

[0165] 403. The encoder sends the bit stream to the decoder.

[0166] As described in the foregoing embodiments, the encoder and decoder have communication interfaces that are connected, and the encoder can send the bit stream to the decoder's communication interface through the communication interface.

[0167] 404. The decoder decodes the bitstream to obtain the first feature map.

[0168] The bitstream corresponds to a first feature map, which is a two-dimensional feature map of multiple channels included in the first feature map. The decoder can decode the bitstream corresponding to each feature element included in the bitstream to obtain the first feature map.

[0169] The decoding process corresponds to the encoding process. Following the aforementioned encoding method based on a probability distribution model, the decoder obtains the context information of the bitstream corresponding to the feature element to be decoded, acquires the probability distribution of the bitstream corresponding to the feature element based on the context information, and then decodes the bitstream corresponding to the feature element based on this probability distribution. After performing the aforementioned process on the bitstream corresponding to each feature element in multiple channels, the first feature map is obtained.

[0170] When the encoder calls a neural network model to encode the first feature map based on a probability distribution model, the decoder can also call a neural network model to decode the bitstream. For example, the context information of the bitstream corresponding to the feature element to be decoded can be input into the same probability estimation network as the encoder side. The probability estimation network outputs the model parameters of the bitstream corresponding to the feature element to be decoded. The model parameters are then input into the same probability distribution model as the encoder side, and the probability distribution model outputs the probability distribution of the bitstream corresponding to the feature element to be decoded. Both the probability distribution model and the probability estimation network can be referenced in process 402, and will not be elaborated further in this embodiment.

[0171] Corresponding to the aforementioned super-prior entropy encoding method, the decoder first performs entropy decoding on the bitstream of the edge information to obtain the edge information. Based on the entropy-decoded edge information, it obtains the probability distribution of the bitstream corresponding to the feature element to be decoded, and then performs entropy decoding on the bitstream corresponding to the feature element to be decoded based on this probability distribution. After performing the aforementioned process on the bitstream corresponding to each feature element in multiple channels, the first feature map is obtained.

[0172] When the encoder calls a neural network model to entropy encode the first feature map using a super-prior entropy encoding method, the decoder can also call a neural network model to entropy decode the bitstream. For example, the side information after entropy decoding can be input into the same probability estimation network as the encoder, and the probability estimation network outputs the probability distribution of the bitstream corresponding to the feature elements to be decoded. The probability estimation network can be referenced in process 402, and will not be elaborated upon here in this embodiment.

[0173] 405. The decoder reconstructs the second feature map based on the first neural network to obtain a reconstructed image. The resolution of both the second feature map and the reconstructed image is the target resolution, which is smaller than the resolution of the original image. The second feature map is the first feature map, or the second feature map is a feature map obtained by processing the first feature map through the second neural network.

[0174] The resolution of the reconstructed image output by the first neural network is the same as the resolution of the input second feature map, both being the target resolution. When the second feature map is the first feature map, the target resolution is equal to the resolution of the first feature map. When the second feature map is a feature map obtained by processing the first feature map through the second neural network, the target resolution is greater than the resolution of the first feature map.

[0175] Optionally, the first neural network may include at least one convolutional layer, with a convolution stride of 1. The first neural network is used to transform the number of channels of the input feature map, and its number can be one or more. When there are multiple first neural networks, the structures of any two first neural networks can be the same or different, and this application embodiment does not limit this.

[0176] The second neural network processes the first feature map to obtain a second feature map. This processing may include convolutional and / or deconvolutional operations, and further, may include normalization operations. Optionally, the second neural network may include one or more output nodes and multiple convolutional layers. The output nodes are located between the output of the first convolutional layer and the input of the last convolutional layer, and are connected to the input of the first neural network. The convolutional layers are used to perform convolutional and / or deconvolutional operations on the input feature map. The first neural network may also include at least one normalization layer, which is used to normalize the input feature map.

[0177] As described in process 401 above, the feature extraction module on the encoder side is used to extract features from the original image to obtain an initial feature map. When the first feature map is the initial feature map, the second neural network can reconstruct the first feature map to obtain a reconstructed image at the original resolution. Therefore, the structure of the second neural network corresponds to the structure of the feature extraction module on the encoder side. The total upsampling factor of the convolutional layers in the second neural network on the first feature map is... Figure 3 The feature extraction module shown includes convolutional layers with the same total downsampling factor as the original image. That is, the second neural network includes convolutional layers that amplify the resolution of the first feature map by the same factor. Figure 3 The feature extraction module shown includes convolutional layers that reduce the resolution of the original image by the same factor. Furthermore, the number of output channels in the last convolutional layer of the second neural network is the same as the actual number of channels in the original image. This allows the second neural network to reconstruct an image at the original resolution.

[0178] When there is only one first neural network, in one example, there is no need to apply a second neural network, and the first neural network can exist independently. In this case, the second feature map is the first feature map, and the first neural network can only output a reconstructed image at one resolution, which is the resolution of the first feature map.

[0179] In another example, the first neural network can be connected to the input of the second neural network and / or to one or more output nodes. In this case, the second feature map is either the first feature map or a feature map obtained by processing the first feature map through the second neural network. The first neural network can output reconstructed images at one or more resolutions.

[0180] When there are multiple first neural networks, the input and / or each output node of the second neural network are connected to the first neural network. In this case, the second feature map is either the first feature map or a feature map obtained by processing the first feature map through the second neural network. The first neural network can output reconstructed images at various resolutions.

[0181] The second neural network is used to process the first feature map to obtain the second feature map. The second feature map is output by the output node and input to the first neural network connected to the output node.

[0182] The following example illustrates a first neural network with one or more nodes, multiple output nodes, and a second neural network whose input and each output node are connected to an input of the first neural network.

[0183] Each output node corresponds to an output resolution. The resolution of the reconstructed image output by the first neural network can include the output resolution of each output node and the resolution of the first feature map. For ease of explanation, all resolutions of the reconstructed image that the first neural network can output are referred to as candidate resolutions. The decoder can first determine the target resolution. When the target resolution is equal to the resolution of the first feature map, the decoder can directly input the first feature map into the first neural network connected to the input end of the second neural network to obtain the reconstructed image output by the first neural network. When the target resolution is not equal to the resolution of the first feature map, the decoder can determine the output node with the target resolution as the target input node. Then, the first feature map is input into the second neural network to obtain the second feature map output by the target output node of the second neural network. The second feature map output from the target output node is then input into the first neural network to obtain the reconstructed image output by the first neural network.

[0184] When there is only one first neural network, the second feature map output by the target output node is directly input into that single network to obtain the reconstructed image output by that single neural network. When there are multiple first neural networks, the second feature map is input into the first neural network connected to the target output node to obtain the reconstructed image output by the first neural network connected to the target output node.

[0185] The target resolution can be determined by the decoder or externally, such as by the external application layer or encoder. External application layers can refer to the aforementioned digital video applications, such as video players, photo albums, and web pages.

[0186] When the target resolution is determined by the decoder, in one example, the decoder can determine the target resolution based on the thumbnail resolution. For example, in conjunction with embodiments of this application, the decoder can determine the target resolution as the resolution among the candidate resolutions that is closest to the thumbnail resolution. The thumbnail resolution is the resolution of the thumbnail that is ultimately displayed in the external application layer of the decoder, which may be the same as or different from the target resolution.

[0187] Optionally, the thumbnail resolution can be indicated by an external application layer or an encoder. The external application layer or encoder can send an identifier indicating the thumbnail resolution to the decoder. When the encoder sends the identifier indicating the thumbnail resolution to the decoder, the identifier can be sent separately or carried in the bitstream. This application embodiment does not limit the method by which the decoder determines the thumbnail resolution.

[0188] In another example, the decoder can determine the target resolution based on the complexity and / or power consumption constraints of the reconstructed image acquisition process for each candidate resolution. For example, the decoder can determine the candidate resolution corresponding to the reconstructed image acquisition process with the least complexity as the target resolution; or determine the candidate resolution corresponding to the reconstructed image acquisition process that is closest to the maximum complexity that the external application layer can bear as the target resolution.

[0189] When the target resolution is determined externally to the decoder, in one example, the target resolution is determined by an external application layer and sent to the decoder. The external application layer can send an identifier indicating the target resolution to the decoder, which then determines the target resolution based on this identifier. The external application layer can determine the target resolution based on candidate resolutions and thumbnail resolutions, or it can determine the target resolution based on the complexity and / or power consumption constraints of the reconstructed image acquisition process for each candidate resolution. Candidate resolutions can be located in the bitstream sent by the decoder as Supplemental Enhancement Information (SEI), or in a file format within the application layer, so that the external application layer can obtain the candidate resolutions. The process of determining the target resolution can refer to the aforementioned process of the decoder determining the target resolution, and will not be elaborated upon here in the embodiments of this application.

[0190] In another example, the target resolution is determined by the encoder and sent to the decoder. The encoder may send an identifier indicating the target resolution to the decoder, which determines the target resolution based on the identifier. The target resolution identifier may be sent separately or carried in the bitstream. The process of determining the target resolution can refer to the aforementioned process of the decoder determining the target resolution, and will not be repeated here in the embodiments of this application.

[0191] For example, please refer to Figure 5 , Figure 5 This is a schematic diagram of the structure of a first neural network and a second neural network provided in an embodiment of this application. Figure 5 Let's take the example of a first neural network having multiple nodes as an example. Figure 5 The structure of the second neural network in the text can be similar to... Figure 3 The structure of the feature extraction module shown corresponds to that of the module. For example... Figure 5 As shown, the second neural network consists of four cascaded convolutional layers and three normalization layers. The kernel size of each convolutional layer is... Figure 3 The convolutional kernels in the feature extraction module shown are all the same, 5×5, with a stride of 2. This means each convolutional layer upsamples the input feature map by a factor of 2 (both width and height are upsampled by a factor of 2). The first three convolutional layers have N output channels, and the last convolutional layer has 3 output channels. The second neural network includes three output nodes a1 to a3: a1 is located at the output of the first convolutional layer, a2 at the output of the second convolutional layer, and a3 at the output of the third convolutional layer.

[0192] Figure 5 Four first neural networks, b1 to b4, are shown. The input of b1 is connected to the input of the second neural network, the input of b2 is connected to a1, the input of b3 is connected to a2, and the input of b4 is connected to a3.

[0193] Figure 5Let's take b1 and b2 as examples where their structures are the same, while b1, b3, and b4 are different. Both b1 and b2 consist of two convolutional layers, each with a 5×5 kernel and a stride of 1. The first convolutional layer has N1 output channels, and the last convolutional layer has P output channels. For example, P can be 1 or 3. When P is 1, the reconstructed image is grayscale; when P is 3, the reconstructed image is a three-channel color image. N1 > P; the larger N1 is, the more feature elements are input to the last convolutional layer, providing more information and resulting in a higher-quality reconstructed image. b3 consists of three convolutional layers, each with a 5×5 kernel and a stride of 1. The first convolutional layer in b3 has N2 output channels, the second has N3, and the third has P. b4 consists of two convolutional layers, both with 5×5 kernels and a stride of 1. The first convolutional layer in b4 has N⁴ output channels, and the second convolutional layer has P output channels. Figure 5 As can be seen from the structures of the first and second neural networks shown, the feature maps input to the input terminals of any two neural networks have different resolutions.

[0194] Figure 5 The resolution of output b1 is the resolution of the first feature map. The resolution of output b2 is the resolution of the feature map output by the first convolutional layer after the first feature map is input into the second neural network. The resolution of output b3 is the resolution of the feature map output by the second convolutional layer after the first feature map is input into the second neural network. The resolution of output b4 is the resolution of the feature map output by the third convolutional layer after the first feature map is input into the second neural network. Assuming the resolution of the first feature map is W / 16×H / 16, then the output resolutions of b1 to b4 are W / 16×H / 16, W / 8×H / 8, W / 4×H / 4, and W / 2×H / 2, respectively.

[0195] Assuming the first feature map has M1 channels and a resolution of W / 16×H / 16, when the target resolution is the same as the first feature map's resolution, the first feature map is directly input into b1. The first convolutional layer of b1 maintains the resolution of the first feature map, performs channel conversion on the first feature map, and outputs a feature map with N1 channels and a resolution of W / 16×H / 16. The second convolutional layer of b1 maintains the resolution of the input feature map, performs channel conversion on the input feature map, and outputs a reconstructed image with P channels and a resolution of W / 16×H / 16.

[0196] When the target output node determined based on the target resolution is a1, the first feature map is input into the second neural network. The first convolutional layer of the second neural network upsamples the resolution of the first feature map by a factor of 2, outputting a second feature map with N channels and a resolution of W / 8×H / 8 from a1. The second feature map is then input into b2, which is connected to a1. The first and second convolutional layers of b2 maintain the resolution of the input feature map and sequentially perform channel transformation on the input feature map, finally outputting a reconstructed image with P channels and a resolution of W / 8×H / 8. The outputs of each convolutional layer in b2 can be referenced from b1, and will not be elaborated further in this embodiment.

[0197] When the target output node determined based on the target resolution is a2, the first feature map is input into the second neural network. The first and second convolutional layers of the second neural network sequentially upsample the resolution of the first feature map by a factor of 2, outputting a second feature map with N channels and a resolution of W / 4×H / 4 from a2. The second feature map is then input into b3, which is connected to a2. The first to third convolutional layers of b3 maintain the resolution of the input feature map and sequentially perform channel transformation on the input feature map, finally outputting a reconstructed image with P channels and a resolution of W / 4×H / 4. The outputs of each convolutional layer in b3 can be referenced from b1, and will not be elaborated further in this embodiment.

[0198] When the target output node determined based on the target resolution is a3, the first feature map is input into the second neural network. The first to third convolutional layers of the second neural network sequentially upsample the resolution of the first feature map by a factor of 2, outputting a second feature map with N channels and a resolution of W / 2×H / 2. The second feature map is then input into b4, which is connected to a3. The first and second convolutional layers of b4 maintain the resolution of the input feature map and sequentially perform channel transformation on the input feature map, finally outputting a reconstructed image with P channels and a resolution of W / 2×H / 2.

[0199] It should be noted that there can be one or more target resolutions. When there are multiple target resolutions, and multiple output nodes are determined based on these multiple target resolutions, the first feature map can be input into the second neural network to obtain second feature maps at multiple resolutions output by the multiple target output nodes of the second neural network. Then, the second feature map output by each target output node is input into the first neural network connected to the target output node to obtain reconstructed images at multiple resolutions output by the first neural networks connected to the multiple target output nodes respectively.

[0200] Each output node can output the second feature map in parallel or serially. When the second feature map is output serially, the second feature maps output by each output node can be shared. For example... Figure 5 As shown, assuming the determined target output nodes include a1, a2, and a3, the first feature map can be input into the first neural network three times in sequence, so that a1, a2, and a3 can output the corresponding second feature map in parallel based on the first feature map input from the input end of the first neural network. This can improve the efficiency of obtaining the reconstructed image.

[0201] Alternatively, the first feature map can be input into the first neural network once. After a1 outputs the second feature map based on the first feature map input from the input end of the first neural network, a2 outputs the second feature map based on the second feature map output by a1. a3 can output the second feature map based on the second feature map output by a1 after a1 outputs the second feature map, or it can output the second feature map based on the second feature map output by a2 after a2 outputs the second feature map. This can reduce the number of calculations in the second neural network, thereby reducing the computational overhead of the second neural network.

[0202] Figure 5 The structures of the first and second neural networks, as well as the connection relationships between them, are illustrative and not limited in this application. For example, in... Figure 5 Based on this, any branch of a neural network may also include one or more convolutional layers, which may be the same or different.

[0203] For example, please refer to Figure 6 , Figure 6 This is a schematic diagram of another first neural network and a second neural network provided in an embodiment of this application. Figure 6 Let's take the first neural network as an example, where it consists of one layer. The second neural network includes two convolutional layers, both with 5×5 kernels and a stride of 1. The structure of the second neural network can be found in [reference needed]. Figure 5 The description of the aforementioned neural network branches will not be repeated here in the embodiments of this application. The first convolutional layer has N1 output channels, and the last convolutional layer has P output channels. A description of the first neural network can be found in [reference needed]. Figure 5 The corresponding descriptions are not repeated here in the embodiments of this application.

[0204] like Figure 6 As shown, the second neural network includes three output nodes c1 to c3. c1 is located at the output of the first normalization layer, c2 is located at the output of the second convolutional layer, and c3 is located at the output of the third normalization layer. c1 to c3 are all connected to the input of the first neural network, and the input of the second neural network is also connected to the input of the first neural network.

[0205] Figure 6The resolution of the first neural network output includes: the resolution of the first feature map; the resolution of the feature map output by the first normalization layer after the first feature map is input into the second neural network; the resolution of the feature map output by the second convolutional layer after the first feature map is input into the second neural network; and the resolution of the feature map output by the third normalization layer after the first feature map is input into the second neural network. Assuming the resolution of the first feature map is W / 16×H / 16, then the resolutions of the first neural network output include W / 16×H / 16, W / 8×H / 8, W / 4×H / 4, and W / 2×H / 2.

[0206] Assuming the first feature map has M1 channels and a resolution of W / 16×H / 16, when the target resolution is the same as the resolution of the first feature map, the first feature map is directly input into the first neural network. The first and second convolutional layers of the first neural network keep the resolution of the input feature map unchanged, and sequentially perform channel transformation on the input feature map, finally outputting a reconstructed image with P channels and a resolution of W / 16×H / 16.

[0207] When the target output node determined based on the target resolution is c1, the first feature map is input into the second neural network. The first convolutional layer of the second neural network upsamples the resolution of the first feature map by a factor of 2, and outputs a second feature map with N channels and a resolution of W / 8×H / 8 from c1. The second feature map is then input into the first neural network, and the first neural network finally outputs a reconstructed image with P channels and a resolution of W / 8×H / 8. The processing procedure of the first neural network for the input feature map can be referred to the foregoing description, and will not be repeated here in the embodiments of this application.

[0208] When the target output point determined according to the target resolution is c2, the first feature map is input into the second neural network. The first and second convolutional layers of the second neural network sequentially upsample the resolution of the first feature map by a factor of 2, and output a second feature map with N channels and a resolution of W / 4×H / 4 from c2. The second feature map is then input into the first neural network, and the first neural network outputs a reconstructed image with P channels and a resolution of W / 4×H / 4. The processing procedure of the first neural network for the input feature map can be referred to the foregoing description, and will not be repeated here in the embodiments of this application.

[0209] When the target output point determined based on the target resolution is c3, the first feature map is input into the second neural network. The first to third convolutional layers of the second neural network sequentially upsample the resolution of the first feature map by a factor of 2, outputting a second feature map with N channels and a resolution of W / 2×H / 2 from c3. The second feature map is then input into the first neural network, which outputs a reconstructed image with P channels and a resolution of W / 2×H / 2. The processing procedure of the first neural network for the input feature map can be referred to the foregoing description, and will not be repeated here in the embodiments of this application.

[0210] It should be noted that the second neural network is an existing neural network used in encoding and decoding in related technologies, used to generate reconstructed images at the original resolution. By setting output nodes at different positions in the second neural network, reconstructed images with various target resolutions can be generated using these output nodes and the existing second neural network. In this process, the second neural network processes the first feature map, and each output node shares layers in the second neural network during the generation of reconstructed images at different target resolutions. This reduces the size of the neural network used to generate the reconstructed images (including the first and second neural networks), reduces the storage space occupied by the neural network in the decoder, and thus reduces the operating overhead and complexity of the neural network in the decoder.

[0211] Optionally, in the embodiments of this application, before reconstructing the feature map of the target resolution, the channels of the first feature map and / or the second feature map may be reduced to reduce the complexity of the subsequent reconstruction process and improve the efficiency of the reconstruction process.

[0212] For example, the second feature map includes two-dimensional feature maps of multiple channels, and channel reduction processing can be performed on the second feature map. Then, the channel-reduced second feature map is reconstructed to obtain a reconstructed image. Two-dimensional feature maps of some channels can be randomly extracted from the two-dimensional feature maps of the multiple channels included in the second feature map, or the two-dimensional feature maps of the first few channels or the last few channels can be extracted. The embodiments of this application do not limit the method of channel reduction processing.

[0213] For example, please refer to Figure 7 , Figure 7 This is a schematic diagram of a channel reduction process provided in an embodiment of this application. Figure 7 In Figure 5 The following explanation uses channel reduction processing based on the first and second neural networks as an example. In each first neural network, channel reduction processing is performed on the second feature map. For example... Figure 7As shown, in b1, the number of channels in the second feature map is reduced from M1 to S1; in b2, it is reduced from N to S2; in b3, it is reduced from N to S3; and in b4, it is reduced from N to S4. S1 is less than M1, and S2, S3, and S4 are all less than N. Taking a1 as an example, S1 two-dimensional feature maps can be randomly extracted from the M1-channel two-dimensional feature maps, or the first S1 channels from the M1 channels can be extracted. Here, the second feature map in b1 is the first feature map, and M1 represents the number of channels in the first feature map.

[0214] In another example, the first feature map comprises a two-dimensional feature map with multiple channels, and channel reduction processing can be performed on the first feature map. The second feature map is the first feature map after channel reduction processing, or the second feature map is a feature map obtained by processing the first feature map after channel reduction processing through a second neural network. As described above. Figure 7 As shown, the first feature map can be processed by channel reduction before inputting the first feature map into the second neural network or b1. This process can be referred to the foregoing example, and will not be described in detail here.

[0215] Furthermore, in this embodiment, after obtaining the reconstructed image, the reconstructed image at the target resolution can be directly output and displayed in the external application layer. Alternatively, the target resolution can be compared with the thumbnail resolution, and further processing can be performed based on the comparison results.

[0216] Optionally, when the target resolution is equal to the thumbnail resolution, the reconstructed image at the target resolution is directly output and displayed. When the target resolution is not equal to the thumbnail resolution, the reconstructed image is scaled to make the target resolution equal to the thumbnail resolution, and then the scaled reconstructed image is output and displayed. The scaling operation includes downsampling and upsampling operations. When the target resolution is less than the thumbnail resolution, the reconstructed image can be upsampled; when the target resolution is greater than the thumbnail resolution, the reconstructed image can be downsampled. For example, the upsampling operation may include bilinear interpolation upsampling, or directly filling in the pixel values ​​of adjacent positions for upsampling, or achieving upsampling through deconvolution of a convolutional layer with a stride greater than 1. The downsampling operation may include bilinear interpolation downsampling, directly removing some pixel values ​​for downsampling, or achieving downsampling through convolution of a convolutional layer with a stride less than 1. The upsampling and downsampling methods in this application embodiment are not limited, as long as the resolution of the reconstructed image can be increased or decreased from the target resolution to the thumbnail resolution.

[0217] Alternatively, when the difference between the target resolution and the thumbnail resolution is less than a difference threshold, the reconstructed image at the target resolution is directly output and displayed. When the difference between the target resolution and the thumbnail resolution is greater than the difference threshold, the reconstructed image is scaled to make the difference between the target resolution and the thumbnail resolution less than the difference threshold, and then the scaled reconstructed image is output and displayed. The scaling operation can be referred to the foregoing description, and will not be repeated here in the embodiments of this application.

[0218] It should be noted that the process of comparing the target resolution with the thumbnail resolution and performing further processing based on the comparison results can be performed by the decoder or by an external module of the decoder. This application embodiment does not limit this.

[0219] In summary, the encoding / decoding method provided in this application involves an encoder extracting features from the original image to obtain an initial feature map, encoding the first feature map to obtain a bitstream, and sending the bitstream to the decoder. The decoder decodes the bitstream to obtain the first feature map, and then reconstructs the second feature map using a first neural network to obtain a reconstructed image. Both the second feature map and the reconstructed image have a target resolution, which is smaller than the resolution of the original image. The second feature map includes the first feature map, or it is a feature map obtained by processing the feature map of the original image using a second neural network. Since the resolution of the reconstructed image obtained after decoding and reconstruction is smaller than that of the original image, there is no need to reconstruct the original image first during the acquisition of the reconstructed image; instead, the reconstructed image is obtained directly, improving the efficiency of acquiring the reconstructed image and thus increasing the speed at which digital video applications display thumbnails of the original image.

[0220] Furthermore, the second neural network includes one or more output nodes, each corresponding to an output resolution. Using these output nodes and the second neural network, reconstructed images with various target resolutions can be generated. In this process, the second neural network processes the first feature map. During the generation of reconstructed images with different target resolutions, each output node shares layers within the second neural network. This reduces the size of the neural network used to generate the reconstructed images (including both the first and second neural networks), decreases the storage space occupied by the neural network in the decoder, and thus reduces the overhead and complexity of the neural network in the decoder.

[0221] The order of the methods provided in the embodiments of this application can be adjusted appropriately, and the process can also be added or removed as appropriate. Any variations that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application, and the embodiments of this application do not limit this.

[0222] This application provides another encoding / decoding method; please refer to [link / reference]. Figure 8 , Figure 8 This is a flowchart illustrating another encoding / decoding method, process 500, provided in an embodiment of this application. Process 500 can be executed by an electronic device (including an encoder and a decoder). Specifically, process 500 can be executed by the electronic device by invoking a neural network model. Process 500 is described as a series of operations; it should be understood that process 500 can be executed in various orders and / or occur simultaneously, and is not limited to... Figure 8 The execution order is shown. Process 500 may include the following processes:

[0223] 501. The encoder extracts features from the original image to obtain an initial feature map. The resolution of the initial feature map is smaller than that of the original image.

[0224] This process can refer to the aforementioned process 401, and will not be repeated here in the embodiments of this application.

[0225] 502. The encoder encodes the first feature map to obtain a bitstream. The first feature map is the initial feature map, or the first feature map includes a two-dimensional feature map of some channels in the initial feature map.

[0226] This process can refer to the aforementioned process 402, and will not be repeated here in the embodiments of this application.

[0227] 503. The encoder sends the bit stream to the decoder.

[0228] This process can be referred to as the aforementioned process 403, and will not be repeated here in the embodiments of this application.

[0229] 504. The decoder decodes the bitstream to obtain the first feature map.

[0230] This process can be referred to in the aforementioned process 404, and will not be repeated here in the embodiments of this application.

[0231] 505. The decoder reconstructs the second feature map of the first resolution based on the first neural network to obtain a reconstructed image of the second resolution. The second resolution is different from the first resolution and is smaller than the resolution of the original image. The second feature map includes the first feature map, or the second feature map is a feature map obtained by processing the first feature map through the second neural network.

[0232] The difference between this process and the aforementioned process 405 is that in the aforementioned process 405, the first neural network maintains the resolution of the second feature map unchanged to generate the reconstructed image, while in process 505, the first neural network can change the resolution of the second feature map.

[0233] The resolution of the reconstructed image output by the first neural network differs from the resolution of the input second feature map. Optionally, the first neural network may include at least one convolutional layer, wherein at least one convolutional layer contains a convolutional layer with a stride other than 1. The first neural network may upsample or downsample the resolution of the input second feature map by using convolutional layers with strides other than 1, thereby changing the resolution of the input second feature map.

[0234] For example, as mentioned above Figure 5 As shown, the stride of the second convolutional layer in b1 can be 1.3, and it is used to downsample the resolution of the second feature map by a factor of 1.3. As previously mentioned... Figure 6 As shown, the stride of the first convolutional layer of the first neural network can be 1.6, and it is used to upsample the resolution of the second feature map by a factor of 1.6.

[0235] Since the second neural network can output values ​​at various resolutions that are fixed, the first neural network connected to one or more output nodes of the second neural network also outputs values ​​at fixed resolutions. In process 505, the first neural network also has the ability to change the resolution of the input feature map. This allows for the output of various numerical resolutions through first neural networks with different structures, reducing the overhead and complexity of the neural network used to generate the reconstructed image in the decoder while increasing the flexibility of generating the reconstructed image.

[0236] The relevant descriptions of the first and second neural networks in process 505 can be found in the aforementioned process 405, and will not be repeated here in the embodiments of this application.

[0237] In summary, the encoding / decoding method provided in this application involves an encoder extracting features from the original image to obtain an initial feature map, then encoding the first feature map to obtain a bitstream and sending the bitstream to the decoder. The decoder decodes the bitstream to obtain the first feature map, and then reconstructs the second feature map at a first resolution using a first neural network to obtain a reconstructed image at a second resolution. The second resolution differs from the first resolution; it is smaller than the resolution of the original image. The second feature map includes the first feature map, or it is a feature map obtained by processing the first feature map using a second neural network. The reconstructed image obtained after decoding and reconstruction has a resolution smaller than the original image. Therefore, in the process of acquiring the reconstructed image, it is not necessary to first reconstruct the original image; instead, the reconstructed image is obtained directly, improving the efficiency of acquiring the reconstructed image and thus increasing the speed at which digital video applications display thumbnails of the original image.

[0238] In addition, the first neural network also has the function of changing the resolution of the input feature map. This allows for the output of various numerical resolutions through first neural networks with different structures. This reduces the overhead and complexity of the neural network used to generate the reconstructed image in the decoder, while improving the flexibility of generating the reconstructed image.

[0239] The order of the methods provided in the embodiments of this application can be adjusted appropriately, and the process can also be added or removed as appropriate. Any variations that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application, and the embodiments of this application do not limit this.

[0240] This application provides another encoding / decoding method; please refer to [reference needed]. Figure 9 , Figure 9 This is a flowchart illustrating process 600 of another encoding / decoding method provided in an embodiment of this application. Process 600 can be executed by an electronic device (including an encoder and a decoder). Specifically, process 600 can be executed by an electronic device by calling a neural network model. Process 600 is described as a series of operations. It should be understood that process 600 can be executed in various orders and / or occur simultaneously, and is not limited to... Figure 8 The execution order is shown. Process 600 may include the following procedures:

[0241] 601. The encoder extracts features from the original image to obtain an initial feature map. The resolution of the initial feature map is smaller than that of the original image, and the number of channels in the initial feature map is M.

[0242] This process can refer to the aforementioned process 401, and will not be repeated here in the embodiments of this application.

[0243] 602. The encoder encodes the feature map to be encoded to obtain a bitstream. The bitstream corresponds to the two-dimensional feature map of M1 channels in the initial feature map, where M1≤M.

[0244] This process can refer to the aforementioned process 402. The feature map to be encoded is equivalent to the first feature map in process 602. The embodiments of this application will not be described in detail here.

[0245] 603. The encoder sends the bit stream to the decoder.

[0246] This process can be referred to as the aforementioned process 403, and will not be repeated here in the embodiments of this application.

[0247] 604. The decoder decodes the bitstream corresponding to the two-dimensional feature maps of M2 out of M1 channels to obtain the first feature map, where M2 < M1, and the first feature map includes the two-dimensional feature maps of M2 channels.

[0248] The bitstream corresponds to M1 channels of two-dimensional feature maps, and the bitstreams corresponding to the M1 channels of two-dimensional feature maps are arranged sequentially. The decoder can decode the bitstreams corresponding to the two-dimensional feature maps of the first M2 channels out of the M1 channels. In this way, the subsequent reconstruction process can be performed after the bitstreams corresponding to the two-dimensional feature maps of the M2 channels are decoded, without having to perform the subsequent reconstruction process after the entire bitstream is decoded. This improves the efficiency of obtaining the third feature map, thereby improving the efficiency of obtaining the reconstructed image.

[0249] The process of decoding the bitstream corresponding to the two-dimensional feature maps of M2 out of M1 channels can be referred to the aforementioned process 404, and will not be repeated here in the embodiments of this application.

[0250] 605. The decoder reconstructs the second feature map based on the first neural network to obtain a reconstructed image. The resolution of both the second feature map and the reconstructed image is the target resolution, which is smaller than the resolution of the original image. The second feature map is the first feature map, or the second feature map is a feature map obtained by processing the first feature map through the second neural network.

[0251] This process can refer to the aforementioned process 405, and will not be repeated here in the embodiments of this application.

[0252] 606. The decoder upsamples the reconstructed image to obtain the first image, which has the same resolution as the original image.

[0253] For example, upsampling can include bilinear interpolation upsampling, or upsampling by directly filling in the pixel values ​​of neighboring locations, or upsampling can be achieved through deconvolution operations of convolutional layers with a stride greater than 1.

[0254] It should be noted that if the feature map obtained by decoding the entire bitstream is reconstructed to generate a reconstructed image at the original resolution, the original resolution image will have a large resolution, and the generation process will be time-consuming, affecting the speed at which digital video applications can display the original resolution image, causing users to experience stuttering when browsing the original resolution image. In this embodiment, only a portion of the bitstream is decoded during the process of obtaining the reconstructed image, resulting in a smaller data volume for the reconstructed image and higher efficiency in obtaining the reconstructed image. Consequently, when obtaining the first image based on the reconstructed image, the efficiency of obtaining the first image is improved, thereby reducing the time spent in the process of obtaining the first image and improving the speed at which digital video applications can display the original resolution image.

[0255] As shown in process 602, the bitstream can be the bitstream of the initial feature map (i.e., M1 = M), or the bitstream can be a two-dimensional feature map corresponding to a portion of the channels in the initial feature map (i.e., M1 < M). When M1 = M, a high-quality image with the original resolution can be reconstructed, and subsequent processes 607 and 608 can be executed. When M1 < M, subsequent processes 607 and 608 do not need to be executed.

[0256] 607. The decoder decodes the bitstream corresponding to the two-dimensional feature maps of the M1 channels excluding the M2 channels to obtain the third feature map, which includes the two-dimensional feature maps of the M1-M2 channels.

[0257] The decoding process can be referred to in process 404, and will not be described in detail in this embodiment of the application.

[0258] 608. The decoder processes the first feature map and the third feature map according to the second neural network to obtain the second image. The resolution of the second image is the same as that of the original image.

[0259] The first feature map and the third feature map together form a two-dimensional feature map with M1 channels (i.e., the initial feature map). This second image is the final output of the second neural network. The structure and processing of the second neural network can be referred to the aforementioned process 405, and will not be repeated here in the embodiments of this application.

[0260] The second image is generated by reconstructing the first and third feature maps. The second image has a larger data volume than the first image, and its image quality is higher. Processes 607 and 608 can be executed simultaneously with process 606. That is, process 606 quickly generates a low-quality first image for initial display by the digital video application, while processes 607 and 608 reconstruct a high-quality second image. Since the reconstruction process of the second image is time-consuming, quickly generating the low-quality first image for display while simultaneously reconstructing the high-quality second image prevents stuttering when the digital video application displays the original resolution image, thus improving the display effect of the original resolution image.

[0261] In summary, the encoding and decoding method provided in this application involves an encoder extracting features from the original image to obtain an initial feature map with M channels. The first feature map is then encoded to obtain a bitstream, which is sent to the decoder. The bitstream corresponds to the two-dimensional feature data of M1 channels in the initial feature map, where M1 ≤ M. The decoder decodes the bitstream corresponding to the two-dimensional feature maps of M2 channels out of the M1 channels to obtain the first feature map. Then, a second feature map is reconstructed using a first neural network to obtain a reconstructed image. Both the second feature map and the reconstructed image have a target resolution, which is smaller than the resolution of the original image. The second feature map is either a second feature map itself or a feature map obtained by processing the second feature map using a second neural network. Since the resolution of the reconstructed image obtained after decoding and reconstruction is smaller than the resolution of the original image, there is no need to reconstruct the original image first during the acquisition of the reconstructed image; instead, the reconstructed image is obtained directly, improving the efficiency of acquiring the reconstructed image and thus increasing the speed at which digital video applications display thumbnails of the original image.

[0262] Furthermore, after obtaining the reconstructed image, it can be upsampled to obtain a first image with the same resolution as the original image. Simultaneously, the bitstreams corresponding to the two-dimensional feature maps of the M1 channels (excluding M2 channels) are decoded to obtain a third feature map. This third feature map includes the two-dimensional feature maps of the M1-M2 channels. The first and third feature maps are then processed using a second neural network to obtain a second image with the same resolution as the original image. The second image has a larger data volume than the first image and higher image quality. Since the reconstruction process of the second image is time-consuming, a low-quality first image is generated quickly for display while a high-quality second image is reconstructed. This prevents stuttering when displaying images at the original resolution in digital video applications, improving the display effect of the original resolution image.

[0263] The order of the methods provided in this application can be adjusted appropriately, and the processes can be added or removed as needed. For example, when the bitstream corresponds to a two-dimensional feature map of some channels in the initial feature map, processes 607 and 608 can be omitted. Any variations that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application, and this application does not limit them.

[0264] This application provides yet another encoding / decoding method; please refer to [link / reference]. Figure 10 , Figure 10This is a flowchart illustrating process 700 of another encoding / decoding method provided in an embodiment of this application. Process 700 can be executed by an electronic device (including an encoder and a decoder). Specifically, process 700 can be executed by the electronic device by invoking a neural network model. Process 700 is described as a series of operations; it should be understood that process 700 can be executed in various orders and / or occur simultaneously, and is not limited to... Figure 10 The execution order is shown. Process 700 may include the following processes:

[0265] 701. The encoder extracts features from the original image to obtain an initial feature map. The resolution of the initial feature map is smaller than that of the original image.

[0266] This process can refer to the aforementioned process 401, and will not be repeated here in the embodiments of this application.

[0267] 702. The encoder encodes the feature map of the original image to obtain the bitstream. The feature map of the original image is the initial feature map, or the feature map of the original image includes two-dimensional feature maps of some channels in the initial feature map.

[0268] This process can refer to the aforementioned process 402, and will not be repeated here in the embodiments of this application.

[0269] 703. The encoder sends the bit stream to the decoder.

[0270] This process can be referred to as the aforementioned process 403, and will not be repeated here in the embodiments of this application.

[0271] 704. The decoder decodes the bitstream to obtain the feature map of the original image.

[0272] This process can be referred to in the aforementioned process 404, and will not be repeated here in the embodiments of this application.

[0273] 705. The decoder reconstructs the feature map of the original image based on the neural network to obtain a reconstructed image at the target resolution, which is less than or equal to the resolution of the feature map of the original image.

[0274] The resolution of the reconstructed image output by the neural network may be the same as or different from the resolution of the input feature map.

[0275] The neural network can be one or more. When there is only one neural network, the decoder can directly input the feature map of the original image into this single neural network to obtain the reconstructed image output by that single neural network. In this case, the neural network can only output a reconstructed image at one resolution.

[0276] When there are multiple neural networks, the structures of any two neural networks can be the same or different, and this application does not limit this. The decoder can determine the target neural network among the multiple neural networks. Then, the feature map of the original image is input into the target neural network to obtain the reconstructed image output by the target neural network.

[0277] Each neural network includes at least one convolutional layer, which processes the feature map of the input raw image. Each neural network corresponds to an output resolution, which is the resolution of the feature map output by the last convolutional layer of any neural network after the feature map of the raw image is input into that neural network. The decoder can first determine the target resolution, and then identify the neural network with the input resolution of the target resolution as the target neural network. The method for determining the target resolution can refer to the aforementioned process 405, which will not be repeated here in the embodiments of this application.

[0278] For example, please refer to Figure 11 , Figure 11 This is a schematic diagram of a neural network structure provided in an embodiment of this application. Figure 11 Four neural networks, d1 to d4, are shown. d1 consists of two convolutional layers, both with a kernel size of 5x5 and a stride of 1. The first convolutional layer in d1 has N1 output channels, and the last convolutional layer has P output channels. d2 consists of three convolutional layers, all with a kernel size of 5x5. The first convolutional layer in d2 has a stride of 2 and N output channels; the second and last convolutional layers in d2 both have a stride of 1. The second convolutional layer in d2 has N1 output channels, and the last convolutional layer has P output channels. d3 consists of four convolutional layers. The first and second convolutional layers in d3 both have a stride of 2 and N output channels; the third and last convolutional layers in d3 both have a stride of 1. The third convolutional layer in d3 has N1 output channels, and the last convolutional layer has P output channels. d4 consists of five convolutional layers. The stride of the first, second, and third convolutional layers in d4 is 2, and the number of output channels is N. The stride of the fourth and last convolutional layers in d4 is 1. The number of output channels of the fourth convolutional layer in d4 is N1, and the number of output channels of the last convolutional layer is P.

[0279] Assuming the feature map of the original image has M channels and a resolution of W / 16×H / 16, the output resolutions of the four neural networks d1 to d4 are W / 16×H / 16, W / 8×H / 8, W / 4×H / 4, and W / 2×H / 2, respectively.

[0280] When the target neural network is d1, the feature map of the original image is input into d1. The first and second convolutional layers of d1 keep the resolution of the input feature map unchanged, and channel transformation is performed on the input feature map in sequence. Finally, a reconstructed image with P channels and a resolution of W / 16×H / 16 is output.

[0281] When the target neural network is d2, the feature map of the original image is input into d2. The first convolutional layer of d2 upsamples the resolution of the feature map by a factor of 2, outputting a feature map with N channels and a resolution of W / 8×H / 8. The second and third convolutional layers of d2 maintain the resolution of the input feature map and sequentially perform channel transformation on the input feature map, finally outputting a reconstructed image with P channels and a resolution of W / 8×H / 8.

[0282] When the target neural network is d3, the feature map of the original image is input into d3. The first and second convolutional layers of d3 sequentially upsample the resolution of the feature map of the original image by a factor of 2, outputting a feature map with N channels and a resolution of W / 4×H / 4. The third and fourth convolutional layers of d3 maintain the resolution of the input feature map and sequentially perform channel transformation on the input feature map, finally outputting a reconstructed image with P channels and a resolution of W / 4×H / 4.

[0283] When the target neural network is d4, the feature map of the original image is input into d4. The first, second, and third convolutional layers of d4 sequentially upsample the resolution of the feature map of the original image by a factor of 2, outputting a feature map with N channels and a resolution of W / 2 × H / 2. The fourth and fifth convolutional layers of d4 maintain the resolution of the input feature map and sequentially perform channel transformation on the input feature map, finally outputting a reconstructed image with P channels and a resolution of W / 2 × H / 2.

[0284] For example, please refer to Figure 12 , Figure 12 This is a schematic diagram of another neural network structure provided in an embodiment of this application. Figure 12Four neural networks, e1 to e4, are shown. e1 consists of two convolutional layers, both with a kernel size of 5x5 and a stride of 1. The first convolutional layer in e1 has N1 output channels, and the last convolutional layer has P output channels. e2 consists of one convolutional layer, with a kernel size of 5x5, a stride of 2, and P output channels. e3 consists of two convolutional layers, both with a stride of 2. The first convolutional layer in e3 has N output channels, and the second convolutional layer in e3 has P output channels. e4 consists of three convolutional layers, all with a stride of 2. The first and second convolutional layers in e4 have N output channels, and the third convolutional layer in e4 has P output channels.

[0285] Assuming the feature map of the original image is M×W / 16×H / 16, and the output resolutions of the four neural networks e1 to e4 are W / 16×H / 16, W / 8×H / 8, W / 4×H / 4, and W / 2×H / 2, respectively, then when the target neural networks are e1 to e4, the reconstructed images output are P×W / 16×H / 16, P×W / 8×H / 8, P×W / 4×H / 4, and P×W / 2×H / 2, respectively.

[0286] It should be noted that there can be one or more target resolutions, and correspondingly, there can also be one or more target neural networks. When there are multiple target neural networks, these multiple target neural networks can simultaneously output reconstructed images of the corresponding target resolutions. The reconstruction process of each target neural network can be referred to the foregoing description, and will not be repeated here in the embodiments of this application.

[0287] Figure 11 and Figure 12 The structure of the neural network is illustrated illustratively, and the embodiments of this application do not limit it. For example, in... Figure 11 or Figure 12 Based on this, any neural network can also include one or more convolutional layers, which can be the same or different.

[0288] Optionally, in this embodiment, before reconstructing the feature map of the original image or outputting the reconstructed image from the last convolutional layer of the neural network, channel reduction can be performed on the feature map of the original image and / or the intermediate feature map output from a certain convolutional layer to reduce the complexity of the subsequent reconstruction process and improve the efficiency of the reconstruction process. This process can refer to the aforementioned process 405, and will not be described in detail here.

[0289] Please refer to Figure 13 , Figure 13This is a schematic diagram illustrating another channel reduction process provided in an embodiment of this application. Figure 13 In Figure 11 Channel reduction is performed on the neural network shown, taking the channel reduction of the intermediate feature map as an example. That is, the number of output channels of the convolutional layer in each neural network is reduced. Figure 13 As shown, in d1, the number of channels in the feature map of the original image is reduced from M1 to T1; in d2, the number of channels in the intermediate feature map output from the first convolutional layer is reduced from N to T2; in d3, the number of channels in the intermediate feature map output from the second convolutional layer is reduced from N to T3; and in d4, the number of channels in the intermediate feature map output from the third convolutional layer is reduced from N to T4. T1 is less than M1, and T2, T3, and T4 are all less than N.

[0290] This process 505 can refer to the aforementioned process 405, and will not be described in detail here in the embodiments of this application.

[0291] In summary, the encoding / decoding method provided in this application involves the encoder extracting features from the original image to obtain a feature map, then encoding the feature map to obtain a bitstream and sending the bitstream to the decoder. The decoder decodes the bitstream to obtain the feature map of the original image, and then reconstructs the feature map to obtain a reconstructed image at a target resolution, where the target resolution is less than or equal to the resolution of the feature map of the original image. Since the resolution of the reconstructed image obtained after decoding and reconstruction is less than the resolution of the original image, the process of acquiring the reconstructed image does not require first reconstructing the original image; instead, the reconstructed image is obtained directly, improving the efficiency of acquiring the reconstructed image and thus increasing the speed at which digital video applications display thumbnails of the original image.

[0292] Furthermore, a reconstructed image can be obtained by reconstructing the feature map of the original image using a neural network. The neural network comprises multiple sub-networks, each corresponding to an output resolution. The target resolution can be determined based on the desired thumbnail resolution, and the neural sub-network whose output resolution matches the target resolution is identified as the target neural sub-network. The reconstructed image is then obtained based on the target neural sub-network. This process of acquiring the reconstructed image allows for the selection of a matching neural sub-network based on the determined target resolution and enables the acquisition of reconstructed images with multiple target resolutions, thus improving the flexibility of image acquisition.

[0293] The order of the methods provided in the embodiments of this application can be adjusted appropriately, and the process can also be added or removed as appropriate. Any variations that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application, and the embodiments of this application do not limit this.

[0294] Alternatively, the embodiments of this application may also include the following five embodiments:

[0295] Example 1

[0296] The main flowchart of this embodiment is as follows: Figure 14 As shown.

[0297] Figure 14 This is a schematic diagram of an encoding and decoding process provided in an embodiment of this application. At the encoding end, the original image x is input to the feature extraction module, which outputs a feature map y. This feature map y is then input to the quantization module to obtain a quantized feature map y0. The entropy encoding module performs entropy encoding on the feature map y0 to obtain a compressed bitstream. The specific technical solution at the encoding end is described below:

[0298] Step 1: Obtain the 3D feature map y.

[0299] The original image x is input to the feature extraction module, which outputs a 3D feature map y. The feature extraction module uses a neural network-based method. For example... Figure 15 As shown, Figure 15 This diagram illustrates the network structure of a feature extraction module provided in this application embodiment. It mainly comprises four convolutional layers and three GDN layers interleaved and cascaded. Each convolutional layer has a 5x5 kernel size, and the last convolutional layer outputs a feature map with M channels. Each convolutional layer has a stride of 2, meaning that the width and height of the input image or feature map are downsampled by a factor of 2. Therefore, when an original image of size W×H is input into this feature extraction network, the output is a three-dimensional feature map y of size M×W / 16×H / 16. This can be understood as feature map y comprising a two-dimensional feature map with M channels and a resolution of W / 16×H / 16.

[0300] Taking the first convolutional layer (conv Nx5x5 / 2↓) as an example, it means that the convolutional kernel is 5x5, N represents the number of channels of the output feature map of the first convolutional kernel, and the number 2 represents the stride of 2, which can be understood as performing a downsampling operation of 2 times on both the width and height of the input image. Therefore, the size of the output feature map of the first convolutional layer is N×W / 2×H / 2.

[0301] Similarly, if the second convolutional layer has N channels and a stride of 2, then the size of the output feature map is N×W / 4×H / 4. After execution... Figure 15 After the four convolutional operations shown, the size of the output feature map y is M×W / 16×H / 16.

[0302] It should be noted that the structure of the feature extraction network is not specifically limited in this step, the number of convolutional layers is not limited, and whether or not GDN is included is not limited; there are no other activation functions; the size of the convolutional kernel is not limited, and it can be 3×3, 5×5, 7×7 or other sizes, which are determined by the specific network design of the encoder and decoder. This application does not impose any limitations. In this application, 5×5 is used as an example for description.

[0303] Step 2: Input the feature map y into the quantization module to obtain the quantized feature map y0.

[0304] The feature quantization module quantizes each feature value (or feature element) in the feature map y, rounding the floating-point feature values ​​to obtain integer feature values, resulting in the quantized feature map y0. Feature map y0 is a three-dimensional integer feature map of size M×W / 16×H / 16.

[0305] It should be noted that this application does not limit the specific quantization method; it could also involve truncating the feature values ​​of floating-point numbers to obtain integer feature values. Alternatively, it could involve performing quantization operations based on a preset quantization step size to obtain quantized feature values.

[0306] Step 3: The entropy coding module performs entropy coding on the feature map y0 to obtain the compressed bitstream.

[0307] When entropy encoding each feature element in feature map y0, one of the following methods can be used, without limitation.

[0308] Method 1: Entropy Encoding Method Based on Probabilistic Model: When entropy encoding each feature element in feature map y0, a probability distribution model is first used for modeling. Then, the context information of the current feature element is input into the probability estimation network to obtain model parameters. The model parameters are then substituted into the probability distribution model to obtain the probability distribution of the current feature element. Entropy encoding is performed based on the probability distribution. The probability distribution model can be: Gaussian single model (GSM), asymmetric Gaussian model, Gaussian mixture model (GMM), or Laplace distribution model. The probability estimation network can be based on deep learning networks, such as recurrent neural networks (RNN) and convolutional neural networks (PixelCNN), etc., without limitation.

[0309] Method 2: Hyperprior Entropy Coding Method

[0310] The feature map y is processed by the edge information extraction module, which outputs edge information z. The edge information z is then quantized to obtain... Will Entropy encoding is performed and written to the bitstream; the encoding end then performs entropy decoding to obtain the decoded bitstream. The input probability estimation module outputs the probability distribution of each feature element in the feature map y0. Entropy encoding followed by decoding ensures synchronization between encoding and decoding. The entropy encoding module entropy-encodes each feature element in the feature map y0 according to the probability distribution of each feature element, resulting in a compressed bitstream. This includes side information. It is also a type of feature information, represented as a three-dimensional feature map, which contains fewer feature elements than the feature map y.

[0311] like Figure 14 The flowchart shown illustrates that at the decoding end, feature map y0 is parsed from the compressed bitstream. y0 is then input into the reconstruction module to obtain a reconstructed image. This reconstructed image includes a low-resolution reconstructed image and may also include a reconstructed image at the original resolution. In end-to-end image encoding / decoding schemes, the structure of the decoding and reconstruction module generally corresponds to that of the encoding feature extraction module. Specifically, it corresponds to the feature extraction module at the encoding end. Figure 15 The network structure shown is as follows: Figure 16 As shown in Figure 16, it is a schematic diagram of the network structure of a reconstruction module provided in an embodiment of this application. The original resolution reconstruction network can output the reconstructed image at the original resolution, and the low resolution reconstruction network branch can have one or more different branches, corresponding to output low resolution reconstructed images at different resolutions.

[0312] The main innovation of this application lies in the reconstruction module. The technical solution for the decoding end is described as follows:

[0313] Step 1: Perform entropy decoding on the compressed bitstream to obtain a three-dimensional feature map y0, which includes a two-dimensional feature map with a resolution of W / k×H / k for M channels.

[0314] The value of k is determined by the network structure of the feature extraction network at the encoding end. Specifically, the value of k is related to the number of convolutional layers in the feature extraction network and the stride of each convolutional layer. Corresponding to the encoding end, k = 16 in this embodiment, and this embodiment is described using k = 16.

[0315] The entropy decoding method corresponds to the encoding end. When entropy encoding each feature element in the feature map y0, one of the following methods can be used, which are briefly described below:

[0316] Method 1: Input the context information of the current feature element into the same probability estimation network as the encoder to obtain the model parameters. Substitute the model parameters into the probability distribution model to obtain the probability distribution of the current feature element. Perform entropy decoding based on the probability distribution to obtain the value of the feature element.

[0317] Method 2: First, decode to obtain the edge information. Then the edge information The input is the same probability estimation module as the encoder, and the output is the probability distribution of the current feature element. Arithmetic decoding is then performed on the current feature element based on its probability distribution to obtain its value.

[0318] Step 2: Obtain the first resolution, which is smaller than the resolution WxH of the original image.

[0319] The first resolution can be obtained using one of the following methods: if the first resolution is specified externally by the decoder, the decoder selects the corresponding reconstruction branch to perform image reconstruction operation based on the specified first resolution. In this case, the decoder may not need to include the step of obtaining the first resolution and can directly execute step 3.

[0320] It should be noted that the first resolution can be multiple resolutions. If multiple resolutions are specified externally for the decoder, the decoder can output the reconstructed images of the corresponding multiple resolutions simultaneously.

[0321] Method 1: Specifying the First Resolution Externally to the Decoder: This requires the application layer outside the decoder to understand the decoder's capabilities, i.e., which resolutions the decoder can output. The information regarding which resolutions a bitstream can output (candidate resolutions) can be carried in the bitstream via SEI (Search Engine Information) or directly in the application layer (file format). The player reads all candidate resolutions, determines the first resolution based on the target resolution, and then informs the decoder of the first resolution information. In this embodiment, candidate resolutions include W / 16×H / 16, W / 8×H / 8, W / 4×H / 4, and W / 2×H / 2. The application layer includes video players, photo albums, web pages, etc.

[0322] Method 2: Determined internally by the decoder: The application layer informs the decoder of the target resolution, and the decoder matches the candidate resolution closest to the target resolution as the first resolution. The candidate resolution is determined internally by the decoder structure. In this embodiment, the candidate resolutions include W / 16×H / 16, W / 8×H / 8, W / 4×H / 4, and W / 2×H / 2.

[0323] Method 3: Determine the first resolution based on decoder complexity / power consumption constraints. Specifically, the complexity index can be calculated for each reconstruction branch of the decoder. This complexity index is known at the upper layer (application layer). The application layer selects the corresponding resolution based on the complexity index.

[0324] Method 4: The target resolution is specified by the encoder and transmitted to the decoder, which directly decodes the bitstream to obtain the target resolution. For example, the encoder directly transmits a first identifier to indicate the target resolution, and the decoder parses the value of the first identifier to obtain the target resolution. Based on the target resolution, the decoder matches the closest candidate resolution from the candidate resolutions as the first resolution.

[0325] Step 3: Select the corresponding reconstruction network branch according to the first resolution, input the feature map y0 into the reconstruction module, and obtain the reconstructed image at the first resolution.

[0326] In this embodiment, the reconstruction module can output multiple low-resolution reconstructed images simultaneously, or it can output only one low-resolution reconstructed image. The specific output result depends on the needs of the application layer. If the first resolution specified by the application layer includes multiple resolutions, then this step can output multiple low-resolution reconstructed images. This embodiment describes the output of one low-resolution reconstructed image as an example.

[0327] Specifically, such as Figure 16 As shown, corresponding to the encoding end, feature map y0 is input into the original resolution reconstruction network of the reconstruction module, and the output is a reconstructed image at the original resolution. Furthermore, different branches can be drawn from the structure of the original resolution reconstruction network, and different convolutional layers can be added without changing the original resolution reconstruction network structure to output different low-resolution reconstructed images. Combined with... Figure 16 The specific description is as follows: The number of channels in the first, second, and third convolutional layers is N, and the stride is 2.

[0328] If the first resolution is W / 16×H / 16, the feature map y0 is directly input into the reconstruction network of branch one, resulting in a reconstructed image A1 of size P×W / 16×H / 16. The reconstruction network of branch one includes a first deconvolution layer with P output channels and a stride of 1. Branch one may also include a second deconvolution network, which includes one or more deconvolution layers with a stride of 1 and N1 output channels. The N1 values ​​for multiple deconvolution layers can be the same or different. P indicates the number of channels in the final output reconstructed image A1, typically taking the value 3 or 1. If a three-channel color image is required, P is set to 3; if a grayscale image is required, P is set to 1. The value of N1 is not limited.

[0329] If the first resolution is W / 8×H / 8, the feature map y0 is input into the reconstruction network at the original resolution, and the first deconvolution operation is performed to obtain a feature map Q2 of size N×W / 8×H / 8. Feature map Q2 is then input into the reconstruction network of branch two to obtain a reconstructed image A2 of size P×W / 8×H / 8. The network structure corresponding to branch two can be the same as or different from that of branch one.

[0330] If the first resolution is W / 4×H / 4, the feature map y0 is input into the reconstruction network at the original resolution. The first and second deconvolution operations are performed to obtain a feature map Q3 of size N×W / 4×H / 4. Feature map Q3 is then input into the reconstruction network of branch three to obtain a reconstructed image A3 of size P×W / 4×H / 4. The network structure corresponding to branch three can be the same as or different from that of branch one or branch two.

[0331] If the first resolution is W / 2 × H / 2, the feature map y0 is input into the reconstruction network at the original resolution. The first, second, and third deconvolution layers are then executed to obtain a feature map Q4 of size N × W / 2 × H / 2. Feature map Q4 is then input into the reconstruction network of branch four to obtain a reconstructed image A4 of size P × W / 2 × H / 2. The network structure corresponding to branch four can be the same as, or different from, branch one, branch two, or branch three.

[0332] Step 4: Process the reconstructed image at the first resolution to obtain the reconstructed image at the target resolution.

[0333] The first resolution is compared with the target resolution. If they are the same, no further processing is needed, and the reconstructed image at the first resolution is directly output and displayed as the reconstructed image at the target resolution.

[0334] If they are different, corresponding scaling, upsampling, or downsampling operations should be performed to obtain a reconstructed image with the same target resolution as the target resolution before outputting it.

[0335] It should be noted that this step can be performed by the player's image scaling module outside the decoder, so the decoding solution may not need to include this step.

[0336] Example 2

[0337] The encoding scheme is the same as in Example 1, and will not be repeated here. Figure 17 As shown, Figure 17 This is a flowchart illustrating a decoding end provided in an embodiment of this application. The specific steps are as follows: only step 3 differs from that in Embodiment 1, and only step 3 will be described in detail here.

[0338] Step 3: Select the corresponding reconstruction network branch according to the first resolution, input the feature map y0 into the reconstruction module, and obtain the reconstructed image at the first resolution.

[0339] like Figure 18 As shown, Figure 18 This is a schematic diagram of the structure of a reconstruction network provided in an embodiment of this application. If the first resolution is W / 16×H / 16, the feature map y0 is input into the first reconstruction network (e.g., Figure 18 A) This yields a target image A1 with a first resolution. The first reconstruction network includes a first deconvolutional network with P output channels and a stride of 1. The first reconstruction network may also include one or more second deconvolutional networks with a stride of 1. The output channels N1 of the multiple second deconvolutional networks may be the same or different.

[0340] If the first resolution is W / 8×H / 8, the feature map y0 is input into the second reconstruction network (e.g., Figure 18 As shown in Figure B, a target image A2 with a first resolution is obtained. The second reconstruction network includes a third deconvolutional network, which has N channels and a stride of 2. The second reconstruction network also includes a first deconvolutional network and a second deconvolutional network.

[0341] If the first resolution is W / 4×H / 4, the feature map y0 is input into the third reconstruction network (e.g., Figure 18 As shown in Figure C, a target image A3 with a first resolution is obtained. The third reconstruction network includes a third deconvolutional network with N channels and a stride of 2. The third reconstruction network also includes the structure of the second reconstruction network.

[0342] If the first resolution is W / 2×H / 2, the feature map y0 is input into the fourth reconstruction network (e.g., Figure 18 As shown in Figure D), a target image A4 with a first resolution is obtained. The fourth reconstruction network includes a fourth deconvolutional network with N channels and a stride of 2. The fourth reconstruction network also incorporates the structure of the third reconstruction network.

[0343] As another implementation method, such as Figure 19 As shown, Figure 19 This is a schematic diagram of another reconstruction network structure provided in an embodiment of this application. The first reconstruction network, second reconstruction network, third reconstruction network, and fourth reconstruction network can also use... Figure 19 The network structure shown is used for implementation, and no specific limitations are imposed here.

[0344] Step 2.3: Process the reconstructed image at the first resolution to obtain the image at the target resolution.

[0345] Example 3

[0346] The difference between this embodiment and Embodiment 1 is that the number of channels in the feature map is reduced before the input feature map is reconstructed, or the number of channels in the feature map obtained after performing one or more deconvolution operations during the reconstruction process is reduced, in order to reduce the decoding complexity.

[0347] The decoding method is as follows:

[0348] Based on Embodiment 1, this embodiment differs from Embodiment 1 only in step 3, and will be described in detail here.

[0349] In one possible way, such as Figure 20 As shown, Figure 20 This is a schematic diagram of a channel reduction process provided in an embodiment of this application. Before inputting feature map y0 into the reconstruction network, the M channels of feature map y0 are reduced to S, and only the S channels of feature map y0 are input, where S is less than the number of channels M of feature map y0. In this implementation, multiple different low-resolution reconstructed images cannot be output simultaneously. The reconstruction network can be implemented using the methods described in Embodiments 1 and 2.

[0350] One possible implementation is described in conjunction with Embodiment Two. For example... Figure 21 As shown, Figure 21 This is a schematic diagram of another channel reduction process provided in an embodiment of this application. S1, S2, S3, and S4 are all less than the number of channels M of feature map y0. The description is based on the example of inputting feature map y0 into the second reconstruction network to obtain target image A2 when the first resolution is W / 8 × H / 8. In this implementation, multiple different low-resolution reconstructed images can be output simultaneously. When the first resolution is W / 8 × H / 8, feature map y0 is input into the second reconstruction network, passes through a deconvolution layer with N channels and a stride of 2, and outputs feature map y1 with N channels. The number of channels in y1 is reduced to obtain feature map S2, which is then input into the next level of the deconvolution network until the target image A2 is output. Specifically, the reduction method involves randomly extracting feature map data from S2 channels, or extracting the feature map data from the first S2 channels out of N channels; this is not limited here.

[0351] In another possible implementation, as described in conjunction with Embodiment 1: (e.g.) Figure 22 As shown, Figure 22 This is a schematic diagram of another channel reduction process provided in an embodiment of this application. Figure 22To reduce the number of feature map channels output by the convolutional layer in the reconstruction network, S1, S2, S3, and S4 are all smaller than the channel number M of the feature map y0. The description is given with the first resolution being W / 8 × H / 8. In this implementation, multiple different low-resolution reconstructed images can be output simultaneously. When the first resolution is W / 8×H / 8, the feature map y0 is input into the original resolution reconstruction network, and the first layer convolution operation is performed to obtain the feature map Q2 with the size of N×W / 8×H / 8. The number of channels of the feature map Q2 is reduced to obtain a feature map with S2 channels, which is input into the reconstruction network of the second branch, so as to obtain the reconstructed image A2 with the size of P×W / 8×H / 8. The specific reduction method is to randomly extract the feature map data of S2 channels, or extract the feature map data of the first S2 channels among the N channels, which is not limited herein.

[0352] Fourth Embodiment

[0353] Step 1: Decode the main bitstream feature map to obtain the feature maps of the first M channels (M<N) among all N channels (resolution W / K×H / K, K=16).

[0354] Step 2: Input the feature maps of M channels into the reconstruction network 1 to obtain a low-resolution image A1 (the resolution of A1 is smaller than the resolution W×H of the original image).

[0355] Method 1: Perform an image upsampling operation on the low-resolution reconstructed image A1 to obtain a target resolution image B1.

[0356] Herein, the method of inputting the feature maps of M channels into the reconstruction network 1 to obtain the low-resolution image A1 is the same as that in the first embodiment, and the low resolution corresponds to the first resolution in the first embodiment. The setting method of the target resolution is not limited herein, and for example, it may be specified by the application side.

[0357] Herein, the image upsampling operation is, for example, bilinear interpolation upsampling, or directly filling the pixel values of adjacent positions for upsampling.

[0358] Method 2: Perform an image upsampling operation on the low-resolution reconstructed image A1 to obtain an original resolution image B3.

[0359] The upsampling operation includes: the image upsampling operation is, for example, bilinear interpolation upsampling, or directly filling the pixel values of adjacent positions for upsampling. Alternatively, it is implemented by a deconvolution layer with a stride greater than 1.

[0360] The resolution of B3 is W×H.

[0361] Step 3: Decode the main bitstream feature map to obtain the feature maps of the remaining N-M channels, and input the feature maps of all N channels into the reconstruction network 2 to obtain the original resolution image B2.

[0362] Beneficial effects: Steps 1 and 2 quickly generate a low-quality image by using less entropy decoding and image reconstruction latency; Step 3 can be executed in parallel with Step 2 to obtain a high-quality (standard-compliant) image with greater latency.

[0363] Compared to the first two embodiments,

[0364] 1) Low-quality image reconstruction can begin after partially decoding the Channel feature map.

[0365] 2) The reconstructed network can contain deconvolutional layers with a stride greater than 1 to achieve the effect of upsampling.

[0366] 3) Contains only 2 rebuild branches.

[0367] Example 5:

[0368] In another possible implementation, the decoding of the entire scheme in Embodiments 1 to 4 can be simplified to the following two steps:

[0369] Step 1: Perform entropy decoding on the compressed bitstream to obtain a three-dimensional feature map y0, which includes a two-dimensional feature map with a resolution of W / k×H / k for M channels.

[0370] Step 2: Input the 3D feature map y0 into the reconstruction network to obtain a reconstructed image at the first resolution. The first resolution is smaller than the resolution W×H of the original image.

[0371] It should be noted that the processes in the foregoing embodiments can be combined arbitrarily. For example, process 605 can be: the decoder reconstructs the third feature map of the first resolution based on the second neural network to obtain a reconstructed image of the second resolution. The first and second resolutions are different from the first resolution, and the second resolution is smaller than the resolution of the original image. The third feature map includes the second feature map and / or the feature map obtained by processing the second feature map through the second neural network. This application does not limit this aspect.

[0372] The above primarily describes the encoding and decoding method provided in this application from the perspective of device-to-device interaction. It is understood that each device, in order to achieve the above functions, includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0373] This application embodiment can divide each device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0374] Figure 23 This is a block diagram of a decoding device provided in an embodiment of this application. The decoding device 800 can be applied to an electronic device (e.g., a decoder), and can be a chip or functional module in the electronic device. When the functional modules are divided according to their respective functions, the decoding device 800 includes a processing module 801 and a reconstruction module 802, and also includes a transceiver module 803 (not shown in the figure). The transceiver module 803 may include a sending module and / or a receiving module, respectively used to perform... Figure 4 , 8 9 or Figure 10 The illustrated embodiment shows the sending and receiving operations performed by the decoder. For example, the decoding device includes:

[0375] The processing module 801 is used to decode the bitstream to obtain a first feature map, wherein the resolution of the first feature map is smaller than the resolution of the original image;

[0376] The reconstruction module 802 is used to reconstruct the second feature map according to the first neural network to obtain a reconstructed image. The resolution of the second feature map and the reconstructed image are both target resolutions, and the target resolution is smaller than the resolution of the original image. The second feature map is the first feature map, or the second feature map is a feature map obtained by processing the first feature map through the second neural network.

[0377] In conjunction with the above solution, when the second feature map is a feature map obtained by processing the first feature map through the second neural network, the second neural network includes one or more output nodes and a plurality of convolution layers, the output nodes are located between the output end of the first convolution layer and the input end of the last convolution layer, the output nodes are connected to the input end of the first neural network, the reconstruction module 802 is further configured to input the first feature map into the second neural network to obtain the second feature map output by the target output node of the second neural network, where the target output node belongs to the one or more output nodes; the reconstruction module 802 is specifically configured to input the second feature map output by the target output node into the first neural network to obtain the reconstructed image output by the first neural network.

[0378] In conjunction with the above solution, when the number of the first neural networks is multiple, the reconstruction module 802 is specifically configured to input the second feature map into the first neural network connected to the target output node, to obtain the reconstructed image output by the first neural network connected to the target output node.

[0379] In conjunction with the above solution, when the number of the target resolutions is multiple and the number of the target output nodes is multiple, the reconstruction module 802 is specifically configured to input the first feature map into the second neural network to obtain second feature maps of multiple resolutions output by the plurality of target output nodes of the second neural network; and input the second feature map output by each of the target output nodes into the first neural network connected to the target output node, to obtain reconstructed images of multiple resolutions output by the first neural networks respectively connected to the plurality of target output nodes.

[0380] In conjunction with the above solution, when the second neural network includes a plurality of output nodes, each of the output nodes corresponds to one output resolution, the reconstruction module 802 is further configured to determine the target resolution; and determine an output node whose output resolution is the target resolution as the target output node.

[0381] In conjunction with the above solution, the first neural network includes at least one convolution layer, and the convolution stride of the at least one convolution layer is 1.

[0382] In conjunction with the above solution, the code stream corresponds to two-dimensional feature maps of M1 channels, and the processing module 801 is specifically configured to decode the code stream corresponding to the two-dimensional feature maps of M2 channels among the M1 channels to obtain the first feature map, where M2<M1, and the first feature map includes the two-dimensional feature maps of the M2 channels.

[0383] In conjunction with the above scheme, the reconstruction module 802 is further configured to perform upsampling processing on the reconstructed image to obtain a first image, wherein the resolution of the first image is the same as the resolution of the original image.

[0384] In conjunction with the above scheme, the bitstream is the bitstream of the initial feature map, which is obtained by feature extraction from the original image. The processing module 801 is further used to decode the bitstream corresponding to the two-dimensional feature maps of the channels other than the M2 channels in the M1 channels to obtain a third feature map, which includes the two-dimensional feature maps of the M1-M2 channels. The reconstruction module 802 is further used to process the first feature map and the third feature map according to the second neural network to obtain a second image, the resolution of which is the same as the resolution of the original image.

[0385] In conjunction with the above scheme, the second feature map includes a two-dimensional feature map with multiple channels. Before reconstructing the second feature map according to the first neural network to obtain the reconstructed image, the processing module 801 is further used to perform channel reduction processing on the second feature map. Specifically, the processing module 801 is used to reconstruct the second feature map after channel reduction processing according to the first neural network to obtain the reconstructed image.

[0386] In conjunction with the above scheme, the first feature map includes two-dimensional feature maps of multiple channels, and the reconstruction module 802 is further used to perform channel reduction processing on the first feature map; the second feature map is the first feature map after the channel reduction processing, or the second feature map is a feature map obtained by processing the first feature map after the channel reduction processing through the second neural network.

[0387] The structure of the encoding device can be referenced. Figure 23 The structure of the decoding device shown is illustrated. The encoding device can be applied to electronic devices (such as encoders), and can be a chip or functional module in the electronic device. When functional modules are divided according to their respective functions, the encoding device includes a processing module and an encoding module. It may further include a transceiver module, which includes a transmitting module and / or a receiving module, respectively used to perform... Figure 4 , 8 9 or Figure 10 The illustrated embodiment shows the transmission and reception operations performed by the encoder. For example, the encoding device includes:

[0388] The processing module is used to extract features from the original image to obtain an initial feature map. The initial feature map includes two-dimensional feature maps of multiple channels, and the resolution of the initial feature map is smaller than the resolution of the original image.

[0389] The encoding module is used to encode the feature map to be encoded to obtain a bitstream, wherein the feature map to be encoded is the initial feature map, or a two-dimensional feature map including some channels of the initial feature map;

[0390] The transceiver module is used to send the bit stream to the decoding device.

[0391] Figure 24 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 900 can be a chip or functional module in an encoder; it can also be a chip or functional module in a decoder. Figure 24 As shown, the electronic device 900 includes a processor 901, a transceiver 902, and a communication line 903.

[0392] The system includes a processor 901, a transceiver 902, and a communication line 903. The processor 901 is used to perform tasks such as... Figure 4 , 8 In the method embodiments shown in 9 and 10, any step performed by the encoder or decoder, and when performing data transmission such as acquisition, may optionally invoke the transceiver 902 and the communication line 903 to complete the corresponding operation.

[0393] Furthermore, the electronic device 900 may also include a memory 904. The processor 901, memory 904, and transceiver 902 can be connected via a communication line 903.

[0394] The processor 901 can be a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 901 can also be other devices with processing capabilities, such as circuits, devices, or software modules, without limitation.

[0395] Transceiver 902 is used to communicate with other devices. Transceiver 902 can be a module, circuit, transceiver, or any device capable of enabling communication.

[0396] The transceiver 902 is mainly used for transmitting and receiving data such as images or bitstreams. It can include a transmitter and a receiver to send and receive data respectively. Operations other than data transmission and reception are implemented by the processor, such as data processing and calculation.

[0397] Communication line 903 is used to transmit information between the various components included in electronic device 900.

[0398] In one design, the processor can be viewed as a logic circuit, and the transceiver as an interface circuit.

[0399] Memory 904 is used to store instructions. These instructions can be computer programs.

[0400] The memory 904 can be volatile memory or non-volatile memory, or a combination of both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM). Memory 904 can also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, digital universal discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices. It should be noted that the memory in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0401] It should be noted that the memory 904 can exist independently of the processor 901, or it can be integrated with the processor 901. The memory 904 can be used to store instructions, program code, or some data, etc. The memory 904 can be located inside or outside the electronic device 900, without limitation. The processor 901 is used to execute the instructions stored in the memory 904 to implement the methods provided in the above embodiments of this application.

[0402] In one example, processor 901 may include one or more CPUs, for example Figure 24 CPU0 and CPU1 in the CPU.

[0403] As an optional implementation, the encoding device 900 includes multiple processors, for example, besides Figure 24 In addition to processor 901, it may also include processor 907.

[0404] As an optional implementation, the electronic device 900 also includes an output device 905 and an input device 906. For example, the input device 906 is a device such as a keyboard, mouse, microphone, or joystick, and the output device 905 is a device such as a display screen or speaker.

[0405] The processor and transceiver described in this application can be implemented on integrated circuits (ICs), analog ICs, radio frequency integrated circuits, mixed-signal ICs, application-specific integrated circuits (ASICs), printed circuit boards (PCBs), electronic devices, etc. The processor and transceiver can also be manufactured using various IC process technologies, such as complementary metal oxide semiconductors (CMOS), n-type metal-oxide-semiconductor (NMOS), p-type metal oxide semiconductors (PMOS), bipolar junction transistors (BJTs), bipolar CMOS (BiCMOS), silicon germanium (SiGe), gallium arsenide (GaAs), etc.

[0406] It should be noted that electronic device 900 can be any type of handheld or fixed device, such as laptops or tablets, mobile phones, smartphones, tablets or tablet computers, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, video game consoles, video streaming devices (e.g., content service servers or content distribution servers), broadcast receiving devices, broadcast transmitting devices, and monitoring devices, etc., and may or may not use an operating system of any type. Electronic device 900 can also be a device in a cloud computing scenario, such as a virtual machine in a cloud computing scenario. In some cases, electronic device 900 may be equipped with components for wireless communication. Therefore, electronic device 900 can be a wireless communication device. Figure 15 Equipment with a similar structure. Furthermore... Figure 24 The structural composition shown does not constitute a limitation on the electronic device 900, except... Figure 24 In addition to the components shown, the electronic device 900 may include more than Figure 24 This may indicate more or fewer components, or a combination of certain components, or a different arrangement of components.

[0407] Electronic device 900 can also be a chip system, which can be composed of chips or include chips and other discrete devices.

[0408] Furthermore, the actions, terms, etc., involved in the various embodiments of this application can be referenced interchangeably without limitation. The message names or parameter names in the messages exchanged between the various devices in the embodiments of this application are merely examples, and other names may be used in specific implementations without limitation.

[0409] As another feasible approach Figure 23 The transceiver module 801 in the middle can be made by Figure 24 The transceiver 902 in the original text can be replaced by a processor 907, which can integrate the functions of the transceiver module 801; the processing module 802 can be replaced by a processor 907, which can integrate the functions of the processing module 802. Furthermore, Figure 23 The decoding device 800 shown may further include a memory (not shown in the figure). When the transceiver module 801 is replaced by a transceiver and the processing module 802 is replaced by a processor, the decoding device 800 or encoding device involved in the embodiments of this application can be... Figure 24 The electronic device shown is 900.

[0410] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to execute any of the methods described in the embodiments of this application.

[0411] This application also provides a chip, which includes at least one processor, a memory, and an interface circuit. The memory, the transceiver, and the at least one processor are interconnected via circuits. The at least one memory stores a computer program. When the computer program is executed by the processor, any of the methods described in this application is implemented.

[0412] This application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be executed by a computer or an information processing device, using computer programs or instructions to control related hardware. The computer program or set of instructions can be stored in the aforementioned computer-readable storage medium. When executed, the computer program or set of instructions can include the processes described in the above method embodiments. The computer-readable storage medium can be an internal storage unit of the electronic device (including encoders and / or decoders) of any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The aforementioned computer-readable storage medium can also be an external storage device of the aforementioned electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the aforementioned electronic device. Further, the aforementioned computer-readable storage medium can include both internal storage units and external storage devices of the aforementioned electronic device. The aforementioned computer-readable storage medium is used to store the aforementioned computer program or instructions, as well as other programs and data required by the aforementioned electronic device. The aforementioned computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0413] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0414] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0415] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0416] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0417] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0418] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0419] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A decoding method, characterized in that, The method includes: The bitstream is decoded to obtain a first feature map, the resolution of which is smaller than that of the original image; Reconstructing a reconstructed image from a second feature map using a first neural network, wherein both the second feature map and the reconstructed image have a target resolution smaller than the resolution of the original image, wherein the second feature map is either the first feature map or a feature map obtained by processing the first feature map using a second neural network. When the second feature map is a feature map obtained by processing the first feature map using the second neural network, the second neural network includes one or more output nodes connected to the input of the first neural network. The reconstructing of the reconstructed image from the second feature map using the first neural network includes: The second feature map output by the target output node is input into the first neural network to obtain the reconstructed image output by the first neural network, wherein the target output node belongs to one or more output nodes.

2. The method according to claim 1, characterized in that, The second neural network further includes multiple convolutional layers, and the output node is located between the output of the first convolutional layer and the input of the last convolutional layer. The method further includes: The first feature map is input into the second neural network to obtain the second feature map output by the target output node of the second neural network.

3. The method according to claim 2, characterized in that, When there are multiple first neural networks, the step of inputting the second feature map output by the target output node into the first neural network to obtain the reconstructed image output by the first neural network includes: The second feature map is input into the first neural network connected to the target output node to obtain the reconstructed image output by the first neural network connected to the target output node.

4. The method according to claim 3, characterized in that, When there are multiple target resolutions and multiple target output nodes, the step of inputting the first feature map into the second neural network to obtain the second feature map output by the target output node of the second neural network includes: The first feature map is input into the second neural network to obtain second feature maps of multiple resolutions output by multiple target output nodes of the second neural network; The step of inputting the second feature map into the first neural network connected to the target output node to obtain the reconstructed image output by the first neural network connected to the target output node includes: The second feature map output by each of the target output nodes is input into the first neural network connected to the target output node to obtain reconstructed images of multiple resolutions output by the first neural networks connected to the multiple target output nodes respectively.

5. The method according to any one of claims 2 to 4, characterized in that, When the second neural network includes multiple output nodes, each output node corresponding to an output resolution, the method further includes: Determine the target resolution; The output node with an output resolution equal to the target resolution is determined as the target output node.

6. The method according to any one of claims 1 to 4, characterized in that, The first neural network includes at least one convolutional layer, wherein the convolutional stride of the at least one convolutional layer is 1.

7. The method according to any one of claims 1 to 4, characterized in that, The code stream corresponds to two-dimensional feature maps of M1 channels, and the decoding the code stream to obtain a first feature map includes: Decode the code stream corresponding to the two-dimensional feature maps of M2 channels among the M1 channels to obtain the first feature map, where M2<M1, and the first feature map includes the two-dimensional feature maps of the M2 channels.

8. The method according to claim 7, characterized in that, The method further includes: Perform upsampling processing on the reconstructed image to obtain a first image, where the resolution of the first image is the same as the resolution of the original image.

9. The method according to claim 8, characterized in that, The code stream is a code stream of an initial feature map, the initial feature map is obtained by performing feature extraction on the original image, and the method further includes: Decode the code stream corresponding to the two-dimensional feature maps of channels other than the M2 channels among the M1 channels to obtain a third feature map, where the third feature map includes two-dimensional feature maps of M1-M2 channels; Process the first feature map and the third feature map according to a second neural network to obtain a second image, where the resolution of the second image is the same as the resolution of the original image.

10. The method according to any one of claims 1 to 4, 8, and 9, characterized in that, The second feature map includes two-dimensional feature maps of a plurality of channels, and before reconstructing the second feature map according to a first neural network to obtain a reconstructed image, the method further includes: Perform channel reduction processing on the second feature map; The reconstructing the second feature map according to a first neural network to obtain a reconstructed image includes: Reconstruct the second feature map after the channel reduction processing according to the first neural network to obtain the reconstructed image.

11. The method according to any one of claims 1 to 4, 8, and 9, characterized in that, The first feature map includes two-dimensional feature maps of a plurality of channels, and the method further includes: Perform channel reduction processing on the first feature map; The second feature map is the first feature map after the channel reduction processing, or the second feature map is a feature map obtained by processing the first feature map after the channel reduction processing through the second neural network.

12. An encoding method, characterized in that, The method includes: Perform feature extraction on an original image to obtain an initial feature map, where the initial feature map includes two-dimensional feature maps of a plurality of channels, and the resolution of the initial feature map is smaller than the resolution of the original image; A bitstream is obtained by encoding a feature map to be encoded, wherein the feature map to be encoded is the initial feature map, or a two-dimensional feature map including some channels of the initial feature map. The bitstream is used by a decoding device to decode a first feature map, the resolution of which is less than the resolution of the original image. The first feature map is used to determine a second feature map, which is either the first feature map or a feature map obtained by processing the first feature map using a second neural network. The second feature map is used to reconstruct a reconstructed image using a first neural network. The resolution of both the second feature map and the reconstructed image is a target resolution, which is less than the resolution of the original image. When the second feature map is a feature map obtained by processing the first feature map using a second neural network, the second neural network includes one or more output nodes, which are connected to the input of the first neural network. The reconstructed image is obtained by inputting the second feature map output by the target output node into the first neural network, and the target output node belongs to the one or more output nodes.

13. A decoding device, characterized in that, The decoding device includes: The processing module is used to decode the bitstream to obtain a first feature map, wherein the resolution of the first feature map is smaller than the resolution of the original image; A reconstruction module is used to reconstruct a second feature map based on a first neural network to obtain a reconstructed image. The resolution of both the second feature map and the reconstructed image is a target resolution, which is smaller than the resolution of the original image. The second feature map is either the first feature map or a feature map obtained by processing the first feature map through a second neural network. When the second feature map is a feature map obtained by processing the first feature map through the second neural network, the second neural network includes one or more output nodes, which are connected to the input of the first neural network. Specifically, the reconstruction module is used to input the second feature map output by the target output node into the first neural network to obtain the reconstructed image output by the first neural network. The target output node belongs to the one or more output nodes.

14. An encoding device, characterized in that, The encoding device includes: The processing module is used to extract features from the original image to obtain an initial feature map. The initial feature map includes two-dimensional feature maps of multiple channels, and the resolution of the initial feature map is smaller than the resolution of the original image. An encoding module is used to encode a feature map to obtain a bitstream. The feature map to be encoded is the initial feature map, or a two-dimensional feature map including some channels of the initial feature map. The bitstream is used by a decoding device to decode a first feature map, the resolution of which is less than the resolution of the original image. The first feature map is used to determine a second feature map, which is either the first feature map or a feature map obtained by processing the first feature map through a second neural network. The second feature map is used to reconstruct a reconstructed image through a first neural network. The resolution of both the second feature map and the reconstructed image is a target resolution, which is less than the resolution of the original image. When the second feature map is a feature map obtained by processing the first feature map through the second neural network, the second neural network includes one or more output nodes. The output nodes are connected to the input of the first neural network. The reconstructed image is obtained by inputting the second feature map output by the target output node into the first neural network. The target output node belongs to the one or more output nodes.

15. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more computer programs or instructions; When the one or more computer programs or instructions are executed by the one or more processors, the one or more processors perform the method as described in any one of claims 1-12.

16. An electronic device, characterized in that, include: A processor for performing the method as described in any one of claims 1-12.

17. A computer-readable storage medium, characterized in that, It includes a computer program or instructions that, when executed on a computer, cause the computer to perform the method of any one of claims 1-12.

Images