Data decoding methods, data encoding methods and related equipment

Abstract

This application provides a data decoding method, a data encoding method, and related equipment. The data decoding method includes: obtaining an entropy segment i from a bitstream, wherein the entropy segment i is one of N entropy segments, the N entropy segments constituting a representation of input data in a latent space, the entropy segment i not overlapping with its adjacent entropy segments, N being a positive integer greater than 1, i = 1, ..., N; determining a synthesized segment i based on the entropy segment i and / or at least one entropy segment adjacent to the entropy segment i, wherein the synthesized segment i includes at least one overlapping region with its adjacent synthesized segments; and determining a reconstructed segment i based on the synthesized segment i by executing a decoding network, wherein the reconstructed segment i not overlapping with its adjacent reconstructed segments. According to the method, pipelined processing can be performed by a decoding device.

Description

Technical Field

[0001] The embodiments of the present invention generally relate to the field of electronic engineering technology, and more specifically, to encoding and decoding data based on a neural network architecture. Specifically, some embodiments relate to methods and apparatus for encoding and decoding images and / or video in a bitstream using multiple processing layers. Background Technology

[0002] Artificial intelligence (AI) is a technological field dedicated to simulating and replicating human intelligence. It encompasses all aspects of building intelligent systems with the goal of enabling these systems to perform tasks and make decisions like humans. AI takes many forms, including machine learning, deep learning, natural language processing, and computer vision.

[0003] AI has a wide range of applications and can play a vital role in many fields. For example, in the field of autonomous driving, AI technology enables cars, airplanes, and other vehicles to achieve automatic navigation and driving functions. In medical diagnosis, AI, through the application of machine learning and image analysis techniques, helps doctors diagnose diseases and interpret images, thereby improving accuracy and efficiency. Furthermore, AI can also be applied to machine translation, voice assistants, and fraud detection in the financial industry.

[0004] AI has also made groundbreaking progress in data compression and encoding, providing more efficient solutions for data processing and transmission. For example, AI technology can be applied to image compression, where deep learning algorithms identify important features in images and achieve lossless or lossy compression, thereby reducing image file size. Similarly, AI can be used to learn correlation and redundancy information and optimize encoding parameters for video and audio compression encoding, making data storage and transmission more efficient. While AI has undoubtedly made significant progress in data compression and encoding, continuous research and innovation are needed to improve encoding and decoding efficiency in order to meet the ever-increasing demands of data processing and transmission.

[0005] For decades, hybrid image and video codecs have been used to compress image and video data. In such codecs, the signal is typically encoded block-by-block by predicting blocks and then further decoded only the differences between the original blocks and their predictions. Specifically, this decoding can include transform, quantization, and bitstream generation, often involving some entropy decoding.

[0006] Typically, the three components of a hybrid decoding method (i.e., transform, quantization, and entropy decoding) are optimized separately. Modern video compression standards such as High-Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), and Essential Video Coding (EVC) also use the transformed representation to encode the residual signal after prediction.

[0007] Recently, neural network architectures have been applied to image and / or video decoding. Typically, these neural network (NN)-based methods can be applied to image and video decoding in various ways. For example, some end-to-end optimized image or video decoding frameworks have been discussed. Furthermore, deep learning has been used to determine or optimize parts of end-to-end decoding frameworks, such as the selection of prediction parameters or compression. Additionally, some neural network-based methods have been discussed for hybrid image and video decoding frameworks, for example, for implementing trained deep learning models for intra-frame or inter-frame prediction in image or video decoding.

[0008] The common thread among the end-to-end optimized image or video decoding applications discussed above is that they generate some feature map data, which is then transmitted between the encoder and decoder.

[0009] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict the output of a received input. Some neural networks include one or more hidden layers in addition to the output layer. Corresponding feature maps can be provided as the output of each hidden layer. These corresponding feature maps of each hidden layer can be used as input to subsequent layers in the network (i.e., subsequent hidden layers or output layers). Each layer of the network generates an output from the received input based on the current values ​​of the corresponding parameter set. In a neural network partitioned between devices (e.g., between encoder and decoder, between device and cloud, or between different devices), the feature map on the output side of the partition location (e.g., the first device) is compressed and transmitted to the remaining layers of the neural network (e.g., the second device).

[0010] The goal is to further improve encoding and decoding using trained network architectures. Summary of the Invention

[0011] This invention provides a method and apparatus for improving data compression. Embodiments of this application provide a data decoding method, a data encoding method, and related equipment. According to this application, pipelined processing can be applied by the decoding equipment.

[0012] The above and other objectives are achieved by the subject matter claimed in the independent claims. Other implementations are apparent from the dependent claims, the specification, and the drawings.

[0013] Specific embodiments are set forth in the appended independent claims, and other embodiments are set forth in the dependent claims.

[0014] According to a first aspect, the present invention relates to a decoding method. The method is performed by an electronic device. According to the first aspect, an embodiment of this application provides a decoding method comprising: obtaining an entropy segment i from a bitstream, wherein the entropy segment i is one of N entropy segments, the N entropy segments constituting a representation of input data in a latent space, the entropy segment i not having an overlapping region with its adjacent entropy segments, N being a positive integer greater than 1, i = 1, ..., N; determining a synthesized segment i based on the entropy segment i and / or at least one entropy segment adjacent to the entropy segment i, wherein the synthesized segment i includes at least one overlapping region with its adjacent synthesized segments; and determining a reconstructed segment i based on the synthesized segment i by executing a decoding network, wherein the reconstructed segment i does not have an overlapping region with its adjacent reconstructed segments.

[0015] By dividing components into multiple segments (e.g., blocks), the memory required for image decoding can be reduced. In the prior art, the entire latent space needs to be parsed / read from the bitstream before the first block can be decoded. Embodiments of the present invention can advantageously support pipelined decoding and synthesis of image regions using neural networks.

[0016] The above and other embodiments may optionally include one or more of the following individual or combined features.

[0017] According to the method provided in the first aspect, pipeline processing can be performed by a decoding device. The decoding device can acquire a first synthesized segment from M synthesized segments and acquire a reconstructed segment corresponding to the first synthesized segment; then, the decoding device can acquire a second synthesized segment from the M synthesized segments and acquire a reconstructed segment corresponding to the second synthesized segment, and so on. Therefore, the decoding device does not need to acquire all synthesized segments to acquire the reconstructed segment.

[0018] In one possible implementation of the first aspect, obtaining entropy segment i from the bitstream includes: determining position information corresponding to the entropy segment i based on an input segment corresponding to the entropy segment i, wherein the input segment corresponding to the entropy segment i is one of M input segments, the M input segments constituting the input data, each of the M input segments including at least one overlapping region with its adjacent input segments, and M being a positive integer greater than 1; and obtaining the entropy segment i from the bitstream based on the position information corresponding to the entropy segment i.

[0019] In one possible implementation of the first aspect, determining the position information corresponding to the entropy segment i based on the input segment corresponding to the entropy segment i includes: determining a shift segment i based on the input segment corresponding to the entropy segment i and the size of the overlapping region; determining a potential shift segment i based on the shift segment i and the alignment parameters of the synthesized transform in the decoding network; and determining the position information corresponding to the entropy segment i based on the potential shift segment i and the shift segment i.

[0020] In one possible implementation of the first aspect, determining the synthesized segment i based on the entropy segment i and / or at least one entropy segment adjacent to the entropy segment i comprises: determining position information of the synthesized segment i based on alignment parameters of the synthesized transform in the decoding network and an input segment corresponding to the synthesized segment i, wherein the input segment corresponding to the synthesized segment i is one of M input segments constituting the input data, each of the M input segments including at least one overlapping region with its adjacent input segments; and determining elements of the synthesized segment i from the entropy segment i and / or at least one entropy segment adjacent to the entropy segment i based on the position information of the synthesized segment i.

[0021] In one possible implementation of the first aspect, before obtaining entropy segment i from the bitstream, the method further includes: obtaining segment identification information from the bitstream, wherein the segment identification information indicates that the bitstream includes N entropy segments.

[0022] In one possible implementation, the method further includes: obtaining a first context segment i based on the entropy segment i and / or at least one entropy segment adjacent to the entropy segment i, wherein the first context segment is obtained from the bitstream and includes at least one overlapping region with the adjacent first context segment; obtaining a second context segment i based on the entropy segment i and / or at least one entropy segment adjacent to the entropy segment i, wherein the second context segment represents an input prediction segment i corresponding to the entropy segment i and includes at least one overlapping region with the adjacent second context segment; inputting the first context segment i and the second context segment i into a context model to form the synthesized segment i; and determining the reconstructed segment i based on the synthesized segment i by inputting the synthesized segment i into the decoding network, wherein the reconstructed segment i does not have an overlapping region with its adjacent reconstructed segments.

[0023] This allows the context model to be used as part of the decoder, while also supporting pipelined processing of the synthesis process. Using the context module enables better predictions, thereby reducing the additional information that needs to be encoded and decoded, i.e., requiring a lower bitrate for similar quality.

[0024] In one possible implementation, the first context fragment i has the same spatial dimension as the synthesized fragment i. This can support the generation of a synthesized fragment that has at least one overlapping region with one or more adjacent synthesized fragments.

[0025] In one possible implementation, the first context fragment i and the second context fragment i each have the same total number of elements. This allows the first and second context fragments to be used as input to a context model to output a synthesized fragment with the same spatial dimensions as the first context fragment.

[0026] In one possible implementation, the input prediction segment is the output of the superdecoder of the variational autoencoder model. Using the input prediction segment from the superdecoder output as input to the context model can improve the accuracy of the final prediction.

[0027] In one possible implementation, the spatial dimension of the second context fragment i is one-quarter of the spatial dimension of the first context fragment I, and the number of channels of the second context fragment i is four times the number of channels of the first context fragment. This allows for the rearrangement of elements in the tensor when the first and second context fragments are input into the context model.

[0028] In one possible implementation, the first context fragment i represents the residual of the input data in the latent space. Using the residual instead of the complete data may reduce the bit rate required for processing.

[0029] In one possible implementation, the at least one overlapping region of the synthesized fragment is ignored in the determination of the reconstructed fragment. This prevents redundant calculations in determining the reconstructed fragment.

[0030] In one possible implementation, the input data is derived from an input image, and each reconstructed segment may correspond to a block of the input image. This allows the method to be used in image reconstruction applications, optionally in conjunction with a context module.

[0031] According to a second aspect, the present invention relates to an encoding method. The method is performed by an electronic device. According to the second aspect, embodiments of this application provide an encoding method comprising: dividing input data into M input segments, wherein each of the M input segments includes at least one overlapping region with its adjacent input segments, and M is a positive integer greater than 1; processing the M input segments using an encoding network to determine M analysis segments, wherein the M analysis segments correspond one-to-one with the M input segments; determining a representation of the input data in a latent space based on the M analysis segments and the M input segments; dividing the representation of the input data into N entropy segments, wherein each of the N entropy segments does not have an overlapping region with its adjacent entropy segments, and N is a positive integer greater than 1; and determining a bitstream by encoding the N entropy segments.

[0032] The above and other embodiments may optionally include one or more of the following individual or combined features.

[0033] According to the methods provided in the first and / or second aspects, overlapping regions in the input segments are removed. Therefore, overlapping regions do not need to be written to the bitstream multiple times. This can shorten the bitstream length and reduce the time consumed in constructing the bitstream. Therefore, overlapping regions do not need to be read from the bitstream multiple times. The decoding device can employ a pipelined method to decode the bitstream.

[0034] In one possible implementation of the second aspect, determining the representation of the input data in the latent space based on the M analysis segments includes: determining M core segments based on the M analysis segments and the M input segments, wherein the j-th core segment among the M core segments is determined based on the j-th analysis segment among the M analysis segments and the j-th input segment among the M input segments, j=1, ...,M, and each core segment among the M core segments does not have an overlapping region with its adjacent core segments; and determining the representation of the input data by constituting the M core segments.

[0035] In one possible implementation of the second aspect, dividing the representation of the input data into N entropy segments includes: determining the position information of each of the N entropy segments; and dividing the representation of the input data into the N entropy segments according to the position information of each of the N entropy segments.

[0036] In one possible implementation of the second aspect, determining the position information of each of the N entropy segments includes: determining N shift segments based on the M input segments; determining N potential shift segments based on the N shift segments and alignment parameters of the analysis transform in the coding network; and determining the position information of each of the N entropy segments based on the N potential shift segments and the N shift segments.

[0037] In one possible design of the second aspect, the method further includes: determining segment identifier information in the bitstream, wherein the segment identifier information indicates that the bitstream includes N entropy segments.

[0038] In one possible implementation, the method further includes: determining a latent segment i, i = 1, ..., M, based on the determined representation of the input data in the latent space, wherein the latent segment i includes at least one overlapping region with its neighboring latent segments; obtaining an input prediction segment i based on the latent segment i, wherein the input prediction segment i includes at least one overlapping region with its neighboring input prediction segments; inputting the latent segment i and the input prediction segment i into a context model to output one of M output segments i, wherein the output segment i does not have an overlapping region with its neighboring output segments; dividing the M output segments into N entropy segments, wherein each of the N entropy segments does not have an overlapping region with its neighboring entropy segments; and determining the bitstream by encoding the N entropy segments. This allows the context model to be used as part of the decoder while supporting pipelined processing of the synthesis process. Using the context module can achieve better prediction, thereby reducing the additional information that needs to be encoded and decoded, i.e., a lower bitrate is required for similar quality.

[0039] In one possible implementation, the potential fragment i has the same spatial dimension as the output fragment i. This can support the generation of an output fragment that has at least one overlapping region with one or more adjacent output fragments.

[0040] In one possible implementation, the latent fragment i and the input predicted fragment i each have the same total number of elements. This allows the latent fragment and the input predicted fragment to be used as input to a context model to produce an output fragment with the same spatial dimensions as the latent fragment.

[0041] In one possible implementation, the input prediction segment is the output of the superdecoder of the variational autoencoder model. Using the input prediction segment from the superdecoder output as input to the context model can improve the accuracy of the final prediction.

[0042] In one possible implementation, the spatial dimension of the input predicted fragment i is one-quarter of the spatial dimension of the latent fragment i. The number of channels of the input predicted fragment i can be four times the number of channels of the latent fragment. This allows for rearranging the elements in the tensor when the input predicted fragment and the latent fragment are fed into the context model.

[0043] In one possible implementation, the output fragment i represents the residual of the input data in the latent space. Using the residual instead of the complete data may reduce the bit rate required for processing.

[0044] In one possible implementation, the at least one overlapping region of the potential segments is ignored in the determination of the output segments. This prevents redundant calculations in determining the output segments.

[0045] In one possible implementation, the input data is derived from an input image, and each potential segment can correspond to a block of the input image. This allows the method to be used in image reconstruction applications, optionally in conjunction with a context module.

[0046] According to a third aspect, the present invention relates to a decoding apparatus. This decoding apparatus can refer to the same advantageous effects as the decoding method described according to the first aspect. Further details are omitted here. The decoding apparatus provides technical means for implementing the operations described in the method according to the first aspect. The function can be implemented in hardware or by hardware executing corresponding software. In one possible implementation, the decoding apparatus includes: an acquisition unit for acquiring an entropy fragment i from a bitstream, wherein the entropy fragment i is one of N entropy fragments constituting a representation of input data in a latent space, the entropy fragment i does not overlap with its adjacent entropy fragments, N is a positive integer greater than 1, and i = 1, ..., N; a processing unit for determining a synthesized fragment i based on the entropy fragment i and / or at least one entropy fragment adjacent to the entropy fragment i, wherein the synthesized fragment i includes at least one overlapping region with its adjacent synthesized fragments; the processing unit is further configured to determine a reconstructed fragment i based on the synthesized fragment i by executing a decoding network, wherein the reconstructed fragment i does not overlap with its adjacent reconstructed fragments.

[0047] These modules can be used to provide corresponding functionality to the method examples described according to the first aspect. See the detailed description in the method examples for further details. It will not be repeated here.

[0048] According to a third aspect, one embodiment of this application provides an electronic device having the function of implementing the method described in the first aspect. The function can be implemented by hardware or by hardware executing corresponding software. The hardware corresponding to the software includes one or more units corresponding to the function.

[0049] The above and other embodiments may optionally include one or more of the following individual or combined features.

[0050] In one possible implementation, the device is further configured to: obtain a first context segment i based on the entropy segment i and / or at least one entropy segment adjacent to the entropy segment i, wherein the first context segment is obtained from the bitstream and includes at least one overlapping region with the adjacent first context segment; obtain a second context segment i based on the entropy segment i and / or at least one entropy segment adjacent to the entropy segment i, wherein the second context segment represents an input prediction segment i corresponding to the entropy segment i and includes at least one overlapping region with the adjacent second context segment; input the first context segment i and the second context segment i into a context model to form the synthesized segment i; wherein the processing unit is further configured to determine the reconstructed segment i based on the synthesized segment i by inputting the synthesized segment i into the decoding network, wherein the reconstructed segment i does not have an overlapping region with its adjacent reconstructed segments. This allows the context model to be used as part of the encoder while supporting pipelined processing of the synthesis process. Using the context module can achieve better prediction, thereby reducing the additional information required for encoding and decoding, i.e., a smaller bitrate is required for similar quality.

[0051] In one possible implementation, the first context fragment i has the same spatial dimension as the synthesized fragment i. This can support the generation of a synthesized fragment that has at least one overlapping region with one or more adjacent synthesized fragments.

[0052] In one possible implementation, the first context fragment i and the second context fragment i each have the same total number of elements. This allows the first and second context fragments to be used as input to a context model to produce a synthesized fragment with the same spatial dimensions as the first context fragment.

[0053] In one possible implementation, the input prediction segment is the output of the superdecoder of the variational autoencoder model. Using the input prediction segment from the superdecoder output on the context model can improve the accuracy of the final prediction.

[0054] In one possible implementation, the spatial dimension of the second context fragment i is one-quarter of the spatial dimension of the first context fragment i, and the number of channels of the second context fragment i is four times the number of channels of the first context fragment. This allows for the rearrangement of elements in the tensor when the first and second context fragments are input into the context model.

[0055] In one possible implementation, the first context fragment i represents the residual of the input data in the latent space. Using the residual instead of the complete data may reduce the bit rate required for processing.

[0056] In one possible implementation, the processing unit is further configured to ignore the at least one overlapping region of the synthesized segment during the determination of the reconstructed segment. This prevents redundant calculations in determining the reconstructed segment.

[0057] In one possible implementation, the input data is derived from an input image, and each reconstructed segment may correspond to a block of the input image. This can support image reconstruction using a pipelined decoding and synthesis process, optionally used in conjunction with a context module.

[0058] According to a fourth aspect, the present invention relates to an encoding apparatus. This encoding apparatus can refer to the same advantageous effects as the encoding method described according to the second aspect. Further details are omitted here. The encoding apparatus provides technical means for implementing the operations described in the second aspect. The function can be implemented in hardware or by hardware executing corresponding software. In one possible implementation, the encoding device includes: a partitioning unit for partitioning input data into M input segments, wherein each of the M input segments includes at least one overlapping region with its adjacent input segments, and M is a positive integer greater than 1; a processing unit for processing the M input segments using an encoding network to determine M analysis segments, wherein the M analysis segments correspond one-to-one with the M input segments; the processing unit is further configured to determine the representation of the input data in a latent space based on the M analysis segments and the M input segments; the processing unit is further configured to partition the representation of the input data into N entropy segments, wherein each of the N entropy segments does not have an overlapping region with its adjacent entropy segments, and N is a positive integer greater than 1; the processing unit is further configured to determine a bitstream by encoding the N entropy segments.

[0059] These modules can be used to provide corresponding functionality to the method examples described according to the second aspect. See the detailed description in the method examples for further details. It will not be repeated here.

[0060] According to a fourth aspect, one embodiment of this application provides an electronic device having the function of implementing the method described in the second aspect. The function can be implemented by hardware or by hardware executing corresponding software. The hardware corresponding to the software includes one or more units corresponding to the function.

[0061] The above and other embodiments may optionally include one or more of the following individual or combined features.

[0062] In one possible implementation, the device is further configured to: determine a latent segment i, i = 1, ..., M, based on the determined representation of the input data in the latent space, wherein the latent segment i includes at least one overlapping region with its neighboring latent segments; obtain an input prediction segment i based on the latent segment i, wherein the input prediction segment i includes at least one overlapping region with its neighboring input prediction segments; input the latent segment i and the input prediction segment i into a context model to output one of M output segments i, wherein the output segment i does not have an overlapping region with its neighboring output segments; divide the M output segments into N entropy segments, wherein each of the N entropy segments does not have an overlapping region with its neighboring entropy segments; and determine the bitstream by encoding the N entropy segments. This allows the context model to be used as part of the encoder while supporting pipelined processing of the synthesis process. Using the context module can achieve better prediction, thereby reducing the additional information that needs to be encoded and decoded, i.e., a smaller bitrate is required for similar quality.

[0063] In one possible implementation, the potential fragment i has the same spatial dimension as the output fragment i. This can support the generation of an output fragment that has at least one overlapping region with one or more adjacent output fragments.

[0064] In one possible implementation, the latent fragment i and the input predicted fragment i each have the same total number of elements. This allows the latent fragment and the input predicted fragment to be used as input to a context model to produce an output fragment with the same spatial dimensions as the latent fragment.

[0065] In one possible implementation, the input prediction segment is the output of the superdecoder of the variational autoencoder model. Using the input prediction segment from the superdecoder output on the context model can improve the accuracy of the final prediction.

[0066] In one possible implementation, the spatial dimension of the input predicted fragment i is one-quarter of the spatial dimension of the latent fragment i, wherein the number of channels of the input predicted fragment i is four times the number of channels of the latent fragment. This allows for rearranging the elements in the tensor when the input predicted fragment and the latent fragment are input into the context model.

[0067] In one possible implementation, the output fragment i represents the residual of the input data in the latent space. Using the residual instead of the complete data may reduce the bit rate required for processing.

[0068] In one possible implementation, the processing unit is further configured to ignore at least one overlapping region of the potential segments during the determination of the output segments. This prevents redundant calculations in determining the output segments. This may reduce the bitrate in the bitstream.

[0069] In one possible implementation, the input data is derived from an input image, and each potential segment can correspond to a block of the input image. This can support image reconstruction using a pipelined decoding and synthesis process, optionally used in conjunction with a contextual model.

[0070] According to a fifth aspect, one embodiment of this application provides a computer-readable storage medium including instructions. When the instructions are executed on an electronic device, they cause the electronic device to perform the method described in the first aspect or any possible implementation thereof.

[0071] According to a sixth aspect, one embodiment of this application provides a computer-readable storage medium including instructions. When the instructions are executed on an electronic device, they cause the electronic device to perform the method described in the second aspect or any possible implementation thereof.

[0072] A computer-readable storage medium stores instructions that, when executed, cause one or more processors to encode video data. The instructions cause the one or more processors to perform the method according to the first aspect or the second aspect, or any possible embodiment of the first aspect or the second aspect.

[0073] According to a seventh aspect, one embodiment of this application provides an electronic device including a processor and a memory. The processor is connected to the memory. The memory is used to store instructions, and the processor is used to execute the instructions. When the processor executes the instructions stored in the memory, the processor performs the method of the first aspect or any possible implementation thereof.

[0074] According to an eighth aspect, one embodiment of this application provides an electronic device including a processor and a memory. The processor is connected to the memory. The memory is used to store instructions, and the processor is used to execute the instructions. When the processor executes the instructions stored in the memory, the processor performs the method of the second aspect or any possible implementation of the second aspect.

[0075] According to a ninth aspect, an embodiment of this application provides a chip system, wherein the chip system includes a memory and a processor, the memory is used to store a computer program, and the processor is used to call the computer program from the memory and run the computer program, such that an electronic device in which the chip system is located performs the method of the first aspect or any possible implementation of the first aspect.

[0076] According to a tenth aspect, an embodiment of this application provides a chip system, wherein the chip system includes a memory and a processor, the memory is used to store a computer program, and the processor is used to call the computer program from the memory and run the computer program, such that an electronic device in which the chip system is located performs the method of the second aspect or any possible implementation of the second aspect.

[0077] According to an eleventh aspect, one embodiment of this application provides a computer program product, wherein when the computer program product is run on an electronic device, the electronic device causes the electronic device to perform the method described in the first aspect or any possible implementation thereof. The computer program product includes program code for performing the method described according to the first aspect or any possible embodiment thereof when executed on a computer.

[0078] According to a twelfth aspect, one embodiment of this application provides a computer program product, wherein when the computer program product is run on an electronic device, the electronic device causes the electronic device to perform the method described in the second aspect or any possible implementation thereof. The computer program product includes program code for performing the method described according to the second aspect or any possible embodiment thereof when executed on a computer.

[0079] The method according to the first aspect of the invention can be performed by the apparatus according to the third aspect of the invention. Other features and implementations of the method according to the first aspect of the invention correspond to the corresponding features and implementations of the apparatus according to the third aspect of the invention. The advantages of the method according to the first aspect can be the same as the advantages of the corresponding implementations of the apparatus according to the third aspect.

[0080] The method according to the second aspect of the invention can be performed by the apparatus according to the fourth aspect of the invention. Other features and implementations of the method according to the second aspect of the invention correspond to the corresponding features and implementations of the apparatus according to the fourth aspect of the invention. The advantages of the method according to the second aspect can be the same as the advantages of the corresponding implementations of the apparatus according to the fourth aspect.

[0081] According to another aspect, the present invention relates to a video stream decoding apparatus, including a processor and a memory. The memory stores instructions that cause the processor to perform the method according to the first aspect.

[0082] According to another aspect, the present invention relates to a video stream encoding apparatus, comprising a processor and a memory. The memory stores instructions that cause the processor to perform the method according to the second aspect.

[0083] According to another aspect, the present invention relates to a bitstream comprising data blocks representing N entropy segments, wherein each of the N entropy segments does not overlap with its adjacent entropy segments, and N is a positive integer greater than 1.

[0084] Another embodiment of this application can provide a system for transmitting bitstreams, comprising: at least one storage medium for storing at least one bitstream as defined above or generated by the encoding method described above; and a video streaming device for acquiring the bitstream from one of the at least one storage mediums and sending the bitstream to a terminal device; wherein the video streaming device includes a content server or a content distribution server.

[0085] In one possible embodiment, the system may further include: one or more processors configured to encrypt at least one bitstream to obtain at least one encrypted bitstream; at least one storage medium configured to store the encrypted bitstream; or, one or more processors configured to convert a bitstream of a first format into a bitstream of a second format; at least one storage medium configured to store the bitstream of the second format. In one possible embodiment, the system may further include: a receiver configured to receive a first operation request; one or more processors configured to, in response to the first operation request, determine a target bitstream in at least one storage medium; and a transmitter configured to transmit the target bitstream to a terminal-side device. In one possible embodiment, one or more processors may further be configured to: encapsulate the bitstream to obtain a transport stream of a first format; and the transmitter may further be configured to: transmit the transport stream of the first format to a terminal-side device for display; or, transmit the transport stream of the first format to storage space for storage.

[0086] In one possible embodiment, an exemplary method for storing a bitstream is provided, the method comprising: obtaining a bitstream according to any of the encoding methods shown above; and storing the bitstream in a storage medium. Optionally, the method further comprises: performing encryption processing on the bitstream to obtain an encrypted bitstream; and storing the encrypted bitstream in the storage medium. It should be understood that any known encryption method can be used.

[0087] In one possible embodiment, an exemplary system for storing a bitstream is provided, the system comprising: a receiver for receiving a bitstream generated by any of the above encoding methods; a processor for performing encryption processing on the bitstream to obtain an encrypted stream; and a computer-readable storage medium for storing the encrypted stream.

[0088] Optionally, the system includes a video streaming device, which may be a content server or a content distribution server. The video streaming device is used to obtain a bitstream from a storage medium and send the bitstream to a terminal device.

[0089] According to another aspect, the present invention relates to a video compression method for a video stream, the method comprising: receiving a bitstream as described above; and decoding the bitstream to form a representation of the state of the video stream in one or more channels.

[0090] According to another aspect, the present invention relates to a video stream decoding apparatus, comprising a processor and a memory. The memory stores instructions that cause the processor to perform the method according to the fourth aspect.

[0091] According to another aspect, the present invention relates to a video stream encoding apparatus, comprising a processor and a memory. The memory stores instructions that cause the processor to perform the method according to the second aspect.

[0092] One or more embodiments will be described in detail in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the specification, drawings, and claims. Attached Figure Description

[0093] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0094] Figure 1 This is a schematic diagram illustrating the channels processed by the layers of a neural network; Figure 2 This is a schematic diagram illustrating the type of autoencoder in a neural network; Figure 3A This is a schematic diagram illustrating an exemplary network architecture that includes an encoder and decoder with a priori model. Figure 3B This is a schematic diagram illustrating a general network architecture that includes an encoding end with a super-prior model; Figure 3C This is a schematic diagram illustrating a general network architecture that includes a decoding end with a priori model; Figure 4 This is a schematic diagram illustrating an exemplary network architecture that includes an encoder and decoder with a priori model. Figure 5It is a block diagram of the structure of a cloud-based solution for machine-based tasks such as machine vision. Figure 6A This is a block diagram of an end-to-end video compression framework based on neural networks; Figure 6B This is a block diagram illustrating some exemplary details of the application of a neural network for sports field compression; Figure 6C This is a block diagram illustrating some exemplary details of the application of neural networks for motion compression; Figure 7 This is a block diagram illustrating an example of an encoding or decoding device; Figure 8 This is a block diagram illustrating another example of an encoding or decoding device; Figure 9 This is a diagram showing the calculation of the total receptive field; Figure 10 This is a diagram illustrating another calculation of the total receptive field; Figure 11 This is a schematic block diagram illustrating a decoding system provided in some embodiments of this application; Figure 12 An example illustrates the VAE framework; Figure 13 The encoder component of the VAE framework is described; Figure 14 It describes the decoder component of the VAE framework; Figure 15 The general principle of compression is illustrated; Figure 16 Another VAE framework is shown; Figure 17 Flowcharts of encoding methods provided by some embodiments of this application are shown; Figure 18 Flowcharts of decoding methods provided by some embodiments of this application are shown; Figure 19 The assembly line process is shown; Figure 20 The diagram shows the blocks used to determine the reconstructed image and the blocks read from the bitstream; Figure 21 This illustrates entropy partitioning, synthesis partitioning, and reconstruction partitioning; Figure 22 The diagram illustrates entropy partitioning, synthesis partitioning, and reconstruction partitioning under normal conditions. Figure 23 The entropy block, synthesis block, and reconstruction block are shown in special cases; Figure 24a The block mesh used for block synthesis is shown; Figure 24b A tiled grid is shown for reading from and writing to the bitstream; Figure 25 The input and output of the context model in the decoder are shown; Figure 26 The diagram shows the context model input to the decoder and the chunks output from the context model in the decoder. Figure 27 The block divisions used during the decoding process using the context model are shown; Figure 28 This illustrates exemplary dimensions of the input and output of the context model in the decoder; Figure 29 The pipeline process, including context determination for each segment, is shown. Figure 30 The block divisions used during the encoding process using the context model are shown; Figure 31 The decoding process is shown; Figure 32 These are schematic block diagrams of an electronic device provided in some embodiments of this application; the device can be used for processing by a neural network-based unit; Figure 33 These are schematic block diagrams of an electronic device provided in some embodiments of this application; the device can be used for processing by a neural network-based unit; Figure 34 These are schematic block diagrams of electronic devices provided in some embodiments of this application; Figure 35 This is a flowchart of an exemplary method for decoding; Figure 36 This is a flowchart of an exemplary method for encoding; Figure 37 The bitstream structure is shown; Figure 38 This is a block diagram illustrating an example of a video decoding system for implementing embodiments of the present invention; Figure 39 This is a block diagram illustrating another example of a video decoding system for implementing embodiments of the present invention; Figure 40 This is a block diagram illustrating an example of an encoding or decoding device; Figure 41 This is a block diagram illustrating another example of an encoding or decoding device; Figure 42 This is a block diagram illustrating another example of an encoding or decoding device; Similar reference numerals and names in different figures can represent similar elements. Detailed Implementation

[0095] In the following description, reference is made to the accompanying drawings, which form part of this invention, illustrating by way of description specific aspects of embodiments of the invention or specific aspects in which embodiments of the invention may be used. It should be understood that embodiments of the invention can be used in other aspects and include structural or logical variations not depicted in the drawings. Therefore, the following detailed description should not be construed as limiting, and the scope of the invention is defined by the appended claims.

[0096] For example, it should be understood that the disclosure relating to the described method is equally applicable to the corresponding device or system used to perform the method, and vice versa. For instance, if one or more specific method steps are described, the corresponding device may include one or more units (e.g., functional units) to perform the described one or more method steps (e.g., one unit performs one or more steps, or multiple units perform one or more of a plurality of steps respectively), even if such one or more units are not explicitly described or illustrated in the drawings. Furthermore, if a specific apparatus is described according to one or more units such as functional units, the corresponding method may include a step to perform the function of one or more units (e.g., one step performs the function of one or more units, or multiple steps perform the function of one or more of a plurality of units respectively), even if such one or more units are not explicitly described or illustrated in the drawings. Moreover, it should be understood that, unless otherwise expressly stated, features of the various exemplary embodiments and / or aspects described herein can be combined with each other.

[0097] The following provides an overview of some of the technical terms used and the framework for using embodiments of the present invention.

[0098] Artificial Neural Networks Artificial neural networks (ANNs), or connectionist systems, are computational systems inspired by the fuzzy logic of the biological neural networks that make up the animal brain. These systems "learn" to perform tasks by example, typically without being programmed using task-specific rules. For instance, in image recognition, these systems might learn to recognize images containing cats by analyzing exemplary images manually labeled "cat" or "no cat" and using the results to identify cats in other images. These systems do this without prior knowledge about cats, such as that they have fur, a tail, whiskers, and a cat's face. Instead, they automatically generate recognizable features from the examples they process.

[0099] ANNs are based on a set of connection units or nodes called artificial neurons, which loosely mimic neurons in a biological brain. Each connection, like a synapse in a biological brain, can transmit signals to other neurons. The artificial neuron that receives the signal processes it and can then send signals to the neurons connected to it.

[0100] In ANN implementations, the "signals" at connections are real numbers, and the output of each neuron is calculated using a nonlinear function of the sum of its inputs. These connections are called edges. Neurons and edges typically have weights that adjust as learning progresses. These weights increase or decrease the strength of the signals at the connections. Neurons can have a threshold such that a signal is only sent when the aggregate signal exceeds that threshold. Typically, neurons are clustered into layers. Different layers can perform different transformations on their inputs. Signals are transmitted from the first layer (input layer) to the last layer (output layer), but this may occur after multiple traversals of these layers.

[0101] The initial goal of ANN methods was to solve problems in the same way the human brain does. Over time, attention shifted to performing specific tasks, leading to a departure from biology. ANNs have been applied to a wide variety of tasks, including computer vision, speech recognition, machine translation, social network filtering, board games and video games, medical diagnosis, and even activities traditionally considered solely human, such as painting.

[0102] The name "convolutional neural network" indicates that the network employs a mathematical operation called convolution. Convolution is a specialized linear operation. A convolutional network is a neural network that uses convolution instead of general matrix multiplication in at least one of its layers.

[0103] Figure 1 This schematically illustrates a general concept processed by neural networks such as CNNs. A convolutional neural network consists of an input layer, an output layer, and multiple hidden layers. The input layer provides input (e.g., ... Figure 1 The hidden layers in a CNN typically consist of a series of convolutional layers that perform convolutions via multiplication or other dot products. The result of a layer is one or more feature maps (represented by empty solid-line rectangles), sometimes also called channels. Some or all layers may involve resampling (e.g., subsampling). Therefore, the feature maps may become smaller, such as... Figure 1As shown. It's worth noting that convolutions with strides can also reduce the size of the input feature map (resampling). The activation functions in CNNs are typically rectified linear unit (ReLU) layers or Leaky ReLU, followed by additional convolutional layers such as pooling layers, fully connected layers, and normalization layers. These layers are called hidden layers because their inputs and outputs are masked by the activation function and the final convolution. While these layers are colloquially called convolutional layers, this is just a convention. Mathematically, convolution is technically a sliding dot product or cross-correlation. This is important for indexing in a matrix because the sliding dot product or cross-correlation affects how weights are determined at a particular index point.

[0104] When programming a CNN to process images, such as Figure 1 As shown, the input is a tensor with shape (number of images) × (image width) × (image height) × (image depth). It should be understood that image depth can be composed of the image's channels. After passing through the convolutional layer, the image is abstracted into a feature map with shape (number of images) × (feature map width) × (feature map height) × (feature map channels). The convolutional layers within the neural network should have the following properties: a convolutional kernel defined by width and height (hyperparameters); and the number of input and output channels (hyperparameters). The depth of the convolutional filter (input channels) should be equal to the number of channels (depth) of the input feature map.

[0105] In the past, traditional multilayer perceptron (MLP) models have been applied to image recognition. However, due to the full connectivity between nodes, MLP models are affected by high dimensionality and cannot scale well when processing high-resolution images. A 1000×1000 pixel image with RGB color channels has 3 million weights, which is too high for efficient and feasible large-scale processing with full connectivity. This network architecture also does not consider the spatial structure of the data, causing input pixels that are far apart to be processed in the same way as pixels that are very close together. This ignores the principle of locality in image data both computationally and semantically. Therefore, the full connectivity of neurons is not useful for purposes such as image recognition dominated by spatially local input patterns.

[0106] Convolutional Neural Networks (CNNs) are biologically inspired variants of multilayer perceptrons, specifically designed to mimic the behavior of the visual cortex. These models mitigate the challenges of MLP architectures by leveraging the strong spatial local correlations present in natural images. Convolutional layers are the core building blocks of CNNs. The parameters of a layer consist of a set of learnable filters (the kernels mentioned above) with small receptive fields that extend across the entire depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, computing the dot product between the entries in that filter and the input, and producing a two-dimensional activation map of that filter. Thus, the network learns the filters that are activated when the network detects a specific type of feature at a spatial location in the input.

[0107] The activation maps of all filters stacked along the depth dimension form the complete output image of the convolutional layer. Therefore, each entry in the output image can also be interpreted as the output of a neuron that looks at a small region of the input and shares parameters with neurons in the same activation map. A feature map, or activation map, is the output activation of a given filter. Feature maps and activations have the same meaning. In some papers, it's called an activation map because it's a mapping corresponding to the activations of different parts of the image; it's also called a feature map because it's a mapping indicating the location of a certain feature in the image. High activation indicates that a certain feature has been found.

[0108] Another important concept in CNNs is pooling, which is a form of non-linear downsampling. Several non-linear functions exist to implement pooling, with max pooling being the most common. Max pooling divides the input image into a set of non-overlapping rectangles and outputs the maximum value for each sub-region.

[0109] Intuitively, the exact location of a feature is less important than its approximate location relative to other features. This is the reason behind the use of pooling in convolutional neural networks. Pooling layers are used to progressively reduce the size of the representation space, decreasing the number of parameters, memory footprint, and computational cost in the network, and thus also controlling overfitting. In CNN architectures, it is common practice to periodically insert pooling layers between consecutive convolutional layers. Pooling operations provide another form of translation invariance.

[0110] Pooling layers operate independently on each depth strip in the input and spatially resize themselves. The most common form is a 2×2 pooling layer that applies two samples along both the width and height of each depth strip in the input with a stride of 2, discarding 75% of the activations. In this case, each max operation is performed on four numbers. The depth dimension remains unchanged. Besides max pooling, pooling units can also use other functions, such as average pooling or... 2-norm pooling. Average pooling was frequently used in the past, but it has been less common recently compared to max pooling, as max pooling generally performs better in practice. Due to the significant reduction in representation size, there has recently been a trend towards using smaller filters or discarding pooling layers entirely. Region of Interest (ROI) pooling is a variant of max pooling where the output size is fixed and the input rectangle is a parameter. Pooling is an important component of convolutional neural networks for object detection based on the Fast R-CNN architecture.

[0111] The ReLU mentioned above stands for Corrected Linear Unit, which uses a non-saturating activation function. ReLU effectively removes negative values ​​from the activation map by setting them to 0, increasing the non-linearity of the decision function and the overall network without affecting the receptive field of the convolutional layers. Other functions are also used to increase non-linearity, such as the saturating hyperbolic tangent and the sigmoid function. ReLU is generally superior to other functions because it trains neural networks several times faster without significantly reducing generalization accuracy.

[0112] Leaky Rectified Linear Unit (Leaky ReLU) is a type of ReLU-based activation function, but it has a small slope for negative values ​​instead of a flat slope. The slope coefficients are determined before training, meaning they are not learned during training. This type of activation function is common in tasks with sparse gradients, such as training generative adversarial networks. Leaky ReLU applies an element-wise function: LeakyReLU(x)=max(0,x)+negative_slope min(0,x), or

[0113] Among them, the parameters are: negative_slope: Controls the angle of the negative slope. Default value: 1e–2 inplace: Optionally, the operation can be performed in place. Default: False.

[0114] After several convolutional and max-pooling layers, high-level inference in a neural network is achieved through fully connected layers. Neurons in a fully connected layer are connected to all activations in the previous layer, as shown in a conventional (non-convolutional) artificial neural network. Therefore, these activations can be computed as affine transformations, which involve matrix multiplication and subsequent bias shifts (vector addition of learned or fixed bias terms).

[0115] A "loss layer" (including the calculation of the loss function) represents how training penalizes the deviation between the predicted (output) and the true label, and is usually the last layer of a neural network. Various loss functions suitable for different tasks can be used. Softmax loss is used to predict a single class among K mutually exclusive classes. Sigmoid cross-entropy loss is used to predict K independent probability values ​​in [0, 1]. Euclidean loss is used to regress to real-valued labels.

[0116] In short, Figure 1 The data flow in a typical convolutional neural network is illustrated. First, the input image is passed through a convolutional layer and abstracted into a feature map, which includes several channels corresponding to multiple filters in a set of learnable filters for that layer. The feature map is then subsampled using, for example, a pooling layer, thereby reducing the size of each channel in the feature map. Next, the data reaches another convolutional layer, which may have a different number of output channels. As mentioned above, the number of input and output channels are hyperparameters of the layers. To establish connections in the network, these parameters need to be synchronized between two connected layers; for example, the number of input channels in the current layer should equal the number of output channels in the previous layer. For the first layer processing input data (e.g., an image), the number of input channels is typically equal to the number of channels used to represent the data, such as 3 channels for an RGB or YUV representation of an image or video, or 1 channel for a grayscale image or video representation. Channels obtained by one or more convolutional layers (and possibly one or more resampling layers) can be passed to the output layer. In some implementations, this output layer can be a convolutional layer or a resampling layer. In exemplary and non-limiting implementations, the output layer is a fully connected layer.

[0117] Autoencoders and Unsupervised Learning An autoencoder is a type of artificial neural network used to learn efficient data decoding in an unsupervised manner. Its schematic diagram is shown below. Figure 2 As shown, the autoencoder includes an encoder 210 and a decoder 250, where the input x is fed into the input layer of the encoder subnet 220, and the output x' is output from the decoder subnet 260. The purpose of the autoencoder is to learn a representation (encoding) 230 of a set of data x by training networks 220 and 260 to ignore signal "noise," typically for dimensionality reduction. Together with the reduction-side (encoding) subnet 220, a reconstruction-side (decoding) subnet 260 is learned, in which the autoencoder attempts to generate a representation x' from the reduced encoding 230 that is as close as possible to its original input x, hence the name. In the simplest case, given a hidden layer, the encoder level of the autoencoder takes the input... and map it to

[0118] .

[0119] The image This is commonly referred to as code 230, latent variable, or latent representation. Here, It is an element activation function, such as the sigmoid function or the modified linear unit. It is a weight matrix. This is the bias vector. Weights and biases are typically initialized randomly and then iteratively updated during training via backpropagation. Afterward, the decoder level of the autoencoder maps h to... Reconstruction of the same shape :

[0120] Among them, the decoder , and Possibly corresponding to the encoder , and Irrelevant.

[0121] Variational autoencoder models make strong assumptions about the distribution of latent variables. They use variational methods for latent representation learning, which introduces an additional loss component, and train the algorithm using a specific estimator called a stochastic gradient variational Bayes (SGVB) estimator. It is assumed that the data is generated by a directed graphical model. The generated data is being processed by the encoder, which is learning the posterior distribution. approximation ,in, and These represent the parameters of the encoder (recognition model) and decoder (generative model), respectively. The probability distribution of the latent vectors in a VAE is typically closer to the probability distribution of the training data than that of a standard autoencoder. The objective of a VAE has the following form:

[0122] Here, This represents the Kullback–Leibler divergence. The priors of the latent variables are typically set to central isotropic multivariate Gaussian. Typically, the shapes of the variational and likelihood distributions are chosen to resemble decomposed Gaussian distributions:

[0123]

[0124] in, It is the encoder output, and and ) is the decoder output.

[0125] Recent advances in the field of artificial neural networks, particularly convolutional neural networks, have sparked researchers' interest in applying neural network-based techniques to image and video compression tasks. For example, an end-to-end optimized image compression method has been proposed, which uses a network based on a variational autoencoder.

[0126] Therefore, data compression is considered a fundamental and well-studied problem in engineering, and is typically aimed at designing codes for a given discrete set of data with minimum entropy. This approach relies heavily on an understanding of the probabilistic structure of the data, thus the problem is closely related to probabilistic source modeling. However, since all practical codes must have finite entropy, continuous-valued data (e.g., vectors of image pixel intensities) must be quantized into a finite set of discrete values, which introduces errors.

[0127] In this context, known as the lossy compression problem, a trade-off must be struck between two competing costs: the entropy (rate) of the discrete representation and the error (distortion) introduced by quantization. Different compression applications, such as data storage or transmission over channels with limited capacity, require different rate-distortion trade-offs.

[0128] Joint optimization of rate and distortion is difficult. The general problem of optimal quantization in high-dimensional spaces is intractable without other constraints. Therefore, most existing image compression methods operate by linearly transforming the data vector into a suitable continuous-valued representation, independently quantizing its elements, and then encoding the generated discrete representation using lossless entropy codes. Due to the central role of the transform, this scheme is called transform-decoding.

[0129] For example, JPEG uses Discrete Cosine Transform (DCT) for pixel blocks, while JPEG 2000 uses Multiscale Orthogonal Wavelet Decomposition (MOD). Typically, the three components of a transform decoding method (i.e., transform, quantization, and entropy decoding) are optimized separately (usually through manual parameter tuning). Modern video compression standards (such as HEVC, VVC, and EVC) also use the transformed representation to decode the predicted residual signal. Several transforms are used for this purpose, such as the Discrete Cosine Transform (DCT) and the Discrete Sine Transform (DST), as well as the Low Frequency Non-Separable Manually Optimized Transform (LFNST).

[0130] Variational image compression The variational autoencoder (VAE) framework can be considered a nonlinear transform decoding model. The transform process can be mainly divided into four parts. This is illustrated in the diagram of the VAE framework. Figure 3A Examples are provided below.

[0131] The transformation process can be mainly divided into four parts: Figure 3A An example illustrates the VAE framework. In Figure 3A In this code, encoder 101 maps the input image x to a latent representation (denoted by y) using the function y = f(x). In the following text, this latent representation may also be referred to as a portion of the "latest space" or a point within the "latest space". The function f() is a transformation function that converts the input signal x into a more compressible representation y. Quantizer 102... Transform the latent representation y into a quantized latent representation with (discrete) values. Where Q represents the quantizer function. The entropy model, or superencoder / decoder (also known as superprior), estimates the quantized latent representation. The distribution is determined to obtain the minimum rate achievable through lossless entropy source decoding.

[0132] The latent space can be understood as a compressed representation of data, where similar data points are closer together. The latent space can be used to learn data features and find simpler data representations for analysis. Arithmetic coding (AE) is used to quantize the super-prior 3 latent representation T. and edge information This is included in bitstream 2 (binarization). Additionally, a decoder 104 is provided, which transforms the quantized latent representation into the reconstructed image. , .Signal This is an estimate of the input image x. We expect x to be as close as possible to... In other words, the reconstruction quality should be as high as possible. However, The higher the similarity to x, the more side information is required for transmission. Side information includes... Figure 3A The bitstreams 1 and 2 shown are generated by the encoder and transmitted to the decoder. Generally, the greater the side information, the higher the reconstruction quality. However, a large amount of side information also indicates a low compression ratio. Therefore, Figure 3A One objective of the system described in the text is to balance reconstruction quality with the amount of side information transmitted in the bitstream.

[0133] exist Figure 3A In this module, component AE 105 is an arithmetic coding module that quantizes the latent representation. and edge information The sample is converted into a binary representation bitstream 1. and The samples can include, for example, integers or floating-point numbers. One purpose of the arithmetic encoding module is to convert the sample values ​​into binary number strings (through a binarization process) (then the binary number strings are included in the bitstream, which may include other parts corresponding to the encoded image or other side information).

[0134] Arithmetic decoding (AD) 106 is the process of restoring the binarization process, in which binary numbers are converted back to sample values. Arithmetic decoding is provided by the arithmetic decoding module 106.

[0135] It should be noted that this invention is not limited to this specific framework. Furthermore, this invention is not limited to image or video compression, but can also be applied to object detection, image generation, and recognition systems.

[0136] exist Figure 3A In this case, two subnets are cascaded together. This type of subnet represents a logical division between different parts of the entire network. For example, in... Figure 3A In this code, modules 101, 102, 104, 105, and 106 are referred to as the "encoder / decoder" subnet. The "encoder / decoder" subnet is responsible for encoding (generating) and decoding (parsing) the first bitstream "bitstream 1". Figure 3A The second network, comprising modules 103, 108, 109, 110, and 107, is called the "super encoder / decoder" subnet. This second subnet is responsible for generating the second bitstream, "bitstream 2". These two subnets serve different purposes.

[0137] The first subnet is responsible for: • Transform the input image x (101) into its latent representation y (this makes it easier to compress x). • Quantize the latent representation y (102) into the quantized latent representation , • Arithmetic coding module 105 uses AE compression to quantize the latent representation To obtain the bitstream "bitstream 1" • Use the arithmetic decoding module 106 to parse the bitstream 1 via AD. • Reconstruct the image using parsed data (104) ).

[0138] The purpose of the second subnet is to obtain the statistical characteristics of the samples in "stream 1" (e.g., the mean, variance, and correlation between the samples in stream 1) so that the compression of stream 1 by the first subnet is more efficient. The second subnet generates a second stream "stream 2" which includes the aforementioned information (e.g., the mean, variance, and correlation between the samples in stream 1).

[0139] The second network includes an encoding section, which includes quantized latent representations. Transform (103) into edge information z, and quantize the edge information z into quantized edge information. and the quantized edge information Encoding (e.g., binarization) (109) results in bitstream 2. In this example, binarization is performed using arithmetic encoding (AE). The decoding portion of the second network includes arithmetic decoding (AD) 110, which transforms the input bitstream 2 into decoded quantized side information. . Can be with The same applies because the decoding operation at the arithmetic encoding end is a lossless compression method. Then, the decoded and quantized side information... Transformed (107) into decoded side information . express Statistical properties (e.g.) (The mean of the samples, or the variance of the sample values, etc.). Then, the latent representation is decoded. Provided to the aforementioned arithmetic encoder 105 and arithmetic decoder 106 for control The probability model.

[0140] Figure 3A An example of a variational autoencoder (VAE) is described, the details of which may vary in different implementations. For example, in a particular implementation, additional components may exist to more efficiently acquire the statistical properties of samples from bitstream 1. In one such implementation, a context modeler may exist, the goal of which is to extract the cross-correlation information of bitstream 1. The statistical information provided by the second subnet can be used by the arithmetic encoder (AE) 105 and the arithmetic decoder (AD) 106 components.

[0141] Figure 3A The encoder and decoder are depicted in a single figure. As will be apparent to those skilled in the art, the encoder and decoder can and often are embedded in different devices.

[0142] Figure 3B and Figure 3C The encoder and decoder components of the VAE framework are described separately. According to some embodiments, the encoder receives an image as input. The input image may include one or more channels, such as color channels or other types of channels, such as depth channels or motion information channels. The encoder's output (e.g., ...) Figure 3B (As shown) are bitstream 1 and bitstream 2. Bitstream 1 is the output of the first subnet of the encoder, and bitstream 2 is the output of the second subnet of the encoder.

[0143] Similarly, in Figure 3C In the process, two bitstreams (bitstream 1 and bitstream 2) are received as inputs, and a result is generated at the output. That is, reconstructing (decoded) the image. As mentioned above, a VAE can be divided into different logical units that perform different actions. This is in Figure 3B and Figure 3C The example illustrates that makes Figure 3B The components involved in signal encoding, such as video, and the encoded information provided are described. This encoded information is then... Figure 3C The decoder component described herein receives data for, for example, encoding. It should be noted that the encoder and decoder components, represented by the numbers 12x and 14x, can functionally correspond to those described above. Figure 3A The component mentioned and represented by the number 10x.

[0144] Specifically, such as Figure 3B As shown, the encoder includes encoder 121, which transforms the input x into a signal y, and then provides the signal y to quantizer 322. Quantizer 122 provides information to arithmetic encoding module 125 and super encoder 123. Super encoder 123 provides the bitstream 2 discussed above to super decoder 147, which in turn provides information to arithmetic encoding module 105 (125).

[0145] The output of the arithmetic encoding module is bitstream 1. Bitstream 1 and bitstream 2 are the outputs of signal encoding, and this output is then provided (transmitted) to the decoding process. Although unit 101 (121) is referred to as an "encoder", it can also be used to... Figure 3B The complete subnet described in the code is called an "encoder". The encoding process typically represents a unit (module) converting an input into an encoded (e.g., compressed) output. From Figure 3B As can be seen, unit 121 can actually be considered the core of the entire subnet, as it performs the transformation of the input x to y, where y is a compressed version of x. Compression in encoder 121 can be achieved, for example, by applying a neural network or any processing network typically having one or more layers. In such a network, compression can be performed through cascaded processing including downsampling, which reduces the size of the input and / or the number of channels. Therefore, the encoder can be referred to as a neural network (NN) based encoder, etc.

[0146] The remaining parts in the diagram (quantization unit, super encoder, super decoder, arithmetic encoder / decoder) are all components that improve the efficiency of the encoding process or are responsible for converting the compressed output y into a series of bits (bit stream). Quantization can be provided to further compress the output of the NN encoder 121 through lossy compression. The AE 125, combined with the super encoder 123 and super decoder 127 used to configure the AE 125, can perform binarization, which can further compress the quantized signal through lossless compression. Therefore, it is also possible to... Figure 3B The entire subnet in the code is called the "encoder".

[0147] Most deep learning (DL) based image / video compression systems reduce the dimensionality of the signal before converting it into binary bits. For example, in a VAE framework, the encoder is a non-linear transformation that maps the input image x to y, where the width and height of y are smaller than the width and height of x. Because y has a smaller width and height, it is smaller in size, reducing the dimensionality (size) of the signal, making it easier to compress. It's important to note that, in general, an encoder doesn't necessarily need to reduce size in two (or often all) dimensions. Instead, some exemplary implementations can provide an encoder that reduces size only in one dimension (or often a subset of dimensions).

[0148] In J. Balle, L. Valero Laparra, and EP Simoncelli (2015), “Density Modeling of Images Using a Generalized Normalization Transformation,” In: arXiv e-prints, Presented at the 4th Int.Conf. for Learning Representations, 2016 (hereinafter referred to as “Balle”), the authors proposed an end-to-end optimization framework for an image compression model based on a nonlinear transformation. The authors optimized for mean squared error (MSE) but used a more flexible transformation constructed from linear convolutions and nonlinear concatenations. Specifically, they used a generalized divisive normalization (GDN) coupled with nonlinearity, inspired by neuron models in biological vision systems, which has proven effective in Gaussianizing image densities. This concatenated transformation is followed by uniform scalar quantization (i.e., each element is rounded to the nearest integer), which effectively implements the parametric form of vector quantization in the original image space. The compressed image is reconstructed from these quantized values ​​using an approximate parametric nonlinear inverse transform.

[0149] Such an example of the VAE framework is as follows: Figure 4 As shown, it utilizes six downsampling layers, labeled 401 to 406. The network architecture includes a super-prior model. (Left side (g)) a g s The image autoencoder architecture is shown on the right (h). a h s This corresponds to the realization of a priori autoencoders. Factoring the prior model affects the analysis and synthesis transformation g. a and g s Using the same architecture. Q represents quantization, and AE and AD represent the arithmetic encoder and arithmetic decoder, respectively. The encoder performs g on the input image x. a This process generates a response y (latent representation) with a spatially varying standard deviation. The encoding g... a It includes multiple convolutional layers, which have subsampling and generalized divisive normalization (GDN) as an activation function.

[0150] Feed the response to h aIn the first step, the distribution of the standard deviation in z is summarized. Then, z is quantized, compressed, and sent as side information. The encoder then uses the quantized vector... To estimate This is used to obtain the spatial distribution of the standard deviation of the probability values ​​(or frequency values) of arithmetic coding (AE), and is used to compress and transmit the quantized image representation. (or potential representation). The decoder first recovers from the compressed signal. Then, the decoder uses h s Get This provides it with the correct probability estimate for successful recovery. Then, the decoder will Feed to g s In order to obtain the reconstructed image.

[0151] Layers that include downsampling are indicated by a down arrow in the layer description. The layer description "Conv N,k1,2↓" indicates that the layer is a convolutional layer with N channels and a kernel size of k1×k1. For example, k1 can be equal to 5, and k2 can be equal to 3. As mentioned above, 2↓ indicates that downsampling with a factor of 2 is performed in this layer. Downsampling with a factor of 2 reduces one dimension of the input signal by half at the output. Figure 4 In the diagram, 2↓ indicates that both the width and height of the input image are reduced by half. Since there are six downsampling layers, if the width and height of the input image 414 (also denoted by x) are represented by w and h, then the width and height of the output signal z 413 are equal to w / 64 and h / 64, respectively. The modules represented by AE and AD are the arithmetic encoder and arithmetic decoder, which will refer to... Figures 3A to 3C Let's explain. Arithmetic encoders and decoders are specific implementations of entropy decoding. AE and AD can be replaced by other entropy decoding methods. In information theory, entropy coding is a lossless data compression scheme used to convert the value of a symbol into a binary representation, a process that can be reversed. Furthermore, the "Q" in the diagram corresponds to the above regarding... Figure 4 The quantization operation is also mentioned and further explained in the "Quantization" section above. Furthermore, the quantization operation and the corresponding quantization unit that is part of component 413 or 415 do not necessarily exist and / or can be replaced by another unit.

[0152] exist Figure 4The diagram also shows a decoder including upsampling layers 407 to 412. Another layer 420, implemented as a convolutional layer but not upsampling the received input, is provided between the upsampling layers 411 and 410 in the order of input processing. A corresponding convolutional layer 430 for the decoder is also shown. Such a layer could be provided in a neural network to perform operations on the input that modify specific characteristics without changing the input size. However, it is not necessary to provide such a layer.

[0153] When viewed from the processing order of bitstream 2 through the decoder, the upsampling layers operate in reverse order, from upsampling layer 412 to upsampling layer 407. Each upsampling layer is shown here to provide an upsampling ratio of 2, indicated by ↑. Of course, not all upsampling layers necessarily have the same upsampling ratio, and other upsampling ratios, such as 3, 4, 8, etc., can also be used. Layers 407 to 412 are implemented as convolutional layers (conv). Specifically, since they may be designed to provide the opposite operation to the encoder on the input, the upsampling layers can apply a deconvolution operation to the received input, increasing its size by a factor corresponding to the upsampling ratio. However, the invention is generally not limited to deconvolution, and upsampling can be performed in any other way, such as by bilinear interpolation between two adjacent samples, or by copying nearest neighbor samples, etc.

[0154] In the first subnet, some convolutional layers (401 to 403) use generalized divisive normalization (GDN) at the encoding end and inverse GDN (IGDN) at the decoding end. In the second subnet, the activation function applied is ReLU. It should be noted that this invention is not limited to this implementation; other activation functions can typically be used instead of GDN or ReLU.

[0155] Cloud solutions for machine tasks Video coding for machines (VCM) is another popular area of ​​computer science today. The main idea behind this approach is to send an encoded representation of image or video information for further processing via computer vision (CV) algorithms, such as object segmentation, detection, and recognition. Unlike traditional image and video decoding designed for human perception, quality features are performance metrics for computer vision tasks, such as object detection accuracy, rather than reconstruction quality. This is in... Figure 5 As shown in the image.

[0156] Machine video coding, also known as collaborative intelligence, is a relatively new paradigm for efficiently deploying deep neural networks in mobile cloud infrastructure. By partitioning the network between the mobile device 510 and the cloud 590 (e.g., a cloud server), the computational workload can be distributed, minimizing the overall energy consumption and / or latency of the system. Typically, collaborative intelligence is a paradigm where the processing of a neural network is distributed across two or more distinct computing nodes; for example, devices, but generally any functionally defined node. Here, the term "node" does not refer to the aforementioned neural network node. Rather, a (computing) node here refers to a separate device / module (physically or at least logically) that implements a portion of the neural network. These devices can be different servers, different end-user devices, combinations of servers and / or user devices and / or the cloud and / or processors, etc. In other words, computing nodes can be thought of as nodes belonging to the same neural network and communicating with each other to transmit decoded data within / for the neural network. For example, to perform complex computations, one or more layers can be executed on a first device (e.g., a device on the mobile device 510) and one or more layers can be executed on another device (e.g., a cloud server on the cloud 590). However, the distribution can also be more granular, with a single layer executed on multiple devices. In this invention, the term "multiple" refers to two or more. In some prior art solutions, a portion of the neural network function is executed on a device (user device or edge device, etc.) or multiple such devices, and then the output (feature map) is passed to the cloud. The cloud is a collection of processing or computing systems located outside the device that runs a portion of the neural network. The concept of collaborative intelligence is also extended to model training. In this case, data flows bidirectionally: from the cloud to the mobile device during backpropagation of training, and from the mobile device to the cloud during forward propagation of training and inference. Figure 5 (as shown in the image).

[0157] Some works present semantic image compression by encoding deep features and then reconstructing the input image from these features. Uniform quantization-based compression is shown, followed by context-based adaptive arithmetic coding (CABAC) for H.264. In some scenarios, it might be more efficient to send the output of the hidden layer (deep feature map) 550 from the mobile part 510 to the cloud 590, rather than sending compressed natural image data to the cloud and performing object detection using the reconstructed image. Therefore, compressing the data (features) generated by the mobile part 510 may be advantageous, and the mobile part 510 may include a quantization layer 520 for this purpose. Thus, the cloud 590 may include an inverse quantization layer 560. Efficient compression of feature maps is beneficial for image and video compression and reconstruction for human perception and machine vision. Entropy decoding methods (such as arithmetic coding) are popular methods for compressing deep features (i.e., feature maps).

[0158] Today, video content accounts for over 80% of internet traffic, and this percentage is expected to rise further. Therefore, establishing an efficient video compression system to generate higher-quality frames within a given bandwidth budget is crucial. Furthermore, most video-related computer vision tasks, such as video object detection or video object tracking, are sensitive to the quality of compressed video, and efficient video compression can benefit other computer vision tasks. Simultaneously, video compression technology also contributes to action recognition and model compression. However, for the past few decades, video compression algorithms have relied on handcrafted modules, such as block-based motion estimation and discrete cosine transform (DCT), to reduce redundancy in video sequences, as mentioned above. While each module is carefully designed, the entire compression system has not been optimized end-to-end. It is hoped that by jointly optimizing the entire compression system, video compression performance can be further improved.

[0159] End-to-end image or video compression DNN-based image compression methods can leverage large-scale end-to-end training and highly nonlinear transformations, which are not used in traditional methods. However, directly applying these techniques to build an end-to-end learning system for video compression is not easy. First, learning how to generate and compress motion information tailored to video compression remains an open question. Video compression methods heavily rely on motion information to reduce temporal redundancy in video sequences.

[0160] A simple approach is to use learning-based optical flow to represent motion information. However, current learning-based optical flow methods aim to generate the flow field as accurately as possible. Accurate optical flow is often not the optimal choice for specific video tasks. Furthermore, compared to motion information in traditional compression systems, optical flow significantly increases the amount of data, and directly applying existing compression methods to compress optical flow values ​​would significantly increase the number of bits required to store motion information. Secondly, it is unclear how to construct a DNN-based video compression system by minimizing the residual and motion information based on rate-distortion objectives. Rate-distortion optimization (RDO) aims to achieve higher quality reconstructed frames (i.e., less distortion) given a certain number of bits (or bitrate) for compression. RDO is crucial for video compression performance. To leverage the end-to-end training capabilities of learning-based compression systems, RDO strategies are needed to optimize the entire system.

[0161] exist Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, Zhiyong Gao of" DVC: An End-to-end Deep Video Compression Framework Framework) " IEEE / CVF International Conference on Computer Vision and Pattern Recognition (CVPR) proceedings, 2019, session number 1 Pages 11006-11015 In this paper, the authors propose an end-to-end deep video compression (DVC) model that jointly learns motion estimation, motion compression, and residual decoding.

[0162] This encoder is in Figure 6A shown in . Specifically, Figure 6A The overall structure of an end-to-end trainable video compression framework is shown. To compress motion information, the CNN is specified to incorporate optical flow. Transform into a corresponding representation suitable for better compression Specifically, an autoencoder-style network is used to compress optical flow. A motion vector (MV) compression network is as follows: Figure 6B As shown. Network architecture and Figure 4 The terms ga / gs are somewhat similar. Specifically, optical flow... It is fed into a series of convolutional operations and nonlinear transformations, including GDN and IGDN. The number of output channels *c* for the convolution (deconvolution) is exemplarily 128 here, except for the last deconvolution layer, which in this example has 2 output channels. The kernel size is *k*, for example, *k=3*. Given an optical flow of size *M* × *N* × 2, the MV encoder will generate a motion representation of size *M / 16* × *N* × 16* × 128. Then, the motion representation is quantized (Q), entropy encoded, and sent to the bitstream as... The MV decoder receives the quantized representation. Reconstruct motion information using MV encoder Typically, the values ​​of k and c can differ from those described in the embodiments known in the art.

[0163] Figure 6C The structure of the motion compensation section is shown. Here, the previously reconstructed frame x is used. t–1 To reconstruct motion information, the warping unit generates warped frames (typically using an interpolation filter, such as a bilinear interpolation filter). Then, a separate CNN with three inputs generates the predicted image. The architecture of a motion-compensated CNN is also as follows... Figure 6C As shown.

[0164] The residual information between the original frame and the predicted frame is encoded by a residual encoder network. A highly nonlinear neural network is used to transform the residual into the corresponding latent representation. Compared with the discrete cosine transform in traditional video compression systems, this method can better utilize the power of nonlinear transformation and achieve higher compression efficiency.

[0165] As can be seen from the overview above, CNN-based architectures can be applied to image and video compression, taking into account different parts of the video framework, including motion estimation, motion compensation, and residual decoding. Entropy decoding is a popular method for data compression, widely adopted in the industry, and is also suitable for feature map compression in human perception or computer vision tasks.

[0166] Machine video encoding Video coding for machines (VCM) is another popular area of ​​computer science today. The main idea behind this approach is to send an encoded representation of image or video information for further processing via computer vision (CV) algorithms, such as object segmentation, detection, and recognition. Unlike traditional image and video decoding designed for human perception, quality features are performance metrics for computer vision tasks, such as object detection accuracy, rather than reconstruction quality.

[0167] A recent study proposes a novel deployment paradigm called collaborative intelligence, in which a deep model is partitioned between mobile and cloud. Extensive experiments across various hardware configurations and wireless connectivity modes have shown that the optimal operating point in terms of energy consumption and / or computational latency involves partitioning the model, typically at some point deep within the network. It has been found that today's common approaches (i.e., models entirely in the cloud or entirely on mobile) are rarely, if any, optimal. The concept of collaborative intelligence is also extended to model training. In this context, data flows bidirectionally: from the cloud to the mobile device during backpropagation in training, and from the mobile device to the cloud during forward propagation in training and inference.

[0168] Against the backdrop of recent object detection deep models, this paper investigates lossy compression of deep feature data based on HEVC intra-frame coding. It should be noted that detection performance degrades with increasing compression levels, and compression-enhanced training is proposed to minimize this loss by generating models more robust to quantization noise in feature values. However, this remains a suboptimal solution due to the highly complex codec used, which is optimized for natural scene compression rather than deep feature compression.

[0169] This approach, using the popular YOLOv2 network for object detection, addresses the problem of deep feature compression in collaborative intelligence, aiming to explore the balance between compression efficiency and recognition accuracy. Here, the term "deep feature" has the same meaning as "feature map." The term "deep" comes from the idea of ​​collaborative intelligence, which involves capturing the output feature maps of some hidden (deep) layers and transmitting them to the cloud for inference. This appears to be more efficient than sending compressed natural image data to the cloud and performing object detection using reconstructed images.

[0170] Efficient feature map compression is beneficial for image and video compression and reconstruction used in human perception and machine vision. The drawbacks of state-of-the-art autoencoder-based compression methods also apply to machine vision tasks.

[0171] Functional modules Variable bit rate module Encoders can output bitstreams at different bitrates. Therefore, in some methods, the output of the encoding network is scaled (e.g., each channel is multiplied by a corresponding scaling factor, also known as the target gain value), and the input of the decoding network is inversely scaled (e.g., each channel is multiplied by the reciprocal of the corresponding scaling factor, also known as the target inverse gain value), such as... Figure 7 As shown. The scaling factor can be preset. Different quality levels or quantization parameters correspond to different target gain values. If the output of the coding network is scaled to a smaller value, the bitstream size can be reduced. Otherwise, the bitstream size can be increased.

[0172] Color format conversion RGB and YUV are common color spaces. Conversion between RGB and YUV can be performed according to equations specified in standards such as CCIR 601 and BT.709.

[0173] Independent structures for luminance and chromaticity Some VAE-based codecs use the YUV color space as the input of the encoder and the output of the decoder, such as... Figure 8As shown, the Y component represents luminance, and the UV components represent chrominance. The resolution of the UV components can be the same as or lower than that of the Y component. Typical formats include YUV4:4:4, YUV4:2:2, and YUV4:2:0. The Y component is converted into a feature map F_Y through a network, and an entropy encoding module generates a bitstream for the Y component based on the feature map F_Y. The UV components are converted into a feature map F_UV through another network, and the entropy encoding module generates a bitstream for the UV components based on the feature map F_UV. In this structure, the feature maps of the Y component and the UV components can be independently quantized, so as to flexibly allocate bits for luminance and chrominance. For example, for a color-sensitive image, the quantization of the feature map of the UV components can be reduced, and the number of bits in the bitstream of the UV components can be increased to improve the reconstruction quality of the UV components and obtain a better visual effect.

[0174] In some other methods, the encoder concatenates the Y component and the UV components and then sends them to a UV component processing module (for converting image information into a feature map). In addition, the decoder concatenates the reconstructed feature map of the Y component and the reconstructed feature map of the UV components and then sends them to a UV component processing module 2 (for converting the feature map into image information). In this method, the correlation between the Y component and the UV components can be used to reduce the bitstream of the UV components.

[0175] In this specification, a "parameter" is a value used in the operation process of each layer forming a neural network. For example, it can include weights used when an input value is applied to a certain operation expression. Here, the parameter can be represented in matrix form. The parameter is a set of values of training results and can be updated with separate training data when necessary.

[0176] The following terms can also be defined as follows: The image size refers to the width w or height h of the image or the width-height pair. The width and height of the image are usually measured by the number of luminance samples.

[0177] As mentioned above, downsampling is a process of reducing the sampling rate (sampling interval) of a discrete input signal. For example, if the input signal is an image with a size of h and w, and the size of the downsampled output is h2 and w2, then at least one of the following holds: h2 < h, w2 < w. In an exemplary implementation, downsampling can be implemented by only retaining every m-th sample and discarding the rest of the input signal (such as an image).

[0178] Upsampling is a process of increasing the sampling rate (sampling interval) of a discrete input signal. For example, if the size of the input image is h and w, and the size of the upsampled output is h2 and w2, then at least one of the following holds: h < h2, w < w2.

[0179] Downsampling and upsampling are both examples of resampling. Resampling is the process of changing the sampling rate (sampling interval) of the input signal.

[0180] During upsampling or downsampling, filtering can be applied to improve the accuracy of the resampled signal and reduce aliasing. Interpolation filtering typically involves a weighted combination of sample values ​​from locations surrounding the resampled position. It can be implemented as follows:

[0181] Where f() is the resampled signal, Here, C(k) represents the resampling coordinates, C(k) represents the interpolation filter coefficients, and s(x,y) represents the input signal. For signals located at... Summation is performed on the nearby (x,y) pairs.

[0182] Cropping the outer edges of a digital image can be called cropping. Cropping can be used to reduce the size of an image (number of samples) and / or change the aspect ratio (length to width) of the image.

[0183] Padding refers to increasing the size of the input (i.e., the input image) by generating new samples at the image boundaries using predefined sample values ​​or sample values ​​from locations within the input image. The generated samples are approximations of actual sample values ​​that do not exist.

[0184] Resizing is a general term for changing the size of an input image. It can be done using either padding or cropping methods, or it can be done through resampling.

[0185] Integer division is division that discards the decimal part (remainder).

[0186] Convolution is given by the following general equation. Here, f() can be defined as the input signal, and g() can be defined as the filter.

[0187]

[0188] A downsampling layer is a layer in a neural network that reduces at least one dimension of the input. Typically, the input may have three or more dimensions, which may include the number of channels, width, and height. A downsampling layer usually refers to a reduction in the width and / or height dimensions. It can be implemented using operations such as convolution, averaging, and max pooling.

[0189] Feature maps are generated by applying filters or feature detectors to the input image or feature map output of previous layers. Feature map visualization provides a deep understanding of the internal representation of the specific inputs of each convolutional layer in the model.

[0190] The latent space is the feature map generated by the neural network at the bottleneck layer.

[0191] An upsampling layer is a layer in a neural network that increases at least one dimension of the input. Typically, the input may have three or more dimensions, which may include the number of channels, width, and height. An upsampling layer usually refers to increasing the width and / or height dimensions. It can be implemented using operations such as deconvolution and copying.

[0192] In the context of neural networks, the receptive field is defined as the size of the region in the input that produces features. Essentially, it's a measure of the correlation between the output features (of any layer) and the input region (piece). It's important to note that the idea of ​​the receptive field applies to local operations (i.e., convolution, pooling). For example, a convolution operation with a kernel size of 3×3 has a receptive field of 3×3 samples in the input layer (the convolution node uses 9 input samples to obtain 1 output sample). The total receptive field is the set of input samples used to obtain a specified set of output samples by applying one or more processing layers.

[0193] Total receptive field can be Figure 9 and Figure 10 Let's illustrate with an example. In Figure 9 The example illustrates processing a one-dimensional input (the 7 samples on the left side of the diagram) using two consecutive transposed convolutional (also called deconvolution) layers. The input is processed from left to right: "Deconvolutional Layer 1" processes the input first, and its output is processed by "Deconvolutional Layer 2." In this example, the kernel size is 3 in both deconvolutional layers. This means that 3 input samples are needed to obtain 1 output sample in each layer. In the example, the output sample set is marked within the dashed rectangle and includes 3 samples. Due to the size of the deconvolution kernel, 7 samples are needed at the input to obtain the output sample set including 3 output samples. Therefore, the total receptive field of the 3 marked output samples is 7 samples at the input.

[0194] exist Figure 9 In this model, there are 7 input samples, 5 intermediate output samples, and 3 output samples. The reduction in the number of samples is due to the fact that the input signal is finite (not extended to infinity in every direction), thus resulting in "missing samples" at the input boundaries. In other words, since the deconvolution operation requires 3 input samples corresponding to each output sample, if the number of input samples is 7, only 5 intermediate output samples can be generated. In reality, the number of output samples that can be generated is (k–1) samples, fewer than the number of input samples, where k is the kernel size. Figure 9 In this example, the input sample size is 7. After the first deconvolution with a kernel size of 3, the number of intermediate samples is 5. After the second deconvolution with a kernel size of 3, the number of output samples is 3.

[0195] Sometimes it is desirable to keep the same number of samples after each operation (convolution or deconvolution or others). In this case, padding can be applied at the boundaries of the input to compensate for the "missing samples". It should be noted that the present invention applies to both cases, since padding is not a mandatory operation for convolutional, deconvolutional or any other processing layer.

[0196] This should not be confused with downsampling. During downsampling, for every M samples, there are N samples at the output, where N < M. The key difference is that M is usually much smaller than the number of inputs. In Figure 9 there is no downsampling, and the reduction in the number of samples is due to the fact that the size of the input is not infinite and there are "missing samples" at the input end. For example, if the number of input samples is 100 and the kernel size is k = 3, then the number of output samples will be 100 – (k – 1) – (k – 1) = 96. If both deconvolutional layers have downsampling (at a ratio of M = 2 and N = 1), then the number of output samples will be = 22.

[0197] The operations of convolution and deconvolution (also known as transposed convolution) are the same from the perspective of mathematical expressions. The difference lies in that the deconvolution operation assumes that a previous convolution operation has occurred. In other words, deconvolution is the process of filtering a signal to compensate for an undesired convolution. The goal of deconvolution is to recover the signal that existed before the convolution occurred. Embodiments of the present invention can be applied to convolution and deconvolution operations (and in fact can also be applied to any other operation with a kernel size greater than 1).

[0198] As Figure 9 shown, the total receptive field of 3 output samples is 7 samples at the input end. By continuously using processing layers with a kernel size greater than 1, the size of the total receptive field increases. Generally, the total receptive field of an output sample set is calculated by tracing the connections of each node from the output layer to the input layer, and then finding the union of all samples in the input that are directly or indirectly (through more than 1 processing layer) connected to the output sample set. For example, in Figure 9 each output sample is connected to 3 samples in the previous layer. The union includes 5 samples in the intermediate output layer, which are connected to 7 samples in the input layer.

[0199] Figure 10 shows another example of explaining how to calculate the total receptive field. In Figure 10 a two-dimensional array of input samples is processed by 2 convolutional layers, each with a kernel size of 3×3. After applying 2 deconvolutional layers, an output array is obtained. The output sample set (array) is marked with a dashed rectangle ("output sample set") and includes 2×2 = 4 samples. The total receptive field of the "output sample set" includes 6×6 = 36 samples. The total receptive field can be calculated as: • Each output sample is connected to 3×3 samples in the intermediate output. The union of all samples in the intermediate output connected to the output sample set contains 4×4=16 samples.

[0200] • Each of the 16 samples in the intermediate output is connected to 3×3 samples in the input. The union of all samples in the input connected to the 16 samples in the intermediate output comprises 6×6=36 samples. Therefore, the total receptive field of the 2×2 output samples is 36 samples at the input.

[0201] A neural network can consist of multiple subnetworks. Each subnetwork has one or more layers. Different subnetworks have different input / output sizes, and therefore different memory requirements / computational complexities.

[0202] A pipeline is a series of subnets used to process specific components of an image. An example could be a system with two pipelines, where the first pipeline processes only the luma component, and the second pipeline processes one or more chroma components. A pipeline can only process one component at a time, but it can use a second component as auxiliary information to aid processing. For example, a pipeline that outputs the chroma component can simultaneously take the latent representations of both the luma and chroma components as input (conditional encoding of the chroma components).

[0203] Conditional color separation (CCS) is a neural network architecture for image / video encoding / processing, in which the primary color component is encoded / processed independently, but the secondary color components are conditionally encoded / processed, using the primary component as an auxiliary input.

[0204] Exemplary methods and apparatus according to specific embodiments of the present invention will now be described in further detail. The technical solutions in this application are described below with reference to the accompanying drawings.

[0205] Figure 11 This is a schematic block diagram illustrating a decoding system provided in some embodiments of this application. (Reference) Figure 11 The decoding system 900 includes a source device 910 for providing encoded image data to a destination device 920 for decoding the encoded image data. For convenience, it is assumed that the decoding system 900 is an image decoding system. It will be apparent to those skilled in the art that the image decoding system is merely an exemplary embodiment of the present invention, and the embodiments of the present invention are not limited thereto.

[0206] The source device 910 may include an encoding unit 911. Optionally, the source device 910 may also include an image source unit 912, a preprocessing unit 913, and a communication unit 914.

[0207] Image source unit 912 may include or may be any type of image capture device, such as an image capture device for capturing real-world images, and / or any type of image generation device such as a computer graphics processor for generating computer-animated images, or any type of device for acquiring and / or providing real-world images, computer-animated images (e.g., screen content, virtual reality (VR) images), and / or any combination thereof (e.g., augmented reality (AR) images). In the following text, all such types of images and any other types of images will be referred to as "images".

[0208] A (digital) image is, or can be viewed as, a two-dimensional array or matrix of samples with intensity values. Samples in an array can also be called pixels (short for image elements). The number of samples in the array or image along the horizontal and vertical directions (or axes) defines the image size and / or resolution. To represent color, three color components are typically used; that is, an image can be represented as, or can include, an array of three samples. In RBG format or color space, an image includes corresponding arrays of red, green, and blue samples. However, in video decoding, each pixel is typically represented in a luminance / chrominance format or color space, such as YCbCr, which includes the luminance component represented by Y (sometimes also L) and the two chrominance components represented by Cb and Cr. The luminance component Y represents the brightness or grayscale intensity (e.g., both are the same in grayscale images), while the two chrominance components Cb and Cr represent the chrominance or color information components. Therefore, an image in YCbCr format consists of a luminance sample array composed of luminance sample values ​​(Y) and two chrominance sample arrays composed of chrominance values ​​(Cb and Cr). An RGB format image can be converted or transformed to YCbCr format, and vice versa. This process is also called color transformation or conversion. If the image is black and white, it may only contain the luminance sample array.

[0209] Image source unit 912 may be a camera for capturing images, a memory (e.g., an image memory) that includes or stores previously captured or generated images, and / or any type of (internal or external) interface for acquiring or receiving images. For example, the camera may be a local camera or an integrated camera integrated into the source device, and the memory may be local memory or, for example, an integrated memory integrated into the source device. The aforementioned interface may be an external interface for receiving images from an external video source, which may be an external image capture device such as a camera, external memory, or external image generation device, such as an external computer graphics processor, computer, or server. The aforementioned interface may be any type of interface, such as a wired interface, wireless interface, or optical interface based on any proprietary or standardized interface protocol. The interface for acquiring image data 931 may be the same interface as communication unit 914, or may be part of communication unit 914.

[0210] To distinguish between the preprocessing unit 913 and the processing performed by the preprocessing unit 913, the image or image data 931 can also be referred to as the original image or original image data 931.

[0211] The preprocessing unit 913 is used to receive (raw) image data 931 and preprocess the image data 931 to obtain a preprocessed image 932 or preprocessed image data.

[0212] The preprocessing performed by the preprocessing unit 913 may include trimming, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. The encoding unit 911 is used to receive the preprocessed image data 932 and provide encoded image data 933.

[0213] The communication unit 914 of the source device 910 can be used to receive the encoded image data 933 and transmit it directly to another device (e.g., destination device 920 or any other device) for storage or direct reconstruction, or to process the encoded image data 933 before storing it and / or transmitting it to another device (e.g., destination device 920 or any other device) for decoding or storage.

[0214] The destination device 920 includes a decoding unit 921 and may additionally (i.e., optionally) include a communication unit 924, a post-processing unit 923, and a display device 922.

[0215] The communication unit 924 of the destination device 920 is used to receive encoded image data 933, for example, directly from the source device 910 or any other source (e.g., a memory such as an encoded image data memory).

[0216] Communication unit 914 and communication unit 924 can be used to send or receive encoded image data 933 via a direct communication link (e.g., a direct wired or wireless connection) between source device 910 and destination device 920 or via any type of network (e.g., a wired network or wireless network or any combination thereof, any type of private network and public network or any combination thereof).

[0217] The communication unit 914 may be used, for example, to package the encoded image data 933 into an appropriate format (e.g., a data packet) for transmission on a communication link or communication network, and may also include data loss protection and data loss recovery.

[0218] The communication unit 924, which forms a corresponding part of the communication unit 914, can be used, for example, to unpack data packets to obtain encoded image data 933, and can also be used to perform data loss protection and data loss recovery, including error hiding, for example.

[0219] Both communication unit 914 and communication unit 924 can be configured as unidirectional communication interfaces, such as Figure 11 The middle arrow points to the encoded image data 933 from the source device 910 to the destination device 920, or it can be bidirectional.

[0220] The communication unit can be used for, for example, sending and receiving messages (e.g., establishing a connection), acknowledging and / or retransmitting data including lost or delayed image data, and exchanging any other information related to the communication link and / or data transmission (e.g., encoded image data transmission).

[0221] Decoder 921 is used to receive encoded image data 933 and provide decoded image data 934.

[0222] The post-processing unit 923 of the destination device 920 is used to post-process the decoded image data 934 to obtain post-processed image data 935. The post-processing performed by the post-processing unit 923 may include color format conversion (e.g., from YCbCr to RGB), color adjustment, trimming or resampling, or any other processing, in order to provide the decoded image data 934 for display by the display unit 922, etc.

[0223] The display unit 922 in the destination device 920 is used to receive post-processed image data 935 for displaying the image to a user or viewer. The display unit 922 can be or may include any type of display for representing the reconstructed image, such as an integrated or external display or screen. For example, the aforementioned display may include a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or any other type of display, such as a projector, a hologram (3D), etc.

[0224] although Figure 11 Source device 910 and destination device 920 are described as separate devices; however, device embodiments may also include two devices or two functions, namely source device 910 or its corresponding function and destination device 920 or its corresponding function. In these embodiments, the source device 910 or its corresponding function and the destination device 920 or its corresponding function may be implemented using the same hardware and / or software, or using separate hardware and / or software, or any combination thereof.

[0225] Based on the description, it is obvious to the technicians that... Figure 11 The presence and (precise) division of different units or functions in the source device 910 and / or destination device 920 shown may vary depending on the actual device and application.

[0226] therefore, Figure 11 The source device 910 and destination device 920 shown are merely exemplary embodiments of the present invention, and the embodiments of the present invention are not limited to those described herein. Figure 11 The systems shown in the diagram.

[0227] The source device 910 and the destination device 920 may include any of a variety of devices, including any type of handheld or fixed device, such as a laptop or laptop computer, mobile phone, smartphone, tablet or tablet computer, camera, desktop computer, set-top box, television, display device, digital media player, video game console, video streaming device, broadcast receiver device, etc., and may or may not use any type of operating system.

[0228] This application involves numerous applications of neural networks. Therefore, for ease of understanding, the related terms and concepts, such as neural networks, involved in the embodiments of this application will be introduced below.

[0229] (1) Neural Network A neural network can include neurons. A neuron can be an operational unit that takes xs and an intercept of 1 as input, and the output of the operational unit can be as follows:

[0230] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neuron. f is the activation function of the neuron, which is used to introduce nonlinear features into the neural network to convert the input signal in the neuron into an output signal. The output signal of the activation function can be used as the input to the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many individual neurons together. Specifically, the output of a neuron can be the input of another neuron. The input of each neuron can be associated with the local receptive field of the previous layer to extract features from the local receptive field. The local receptive field can be a region that includes several neurons.

[0231] (2) Deep Neural Networks Deep neural networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers. There's no specific metric for "many." DNNs are categorized based on the location of the layers; the neural network in a DNN can be divided into three types: input layer, hidden layer, and output layer. Typically, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected. Specifically, any neuron in layer i is guaranteed to be connected to any neuron in layer (i+1). Although DNNs appear complex, their operation at each layer is actually not complicated and can be simply represented by the following linear relationship expression: ,in, It is the input vector. It is the output vector. is the bias vector, W is the weight matrix (also called coefficients), and α() is the activation function. At each layer, the output vector... By processing the input vector This is obtained by performing this simple operation. Because there are many layers in a DNN, there are also many coefficients W and bias vectors. The parameters in a DNN are defined as follows: Taking the coefficient W as an example. Assuming a three-layer DNN, the linear coefficient from the fourth neuron in the second layer to the second neuron in the third layer is defined as... The superscript 3 indicates the layer where coefficient W resides, and the subscript corresponds to the third layer index 2 for output and the second layer index 4 for input. In summary, the coefficients from the k-th neuron in layer (L–1) to the j-th neuron in layer L are defined as... It should be noted that the input layer has no parameter W. In deep neural networks, more hidden layers enable the network to better describe complex real-world cases.

[0232] In theory, a model with more parameters has higher representation complexity and greater "capacity," indicating that the model can perform more complex learning tasks. Training a deep neural network is the process of learning the weight matrix. The ultimate goal of training is to obtain the weight matrix of all layers of the trained deep neural network (including the weight matrix of the vector W of many layers).

[0233] (3) Convolutional Neural Network A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN includes a feature extractor, which comprises convolutional layers and subsampling layers. The feature extractor can be viewed as a filter. The convolution process can be considered as performing convolution on an input image or a convolutional feature plane (feature map) using a trainable filter. A convolutional layer is a layer of neurons in a CNN where the input signal is processed by convolution. In a convolutional layer of a CNN, a neuron may only be connected to some neurons in neighboring layers. A convolutional layer typically includes multiple feature planes, each of which may include a number of neurons arranged in a rectangular pattern. Neurons in the same feature plane share weights. These shared weights are the convolutional kernels. Weight sharing can be understood as the way image information is extracted being location-independent. An implicit principle here is that the statistical information of one part of an image is the same as the statistical information of other parts. This means that image information learned in one part can also be used in another part. Therefore, the same learned image information can be used for all locations in the image. In the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0234] Convolutional kernels can be initialized as matrices of random size. During the training of a convolutional neural network, the kernels can learn appropriate weights. Furthermore, a direct benefit of weight sharing is that it reduces the connections between layers in the convolutional neural network and lowers the risk of overfitting.

[0235] (4) Recurrent Neural Network Recurrent neural networks (RNNs) are used to process sequential data. In traditional neural network models, each layer from the input layer to the hidden layer and then to the output layer is fully connected, but the nodes in each layer are not connected. This common type of neural network solves many intractable problems, but it still cannot solve many others. For example, if you want to predict a word in a sentence, you usually need to use the previous word because adjacent words in a sentence are not independent. The reason RNNs are called recurrent neural networks is that the current output of a sequence is also related to the previous outputs of the sequence. A concrete representation is that the network remembers previous information and applies that information to the calculation of the current output. Specifically, the nodes in the hidden layer are connected, and the input to the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous time step. Theoretically, RNNs can handle sequential data of any length. The training of RNNs is the same as that of traditional CNNs or DNNs. The backpropagation algorithm is also used, but the difference is that if the RNN is expanded, the parameters such as W of the RNN will be shared. This is different from the traditional neural networks described in the above example. Furthermore, when using the gradient descent algorithm, the output of each step depends not only on the network in the current step but also on the network state in the previous few steps. The learning algorithm is called the back propagation through time (BPTT) algorithm.

[0236] Since we have convolutional neural networks (CNNs), why do we need recurrent neural networks (RNNs)? The reason is simple. In CNNs, elements are assumed to be independent of each other, and inputs and outputs are also independent, like cats and dogs. However, in the real world, many elements are interconnected. For example, stocks change over time. Another example is someone saying, "I like to travel, and my favorite place is Yunnan. If I have the opportunity, I will go." If given the chance to fill in the blank, people should know that the blank should be filled with "Yunnan." This is because people can infer the answer from the context. But how can machines do this? That's where RNNs come in. The purpose of RNNs is to enable machines to memorize like humans. Therefore, the output of an RNN needs to rely on both the current input information and historical memory information.

[0237] (5) Loss function During the training of a deep neural network, since the goal is for the network's output to be as close as possible to the actual expected predicted value, the current network's predicted value is compared to the actual expected target value. The weight vector of each layer in the neural network is then updated based on the difference between the predicted and target values ​​(of course, there is usually an initialization process before the first update; specifically, all layers of the deep neural network have pre-configured parameters). For example, if the network's predicted value is large, the weight vector is adjusted to decrease the predicted value, and this adjustment continues until the deep neural network can predict the actual expected target value or a value very close to the actual expected target value. Therefore, "how to obtain the difference between the predicted and target values ​​through comparison" needs to be predefined. This is the loss function or objective function. The loss function and objective function are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, the higher the output value (loss) of the loss function, the larger the difference. Therefore, training a deep neural network is a process of minimizing loss as much as possible.

[0238] (6) Backpropagation algorithm Convolutional neural networks can use the back propagation (BP) algorithm to correct the parameter values ​​in the initial super-resolution model during training, thereby reducing the error loss in the reconstructed super-resolution model. Specifically, the input signal is propagated forward until an error loss occurs at the output, and the parameters in the initial super-resolution model are updated based on the backpropagation error loss information to bring the error loss to converge. The backpropagation algorithm is a backpropagation process centered on the error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

[0239] A variational autoencoder (VAE) is a generative model that combines the ideas of autoencoders and probabilistic graphical models to learn the underlying distribution of input data. In a VAE, it is assumed that input data x is generated by latent variables y. In the encoder, input data x is mapped to the distribution of latent variables y.

[0240] Figure 12 An example illustrates the VAE framework. In the figure, encoder 1001 maps image x to a latent representation using the function y=f(x). The latent representation is then transformed into discrete values ​​( The quantizer 1002, entropy model, or superencoder / decoder (also known as superprior) estimate of Q(y)). The distribution of the data to obtain the minimum rate achievable through lossless entropy source decoding; and the use of arithmetic decoding (and a decoder that transforms the quantized latent representation into an image), =g( t)) will be beyond the prior and edge information Included in the bitstream.

[0241] although Figure 12 The input data is an image, but this application is not limited to image or video compression, and can also be applied to object detection, image generation and recognition systems.

[0242] exist Figure 12 In this network, two subnets are cascaded together. The first network consists of processing units 1001, 1002, 1004, 1005, and 1006. Units 1001, 1002, and 1005 are referred to as an autoencoder / decoder or simply an encoder / decoder network. The second subnet consists of units 1003 and 1007 and is called a superencoder / decoder.

[0243] Figure 12 The encoder and decoder are described in a single diagram. Figure 13 The encoder is described. Figure 14 The decoder components of the VAE framework are described. The encoder outputs are bitstream 1 and bitstream 2, where bitstream 1 is the output of the encoder's first subnet and bitstream 2 is the output of the encoder's second subnet.

[0244] Similarly, in Figure 14 In the process, two bitstreams are received as input, and a result is generated at the output. That is, to reconstruct (decoded) the image.

[0245] Most deep learning-based image / video compression systems reduce the dimensionality of the signal before converting it into binary numbers (bits). For example, in a VAE framework, the encoder is a non-linear transformation that maps the input image x to y, where the width and height of y are smaller than the width and height of x. Because y has a smaller width and height, it is smaller in size, the signal size is reduced, and therefore it is easier to compress the signal y. The general principle of compression is... Figure 15 The following example illustrates this. The latent space is the output of the encoder and the input of the decoder, representing the compressed data. It should be noted that the size of the latent space is much smaller than the size of the input signal.

[0246] The decrease in the magnitude of the input signal is as follows Figure 15 For example, Figure 15 A deep learning-based encoder and decoder are shown. Figure 15 In this diagram, the input image x corresponds to the input data, i.e., the encoder's input. The transformed signal y corresponds to the latent space, which has a smaller dimension than the input signal. Each column of circles represents a layer in the encoder or decoder's processing chain. The number of circles in each layer indicates the magnitude or dimension of the signal at that layer.

[0247] from Figure 15As can be seen, the encoding operation corresponds to reducing the size of the input signal, while the decoding operation corresponds to reconstructing the image of the original size.

[0248] Furthermore, the encoding operation also depends on the number of channels. For example, for an encoder with one luminance input channel, the potential space typically has 128 to 160 channels. The number of elements is usually reduced by encoding operations (width multiplied by height multiplied by channels).

[0249] One method to reduce signal size is downsampling. Downsampling is the process of reducing the sampling rate of the input signal. For example, if the size of the input image is h and w, and the downsampled output is h² and w², then at least one of the following holds true: h2 <h w2 <w The reduction in signal size typically occurs incrementally along the chain of processing layers, rather than all at once. For example, if the input image x has dimensions h and w (representing height and width), and the latent space y has dimensions h / 16 and w / 16, then during encoding, the size reduction may occur across four layers, where each layer halves the signal size in each dimension.

[0250] State-of-the-art deep learning-based video / image compression methods use multiple downsampling layers. For example, Figure 16 The VAE framework utilizes six downsampling layers, labeled 1401 to 1406. Layers involving downsampling are indicated by down arrows in their layer descriptions. The layer description "Conv N×5×5 / 2↓" indicates that the layer is a convolutional layer with N channels and a kernel size of 5×5. As mentioned above, 2↓ indicates that downsampling by a factor of 2 is performed in this layer. Downsampling by a factor of 2 reduces one dimension of the input signal by half at the output. Figure 16 In the diagram, 2↓ indicates that both the width and height of the input image are reduced by half. Due to the presence of six downsampling layers, if the width and height of the input image x(1414) are represented by w and h, then the output signal... The width and height of (1413) are equal to w / 64 and h / 64, respectively.

[0251] Feed the response to h a In the first step, the distribution of the standard deviation in z is summarized. Then, z is quantized, compressed, and sent as side information. The encoder then uses the quantized vector... To estimate This is used to obtain the spatial distribution of the standard deviation of the probability values ​​(or frequency values) of arithmetic coding (AE), and is used to compress and transmit the quantized image representation. (or potential representation). The decoder first recovers from the compressed signal. Then, the decoder uses h s Get This provides it with the correct probability estimate for successful recovery. Then, the decoder will Feed to g s In order to obtain the reconstructed image.

[0252] exist Figure 16 The diagram also shows a decoder including upsampling layers 407 to 412. Another layer 420, implemented as a convolutional layer but not upsampling the received input, is provided between the upsampling layers 411 and 410 in the order of input processing. A corresponding convolutional layer 430 for the decoder is also shown. Such a layer could be provided in a neural network to perform operations on the input that modify specific characteristics without changing the input size. However, it is not necessary to provide such a layer.

[0253] When viewed from the processing order of bitstream 2 through the decoder, the upsampling layers operate in reverse order, from upsampling layer 1412 to upsampling layer 407. Each upsampling layer is shown here to provide an upsampling ratio of 2, indicated by ↑. Of course, not all upsampling layers necessarily have the same upsampling ratio, and other upsampling ratios, such as 3, 4, 8, etc., can also be used. Layers 1407 to 1412 are implemented as convolutional layers (conv). Specifically, since they may be designed to provide the opposite operation to the encoder on the input, the upsampling layers can apply a deconvolution operation to the received input, increasing its size by a factor corresponding to the upsampling ratio. However, the invention is generally not limited to deconvolution, and upsampling can be performed in any other way, such as by bilinear interpolation between two adjacent samples, or by copying nearest neighbor samples, etc.

[0254] This application is applicable to both end-to-end AI codecs and hybrid AI codecs. For example, in a hybrid AI codec, filtering operations (filtering for reconstructed images) can be performed using a neural network (NN). This application is applicable to such NN-based processing modules. Typically, this application can be applied to all or part of a compression and decompression process if at least a part of the process includes an NN, and if the NN includes convolution or transposed convolution operations.

[0255] In image and video compression systems, compression and decompression of very large input images are typically performed by dividing the input image into multiple parts. VVC and HEVC employ such partitioning methods, for example, dividing the input image into blocks or wavefront processing units.

[0256] In traditional video decoding systems using chunking, the input image is typically divided into multiple rectangular portions. These portions can be processed independently, and the bitstream used to decode each portion can be encapsulated into independent decodeable units. Therefore, the decoder can independently parse (i.e., obtain the syntax elements needed for sample reconstruction) each bitstream (corresponding to portion 1 and portion 2) and can independently reconstruct samples for each portion.

[0257] Using chunks allows decoding operations to be performed independently of each other, either in whole or in part. The advantage of independent processing is that multiple identical processing cores can be used to process the entire image, thus increasing processing speed. If the processing power of a single core is insufficient to handle a large image, the image can be divided into multiple parts, each requiring fewer resources to process. In this case, even if the entire image cannot be processed due to resource constraints, the less powerful processing unit can still handle each part.

[0258] According to the encoding method provided in this application, the input data is divided into several segments (hereinafter referred to as "input segments"). These input segments can be processed individually. For example, the input segments can be processed by an encoding network. More than one identical encoding network can be used to process the input segments.

[0259] Figure 17 A flowchart illustrating an encoding method provided by some embodiments of this application is shown. In some embodiments, Figure 17 The method shown can be performed by an electronic device. The electronic device is used to encode input data. Therefore, in some embodiments, the electronic device may be referred to as an encoding device. In other embodiments, Figure 17 The method shown can be executed by components of the encoding device. For example, Figure 17 The method shown can be implemented by a processor, chip, system-on-chip (SoC), or processing circuitry of an encoding device. For simplicity, Figure 17 The method shown will be described as being performed by the encoding device.

[0260] 701. The encoding device divides the input data into M input segments. M is a positive integer greater than 1. Each of the M input segments has an overlapping region with its adjacent input segments. For example, input segment 1 is the first input segment of the M input segments, input segment 2 is the second input segment of the M input segments, and input segment 3 is the third input segment of the M input segments. In other words, input segment 1 is adjacent to input segment 2, and input segment 2 is adjacent to both input segment 1 and input segment 3. Input segment 1 may include regions 1 and 2, input segment 2 may include regions 2, 3, and 4, and input segment 3 may include regions 4 and 5. For input segment 1 and input segment 2, region 2 is an overlapping region. For input segment 2 and input segment 3, region 4 is an overlapping region. Therefore, it can be described that input segment 1 has one overlapping region with its adjacent input segment (i.e., input segment 2), input segment 2 has two overlapping regions with its adjacent input segments (i.e., input segment 1 and input segment 3), and input segment 3 has one overlapping region with its adjacent input segment (i.e., input segment 2).

[0261] In some embodiments, the size of the input segments is fixed. That is, any two input segments out of the M segments are the same size. In other embodiments, the size of the input segments is not fixed. In other words, any two input segments out of the M segments can have different sizes.

[0262] Furthermore, in some embodiments, the size of the input segment can be predetermined. In other embodiments, the size of the input segment can be set by the user. In still other embodiments, the size of the input segment can be negotiated between the encoding device and the corresponding decoding device. In yet another embodiment, the size of the input segment can be determined according to preset rules. For example, the size of the input segment can be determined based on the size of the input data. The size of the input segment and the method for determining the size of the input segment are not limited thereto.

[0263] 702, The encoding device processes M input segments using an encoding network to determine M analysis segments.

[0264] For example, in some embodiments, the encoding network may include Figure 13 The encoder 1001 and quantizer 1002 are shown. In some embodiments, the encoder 1001 can perform analysis and transformation on each of the M input segments and output M transformed input segments. The quantizer 1002 can transform the resulting M transformed input segments into M discrete value sets, where the M discrete value sets are M analysis segments.

[0265] In other embodiments, the encoding network may consist only of analytic transform operations. In other words, the encoding device can perform analytic transforms on M input segments, resulting in M ​​analytic segments.

[0266] In other embodiments, in addition to encoder 1001 and quantizer 1002, the encoding network may also include one or more additional processing units. For example, one of the additional processing units is used to correct, remove, or add one or more components in one or more segments.

[0267] 703, The encoding device determines the representation of the input data in the latent space based on M analysis segments and M input segments.

[0268] As mentioned earlier, the encoding network can include analytic transformation operations, which can transform input segments into the latent space. Therefore, M analytic segments can represent M input segments in the latent space.

[0269] In some embodiments, the encoding device can determine M core segments based on M analysis segments and M input segments. The j-th core segment (j=1, ..., M) is determined from the j-th analysis segment and the j-th input segment among the M core segments. The encoding device then determines the representation of the input data by assembling the M core segments. Each core segment in the M core segments has no overlapping region with its adjacent core segments. In other words, the encoding device prunes the analysis segments to obtain the core segments. The encoding device then concatenates the core segments into a representation.

[0270] 704, the encoding device divides the representation of the input data into N entropy segments.

[0271] Each of the N entropy segments has no overlapping regions with its neighboring entropy segments. N is a positive integer greater than 1.

[0272] In some embodiments, the encoding device can determine N shift segments based on M input segments, determine N potential shift segments based on the N shift segments and the alignment parameters of the analysis transform, and determine the position information of each entropy segment among the N potential shift segments and N shift segments.

[0273] 705. The encoding device determines the bitstream by encoding N entropy segments.

[0274] In other words, the bitstream can be used to send N entropy segments to the decoding device. The bitstream can also include segment identification information. Segment flatness information is used to indicate that the bitstream includes N entropy segments.

[0275] For example, a 1-bit field can be used to carry segment flatness information. If the value of this field is 1, the bitstream carries N entropy segments. If the value of this field is 0, the bitstream does not carry N entropy segments. In other words, if the value of this field is 1, the input data is divided into several segments, and these segments are processed according to the embodiments provided in this application. If the value of this field is 0, the input data is not processed according to the embodiments provided in this application.

[0276] In addition to information about the flatness of the segments, the bitstream also carries some characteristics of the input data. For example, the bitstream can carry information such as the size of the input segments, the size of the input data, the size of the overlapping regions, and the alignment parameters of the analysis transform.

[0277] In some embodiments, the bitstream carrying N entropy fragments can be Figure 12 , Figure 13 , Figure 14 or Figure 16 The bitstream shown is 1.

[0278] Figure 18 A flowchart of a decoding method provided by some embodiments of this application is shown. In some embodiments, Figure 18 The method shown can be performed by an electronic device. The electronic device is used to decode the received bitstream. Therefore, in some embodiments, the electronic device may be referred to as a decoding device. In other embodiments, Figure 18 The method shown can be executed by components of the decoding device. For example, Figure 18 The method shown can be implemented by a processor, chip, system-on-chip (SoC), or processing circuitry of the decoding device. For simplicity, Figure 18 The method shown will be described as being performed by the encoding device.

[0279] 801, the decoding device obtains the entropy fragment i from the bitstream.

[0280] The bitstream is determined by the encoding device. For example, the encoding device can perform... Figure 17 The encoding method is shown, and the bitstream is determined. The decoding device receives the bitstream from the encoding device and parses the bitstream to obtain the entropy fragment i.

[0281] Entropy segment i is one of the N entropy segments carried in the bitstream. N is a positive integer greater than 1, and i = 1, ..., N. Entropy segment i does not overlap with its neighboring entropy segments. In other words, entropy segment i is any one of the N entropy segments. Each of the N entropy segments does not overlap with its neighboring entropy segments. The N entropy segments can constitute a representation of the input data in the latent space.

[0282] In some embodiments, the decoding device can determine the position information corresponding to entropy segment i based on the input segment corresponding to entropy segment i, wherein the input segment corresponding to entropy segment i is one of M input segments, the M input segments constitute the input data, each of the M input segments includes at least one overlapping region with its adjacent input segments, and M is a positive integer greater than 1. Then, the decoding device can obtain entropy segment i from the bitstream based on the position information corresponding to entropy segment i.

[0283] In some embodiments, the decoding device can determine the shift segment i based on the size of the input segment corresponding to the entropy segment i and the overlapping region, determine the potential shift segment i based on the alignment parameters of the shift segment i and the synthesis transform, and determine the position information corresponding to the entropy segment i based on the potential shift segment i and the shift segment i.

[0284] 802, the decoding device determines the synthesized segment i based on the entropy segment i and / or at least one entropy segment adjacent to the entropy segment i.

[0285] Synthetic fragment i includes at least one overlapping region with its adjacent synthetic fragments.

[0286] For example, in some embodiments, the decoding device can determine the position information of the synthesized segment i based on the alignment parameters of the synthesized transform and the input segment corresponding to the synthesized segment i, wherein the input segment corresponding to the synthesized segment i is one of M input segments constituting the input data, and each of the M input segments includes at least one overlapping region with its neighboring input segments. Then, based on the position information of the synthesized segment i, the decoding device determines elements of the synthesized segment i from the entropy segment i and / or at least one entropy segment adjacent to the entropy segment i.

[0287] 803, the decoding device can determine the reconstructed fragment i by executing the decoding network based on the synthesized fragment i.

[0288] A decoding network is a network corresponding to an encoding network. A decoding network includes one or more operations corresponding to one or more operations included in the encoding network. For example, an encoding network includes an analysis transform, while a decoding network includes a synthesis transform operation corresponding to the analysis transform operation. For example, a decoding device can perform a synthesis transform operation on a synthesized segment i to obtain a reconstructed segment.

[0289] In some embodiments, the decoding network may include one or more additional processing units. For example, the decoding network may include a post-filter. The post-filter can acquire the output of the synthesis transform operation and process the acquired segment. Under this condition, the output of the post-filter is the reconstructed segment. The main purpose of the synthesis transform operation is to transform the synthesized segment in the latent space to the signal space. The synthesized segment includes overlapping regions. The synthesis transform operation does not change this characteristic. Therefore, the transformation result of the synthesized segment (hereinafter referred to as the transformed synthesized segment) may also include overlapping regions. The post-filter can be used to remove overlapping regions from the transformed synthesized segment. Therefore, the reconstructed segment will not have overlapping regions.

[0290] according to Figure 18 The method shown allows the pipelined processing to be performed by a decoding device. The decoding device can acquire a first synthesized segment from M synthesized segments and acquire the reconstructed segment corresponding to the first synthesized segment. Then, the decoding device can acquire a second synthesized segment from the M synthesized segments and acquire the reconstructed segment corresponding to the second synthesized segment, and so on. Therefore, the decoding device does not need to acquire all synthesized segments to acquire the reconstructed segment.

[0291] Figure 19 The assembly line process is shown.

[0292] The decoding device can acquire M reconstructed segments. Each reconstructed segment has no overlapping region with its adjacent reconstructed segments. The decoding device can stitch the M reconstructed segments to obtain reconstructed data corresponding to the input data. In other words, the decoding device can trim overlapping regions based on the synthesized segments and stitch the trimmed segments to obtain the reconstructed data.

[0293] To help those skilled in the art better understand the technical solution of this application, the following is an embodiment using images as input data. The input image can be divided into several image blocks.

[0294] In some embodiments, the decoding device can obtain the block size, the input image size, and the overlap size, and determine a number of image blocks based on the block size, the input image size, and the overlap size. For example, image blocks can be determined as follows (hereinafter referred to as the "first block determination operation"): for tile_start_y in range(0, image_height - overlap, tile_height -overlap): for tile_start_x in range(0, image_width - overlap, tile_width -overlap): height = min(tile_height, image_height - tile_start_y) width = min(tile_width, image_width - tile_start_x) im_tile i =(tile_start_x, tile_start_y, width, height) im_tile i This refers to the i-th image block among several image blocks. The first block determination operation can also be performed by the encoding device. For example, the encoding device can perform a first block determination operation to divide the input image into multiple image blocks. The potential blocks corresponding to the image blocks can then be obtained by processing the image blocks in parallel using an encoding network.

[0295] For example, if the input image size is 3120×2424, then the overlap is 64, and the image block size is 1024×1024. Table 1 shows the image blocks corresponding to the input image.

[0296] Table 1

[0297] After determining the image blocks, the encoding device can determine the potential blocks (i.e., analysis segments) corresponding to the image blocks. Each image block in the signal space has a corresponding potential block in the potential space. The potential block corresponding to the i-th image block can be determined as follows (hereinafter referred to as the "second block determination operation"): image_tile = im_tile i lat_tile_start_y = image_tile.position.y / / alignment_size lat_tile_start_x = image_tile.position.x / / alignment_size if image_tile.size.height % alignment_size: height = math.ceil(image_tile.size.height / alignment_size) else: height = image_tile.size.height / / alignment_size if image_tile.size.width % alignment_size: width = math.ceil(image_tile.size.width / alignment_size) else: width = image_tile.size.width / / alignment_size lat_tile i = (lat_tile_start_x, lat_tile_start_y, width, height) lat_tile i It is the i-th potential block.

[0298] Using lat_tile, the corresponding region of the latent space is extracted and read from / written to the bitstream. For this purpose, overlapping portions of the blocks are ignored to avoid reading / writing the same element multiple times.

[0299] As mentioned earlier, each potential block corresponds to an image block. Therefore, when the input image is divided into M image blocks, the encoding device can determine M potential blocks.

[0300] For example, when alignment_size=16, the potential blocks corresponding to image blocks shown in Table 1 are shown in Table 2.

[0301] Table 2

[0302] After determining the potential blocks, the encoding device can determine the core blocks (i.e., core segments) based on the potential blocks and image blocks. The core blocks do not include overlapping regions. The encoding device can then stitch the core blocks together to obtain a representation of the input image in the latent space.

[0303] The following method extracts the non-overlapping portions of image blocks. The input is the image block `im_tile` obtained from `tg_bitstream` and `tg_lat_bitstream`. i and its corresponding potential spatial block lat_tile i .

[0304] The core block can be determined as follows (hereinafter referred to as the "third block determination operation"): img_height = xxx # Height of the image img_width = xxx # Width of the image half_overlap = xxx # Half of the block overlap amount alignment_size = xxx # Alignment size described above (lat_tile_start_x, lat_tile_start_y, lat_tile_width, lat_tile_height)= lat_tile i (tile_start_x, tile_start_y, tile_width, tile_height) = im_tile i overlap_left = 0 if tile_start_x == 0 else half_overlap / / alignment_size overlap_top = 0 if tile_start_y == 0 else half_overlap / / alignment_size overlap_right = 0 if tile_start_x + tile_width>= img_width else half_overlap / / alignment_size overlap_bottom = 0 if tile_start_y + tile_height>= img_height elsehalf_overlap / / alignment_size core_tile =( lat_tile_start_x + overlap_left, lat_tile_start_y + overlap_top, lat_tile_width - overlap_right - overlap_left, lat_tile_height - overlap_bottom - overlap_top, ) Based on the image blocks shown in Table 1 and the potential blocks shown in Table 2, the core blocks shown in Table 3 can be determined.

[0305] Table 3

[0306] The third block determination operation is used to determine the core blocks of potential blocks, which the encoding device uses to trim potential blocks. The third determination operation can also be performed by the decoding device. As mentioned earlier, the transformation result of a composite block (composite segment) can also include overlapping regions. The decoding device can perform the third block determination operation on the transformed composite block (i.e., the transformation result of the composite block) to trim overlapping regions. When the decoding device performs the third block determination operation to trim overlapping regions, the third block determination operation can be adjusted as follows: lat_tile_i needs to be the same as im_tile_i, and alignment_size=1.

[0307] The blocks used to obtain the reconstructed image are different from the blocks read from the bitstream. Figure 20 The diagram shows the blocks used to determine the reconstructed image and the blocks read from the bitstream.

[0308] Figure 20 The solid line blocks shown are used to determine the reconstructed image.

[0309] Dashed-line blocks are blocks read from the bitstream. Pipeline processing is possible based on dashed-line blocks, starting from the top left corner and processing each line. After reading the dashed-line blocks from the bitstream, the synthesis of the corresponding core blocks can begin directly without causing errors in reconstruction. Dashed-line blocks are shifted versions of the blocks used for synthesis. Special cases may occur at image boundaries: the first block may be larger, and the last block may be empty.

[0310] The dashed line blocks can be determined based on the image blocks. The decoding device can determine the shifted blocks corresponding to the image blocks. Then, the decoding device determines the potential blocks corresponding to the shifted blocks (hereinafter referred to as shifted potential blocks). Finally, the decoding device can determine the dashed line blocks based on the shifted blocks and the shifted potential blocks. The method for determining the dashed line blocks based on the shifted blocks and the shifted potential blocks is the same as the method for determining the core blocks based on the image blocks and the potential blocks. For the sake of simplicity, it will not be described again here. The dashed line blocks can also be called shifted core blocks. The method for determining the shifted potential blocks based on the shifted blocks is the same as the method for determining the potential blocks based on the image blocks. For the sake of simplicity, it will not be described again here.

[0311] Shift blocks can be determined based on image block division. One possible method for determining shift blocks is as follows: img_height = xxx # Image height img_width = xxx # Image width half_overlap = xxx # Half of the overlap shifted_image_tiles = [] reached_bottom_border = False for tiles_row in tg_synth: shifted_tile_row = [] reached_right_border = False if reached_bottom_border: shifted_tile_row = [None for im_tile in tiles_row] # None indicates that this tile will not write / read anything to / from the bitstream. This has already been done by the second-to-last tile in the row / column (which may also be the first and only tile).

[0312] shifted_image_tiles.append(shifted_tile_row) continue for im_tile i in tiles_row: (tile_start_x, tile_start_y, tile_width, tile_height) = im_tilei if reached_right_border: shifted_tile_row.append(None) continue if tile_start_x == 0: shifted_start_x = 0 if tile_width + half_overlap <img_width: width = tile_width + half_overlap else: width = img_width reached_right_border = True else: shifted_start_x = tile_start_x + half_overlap if tile_width<img_width - shifted_start_x: width = tile_width else: width = img_width - shifted_start_x reached_right_border = True if tile_start_y == 0: shifted_start_y = 0 if tile_height + half_overlap<img_height: height = tile_height + half_overlap else: height = img_height reached_bottom_border = True else: shifted_start_y = tile_start_y + half_overlap if tile_height<img_height - shifted_start_y: height = tile_height else: height =img_height - shifted_start_y reached_bottom_border = True shifted_tile_i =(shifted_start_x, shifted_start_y, width, height) shifted_tile_row.append(shifted_tile_i) shifted_image_tiles.append(shifted_tile_row) Table 4 shows the shift blocks determined based on the image blocks shown in Table 1.

[0313] Table 4

[0314] Table 5 shows the potential shift blocks determined according to the shift blocks shown in Table 4.

[0315] Table 5

[0316] Table 6 shows the shift core blocks determined based on the shift blocks shown in Table 4 and the shift potential blocks shown in Table 5.

[0317] Table 6

[0318] Encoding devices perform a similar process to determine the blocks to be written into the bitstream (i.e., shift core blocks). For the sake of simplicity, this will not be elaborated upon here.

[0319] Figure 21 The diagram illustrates entropy blocks (i.e., shifted core blocks), synthetic blocks (i.e., potential blocks), and reconstructed blocks (i.e., core blocks).

[0320] Figure 22 The diagram illustrates entropy partitioning, synthesis partitioning, and reconstruction partitioning under normal conditions.

[0321] Figure 23 The entropy block, synthesis block, and reconstruction block are shown in special cases.

[0322] Therefore, in this embodiment of the invention, segmentation is also used in entropy decoding. The data required to decode the segments is encoded separately for each segment. This allows the decoder to perform pipelined processing. Here, pipelined processing includes reading segments from the bitstream and then synthesizing those segments, such as... Figure 19 As shown (and later in context) Figure 29 (As shown in the image).

[0323] The latent spatial blocks used for image synthesis overlap with each other through one or more latent spatial regions. This overlap avoids block boundaries visible in the decoded image and has no impact on decoding performance. Because of this overlap, the same block set cannot be directly used to determine which samples the encoder should write to the bitstream. This would result in writing the same samples multiple times, increasing the bitrate. This problem is solved by deriving a new set of "entropy blocks" (entropy fragments) based on the block set used for synthesizing the image (synthetic segment). For each synthetic segment, an entropy block is derived that defines which samples need to be written / read from the bitstream in order to decode the corresponding synthetic segment. This is done in such a way that each latent spatial sample only needs to be written / read from the bitstream once.

[0324] In the decoder, luminance and chrominance are decoded separately. However, decoding the chrominance component requires chrominance and luminance latent spaces (i.e., CCS).

[0325] The following text first derives the synthetic block graph (or block grid). It is then used as the basis for deriving the block grid used in entropy decoding.

[0326] Several methods are possible for obtaining the composite patch graph. Some exemplary implementations are: 1. Only the block diagram is explicitly indicated for the principal components. The same block diagram is used for other components (for YUV420 and CCS, the principal component will be luminance / Y, and the other components will be chrominance / UV).

[0327] 2. Explicitly indicate the block diagram for each component of the image.

[0328] When a block diagram is explicitly specified, this can be done in one of the following exemplary ways: 1. Use a regular grid with blocks of the same size (except for the bottom and right boundaries). Indicate the offset and size values ​​from which the block diagram can be derived (see below).

[0329] 2. Use a regular grid with tiles of the same size (except for the bottom and right boundaries). Use, but do not directly indicate, offset and size values. Instead, size and offset values ​​are derived from the already decoded level definitions. The tile map can be derived from the size and offset values ​​(see below).

[0330] 3. Use an arbitrary grid with chunks. First, indicate the number of chunks, then indicate the position and size of each chunk (overlap will be implicitly included in this indication).

[0331] If the block diagram is indicated by values ​​for block size (block width equals height) and overlap (expressed as signal space size / coordinates), then the N overlapping regions can be derived as follows: for tile_start_y in range(0, image_height - overlap, tile_height -overlap): for tile_start_x in range(0, image_width - overlap, tile_width -overlap): height = min(tile_height, image_height - tile_start_y) width = min(tile_width, image_width - tile_start_x) im_tile i =(tile_start_x, tile_start_y, width, height) Each block used for reading / writing the bitstream can be synthesized independently. Overlapping regions should not be encoded twice (as this would increase the bitrate). Furthermore, overlapping regions have already been read from the bitstream before a specific block is synthesized.

[0332] Figure 24(a) summarizes the tiled grid used for compositing. The image boundary is shown in the upper left corner. Solid lines represent core tiles (i.e., the areas that will be retained after compositing). Dashed lines indicate overlapping areas. Dark fill indicates the first core tile. Light fill indicates the first overlapping tile included for compositing. If core tiles are used for reading / writing the bitstream, not all necessary elements are available during compositing when using pipelined processing: samples from overlapping areas are also used. However, if overlapping areas are used for reading / writing the bitstream, elements will be written / read multiple times (increasing the bitrate).

[0333] Figure 24(b) summarizes the block grid used for reading / writing the bitstream. The image boundary is shown in the upper left corner. Solid lines represent core blocks. Dashed lines indicate blocks used for reading / writing the bitstream. Pipeline processing is possible, starting from the upper left corner and processing each line. After reading the dashed block from the bitstream, the synthesis of the corresponding core block can begin directly without errors in reconstruction. Essentially, this block grid is a shifted version of the block grid used for synthesis. However, a special case arises at the image boundary. The first block is large, and the last block may be empty (the penultimate block has already been read from / written to the bitstream).

[0334] Therefore, the arrangement of the chunks used for synthesis (which can be called the chunk grid) is slightly different from the chunk grid used for reading / writing the bitstream. Roughly, the latter is a bottom-right shifted version of the former. More precisely, the chunk grid used for reading / writing the bitstream (tg_bitstream) is derived from the chunk grid used for synthesis: tg_synth[y][x] Here, tg_synth is a 2D array used to store the parameters of each block (im_tile). i The derivation of the shift block can be described as above.

[0335] In one example, the blocks used in entropy decoding can be indicated as shown in Tables 7 and 8 below. A flag can be added to the block header of each component. If the flag is equal to 1, the block is used in entropy decoding.

[0336] Table 7

[0337] Table 8

[0338] In another architecture, the decoder can include a context module. Using a context module enables better predictions, thereby reducing the amount of additional information that needs to be encoded, i.e., requiring a lower bitrate for similar quality.

[0339] like Figure 25 The diagram schematically illustrates that the inputs to the context model 2301 (MCM) in the decoder are the residual 2302 read from the bitstream and the prediction information 2303 (from the superdecoder). The output of the context model is... (Image transformed to latent space).

[0340] like Figure 26 As shown, when the context module is part of the pipeline in a decoder that uses chunks, the chunks output by the context module include one or more overlapping regions used by the synthesis transform. The data in the overlapping regions of adjacent chunks may differ. Therefore, this data is not stored as a large tensor but remains independent. The reconstructed chunks are only merged into a single image after the synthesis transform is performed.

[0341] In traditional methods, chunking is not used within context modules. This breaks the pipeline because... Generated by the context module. Previously, it was simply a matter of adding residuals and forecasts (means) instead of context, which was a pointwise operation that didn't require special attention to the pipeline.

[0342] In embodiments of the invention, the same blocks as the synthesis transform are used as input to the context module. These blocks include one or more overlapping regions with their adjacent blocks. This allows for the use of a pipelined process. Only those generated by the context... The non-overlapping parts are used to form reconstruction blocks.

[0343] Figure 27 The diagram schematically illustrates the shape of three distinct blocks as they pass through the pipeline in the decoder. In the top row, the residual block is read from the bitstream. The shaded area shows the previously parsed data. In the second row, the shaded area shows the blocks used to extract the residual and prediction inputs for the context module. In the third row, the shaded area shows the blocks used for the synthesis transform. The image is divided into blocks. In the bottom row, the shaded areas represent reconstructed samples stored in the image domain, with overlapping areas ignored. The data associated with the shaded blocks shown in the first, second, and fourth rows is used for entropy blocks, residuals, prediction inputs, and reconstruction blocks, and can be stored as a large tensor / image. This is particularly useful when using context modules. It is impossible to correlate data across blocks. Here, the overlapping areas may differ for different blocks.

[0344] In one example, the context module ( Figure 25 and Figure 26 MCM 2301 in k stages (MCM 2301) k The process runs in a context module, where k = 0, 1, ..., k. Before using the context module, the tensor with the explicit prediction input μ and the residual tensor r are uniformly partitioned along the channel dimension into multiple tensors. and The context of the phase uses the already reconstructed data from the previous phase. The first average is predicted from a portion of the data. Then, the predicted average for the current stage is obtained by adding the predicted average for the current stage to the decoded / parsed residual for the current stage. The part.

[0345] In this example, in the first stage of the context, the context model MCM0 is used to predict the mean based on the explicit prediction of input 0 and a tensor of the same size initialized with 0. The mean is added to r0 to obtain... 0.

[0346] In the second phase of the context, the context model MCM1 is used based on explicit prediction of input 1 and tensor. 0. Predicted average. The average is added to r1 to obtain... 1.

[0347] In the third phase of the context, the context model MCM2 is used based on explicitly predicted input 2 and the spliced ​​tensor (along the channel dimension). 0, 1] Predict the average value. The average value is obtained by adding r². 2.

[0348] In the fourth phase of the context, the context model MCM3 is used based on explicitly predicted input 3 and the spliced ​​tensor (along the channel dimension). 0, 1, 2] Predict the average value. The average value is obtained by adding r3. 3.

[0349] There can be further stages (i.e., k > 3). For simplicity, four stages (k = 3) are described here.

[0350] The context model can also be included in the encoder. The encoder and decoder use the same context model neural network. However, as mentioned earlier (for the decoder), the model takes explicit prediction parameters and residuals as input and generates... In order to obtain input In addition to explicitly predicting parameters and generating residuals, the encoder also performs the following operations: In each stage, the average value is based on the explicit predicted input and the previous reconstruction. Predicted. Generated by previous stages of the context module. This can differ slightly from y because quantization is applied here. The y produced by the analytical transformation has not yet been quantized. These averages are then subtracted from y to obtain the residual for the current stage of the context.

[0351] Figure 28 This shows that the two blocks (residual and prediction input) input to the context module have the same total number of elements. That is, the residual block and the prediction input block have the same number of elements. Figure 28 In the example shown, the spatial dimension of the second context fragment i is one-quarter of the spatial dimension of the first context fragment i, but the number of channels of the second context fragment i is four times that of the first context fragment.

[0352] When the encoder and / or decoder includes a context module, the context is also part of the pipeline, such as... Figure 29As shown in the figure, residual block 0 is read from the bitstream, and its prediction information is obtained simultaneously. In the next stage of the pipeline, the context of block 0 is determined by inputting the residual and prediction information of block 0 into the context model. Simultaneously, residual block 1 is read from the bitstream, and its prediction information is obtained. In the next stage, block 0 is synthesized, and its context is determined by inputting the residual and prediction information of block 1 into the context model.

[0353] Figure 30 The diagram schematically illustrates the shape of three distinct blocks as they pass through the pipeline in the encoder. In the bottom row, a residual block is read from the bitstream, comprising one or more overlapping regions with one or more adjacent residual blocks. The shaded area represents previously parsed data. In the second row, the shaded area represents the blocks used for extracting the residual and prediction inputs used by the context module. In the third row, the shaded area represents the blocks used for the synthesis transform. The image is divided into blocks. In the bottom row, the shaded areas show the reconstructed samples in the image domain stored in the image, and overlapping areas are ignored.

[0354] At the decoding end, the input to the context module is the prediction information (from the superdecoder) and the residual read from the bitstream. These inputs are referred to in this paper as the first and second context segments / blocks. The output of the context module at the decoding end is... (Image transformed into the latent space). The modification to the decoder process is that the context is not applied to the entire latent space all at once, but rather block by block. The first and second context blocks are the inputs to the context module.

[0355] At the encoding end, the input to the context module is the prediction information (from the superdecoder) and (Input image transformed into the latent space). The output of the context module at the encoder is the residual, which is then encoded into the bitstream. The modification to the encoder process is that the context is not applied to the entire latent space all at once, but rather applied block by block.

[0356] like Figure 25 As shown, both inputs to the context module have one or more overlapping regions in the blocks, and the same is true for the output blocks. That is, the synthesized blocks ( Both the prediction input block and the residual block have one or more overlapping regions with one or more adjacent blocks of the same type. Overlapping regions are ignored when determining the reconstruction block (in the decoder) and the entropy block (encoded in the bitstream).

[0357] Therefore, this method supports pipelined reading and writing of the bitstream, application of context models to blocks, and synthesis of blocks.

[0358] Using chunking reduces the memory required to decode an image. In existing techniques, the entire latent space needs to be parsed / read from the bitstream before the first chunk can be decoded. Embodiments of this invention advantageously support pipelined decoding and synthesis of image regions using neural networks.

[0359] Figure 31 The decoding process is illustrated. The decoding device can determine four potential blocks ( Figure 31 (Only two potential blocks are shown), four image blocks and four core blocks are identified, and the reconstructed image is determined based on these four core blocks.

[0360] Figure 32 This is a schematic block diagram of an electronic device 1500 provided in some embodiments of this application. See also Figure 32 The electronic device 1500 includes an acquisition unit 1501 and a processing unit 1502.

[0361] The acquisition unit 1501 is used to acquire entropy segment i from the bit stream, wherein entropy segment i is one of N entropy segments, the N entropy segments constitute the representation of the input data in the latent space, entropy segment i and its adjacent entropy segments do not have overlapping regions, N is a positive integer greater than 1, i=1, ...,N.

[0362] The processing unit 1502 is used to determine a synthetic fragment i based on the entropy fragment i and / or at least one entropy fragment adjacent to the entropy fragment i, wherein the synthetic fragment i includes at least one overlapping region with the synthetic fragments adjacent to it.

[0363] The processing unit 1502 is further configured to determine the reconstructed fragment i by executing a decoding network based on the synthesized fragment i, wherein the reconstructed fragment i does not have an overlapping region with its adjacent reconstructed fragments.

[0364] The processing unit 1502 may be implemented by a processor. In some embodiments, the acquisition unit 1501 may be implemented by a processor.

[0365] For details on how to process the input image, please refer to the above embodiments, which will not be repeated here.

[0366] Figure 33 This is a schematic block diagram of an electronic device 1600 provided in some embodiments of this application. See also Figure 33 The electronic device 1600 includes a partitioning unit 1601 and a processing unit 1602.

[0367] The partitioning unit 1601 is used to partition the input data into M input segments, wherein each of the M input segments includes at least one overlapping region with its adjacent input segments, and M is a positive integer greater than 1.

[0368] The processing unit 1602 is used to process M input segments using an encoding network to determine M analysis segments, wherein the M analysis segments correspond one-to-one with the M input segments.

[0369] The processing unit 1602 is also configured to determine the representation of the input data in the latent space based on the M analysis segments and the M input segments.

[0370] The processing unit 1602 is further configured to divide the representation of the input data into N entropy segments, wherein each of the N entropy segments does not have an overlapping region with its adjacent entropy segments, and N is a positive integer greater than 1.

[0371] The processing unit 1602 is also used to determine the bitstream by encoding N entropy segments.

[0372] The processing unit 1602 may be implemented by a processor. In some embodiments, the partitioning unit 1601 may be implemented by a processor.

[0373] For details on how to process the input image, please refer to the above embodiments, which will not be repeated here.

[0374] like Figure 34 As shown, the electronic device 1700 may include a receiver 1701, a processor 1702, and a memory 1703. The memory 1703 may be used to store code, instructions, etc., executed by the processor 1702.

[0375] It should be understood that the processor 1702 can be an integrated circuit chip with signal processing capabilities. In implementation, the various steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a system-on-chip (SoC), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor, or it can be any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly executed and completed by the processor, or executed and completed by using a combination of hardware and software modules in the processor. The software modules can reside in mature storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in memory, the processor reads information from the memory, and, in conjunction with the hardware in the processor, completes the steps of the above methods executed by the encoding or decoding device.

[0376] It is understood that the memory 1703 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0377] It should be noted that the memory of the electronic devices and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0378] Figure 35This is a flowchart illustrating an exemplary method 3500 for decoding images based on a neural network architecture. This method can be used when the decoder includes a context model. In step 3501, the method includes: obtaining a first context segment i based on an entropy segment i and / or at least one entropy segment adjacent to the entropy segment i, wherein the first context segment is obtained from a bitstream and includes at least one overlapping region with the adjacent first context segment. In step 3502, the method includes: obtaining a second context segment i based on an entropy segment i and / or at least one entropy segment adjacent to the entropy segment i, wherein the second context segment represents an input predicted segment i corresponding to the entropy segment i and includes at least one overlapping region with the adjacent second context segment. In step 3503, the method includes: inputting the first context segment i and the second context segment i into the context model to form a synthesized segment i. In step 3504, the method includes: determining a reconstructed segment i based on the synthesized segment i by inputting the synthesized segment i into a decoding network, wherein the reconstructed segment i does not have an overlapping region with its adjacent reconstructed segments.

[0379] Figure 36 This is a flowchart of an exemplary method 3600 for encoding. According to... Figure 36 The embodiments can be used to provide through reference Figure 35 The described decoding method is easy to decode the output. Figure 36 Method 3600 will be described as being performed by a neural network system of one or more computers located in one or more locations. For example, a system for performing image compression, such as... Figure 1 The neural network can execute method 3600. This method can be used when the encoder includes a context model. In step 3601, the method includes: determining a latent segment i, i = 1, ..., M, based on a determined representation of the input data in the latent space, wherein the latent segment i includes at least one overlapping region with its neighboring latent segments. In step 3602, the method includes: obtaining an input predicted segment i based on the latent segment i, wherein the input predicted segment i includes at least one overlapping region with its neighboring input predicted segments. In step 3603, the method includes: inputting the latent segment i and the input predicted segment i into the context model to output one of M output segments i, wherein the output segment i does not have an overlapping region with its neighboring output segments. In step 3604, the method includes: dividing the M output segments into N entropy segments, wherein each of the N entropy segments does not have an overlapping region with its neighboring entropy segments. In step 3605, the method includes: determining a bitstream by encoding the N entropy segments.

[0380] Although the accompanying drawings depict operations in a specific order, this should not be construed as requiring such operations to be performed in the specific order shown or sequentially, or requiring the execution of all the operations shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

[0381] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the operations described in the claims can be performed in a different order and still achieve the desired result. For example, the processes depicted in the drawings do not necessarily require execution in the specific order shown to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0382] Figure 37 An example of a bitstream is shown. The bitstream may include a marker indicating the start of the bitstream, header data including, for example, image header data or tool header data, entropy-encoded data, and optionally padding including a bitstream end marker. In a particular example, the bitstream comprises data blocks. Each data block can be decoded by a decoder. The decoder can extract entropy fragments from the bitstream, as described herein. The bitstream may be non-transient. The bitstream may be stored on a data carrier.

[0383] Some exemplary implementations in hardware and software Figure 38 A corresponding system is shown that can be deployed with the above encoder-decoder processing chain. Figure 38 This is a schematic block diagram illustrating an exemplary decoding system (e.g., a video, image, audio, and / or other decoding system) that can use the techniques described in this application. The video encoder 3720 (or simply encoder 3720) and video decoder 3730 (or simply decoder 3730) in the video decoding system 3710 are examples of devices that can be used to perform various techniques according to the various examples described in this application. For example, video encoding and decoding can use a neural network, which can be distributed, and the aforementioned bitstream parsing and / or bitstream generation can be applied to transfer feature maps between distributed computing nodes (two or more).

[0384] like Figure 38 As shown, the decoding system 3710 includes a source device 3712, which provides encoded image data 3721 to a destination device 3714 or the like for decoding the encoded image data 3713.

[0385] The source device 3712 includes an encoder 3720 and may additionally (optionally) include an image source 3716, a preprocessor (or preprocessing unit) 3718 (e.g., an image preprocessor 3718), and a communication interface or communication unit 3722.

[0386] Image source 3716 may include or may be any type of image capture device such as a camera for capturing real-world images, and / or any type of image generation device such as a computer graphics processor for generating computer-generated animated images, or any other device of any type for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images) and / or any combination thereof (e.g., augmented reality (AR) images). Image source may be any type of memory / storage for storing any of the aforementioned images.

[0387] To distinguish between the preprocessor 3718 and the processing performed by the preprocessing unit 3718, the image or image data 3717 can also be referred to as the raw image or raw image data 3717.

[0388] The preprocessor 3718 receives (raw) image data 3717 and performs preprocessing on the image data 3717 to obtain a preprocessed image 3719 or preprocessed image data 3719. The preprocessing performed by the preprocessor 3718 may include trimming, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It should be understood that the preprocessing unit 3718 may be an optional component. It should be noted that preprocessing may also utilize neural networks indicated by presence indicators (e.g., in...). Figures 1 to 7 (in any one of them).

[0389] The video encoder 3720 is used to receive preprocessed image data 3719 and provide encoded image data 3721.

[0390] The communication interface 3722 in the source device 3712 can be used to receive encoded image data 3721 and send the encoded image data 3721 (or data obtained after further processing of the encoded image data 3721) to another device, such as the destination device 3714 or any other device, via the communication channel 3713 for storage or direct reconstruction.

[0391] Destination device 3714 includes decoder 3730 (e.g., video decoder 3730) and may additionally (optionally) include communication interface or communication unit 3728, post-processor 3732 (or post-processing unit 3732) and display device 3734.

[0392] The communication interface 3728 in the destination device 3714 is used to directly receive encoded image data 3721 (or data obtained after further processing of encoded image data 3721) from the source device 3712 or from any other source such as a storage device (e.g., an encoded image data storage device) and to provide the encoded image data 3721 to the decoder 3730.

[0393] Communication interfaces 3722 and 3728 can be used to send or receive encoded image data 3721 or encoded data 3713 via a direct communication link (e.g., a direct wired or wireless connection) between source device 3712 and destination device 3714, or via any type of network (e.g., a wired network or wireless network or any combination thereof, any type of private network and public network or any combination thereof).

[0394] For example, the communication interface 3722 can be used to encapsulate the encoded image data 3721 into a suitable format (e.g., a data packet) and / or process the encoded image data through any type of transmission encoding or processing method for transmission over a communication link or communication network.

[0395] The corresponding part of the communication interface 3722, the communication interface 3728, can be used, for example, to receive transmitted data and process the transmitted data using any kind of corresponding transmission decoding or processing and / or unpacking to obtain encoded image data 3721.

[0396] Both communication interfaces 3722 and 3728 can be used for Figure 38 The arrow pointing from the source device 3712 to the destination device 3714 indicates a one-way communication interface, or a two-way communication interface, which can be used to send and receive messages, establish connections, acknowledge and exchange any other information related to the communication link and / or data transmission (e.g., encoded image data transmission). The decoder 30 is used to receive encoded image data 3721 and provide decoded image data 3731 or a decoded image 3731.

[0397] The post-processor 3732 in the destination device 3714 is used to post-process the decoded image data 3731 (also known as reconstructed image data) (e.g., decoded image 3731) to obtain post-processed image data 3733 (e.g., post-processed image 3733). The post-processing performed by the post-processing unit 3732 may include color format conversion (e.g., from YcbCr to RGB), color correction, trimming or resampling, or any other processing to provide the decoded image data 3731 for display by the display device 3734, etc.

[0398] The display device 3734 in the destination device 3714 is used to receive post-processed image data 3733 for displaying the image to a user or viewer. The display device 3734 can be or may include any type of display for representing the reconstructed image, such as an integrated or external display or screen. For example, the display may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display.

[0399] although Figure 38 Source device 3712 and destination device 3714 are described as separate devices; however, device embodiments may also include two devices or two functions, namely source device 3712 or its corresponding function and destination device 3714 or its corresponding function. In these embodiments, the source device 3712 or its corresponding function and the destination device 3714 or its corresponding function may be implemented using the same hardware and / or software, or using separate hardware and / or software, or any combination thereof.

[0400] If it is obvious to a person skilled in the art based on the description, such as Figure 38 The presence and (precise) division of different units or functions within the source device 3712 and / or destination device 3714 shown may vary depending on the actual device and application.

[0401] The encoder 3720 (e.g., a video encoder 3720) or decoder 3730 (e.g., a video decoder 3730), or both encoder 3720 and decoder 3730, may be implemented by processing circuitry, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video decoding processors, or any combination thereof. The encoder 3720 may be implemented by processing circuitry 3746 to embody various modules including neural networks or portions thereof. The decoder 3730 may be implemented by processing circuitry 3746 to embody any decoding system or subsystem described herein. The processing circuitry may be used to perform various operations discussed below. If these techniques are partially implemented in software, a device may store the instructions of that software in a suitable non-transitory computer-readable storage medium, and these instructions may be executed in hardware by one or more processors to perform the techniques of this invention. Either the video encoder 3720 or the video decoder 3730 can be integrated into a single device as part of a combined codec (encoder / decoder, CODEC), such as... Figure 39 As shown.

[0402] Source device 3712 and destination device 3714 can include any of a variety of devices, including any type of handheld or fixed device, such as a laptop or tablet computer, mobile phone, smartphone, tablet / tabletcomputer, camera, desktop computer, set-top box, television, display device, digital media player, video game console, video streaming device (such as a content service server or content distribution server), broadcast receiver device, broadcast transmitter device, etc., and may or may not use any type of operating system. In some cases, source device 3712 and destination device 3714 can be equipped for wireless communication. Therefore, source device 3712 and destination device 3714 can be wireless communication devices.

[0403] In some cases, Figure 38The video decoding system 3710 shown is merely an example, and the technology of this application can be applied to video decoding setups (e.g., video encoding or video decoding) that do not necessarily involve any data communication between the encoding and decoding devices. In other examples, data is retrieved from local memory, streamed over a network, etc. The video encoding device can encode data and store it in memory, and / or the video decoding device can retrieve data from memory and decode it. In some examples, encoding and decoding are performed by devices that do not communicate with each other but merely encode data into memory and / or retrieve data from memory and decode it.

[0404] Figure 40 This is a schematic diagram of a video decoding device 8000 provided in an embodiment of the present invention. The video decoding device 8000 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 8000 may be a decoder (such as...) Figure 38 Video decoder 3730) or encoder (such as Figure 38 (Video encoder 3720).

[0405] The video decoding device 8000 includes an input port 8010 (or input port 8010) and a receiving unit (Rx) 8020 for receiving data; a processor, logic unit, or central processing unit (CPU) 8030 for processing data; a transmitting unit (Tx) 8040 and an output port 8050 (or output port 8050) for transmitting data; and a memory 8060 for storing data. The video decoding device 8000 may also include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the input port 8010, receiving unit 8020, transmitting unit 8040, and output port 8050 for the input or output of signals or electrical signals.

[0406] Processor 8030 is implemented through hardware and software. Processor 8030 can be implemented as one or more CPU chips, one or more cores (e.g., a multi-core processor), FPGA, ASIC, and DSP. Processor 8030 communicates with ingress port 8010, receiver unit 8020, transmitter unit 8040, egress port 8050, and memory 8060. Processor 8030 includes a neural network-based codec 8070. The neural network-based codec 8070 implements the embodiments disclosed above. For example, the neural network-based codec 8070 performs, processes, prepares, or provides various decoding operations. Therefore, including the neural network-based codec 8070 provides a substantial improvement to the functionality of the video decoding device 8000 and affects the transitions of the video decoding device 8000 to different states. Optionally, the neural network-based codec 8070 is implemented with instructions stored in memory 8060 and executed by processor 8030.

[0407] The memory 8060 may include one or more disks, one or more tape drives, and one or more solid-state drives, and may be used as an overflow data storage device to store programs as selected for execution, as well as instructions and data read during program execution. For example, the memory 8060 may be volatile and / or non-volatile, and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0408] Figure 41 This is a simplified block diagram of an apparatus provided in an exemplary embodiment, which can be used as... Figure 38 The source device 3712 and the destination device 3714 are either or both of them.

[0409] The processor 9002 in device 9000 may be a central processing unit. Alternatively, the processor 9002 may be any other type of device or multiple devices, existing or to be developed in the future, capable of manipulating or processing information. While the disclosed implementation may be carried out using a single processor such as the processor 9002 shown in the figure, using more than one processor can improve speed and efficiency.

[0410] In one implementation, the memory 9004 in the device 9000 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as memory 9004. Memory 9004 may include code and data 9006 accessed by processor 9002 via bus 9012. Memory 9004 may also include an operating system 9008 and an application program 9010, which includes at least one program that causes processor 9002 to perform the methods described herein. For example, application program 9010 may include applications 1 to N, including a video decoding application that performs the methods described herein.

[0411] The device 9000 may also include one or more output devices, such as a display 9018. In one example, the display 9018 may be a touch-sensitive display that combines a display with a touch-sensitive element capable of sensing touch input. The display 9018 may be coupled to the processor 9002 via a bus 9012.

[0412] Although bus 9012 of device 9000 is described herein as a single bus, bus 9012 may include multiple buses. Furthermore, auxiliary memory may be directly coupled to other components in device 9000 or accessible via a network, and may include a single integrated unit (e.g., a memory card) or multiple units (e.g., multiple memory cards). Therefore, device 9000 can be implemented in a variety of configurations.

[0413] Figure 42 This is a block diagram of a video decoding system 10000 provided in an embodiment of the present invention. The platform 10002 in system 10000 can be a cloud server or a local server. Alternatively, platform 10002 can be any other type of device, or multiple devices, capable of computation, storage, transcoding, encryption, rendering, decoding, or encoding. Although the disclosed implementation can be implemented using a single platform such as the processor 10002 shown in the figure, using more than one platform can improve speed and efficiency.

[0414] The content delivery network (CDN) 10004 in system 10000 can be a group of geographically distributed servers. Alternatively, CDN 10004 can be any other type of device, or multiple devices, capable of data buffering, scheduling, propagation, or accelerating the distribution of web page content by bringing web page content closer to the user's location. Although the disclosed implementation can be carried out using a single CDN such as CDN 10004 as shown in the figure, using more than one CDN can improve speed and efficiency.

[0415] The terminal 10006 in device 10000 can be a mobile phone, computer, television, laptop, or camera. Alternatively, the terminal 10006 can be any other type of device, or multiple devices capable of displaying video or images.

[0416] This application provides a computer-readable storage medium including instructions. When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the above-described method.

[0417] This application provides a chip system. The chip system includes a memory for storing a computer program and a processor for retrieving and running the computer program from the memory, causing an electronic device equipped with the chip system to perform the methods described above.

[0418] This application provides a computer program product. When the computer program product is run on an electronic device, it causes the electronic device to perform the above-described method.

[0419] In embodiments of this application, "at least one" means one or more, and "multiple" means two or more. The term "and / or" describes the association between associated objects and indicates that three relationships may exist. For example, A and / or B can represent the following three cases: only A exists, both A and B exist, and only B exists, where A and B can be singular or plural. The character " / " generally represents an "OR" relationship between associated objects. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of one or more. For example, at least one of a, b, and c can represent: a, b, c; a and b, a and c, b and c, or a, b, and c, where a, b, and c can be singular or plural.

[0420] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0421] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0422] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the unit division is only a logical functional division, and other division methods may be used in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be implemented through some interfaces. Indirect coupling or communication connection between devices or units can be implemented electronically, mechanically, or in other forms.

[0423] The units described as individual components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0424] Furthermore, in the various embodiments of this application, each functional unit can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0425] If implemented as software functional units and sold or used as independent products, these functions can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be implemented in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to instruct a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0426] The above descriptions are merely some specific implementations of this application and are not intended to limit the scope of protection of this application. Any variations or substitutions easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A decoding method, characterized in that, include: Obtain entropy fragment i from the bitstream, wherein entropy fragment i is one of N entropy fragments, the N entropy fragments constitute the representation of the input data in the latent space, and entropy fragment i does not have overlapping regions with its adjacent entropy fragments, N is a positive integer greater than 1, i = 1, ..., N; Based on the entropy fragment i and / or at least one entropy fragment adjacent to the entropy fragment i, a synthetic fragment i is determined, wherein the synthetic fragment i includes at least one overlapping region with the synthetic fragments adjacent to it; Based on the synthesized fragment i, the reconstructed fragment i is determined by executing a decoding network, wherein the reconstructed fragment i does not have an overlapping region with its neighboring reconstructed fragments.

2. The method according to claim 1, characterized in that, The process of obtaining entropy fragment i from the bitstream includes: Based on the input segment corresponding to the entropy segment i, determine the position information corresponding to the entropy segment i, wherein the input segment corresponding to the entropy segment i is one of M input segments, the M input segments constitute the input data, each of the M input segments includes at least one overlapping region with its adjacent input segments, and M is a positive integer greater than 1; The entropy segment i is obtained from the bitstream based on the position information corresponding to the entropy segment i.

3. The method according to claim 2, characterized in that, The step of determining the position information corresponding to the entropy segment i based on the input segment corresponding to the entropy segment i includes: The shift segment i is determined based on the input segment corresponding to the entropy segment i and the size of the overlapping region; The potential shift segment i is determined based on the shift segment i and the alignment parameters of the synthesized transform in the decoding network; Based on the potential shift fragment i and the shift fragment i, the position information corresponding to the entropy fragment i is determined.

4. The method according to any one of claims 1 to 3, characterized in that, Determining the synthetic fragment i based on the entropy fragment i and / or at least one entropy fragment adjacent to the entropy fragment i includes: The position information of the synthesized segment i is determined based on the alignment parameters of the synthesized transform in the decoding network and the input segment corresponding to the synthesized segment i. The input segment corresponding to the synthesized segment i is one of M input segments, the M input segments constitute the input data, and each of the M input segments includes at least one overlapping region with its adjacent input segments. Based on the position information of the synthesized fragment i, elements of the synthesized fragment i are determined from the entropy fragment i and / or at least one entropy fragment adjacent to the entropy fragment i.

5. The method according to any one of claims 1 to 4, characterized in that, Before obtaining the entropy fragment i from the bitstream, the method further includes: Obtain segment identifier information from the bitstream, wherein the segment identifier information indicates that the bitstream includes N entropy segments.

6. The method according to any one of the preceding claims, characterized in that, The method further includes: A first context segment i is obtained based on the entropy segment i and / or at least one entropy segment adjacent to the entropy segment i, wherein the first context segment is obtained from the bitstream and includes at least one overlapping region with the adjacent first context segment; A second context segment i is obtained based on the entropy segment i and / or at least one entropy segment adjacent to the entropy segment i, wherein the second context segment represents the input prediction segment i corresponding to the entropy segment i and includes at least one overlapping region with the adjacent second context segment; The first context fragment i and the second context fragment i are input into the context model to form the synthesized fragment i; Based on the synthesized fragment i, the reconstructed fragment i is determined by inputting the synthesized fragment i into the decoding network, wherein the reconstructed fragment i does not have an overlapping region with its neighboring reconstructed fragments.

7. The method according to claim 6, characterized in that, The first context fragment i and the synthesized fragment i have the same spatial dimension.

8. The method according to claim 6 or 7, characterized in that, The first context fragment i and the second context fragment i each have the same total number of elements.

9. The method according to any one of claims 6 to 8, characterized in that, The input prediction segment is the output of the superdecoder of the variational autoencoder model.

10. The method according to any one of claims 6 to 9, characterized in that, The spatial dimension of the second context fragment i is one-quarter of the spatial dimension of the first context fragment i, and the number of channels of the second context fragment i is four times the number of channels of the first context fragment.

11. The method according to any one of claims 6 to 10, characterized in that, The first context fragment i represents the residual of the input data in the latent space.

12. The method according to any one of the preceding claims, characterized in that, The at least one overlapping region of the synthesized fragment is ignored in the determination of the reconstructed fragment.

13. The method according to any one of the preceding claims, characterized in that, The input data is derived from the input image, and each reconstructed segment corresponds to a block of the input image.

14. An encoding method, characterized in that, include: The input data is divided into M input segments, wherein each of the M input segments includes at least one overlapping region with its adjacent input segments, and M is a positive integer greater than 1. By processing the M input segments using an encoding network, M analysis segments are determined, wherein the M analysis segments correspond one-to-one with the M input segments; Based on the M analysis segments and the M input segments, determine the representation of the input data in the latent space; The representation of the input data is divided into N entropy segments, wherein each of the N entropy segments does not overlap with its adjacent entropy segments, and N is a positive integer greater than 1; The bitstream is determined by encoding the N entropy segments.

15. The method according to claim 6, characterized in that, Determining the representation of the input data in the latent space based on the M analysis segments includes: Based on the M analysis segments and the M input segments, M core segments are determined, wherein the j-th core segment among the M core segments is determined based on the j-th analysis segment among the M analysis segments and the j-th input segment among the M input segments, j=1, ...,M, and each core segment among the M core segments has no overlapping region with its adjacent core segments; The representation of the input data is determined by constructing the M core segments.

16. The method according to claim 6 or 7, characterized in that, The step of dividing the representation of the input data into N entropy segments includes: Determine the position information of each of the N entropy segments; Based on the position information of each of the N entropy segments, the representation of the input data is divided into the N entropy segments.

17. The method according to claim 8, characterized in that, Determining the position information of each of the N entropy segments includes: Based on the M input segments, determine N shift segments; Based on the N shift segments and the alignment parameters of the analysis transform in the coding network, N potential shift segments are determined; Based on the N potential shift segments and the N shift segments, determine the position information of each of the N entropy segments.

18. The method according to any one of claims 6 to 9, characterized in that, The method further includes: Determine the segment identifier information in the bitstream, wherein the segment identifier information indicates that the bitstream includes N entropy segments.

19. The method according to any one of claims 14 to 18, characterized in that, The method further includes: Based on the determined representation of the input data in the latent space, determine the potential segment i, i = 1, ..., M, wherein the potential segment i includes at least one overlapping region with its neighboring potential segments; The input prediction segment i is obtained based on the potential segment i, and the input prediction segment i includes at least one overlapping region with the adjacent input prediction segments; The potential fragment i and the input predicted fragment i are input into the context model to output one of the M output fragments, wherein the output fragment i does not have an overlapping region with its neighboring output fragments; The M output segments are divided into N entropy segments, wherein each of the N entropy segments does not overlap with its adjacent entropy segments; The bitstream is determined by encoding the N entropy segments.

20. The method according to claim 19, characterized in that, The potential fragment i has the same spatial dimension as the output fragment i.

21. The method according to claim 19 or 20, characterized in that, The potential fragment i and the input predicted fragment i each have the same total number of elements.

22. The method according to any one of claims 19 to 21, characterized in that, The input prediction segment is the output of the superdecoder of the variational autoencoder model.

23. The method according to any one of claims 19 to 22, characterized in that, The spatial dimension of the input predicted fragment i is one-quarter of the spatial dimension of the potential fragment i, and the number of channels of the input predicted fragment i is four times the number of channels of the potential fragment.

24. The method according to any one of claims 19 to 23, characterized in that, The output segment i represents the residual of the input data in the latent space.

25. The method according to any one of claims 19 to 24, characterized in that, The at least one overlapping region of the potential fragment is ignored in the determination of the output fragment.

26. The method according to any one of claims 19 to 25, characterized in that, The input data is derived from the input image, and each potential segment corresponds to a block of the input image.

27. An electronic device, characterized in that, include: An acquisition unit is used to acquire entropy fragment i from the bitstream, wherein the entropy fragment i is one of N entropy fragments, the N entropy fragments constitute the representation of the input data in the latent space, the entropy fragment i does not have an overlapping region with its adjacent entropy fragments, N is a positive integer greater than 1, i = 1, ..., N; A processing unit is configured to determine a synthetic fragment i based on the entropy fragment i and / or at least one entropy fragment adjacent to the entropy fragment i, wherein the synthetic fragment i includes at least one overlapping region with the synthetic fragments adjacent to it; The processing unit is further configured to determine the reconstructed fragment i by executing a decoding network based on the synthesized fragment i, wherein the reconstructed fragment i does not have an overlapping region with its adjacent reconstructed fragments.

28. The electronic device according to claim 27, characterized in that, The acquisition unit is specifically used for: Based on the input segment corresponding to the entropy segment i, determine the position information corresponding to the entropy segment i, wherein the input segment corresponding to the entropy segment i is one of M input segments, the M input segments constitute the input data, each of the M input segments includes at least one overlapping region with its adjacent input segments, and M is a positive integer greater than 1; The entropy segment i is obtained from the bitstream based on the position information corresponding to the entropy segment i.

29. The electronic device according to claim 28, characterized in that, The acquisition unit is specifically used for: The shift segment i is determined based on the input segment corresponding to the entropy segment i and the size of the overlapping region; The potential shift segment i is determined based on the shift segment i and the alignment parameters of the synthesized transform in the decoding network; Based on the potential shift fragment i and the shift fragment i, the position information corresponding to the entropy fragment i is determined.

30. The electronic device according to any one of claims 27 to 29, characterized in that, The processing unit is specifically used for: The position information of the synthesized segment i is determined based on the alignment parameters of the synthesized transform in the decoding network and the input segment corresponding to the synthesized segment i. The input segment corresponding to the synthesized segment i is one of M input segments, the M input segments constitute the input data, and each of the M input segments includes at least one overlapping region with its adjacent input segments. Based on the position information of the synthesized fragment i, elements of the synthesized fragment i are determined from the entropy fragment i and / or at least one entropy fragment adjacent to the entropy fragment i.

31. The electronic device according to any one of claims 27 to 30, characterized in that, The acquisition unit is further configured to acquire segment marker information from the bitstream, wherein the segment marker information indicates that the bitstream includes N entropy segments.

32. The electronic device according to any one of claims 27 to 31, characterized in that, The acquisition unit is also used for: A first context segment i is obtained based on the entropy segment i and / or at least one entropy segment adjacent to the entropy segment i, wherein the first context segment is obtained from the bitstream and includes at least one overlapping region with the adjacent first context segment; A second context segment i is obtained based on the entropy segment i and / or at least one entropy segment adjacent to the entropy segment i, wherein the second context segment represents the input prediction segment i corresponding to the entropy segment i and includes at least one overlapping region with the adjacent second context segment; The first context fragment i and the second context fragment i are input into the context model to form the synthesized fragment i; The processing unit is further configured to determine the reconstructed segment i by inputting the synthesized segment i into the decoding network, wherein the reconstructed segment i does not have an overlapping region with its adjacent reconstructed segments.

33. The electronic device according to claim 32, characterized in that, The first context fragment i and the synthesized fragment i have the same spatial dimension.

34. The electronic device according to claim 32 or 33, characterized in that, The first context fragment i and the second context fragment i each have the same total number of elements.

35. The electronic device according to any one of claims 32 to 34, characterized in that, The input prediction segment is the output of the superdecoder of the variational autoencoder model.

36. The electronic device according to any one of claims 32 to 35, characterized in that, The spatial dimension of the second context fragment i is one-quarter of the spatial dimension of the first context fragment i, and the number of channels of the second context fragment i is four times the number of channels of the first context fragment.

37. The electronic device according to any one of claims 32 to 36, characterized in that, The first context fragment i represents the residual of the input data in the latent space.

38. The electronic device according to any one of claims 32 to 37, characterized in that, The processing unit is also configured to ignore at least one overlapping region of the synthesized fragment in the determination of the reconstructed fragment.

39. The electronic device according to any one of claims 32 to 38, characterized in that, The input data is derived from the input image, and each reconstructed segment corresponds to a block of the input image.

40. An electronic device, characterized in that, include: A partitioning unit is used to divide input data into M input segments, wherein each of the M input segments includes at least one overlapping region with its adjacent input segments, and M is a positive integer greater than 1. The processing unit is configured to process the M input segments using an encoding network to determine M analysis segments, wherein the M analysis segments correspond one-to-one with the M input segments; The processing unit is further configured to determine the representation of the input data in the latent space based on the M analysis segments and the M input segments; The processing unit is further configured to divide the representation of the input data into N entropy segments, wherein each of the N entropy segments does not have an overlapping region with its adjacent entropy segments, and N is a positive integer greater than 1. The processing unit is also used to determine the bitstream by encoding the N entropy segments.

41. The electronic device according to claim 40, characterized in that, The processing unit is specifically used for: Based on the M analysis segments and the M input segments, M core segments are determined, wherein the j-th core segment among the M core segments is determined based on the j-th analysis segment among the M analysis segments and the j-th input segment among the M input segments, j=1, ...,M, and each core segment among the M core segments has no overlapping region with its adjacent core segments; The representation of the input data is determined by constructing the M core segments.

42. The electronic device according to claim 40 or 41, characterized in that, The processing unit is specifically used for: Determine the position information of each of the N entropy segments; Based on the position information of each of the N entropy segments, the representation of the input data is divided into the N entropy segments.

43. The electronic device according to claim 42, characterized in that, The processing unit is specifically used for: Based on the M input segments, determine N shift segments; Based on the N shift segments and the alignment parameters of the analysis transform in the coding network, N potential shift segments are determined; Based on the N potential shift segments and the N shift segments, determine the position information of each of the N entropy segments.

44. The electronic device according to any one of claims 40 to 43, characterized in that, The processing unit is further configured to determine segment marker information in the bitstream, wherein the segment marker information indicates that the bitstream includes N entropy segments.

45. The electronic device according to any one of claims 40 to 44, characterized in that, The device is also used for: Based on the determined representation of the input data in the latent space, determine the potential segment i, i = 1, ..., M, wherein the potential segment i includes at least one overlapping region with its neighboring potential segments; The input prediction segment i is obtained based on the potential segment i, and the input prediction segment i includes at least one overlapping region with the adjacent input prediction segments; The potential fragment i and the input predicted fragment i are input into the context model to output one of the M output fragments, wherein the output fragment i does not have an overlapping region with its neighboring output fragments; The M output segments are divided into N entropy segments, wherein each of the N entropy segments does not overlap with its adjacent entropy segments; The bitstream is determined by encoding the N entropy segments.

46. ​​The electronic device according to claim 45, characterized in that, The potential fragment i has the same spatial dimension as the output fragment i.

47. The electronic device according to claim 45 or 46, characterized in that, The potential fragment i and the input predicted fragment i each have the same total number of elements.

48. The electronic device according to any one of claims 45 to 47, characterized in that, The input prediction segment is the output of the superdecoder of the variational autoencoder model.

49. The electronic device according to any one of claims 45 to 48, characterized in that, The spatial dimension of the input predicted fragment i is one-quarter of the spatial dimension of the potential fragment i, and the number of channels of the input predicted fragment i is four times the number of channels of the potential fragment.

50. The electronic device according to any one of claims 45 to 49, characterized in that, The output segment i represents the residual of the input data in the latent space.

51. The electronic device according to any one of claims 45 to 50, characterized in that, The processing unit is also configured to ignore at least one overlapping region of the potential segment in the determination of the output segment.

52. The electronic device according to any one of claims 45 to 51, characterized in that, The input data is derived from the input image, and each potential segment corresponds to a block of the input image.

53. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the method according to any one of claims 1 to 13.

54. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the method according to any one of claims 14 to 26.

55. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to retrieve the computer program from the memory and run the computer program, causing the electronic device to perform the method according to any one of claims 1 to 13.

56. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to retrieve the computer program from the memory and run the computer program, causing the electronic device to perform the method according to any one of claims 14 to 26.

57. A chip system, characterized in that, The system includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to retrieve the computer program from the memory and run the computer program, causing the electronic device in which the chip system resides to perform the method according to any one of claims 1 to 13.

58. A chip system, characterized in that, The system includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to retrieve and run the computer program from the memory, causing the electronic device in which the chip system resides to perform the method according to any one of claims 14 to 26.

59. A computer program product, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method according to any one of claims 1 to 13.

60. A computer program product, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method according to any one of claims 14 to 26.

Images