Neural network image compression method and device with adaptive intra-frame prediction

Abstract

The present application discloses a neural network image compression method, apparatus, computer device, and storage medium with adaptive intra-frame prediction. The method comprises: receiving an optimal partitioning method; receiving a compressed representation of an input, the input comprising a first set of blocks; for each block in the first set of blocks, receiving a block selection signal, the block selection signal indicating that one of a first recovery block and a second recovery block is used as a current recovery block, and performing one of the first recovery and the second recovery based on the received block selection signal; and merging the current recovery blocks to obtain a reconstructed image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. provisional application No. 63 / 138,963, filed with the U.S. Patent Office on January 19, 2021, and U.S. formal application No. 17 / 488,532, filed with the U.S. Patent Office on September 29, 2021, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The embodiments of the present application relate to data processing technology, and in particular to a neural network image compression method, apparatus, computer equipment, and storage medium with adaptive intra-frame prediction. Background Art

[0004] Standards groups and companies have been actively exploring potential requirements for standardizing future video codec technologies. These groups have established the JPEG-AI group, which focuses on implementing AI-based, end-to-end neural network image compression using neural networks (NN). The recent success of advanced neural network image and video compression methods has generated growing industry interest.

[0005] Given an input image x, the goal of NIC is to use the image x as the input of the NN encoder and calculate the compressed representation The compression is compact for storage and transmission, and then, using As input to the NN decoder, reconstruct the image Previous NIC methods used a variational autoencoder (VAE) structure, in which the NN encoder directly uses the entire image x as its input and calculates the output representation through a set of network layers that work like a black box. Accordingly, the NN decoder uses the entire representation As its input, through another set of network layers that work like another black box, the reconstructed

[0006] A block-based intra-frame prediction and residual codec mechanism encodes the residual between the predicted block and the original block, rather than directly encoding the original complete image. This mechanism has been proven to be very effective in image frame compression in modern video codec standards such as HEVC and VVC. The entire image is partitioned into blocks of various sizes, and the boundary pixels of the previously compressed block are copied along various angles to generate a predicted block. The residual between the original block and the predicted block is then compressed.

[0007] However, different block sizes directly affect the compression performance, and the optimal block size usually depends on the specific image. Therefore, a flexible scheme is needed that can adapt to various block sizes and original images. Summary of the Invention

[0008] Embodiments of the present application relate to a neural network image compression method, apparatus, computer equipment, and storage medium with adaptive intra-frame prediction, providing a flexible and universal architecture that can adapt to various block sizes, intra-frame prediction methods, neural compression methods, and block partitioning methods.

[0009] According to an embodiment of the present application, a neural network image compression method with adaptive intra-frame prediction is provided, comprising: receiving an optimal partitioning method; receiving a compressed representation of an input, the input comprising a first group of blocks; for each block in the first group of blocks, receiving a block selection signal, the block selection signal indicating that one of a first recovery block and a second recovery block is used as a current recovery block, and performing one of a first recovery and a second recovery based on the received block selection signal; and merging the current recovery block to obtain a reconstructed image; wherein the first recovery comprises: using a first neural network to compress blocks in the first group of blocks to calculate a first compressed representation; using a second neural network to decompress the first compressed representation to calculate the first recovery block; the second recovery comprises: calculating a first prediction block based on a group of previously recovered blocks and a group of previously recovered microblocks; calculating a first residual based on the current block in the first group of blocks and the first prediction block; generating a recovery residual based on the first residual; partitioning the first prediction block and adding the recovery residual to obtain the second recovery block.

[0010] According to an embodiment of the present application, a neural network image compression device with adaptive intra-frame prediction is also provided, comprising: a first receiving module for receiving an optimal partitioning method; a second receiving module for receiving a compressed representation of an input, wherein the input comprises a first group of blocks; a third receiving module for receiving a block selection signal for each block in the first group of blocks, wherein the block selection signal indicates that one of a first restored block and a second restored block is used as a current restored block, and based on the received block selection signal, performing one of a first restoration and a second restoration; and a merging module for merging the current restored blocks to obtain a reconstructed image; wherein the first restoration comprises: a first compression module, Used to compress blocks in the first group of blocks using a first neural network to calculate a first compressed representation; a first decompression module, used to decompress the first compressed representation using a second neural network to calculate the first recovered block; the second recovery includes: a first prediction module, used to calculate a first prediction block based on a group of previously recovered blocks and a group of previously recovered microblocks; a first residual module, used to calculate a first residual based on a current block in the first group of blocks and the first prediction block; a first generation module, used to generate a recovered residual based on the first residual; a first partitioning module, used to partition the first prediction block and add the recovered residual to obtain the second recovered block.

[0011] According to an embodiment of the present application, a computer device is also provided, including a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above-mentioned neural network image compression method with adaptive intra-frame prediction.

[0012] According to an embodiment of the present application, a non-volatile computer-readable medium is also provided, characterized in that instructions are stored thereon, and when the instructions are executed by at least one processor for neural network image compression with adaptive intra-frame prediction, the at least one processor implements the above-mentioned neural network image compression method with adaptive intra-frame prediction.

[0013] According to an embodiment of the present application, a computer program product or computer program is also provided. The computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described neural network image compression method with adaptive intra-frame prediction.

[0014] It can be seen from the above technical solution that according to the method provided in an embodiment of the present invention, a NIC framework is proposed, which uses a block-based intra-frame prediction mechanism and an adaptive block size as a flexible and universal architecture that can adapt to different intra-frame prediction methods, different neural compression methods for both residual and original image blocks, and different micro-block and block partitioning methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram showing an environment for implementing the methods, devices, and systems described herein according to an embodiment of the present application is shown;

[0016] Figure 2 Shown Figure 1 a block diagram of example components of at least one computer device;

[0017] Figure 3 FIG2 shows a block diagram of a NIC encoder device in a testing phase according to an embodiment of the present application;

[0018] Figure 4 The test phase according to the embodiment of the present application is shown. Figure 3 Detailed workflow of the partition selection module;

[0019] Figure 5 FIG2 shows a block diagram of a NIC decoder device in a testing phase according to an embodiment of the present application;

[0020] Figure 6 The figure shows the workflow of the NIC intra-frame prediction apparatus in the training phase according to an embodiment of the present application;

[0021] Figure 7 A flowchart of a neural network image compression method with adaptive intra-frame prediction according to an embodiment of the present application is shown;

[0022] Figure 8 A block diagram of a neural network image compression device with adaptive intra-frame prediction according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] This application proposes a neural network image compression (NIC) framework that uses a block-based intra-frame prediction mechanism and adaptive block size to compress input images through a neural network (DNN). Exemplary embodiments are described below with reference to the accompanying drawings. In the drawings, identical modules are labeled identically, and therefore, repeated descriptions are omitted where necessary. Figure 1 is a schematic diagram of an environment for implementing the methods, devices, and systems described herein according to an embodiment of the present application.

[0024] like Figure 1As shown, environment 100 may include user device 110, platform 120, and network 130. The devices of environment 100 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0025] User device 110 includes one or more devices capable of receiving, generating, storing, processing, and / or providing information associated with platform 120. For example, user device 110 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smartphone, a wireless phone, etc.), a wearable device (e.g., a pair of smart glasses or a smart watch), or the like. In some implementations, user device 110 may receive information from platform 120 and / or transmit information to platform 120.

[0026] The platform 120 includes one or more devices capable of generating audio output signals via a multi-band synchronized neural vocoder, as described elsewhere herein. In some implementations, the platform 120 may include a cloud server or a group of cloud servers. In some implementations, the platform 120 may be designed to be modular so that certain software components can be swapped in or out based on specific needs. In this way, the platform 120 can be easily and / or quickly reconfigured for different uses.

[0027] In some implementations, as shown, the platform 120 can be hosted in a cloud computing environment 122. It is worth noting that although the implementations described herein describe the platform 120 as being hosted in a cloud computing environment 122, in some implementations, the platform 120 is not cloud-based (i.e., can be implemented outside of a cloud computing environment) or can be partially cloud-based.

[0028] The cloud computing environment 122 includes an environment that hosts the platform 120. The cloud computing environment 122 can provide computing, software, data access, storage, and other services that do not require an end user (e.g., user device 110) to be aware of the physical location and configuration of one or more systems and / or devices hosting the platform 120. As shown, the cloud computing environment 122 can include a set of computing resources 124 (collectively, "computing resources 124" and individually, "computing resource 124").

[0029] Computing resources 124 include one or more personal computers, workstation computers, server devices, or other types of computing and / or communication devices. In some implementations, computing resources 124 may be hosting platforms 120. Cloud resources may include computing instances executed in computing resources 124, storage devices provided in computing resources 124, data transfer devices provided by computing resources 124, and the like. In some implementations, computing resources 124 may communicate with other computing resources 124 via wired connections, wireless connections, or a combination of wired and wireless connections.

[0030] like Figure 1 As further shown in FIG, the computing resources 124 include a set of cloud resources, such as one or more application programs ("APP") 124-1, one or more virtual machines ("VM") 124-2, virtualized storage ("VS") 124-3, one or more hypervisors ("HYP") 124-4, etc.

[0031] Applications 124-1 include one or more software applications that can be provided to or accessed by user device 110 and / or sensor device 120. Applications 124-1 can eliminate the need to install and execute software applications on user device 110. For example, applications 124-1 can include software associated with platform 120 and / or any other software that can be provided via cloud computing environment 122. In some implementations, one application 124-1 can send and receive information to and from one or more other applications 124-1 via virtual machine 124-2.

[0032] Virtual machine 124-2 comprises a software implementation of a machine (e.g., a computer) that executes programs like a physical machine. Virtual machine 124-2 can be a system virtual machine or a process virtual machine, depending on the use of virtual machine 124-2 and the degree of correspondence with any real machine. A system virtual machine can provide a complete system platform that supports the execution of a complete operating system ("OS"). A process virtual machine can execute a single program and can support a single process. In some implementations, virtual machine 124-2 can execute on behalf of a user (e.g., user device 110) and can manage the infrastructure of cloud computing environment 122, such as data management, synchronization, or long-duration data transfer.

[0033] Virtualized storage 124-3 includes one or more storage systems and / or one or more devices that use virtualization technology within the storage system or device of the computing resource 124. In some implementations, in the context of a storage system, the types of virtualization may include block virtualization and file virtualization. Block virtualization may refer to the abstraction (or separation) of logical storage from physical storage, such that the storage system may be accessed without regard to physical storage or heterogeneous structures. Separation may allow administrators of the storage system flexibility in how the administrator manages storage for end users. File virtualization may eliminate the dependency between data accessed at the file level and where the file is physically stored. This may enable optimization of storage usage, server consolidation, and / or performance of non-disruptive file migration.

[0034] Hypervisor 124-4 can provide hardware virtualization technology that allows multiple operating systems (e.g., "guest operating systems") to execute concurrently on a host computer such as computing resource 124. Hypervisor 124-4 can present a virtual operating platform to the guest operating systems and can manage the execution of the guest operating systems. Multiple instances of various operating systems can share virtualized hardware resources.

[0035] The network 130 includes one or more wired and / or wireless networks. For example, the network 130 may include a cellular network (e.g., a fifth generation (5G) network, a long term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-optic-based network, etc., and / or a combination of these or other types of networks.

[0036] Figure 1 The number and arrangement of devices and networks shown in FIG are provided as examples. In practice, there may be more than Figure 1 The devices and / or networks shown may include more devices and / or networks, fewer devices and / or networks, different devices and / or networks, or differently arranged devices and / or networks. Figure 1 Two or more of the devices shown may be implemented in a single device, or Figure 1 The single device shown may be implemented as multiple distributed devices. Additionally or alternatively, one set of devices (eg, one or more devices) of environment 100 may perform one or more functions described as being performed by another set of devices of environment 100.

[0037] Figure 2 Shown Figure 1 A block diagram of example components of at least one computer device.

[0038] Computer device 200 may correspond to user device 110 and / or platform 120. Figure 2 As shown, computer device 200 may include a bus 210 , a processor 220 , a memory 230 , a storage component 240 , an input component 250 , an output component 260 , and a communication interface 270 .

[0039] The bus 210 includes components that allow communication between components of the computer device 200. The processor 220 is implemented in hardware, firmware, or a combination of hardware and software. The processor 220 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or another type of processing component. In some implementations, the processor 220 includes one or more processors that can be programmed to perform functions. The memory 230 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by the processor 220.

[0040] The storage component 240 stores information and / or software related to the operation and use of the computer device 200. For example, the storage component 240 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a magnetic tape, and / or other types of non-volatile computer-readable media, and corresponding drives.

[0041] Input components 250 include components that allow computer device 200 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). Additionally or alternatively, input components 250 may include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). Output components 260 include components that provide output information from computer device 200 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs)).

[0042] The communication interface 270 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the computer device 200 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 270 can allow the computer device 200 to receive information from another device and / or provide information to another device. For example, the communication interface 270 can include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0043] Computer device 200 can perform one or more of the processes described herein. Computer device 200 can perform these processes in response to processor 220 executing software instructions stored by a non-volatile computer-readable medium, such as memory 230 and / or storage component 240. Computer-readable media is defined herein as a non-volatile memory device. A memory device includes memory space within a single physical storage device or memory space distributed across multiple physical storage devices.

[0044] The software instructions may be read into the memory 230 and / or storage component 240 from another computer-readable medium or from another device via the communication interface 270. When executed, the software instructions stored in the memory 230 and / or storage component 240 may cause the processor 220 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, the implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0045] Figure 2 The number and arrangement of components shown in FIG are provided as examples. In practice, the computer device 200 may include Figure 2 The components shown may be more components, fewer components, different components, or components arranged differently. Additionally or alternatively, one set of components (e.g., one or more components) of computer device 200 may perform one or more functions described as being performed by another set of components of computer device 200.

[0046] A method and apparatus for using a block-based intra prediction mechanism and adaptive block size in NIC will be described in detail below.

[0047] This application proposes a NIC framework that uses a block-based intra-frame prediction mechanism and adaptive block size. Instead of encoding the original pixels, the residual between the predicted block and the original block is encoded, and the block size is adaptively determined based on the compression quality, such as the rate-distortion (RD) loss.

[0048] Figure 3 Shown is a block diagram of the NIC encoder 300 apparatus in the test phase according to an embodiment of the present application.

[0049] like Figure 3 As shown, the encoder 300 includes a partition module 310 and a partition selection module 320 .

[0050] At the encoder side, given an input image x, the partition module 310 partitions the input image x into k microblocks of size (w m , h m ), Among them, m i Represents the i-th microblock. Each microblock m i can be further partitioned into blocks b i,1 ,…,b i,n , where b i,j is the microblock m i For different blocks, block b i,j The size of the CTU can vary. In an exemplary embodiment, in current video codec tools, microblocks are aligned with CTU partitions. Each CTU microblock can be further partitioned into 2×2, 4×4, 8×8, 16×16, 32×32, or 64×64 blocks. The embodiments do not impose any restrictions on the size of the CTU or how the blocks in the CTU are partitioned.

[0051] Assume there are P different ways to convert each microblock m i Partitioning into multiple blocks. The workflow of how to determine the best partitioning method in the partition selection module 320 will now be described in detail.

[0052] Figure 4 The test phase according to the embodiment of the present application is shown. Figure 3 Detailed workflow of the partition selection module 320.

[0053] like Figure 4 As shown, the partition selection module 320 includes an intra-frame prediction module 410, a residual neural compression module 420, a residual neural decompression module 430, a residual compression loss calculation module 440, a neural compression module 450, a neural decompression module 460, a compression loss calculation module 470, a block selection module 480 and a partition loss calculation module 490.

[0054] The output of the partition selection module 320 includes the optimal partitioning scheme p * , a group of block selection signals and a set of compressed representations Typically, after further compression through quantization and entropy coding, these outputs are sent to the decoder side (e.g. Figure 5 decoder 500 described in detail in ).

[0055] make Represents the block obtained by the p-th partitioning method. The block n used for this partition p The total number of microblocks m i The size of the partition and the size of the block are automatically determined. For each partition block b i,p,j , the predicted block can be calculated by the intra prediction module 410 based on the prediction network The prediction network takes as input a set of image pixels selected from x, where the selected pixels can come from two sources: i Previously encoded microblocks and from partition block b i,p,j Previously coded microblock m i Blocks in There are many ways to select pixels and form the input of the prediction network. For example, the partition b in the spatially closest context region i,p,j The adjacent pixels can be organized in a certain order (stacked, concatenated, transformed in space, etc.) to form the input of the prediction network. Through inference calculation, the prediction network outputs the prediction block The prediction network can have various architectures. For each partitioning method, the prediction network can use different NN models for prediction. Convolutional layers and fully connected layers are generally used. The embodiment does not impose any restrictions on the size and shape of the context region used for pixel selection, the method of converting pixels into input of the prediction network, or the network architecture of the prediction network.

[0056] For partition block b i,p,j , in the calculation prediction block Afterwards, we can use the partition block b i,p,j and prediction blocks To calculate the residual r i,p,j , for example by subtraction. Let represents the entire i-th microblock m partitioned in the p-th way i The residual can be Repartitioned into a set of q residual blocks Note that the repartitioning of the residual block can be done with the prediction block The original partitions are the same or different. When q=1, the entire microblock will be processed as a whole. The corresponding microblocks can also be processed in the same way. and prediction blocks Repartition them separately to get repartitioned microblocks and the repartitioned prediction blocks The residual neural compression module 420 can perform Compress and calculate the compressed residual representation The residual neural decompression module 430 represents the compressed residual Decompress and calculate the recovered residual block The recovered residual block can be Then add it to the corresponding repartitioned prediction block Thus, the reconstructed block is obtained The residual compression loss calculation module 440 is based on the repartitioned microblocks Reconstructed blocks and compressed residual representation Calculate residual compression quality loss For example, in an exemplary embodiment, rate-distortion (RD) loss may be calculated as a quality measure (residual quality loss) as follows:

[0057]

[0058] in, is the repartitioned microblock With the reconstructed block The distortion between. is the rate loss, used to measure the compression residual representation The bit consumption of λ is a trade-off hyperparameter that balances the importance of different terms. Other compression quality losses can certainly be used here. The embodiments do not impose any restrictions on the specific measurement used for compression quality loss, distortion or rate loss.

[0059] At the same time, each original block It can be directly compressed by the neural compression module 450 to calculate the compressed representation The neural decompression module 460 decompresses the compressed representation Decompress and directly calculate the recovered block In the calculation compression loss module 470, the original block can be Reconstructed blocks and compressed representation With residual quality loss In the same way, the compression quality loss is calculated Compression quality loss and residual quality loss The block selection module 480 generates a selection signal s i,p,j , to indicate whether to use residual blocks or original To generate the compressed residual representation or compressed representation For example by choosing the option with less quality loss. This gives the compression of the current j-th block Best quality loss For example The calculation partition loss module 490 calculates the partition loss for the microblock m i The total quality loss L of the p-th partition i,p , as shown below:

[0060]

[0061] Among them, each w i,p,j Is the original block By simply setting all weights to 1, all blocks are treated equally. Some blocks can be treated with more attention than others, and an attention map (or significance map) can be used to obtain the weights.

[0062] By repeating the same process for all P ways of partitioning, the quality loss L can be obtained i,p ,p=1,…,P. Then we can choose the best partitioning method p * , for example, as the one with the best loss (i.e., p * =argmin p L i,p 、 ) partition. The corresponding block selection signal It can also be determined as the output of the partition selection module 320.

[0063] make Represented as microblock m i Select the best partition block. According to the block selection signal s i,p*,j , the corresponding compressed residual is expressed as or compressed representation Can also be identified as a block The actual compressed representation of This group of compressed representations Also output from the partition selection module 320. The best partition p * , compressed representation and block select signals is further encoded, such as by quantization and entropy coding, to generate an encoded stream and sent to the decoder side (at Figure 5 (detailed in ).

[0064] The neural compression module 450 and the residual neural compression module 420 may use any neural compression method. Embodiments do not impose any limitations on the specific methods or network architectures used for these two modules.

[0065] Figure 5 FIG. 4 shows a block diagram of a NIC decoder 500 device in a testing phase according to an embodiment of the present application.

[0066] like Figure 5 As shown, the decoder 500 includes an intra-frame prediction module 410, a residual neural decompression module 430, a neural decompression module 460 and a merging module 510.

[0067] At the decoder 500 side, the system receives the best partition p * , compressed representation and block select signals (Usually recovered from the received bitstream by entropy decoding and dequantization.) Select the signal based on each block The system chooses one of the following methods to calculate the recovered blocks: If you select signal Indicates that the recovered block is from the compressed representation based on the encoder 300 side The neural decompression module 460 will be used to calculate the recovered block If you select signal Indicates that the recovered block comes from the compressed residual representation at the encoder 300 side The decoded block is obtained, and the residual neural decompression module 430 is used to calculate the recovered residual When using the recovered residuals In the case of

[0068]

[0069] and previously recovered microblocks The prediction block is calculated by using the prediction network is calculated in the same way in the encoder 300. The only difference is that on the encoder 300 side, the input to the prediction network is formed by the pixels of the original input image x. On the decoder 500 side, the input comes from the corresponding recovered blocks and micro-blocks. The recovered residual can then be Add back (in the same way as encoder 300) the repartitioned prediction block To get the recovered blocks The calculated recovered block ( or ) will give the actual recovered blocks The decoder continues to process the next block. Finally, in the merging module 510, the microblock m i The recovered block is gathered into the reconstructed image In some embodiments, the merging module 510 may further process the recovered blocks to remove artifacts, such as deblocking, denoising, etc. Embodiments do not describe how to aggregate the recovered blocks into the reconstructed image. No restrictions are imposed on the specific methods in the

[0070] The NIC intra prediction training process will now be described. Figure 6 FIG. 6 is a workflow of the NIC intra prediction apparatus 600 during the training phase according to an embodiment.

[0071] like Figure 6 As shown, the NIC intra-frame prediction training device 600 includes a partition module 310, a partition selection module 320, an intra-frame prediction module 410, a residual neural decompression module 430, a neural decompression module 460, a merging module 510, a calculation of overall loss module 610 and a calculation of additional loss module 620.

[0072] The goal of the training process is to learn the prediction network, the neural compression module 450, the neural decompression module 460, the residual neural compression module 420, and the residual neural decompression module 430. In the case of using the learnable merging module 510 and the block selection module 480, for example, when the NN is used to aggregate the restored blocks into the restored image, the corresponding learnable parameters can also be learned during the training process. During the training process, the weight coefficients of the above-mentioned networks and modules to be learned are initialized, for example, by using a pre-trained model, or by setting their parameters to random numbers. Then, given an input training image x, it is obtained by Figure 3 The encoder 300 described in Figure 5 The decoder 500 described in the above embodiment calculates the optimal partition p * , compressed representation

[0073]

[0074] and for each microblock m i Block select signal And the final reconstructed image Distortion loss can be calculated Such as traditional PSNR, MS-SSIM or a weighted combination of the two. Rate loss can be calculated Expressed as measured compression Therefore, the total RD loss can be calculated in the calculation total loss module 610

[0075]

[0076] Among them, α, β i is a hyperparameter that balances the importance of different terms.

[0077] Other forms of loss, such as the recovered residual, may also be calculated in the calculate additional loss module 620 (e.g., MSE or SSIM measures). With the original residual Distortion loss between and distortion loss You can also choose and With the entire RD loss The final loss for the entire system is obtained by combining the gradients of the final loss and backpropagating them to update the learnable parameters in the system. Note that different components (i.e., networks or modules) can be updated at different times with different update frequencies. In some embodiments, some components or some parameters within some components can be pre-trained and fixed, with only the remaining parameters being updated during training.

[0078] Figure 7 is a flowchart of a neural network image compression method with adaptive intra-frame prediction according to an embodiment.

[0079] In some embodiments, Figure 7 One or more process blocks of may be performed by platform 120. In some embodiments, Figure 7 One or more of the process blocks may be performed by another device or group of devices, such as user device 110, that is separate from or includes platform 120. Figure 7 Example blocks of the method are shown, but in some embodiments, the method may include comparing Figure 7 The method may include additional blocks, fewer blocks, different blocks, or differently arranged blocks than those depicted in the method. Additionally or alternatively, two or more blocks of the method may be executed in parallel.

[0080] like Figure 7 As shown, in operation 701, the method includes receiving an optimal partitioning scheme and receiving a compressed representation of an input, wherein the input includes a first set of blocks. For each block in the first set of blocks, operations 702 to 709 are performed.

[0081] In operation 702, Figure 7 The method includes receiving a block selection signal, wherein the block selection signal indicates that one of a first recovery block and a second recovery block is used as a current recovery block.

[0082] In operation 703 , based on the block selection signal, the method proceeds to one of operation 704 to operation 705 or operation 706 to operation 709 .

[0083] In operation 704, Figure 7 The method includes compressing blocks in a first set of blocks using a first neural network to calculate a first compressed representation.

[0084] In operation 705, Figure 7 The method includes decompressing the first compressed representation using a second neural network to compute a first recovered block.

[0085] In operation 706, Figure 7 The method includes calculating a first predicted block based on a set of previously restored blocks and a set of previously restored microblocks.

[0086] In operation 707, Figure 7 The method includes calculating a first residual based on a current block and a first prediction block in a first group of blocks.

[0087] In operation 708, Figure 7 The method includes generating a restored residual based on the first residual.

[0088] In operation 709, Figure 7 The method includes partitioning a first prediction block and adding a recovery residual to obtain a second recovery block.

[0089] In operation 710, Figure 7 The method comprises merging each of the current restored blocks to obtain a reconstructed image.

[0090] Figure 8 is a block diagram of an apparatus for neural network image compression with adaptive intra-frame prediction according to an embodiment.

[0091] like Figure 8 As shown, the apparatus includes a first receiving code 801, a second receiving code 802, a third receiving code 803, a first compression code 804, a first decompression code 805, a first prediction code 806, a first residual code 807, a first generated code 808, a first partition code 809 and a merge code 810.

[0092] The first receiving code 801 is configured to enable at least one processor to receive the optimal partitioning method.

[0093] The second receiving code 802 is configured to cause at least one processor to receive a compressed representation of an input comprising a first set of blocks, and for each block in the first set of blocks.

[0094] The third receiving code 803 is configured to enable at least one processor to receive a block selection signal, where the block selection signal indicates to use one of the first recovery block and the second recovery block as a current recovery block.

[0095] The first compression code 804 is configured to cause the at least one processor to compress blocks in the first set of blocks using a first neural network to calculate a first compressed representation.

[0096] The first decompression code 805 is configured to cause the at least one processor to decompress the first compressed representation using the second neural network to calculate a first recovered block.

[0097] The first prediction code 806 is configured to cause at least one processor to predict a first prediction block based on a set of previously restored blocks and a set of previously restored microblocks.

[0098] The first residual code 807 is configured to enable at least one processor to calculate a first residual based on the current block and the predicted block in the first group of blocks.

[0099] The first generating code 808 is configured to enable at least one processor to generate a restored residual based on the first residual.

[0100] The first partitioning code 809 is configured to enable at least one processor to partition the first prediction block and add the restored residuals to obtain a second restored block.

[0101] The merging code 810 is configured to cause at least one processor to merge each current restored block to obtain a reconstructed image.

[0102] Although Figure 8 Example blocks of an apparatus are shown, but in some implementations, the apparatus may include Figure 4 More blocks, fewer blocks, different blocks, or differently arranged blocks than those depicted in FIG. Additionally or alternatively, two or more blocks of a device may be combined.

[0103] The present invention describes a method for adaptive block partitioning and block compression method selection using intra-frame prediction and original image pixels, as well as a method for using different block sizes for intra-frame prediction residual generation and block-based neural compression. This INC codec method facilitates a flexible and general architecture that can accommodate different intra-frame prediction methods, different neural compression methods for both residual and original image blocks, and different microblock and block partitioning.

[0104] The proposed INC encoding and decoding methods can be used individually or combined in any order. Furthermore, each method (or embodiment), encoder, and decoder can be implemented by a processing circuit (e.g., at least one processor or at least one integrated circuit). In one example, the at least one processor executes a program stored in a non-volatile computer-readable medium.

[0105] The above disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations.

[0106] As used in this application, the term component is intended to be broadly interpreted as hardware, firmware, or a combination of hardware and software.

[0107] It is apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not a limitation of the implementation. Therefore, this application describes the operation and behavior of the systems and / or methods without reference to specific software code - it is understood that software and hardware can be designed to implement the systems and / or methods based on the description of this application.

[0108] Even though particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of possible implementations includes each dependent claim in combination with all other claims in the claim set.

[0109] Unless expressly stated otherwise, no element, act, or instruction used herein should be construed as critical or essential. Furthermore, as used herein, the term "group" is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, etc.) and is used interchangeably with "one or more." Furthermore, unless expressly stated otherwise, the phrase "based on" is intended to mean "based at least in part on."

Claims

1. An image coding method, characterized in that: include: Input an image, partition the image into multiple microblocks, use multiple partitioning methods to partition each microblock into multiple first partition blocks, wherein the first partition blocks are obtained by the pth partitioning method. The first partition block, p, is a positive integer; Regarding the For each first partition block in the first partition blocks, calculating a first prediction block of the first partition block based on the prediction network, and calculating a first residual block based on the first partition block and the first prediction block; right repartitioning the first residual blocks to obtain q second residual blocks, where q is a positive integer; right repartitioning the first prediction blocks to obtain q second prediction blocks; Regarding the repartitioning the first partition blocks to obtain q second partition blocks, compressing the j-th second partition block to obtain a j-th compressed representation, and decompressing the j-th compressed representation to obtain a j-th restored block; calculating a j-th compression quality loss based on the j-th second partition block, the j-th restored block, and the j-th compressed representation, where j is a positive integer, j=1, …, q; compressing a j-th second residual block among the q second residual blocks to obtain a j-th compressed residual representation, decompressing the j-th compressed residual representation to obtain a j-th restored residual block, and adding the j-th restored residual block to the j-th second prediction block to obtain a j-th reconstructed block; calculating a j-th residual compression quality loss based on the j-th second partition block, the j-th reconstructed block, and the j-th compressed residual representation; and, Based on the j-th residual compression quality loss and the j-th compression quality loss, a j-th block selection signal is generated, where the j-th block selection signal is used to indicate whether to use the second residual block to generate a compressed residual representation or to use the second partition block to generate a compressed representation.

2. The method according to claim 1, characterized in that Also includes: Calculating a j-th optimal quality loss based on the j-th residual compression quality loss and the j-th compression quality loss; Calculate the total quality loss of the p-th partitioning method based on the q best quality losses; The partitioning method with the smallest total mass loss among the multiple partitioning methods is determined as the optimal partitioning method.

3. The method according to claim 1 or 2, characterized in that The generating a j-th block selection signal based on the j-th residual compression quality loss and the j-th compression quality loss includes: When the j-th residual compression quality loss is less than the j-th compression quality loss, the block selection signal indicates to use the second residual block to generate a compressed residual representation; When the j-th compression quality loss is less than the j-th residual compression quality loss, the block selection signal indicates to use the second partition block to generate the compressed representation.

4. The method according to any one of claims 1 to 3, characterized in that compressing the j-th second partition block using a first neural network to obtain the j-th compressed representation; decompressing the j-th compressed representation using a second neural network to obtain the j-th restored block; compressing the j-th second residual block using a third neural network to obtain the j-th compressed residual representation; Using a fourth neural network, decompressing the j-th compressed residual representation to obtain the j-th restored residual block.

5. The method according to claim 4, characterized in that Also includes: Training each of the first neural network, the second neural network, the third neural network, and the fourth neural network by: Initialize the weight coefficients; Compute the total loss and propagate the gradient of the total loss backward to update the learnable parameters.

6. The method according to any one of claims 1 to 5, characterized in that The input of the prediction network includes: a plurality of previously encoded microblocks and a plurality of previously encoded first partition blocks in the current microblock.

7. An image encoding device, characterized in that include: An input module is used to input an image, partition the image into a plurality of microblocks, and use a plurality of partitioning methods to partition each microblock into a plurality of first partition blocks, wherein the first partition blocks are obtained by the pth partitioning method. The first partition block, p, is a positive integer; A calculation module for For each first partition block in the first partition blocks, calculate a first prediction block of the first partition block based on the prediction network, and calculate a first residual block based on the first partition block and the first prediction block; Repartition the first residual blocks to obtain q second residual blocks, where q is a positive integer; Repartition the first prediction blocks to obtain q second prediction blocks; repartitioning the first partition blocks to obtain q second partition blocks, compressing the j-th second partition block to obtain a j-th compressed representation, decompressing the j-th compressed representation to obtain a j-th restored block; calculating a j-th compression quality loss based on the j-th second partition block, the j-th restored block, and the j-th compressed representation, where j is a positive integer, j=1, ..., q; compressing the j-th second residual block among the q second residual blocks to obtain a j-th compressed residual representation, decompressing the j-th compressed residual representation to obtain a j-th restored residual block, adding the j-th restored residual block to the j-th second prediction block to obtain a j-th reconstructed block; calculating a j-th residual compression quality loss based on the j-th second partition block, the j-th reconstructed block, and the j-th compressed residual representation; and, A generating module is configured to generate a jth block selection signal based on the jth residual compression quality loss and the jth compression quality loss, wherein the jth block selection signal is used to indicate whether to use the second residual block to generate a compressed residual representation or to use the second partition block to generate a compressed representation.

8. The device according to claim 7, characterized in that Also includes: a determination module, configured to calculate a jth optimal quality loss based on the jth residual compression quality loss and the jth compression quality loss; The total quality loss of the p-th partitioning method is calculated based on the q optimal quality losses; and the partitioning method with the smallest total quality loss among the multiple partitioning methods is determined as the optimal partitioning method.

9. The device according to claim 7 or 8, characterized in that The generation module is configured to, when the j-th residual compression quality loss is less than the j-th compression quality loss, indicate, by the block selection signal, to use the second residual block to generate a compressed residual representation; and when the j-th residual compression quality loss is less than the j-th residual compression quality loss, indicate, by the block selection signal, to use the second partition block to generate a compressed representation.

10. The device according to any one of claims 7 to 9, characterized in that The computing module is configured to compress the j-th second partition block using a first neural network to obtain the j-th compressed representation; decompress the j-th compressed representation using a second neural network to obtain the j-th restored block; compress the j-th second residual block using a third neural network to obtain the j-th compressed residual representation; and decompress the j-th compressed residual representation using a fourth neural network to obtain the j-th restored residual block.

11. The device according to claim 10, characterized in that Also includes: a training module, configured to train each of the first neural network, the second neural network, the third neural network, and the fourth neural network by: Initialize the weight coefficients; Compute the total loss and propagate the gradient of the total loss backward to update the learnable parameters.

12. The device according to any one of claims 7 to 11, characterized in that The input of the prediction network includes: a plurality of previously encoded microblocks and a plurality of previously encoded first partition blocks in the current microblock.

13. A method for transmitting a code stream, characterized in that: Execute the image encoding method according to any one of claims 1 to 6 to generate the code stream; and transmit the code stream.

14. A computer device, characterized in that: The system comprises a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the method according to any one of claims 1 to 6.

15. A non-volatile computer-readable medium, characterized in that Instructions and code streams are stored thereon, and when the instructions are executed by at least one processor, the method according to any one of claims 1 to 6 is implemented to generate the code stream.

Images