Method, encoding method, decoding method, and electronic device for training an image processing network

Abstract

[0003] The present application provides a method, an encoding method, a decoding method, and an electronic device for training an image processing network. The training method includes the steps of: obtaining a first training image and a first predicted image, and obtaining a period of a checkerboard effect, where the first predicted image is generated by performing image processing on the first training image based on an image processing network; dividing the first training image into M first image blocks and the first predicted image into M second image blocks based on the period, where both the size of the first image block and the size of the second image block are related to the period; determining a first loss based on the M first image blocks and the M second image blocks; and then training the image processing network based on the first loss. After the image processing network is trained based on the training method of the present application, the checkerboard effect of an image obtained by processing performed using the trained image processing network can be removed to a certain extent, and the visual quality of the image obtained by processing performed using the trained image processing network can be improved.

Description

[Technical Field]

[0001] This application claims priority to Chinese Patent Application No. 202210945704.0, entitled "Method, encoding method, decoding method, and electronic device for training an image processing network," filed with the State Intellectual Property Office of China on August 8, 2022, the entire contents of which are incorporated herein by reference.

[0002] TECHNICAL FIELD Embodiments of the present application relate to the field of image processing, and in particular to methods, encoding methods, decoding methods, and electronic devices for training image processing networks. [Background technology]

[0003] Deep learning has also been applied to various image processing tasks, such as image compression, image restoration, and image super-resolution, based on the performance of deep learning, which far exceeds that of traditional image algorithms in many areas, such as image recognition and target detection.

[0004] Typically, in many image processing scenarios (e.g., image compression, image restoration, and image super-resolution), a checkerboard effect appears in the image obtained by processing performed using a deep learning network. In other words, a grid that closely resembles a checkerboard appears in a partial or entire region of the obtained image, resulting in a significant reduction in the visual quality of the image obtained by image processing. Summary of the Invention [Means for solving the problem]

[0005] The present application provides a method for training an image processing network, an encoding method, a decoding method, and an electronic device. After the image processing network is trained based on the training method, the checkerboard effect in an image obtained by processing performed using the trained image processing network can be removed to a certain extent.

[0006] According to a first aspect, an embodiment of the present application provides a method for training an image processing network, the method including: first obtaining a first training image and a first predicted image, and obtaining a period of a checkerboard effect, where the first predicted image is generated by performing image processing on the first training image based on an image processing network; dividing the first training image into M first image blocks and the first predicted image into M second image blocks based on the period, where both the size of the first image block and the size of the second image block are related to the period, and M is an integer greater than 1; determining a first loss based on the M first image blocks and the M second image blocks; and then training the image processing network based on the first loss.

[0007] Because the checkerboard effect is periodic, in this application, images before and after processing performed using an image processing network (the first training image is the image before processing performed using the image processing network, and the first predicted image is the image after processing performed using the image processing network) are divided into image blocks based on the period of the checkerboard effect. Then, a loss is calculated by comparing the differences between the image blocks before and after processing performed using the image processing network (the M first image blocks are the image blocks before processing performed using the image processing network, and the M second image blocks are the image blocks after processing performed using the image processing network). The image processing network is trained based on the loss to effectively compensate for each image block processed using the image processing network, thereby reducing the difference between each first image block and the corresponding second image block. Both the size of the first image block and the size of the second image block are related to the period of the checkerboard effect. As the difference between each first image block and the corresponding second image block becomes smaller, the checkerboard effect in each period is also eliminated to a certain extent. In this way, after the image processing network is trained based on the training method of the present application, the checkerboard effect of the image obtained by processing performed using the trained image processing network can be removed to a certain extent, thereby improving the visual quality of the image obtained by processing performed using the trained image processing network.

[0008] For example, a checkerboard effect is a periodic noise on an image (periodic noise is noise associated with a spatial domain and a particular frequency), and the period of the checkerboard effect is the period of the noise.

[0009] For example, the period of the checkerboard effect is two-dimensional (the checkerboard effect is a phenomenon in which a grid that closely resembles a checkerboard appears in a partial or entire area of ​​an image, i.e., the checkerboard effect is two-dimensional, and two-dimensional periodicity means that the period of the checkerboard effect contains two-dimensional values, with the two dimensions corresponding to the length and width of the image). The number of periods of the checkerboard effect is M.

[0010] For example, the shape of the period of the checkerboard effect may be rectangular, and the size of the period of the checkerboard effect may be expressed as p*q, where p and q are positive integers, and the units of p and q are px (pixels), and p and q may or may not be equal, which is not limited in this application.

[0011] It should be understood that the period of the checkerboard effect may alternatively be another shape (e.g., triangular, elliptical, or irregular), and this is not a limitation of the present application.

[0012] For example, the first loss is used to compensate for the checkerboard effect in the image.

[0013] For example, the sizes of the M first image blocks may be the same or different, which is not limited in this application.

[0014] For example, the sizes of the M second image blocks may be the same or different, which is not limited in this application.

[0015] For example, the sizes of the first image block and the second image block may be the same or different, which is not limited in this application.

[0016] For example, the size of the first image block and the size of the second image block related to the period may indicate that the size of the first image block and the size of the second image block are determined based on the period of the checkerboard effect. For example, the size of the first image block and the size of the second image block may be larger, smaller, or equal to the period of the checkerboard effect.

[0017] For example, image processing networks can be applied to image super-resolution, which is the restoration of a low-resolution image or video into a high-resolution image or video.

[0018] For example, image processing networks can be applied to image restoration, which is restoring an image or video with blurry subregions to an image or video with clear details of those subregions.

[0019] For example, an image processing network may be applied to encoding and decoding images.

[0020] For example, when an image processing network is applied to image encoding and decoding, the bit rate point at which the checkerboard effect appears can be reduced. In other words, compared with the prior art, it can be found that the bit rate point at which the checkerboard effect appears is fewer in images obtained by encoding and decoding performed using an image processing network trained according to the training method of the present application. In addition, in the case of a medium bit rate (for example, the bit rate may be 0.15 Bpp (bits per pixel) to 0.3 Bpp, and may be specifically set according to requirements), images obtained by encoding and decoding performed using an image processing network trained according to the training method of the present application have higher quality.

[0021] For example, when the image processing network includes upsampling layers, the period of the checkerboard effect may be determined based on the number of upsampling layers.

[0022] In the first aspect, determining a first loss based on M first image blocks and M second image blocks includes: acquiring a first feature block based on the M first image blocks, where the characteristic value of the first feature block is obtained by calculation based on pixels at corresponding positions in the M first image blocks; acquiring a second feature block based on the M second image blocks, where the characteristic value of the second feature block is obtained by calculation based on pixels at corresponding positions in the M second image blocks; and determining a first loss based on the first feature block and the second feature block. In this way, information from the M first image blocks is aggregated into one or more first feature blocks, and information from the M second image blocks is aggregated into one or more second feature blocks. Next, a first loss is calculated by comparing the first feature block with the second feature block to more intentionally compensate for the periodic checkerboard effect and thereby achieve a better effect of eliminating the checkerboard effect.

[0023] It should be understood that the number of first feature blocks and the number of second feature blocks are not limited in the present application.

[0024]

[0013] Relating to the first aspect or any one of the embodiments of the first aspect, the step of obtaining a first feature block based on M first image blocks includes the step of obtaining a first feature block based on N first image blocks of the M first image blocks, where characteristic values ​​of the first feature block are obtained by calculation based on pixels at corresponding positions in the N first image blocks, where N is a positive integer less than or equal to M. The step of obtaining a second feature block based on M second image blocks includes the step of obtaining a second feature block based on N second image blocks of the M second image blocks, where characteristic values ​​of the second feature block are obtained by calculation based on pixels at corresponding positions in the N second image blocks. In this way, calculations may be performed based on pixels at corresponding positions in some or all of the first image blocks to obtain the first feature block, and calculations may be performed based on pixels at corresponding positions in some or all of the second image blocks to obtain the second feature block. When the first feature block is obtained by calculation based on pixels at corresponding positions of a portion of the first image block and the second feature block is obtained by calculation based on pixels at corresponding positions of a portion of the second image block, less information is used to calculate the first loss, thereby improving the efficiency of calculating the first loss. When the first feature block is obtained by calculation based on pixels at corresponding positions of all the first image blocks and the second feature block is obtained by calculation based on pixels at corresponding positions of all the second image blocks, more comprehensive information is used to calculate the first loss, thereby improving the accuracy of the first loss.

[0025]

[0013] In relation to the first aspect or any one of the embodiments of the first aspect, the step of obtaining a first feature block based on the M first image blocks includes: performing a calculation based on first target pixels at corresponding first positions in all of the M first image blocks to obtain feature values ​​at corresponding first positions in the first feature block, where the number of first target pixels is less than or equal to a total number of pixels included in the first image blocks, and one feature value is correspondingly obtained for the first target pixels at the same first positions in the M first image blocks. The step of obtaining a second feature block based on the M second image blocks includes performing a calculation based on second target pixels at corresponding second positions in all of the M second image blocks to obtain feature values ​​at corresponding second positions in the second feature block, where the number of second target pixels is less than or equal to the total number of pixels included in the second image blocks, and one feature value is correspondingly obtained for the second target pixels at the same second positions in the M second image blocks. In this way, a feature block (first feature block / second feature block) can be calculated based on some or all of the pixels in an image block (first image block / second image block). When the calculation is performed based on some of the pixels in the image block, less information is used to calculate the first loss, thereby improving the efficiency of calculating the first loss. When the calculation is performed based on all the pixels in the image block, more comprehensive information is used to calculate the first loss, thereby improving the accuracy of the first loss.

[0026] For example, when N is less than M, the step of obtaining a first feature block based on N first image blocks among the M first image blocks may include the step of performing a calculation based on first target pixels at corresponding first positions in all of the N first image blocks to obtain feature values ​​at corresponding first positions in the first feature block, wherein one feature value is correspondingly obtained for the first target pixel at the same first position in the N first image blocks; and the step of obtaining a second feature block based on N second image blocks among the M second image blocks may include the step of performing a calculation based on second target pixels at corresponding second positions in all of the N second image blocks to obtain feature values ​​at corresponding second positions in the second feature block, wherein one feature value is correspondingly obtained for the second target pixel at the same second position in the N second image blocks.

[0027] In relation to the first aspect or any one of the embodiments thereof, the step of acquiring a first feature block based on M first image blocks includes a step of determining a feature value at a corresponding position in the first feature block based on average values ​​of pixels at corresponding positions in all of the M first image blocks, where one feature value is corresponding to the average value of pixels at the same positions in the M first image blocks. The step of acquiring a second feature block based on M second image blocks includes a step of determining a feature value at a corresponding position in the second feature block based on average values ​​of pixels at corresponding positions in all of the M second image blocks, where one feature value is corresponding to the average value of pixels at the same positions in the M second image blocks. In this way, the feature value of the feature block (first feature block / second feature block) is determined by a method of calculating pixel average values. This is computationally simple, thereby improving the efficiency of calculating the first loss.

[0028] It should be noted that in this application, in addition to the above-mentioned method of calculating pixel average values, another linear calculation method (e.g., linear weighting) may be used to perform calculations on pixels to determine the characteristic values ​​of feature blocks, which is not limited in this application.

[0029] It should be noted that in the present application, the calculation may alternatively be performed on pixels in a non-linear calculation manner to determine the characteristic value of the feature block. For example, the non-linear calculation may be performed according to the following formula: F i =A1*e 1i 2 +A2*e 2i 2 +A3*e 3i 2 +…+AM*e Mi 2

[0030] where F i represents the feature value at the ith position in the first feature map. 1i represents the pixel at the i-th position in the first image block, and e 2i represents the pixel at the ith position in the second first image block, and e 3i represents the pixel at the i-th position in the third first image block, ..., e Mi represents the pixel at the ith position in the Mth first image block, A1 represents the weighting factor corresponding to the pixel at the ith position in the first first image block, A2 represents the weighting factor corresponding to the pixel at the ith position in the second first image block, A3 represents the weighting factor corresponding to the pixel at the ith position in the third first image block, ..., AM represents the weighting factor corresponding to the pixel at the ith position in the Mth first image block, where i is an integer ranging from 1 to M (i may be equal to 1 or M).

[0031] It should be noted that in the present application, calculations can be performed on pixels based on a convolutional layer to determine the characteristic values ​​of the feature blocks. For example, M first image blocks can be input to a convolutional layer (one or more layers) and a fully connected layer to obtain a first feature block output by the fully connected layer, and M second image blocks can be input to a convolutional layer (one or more layers) and a fully connected layer to obtain a second feature block output by the fully connected layer.

[0032] It should be understood that other calculation methods may alternatively be used to perform calculations on pixels to determine the characteristic values ​​of feature blocks, and this is not a limitation of this application.

[0033] For example, when N is less than M, the step of obtaining a first feature block based on N first image blocks out of the M first image blocks may include the step of determining a characteristic value at a corresponding position in the first feature block based on an average value of pixels at corresponding positions in all of the N first image blocks, where one characteristic value is correspondingly obtained for the average value of pixels at the same positions in the N first image blocks; and the step of obtaining a second feature block based on N second image blocks out of the M second image blocks may include the step of determining a characteristic value at a corresponding position in the second feature block based on an average value of pixels at corresponding positions in all of the N second image blocks, where one characteristic value is correspondingly obtained for the average value of pixels at the same positions in the N second image blocks.

[0034]

[0023] Relating to the first aspect or any one of the embodiments of the first aspect, determining the first loss based on the first feature block and the second feature block includes determining the first loss based on a point-to-point loss between the first feature block and the second feature block.

[0035] For example, the point-to-point loss (i.e., point-based loss) may include an Ln distance (e.g., an L1 distance (Manhattan distance), an L2 distance (Euclidean distance), or an L-Inf distance (Chebyshev distance)). This is not limited in this application.

[0036]

[0023] Relating to the first aspect or any one of the embodiments of the first aspect, determining the first loss based on the first feature block and the second feature block includes determining the first loss based on a feature-based loss between the first feature block and the second feature block.

[0037] For example, the feature-based loss may include SSIM (Structural Similarity), MSSSIM (Multi-Scale Structural Similarity), and LPIPS (Learned Perceptual Image Patch Similarity), which is not limited in this application.

[0038] For example, the first feature block and the second feature block may be further input into a neural network (e.g., a convolutional network or a VGG (Visual Geometry Group Network) network), which outputs a first feature of the first feature block and a second feature of the second feature block. Then, the distance between the first feature and the second feature is calculated to obtain a feature-based loss.

[0039] According to the first aspect or any one of the embodiments of the first aspect, the image processing network includes an encoding network and a decoding network. Before obtaining the first training image and the first predicted image, the method further includes obtaining a second training image and a second predicted image, where the second predicted image is obtained by encoding the second training image based on an untrained encoding network and then decoding the encoding result of the second training image based on an untrained decoding network; determining a second loss based on the second predicted image and the second training image; and pre-training the untrained encoding network and the untrained decoding network based on the second loss. In this way, by pre-training the image processing network, the image processing network can converge more quickly and better in a subsequent training process.

[0040] For example, to obtain the second loss, a bitrate loss and a mean squared error loss may be determined based on the second predicted image and the second training image, and then the bitrate loss and the mean squared error loss are weighted.

[0041] The bitrate loss indicates the size of the bitstream. The mean squared error loss may be the mean squared error between the second predicted image and the second training image, and may be used to improve an objective measure of the image (e.g., PSNR (Peak Signal to Noise Ratio)).

[0042] For example, the weighting factors corresponding to the bit rate loss and the mean square error loss may be the same or different, which is not limited in this application.

[0043] According to the first aspect or any one of the embodiments of the first aspect, the first predicted image is obtained by encoding a first training image based on a pre-trained encoding network, and then decoding the encoding result of the first training image based on a pre-trained decoding network. The step of training the image processing network based on the first loss includes the steps of: determining a third loss based on a classification result obtained by a classification network for the first training image and the first predicted image, the third loss being a generative adversarial network (GAN) loss, the GAN network including a classification network and a decoding network; and training the pre-trained encoding network and the pre-trained decoding network based on the first loss and the third loss. In this way, checkerboard effect compensation can be better performed by referring to the first loss and the third loss to eliminate the checkerboard effect to a higher extent.

[0044] For example, the decoding network may include an upsampling layer, and the period of the checkerboard effect may be determined based on the upsampling layer in the decoding network.

[0045]

[0013] In relation to the first aspect or any one of the embodiments of the first aspect, training the image processing network based on the first loss further includes determining a fourth loss, where the fourth loss includes at least one of the following: an L1 loss, a bitrate loss, a perceptual loss, and an edge loss. Training the pre-trained encoding network and the pre-trained decoding network based on the first loss and the third loss includes training the pre-trained encoding network and the pre-trained decoding network based on the first loss, the third loss, and the fourth loss.

[0046] The L1 loss can be used to improve an objective indicator of an image (e.g., PSNR (Peak Signal to Noise Ratio)). The bitrate loss indicates the size of the bitstream. The perceptual loss can be used to improve the visual effect of an image. The edge loss can be used to prevent edge distortion. In this way, by training an image processing network with reference to multiple losses, the quality of the image processed by the trained image processing network (including objective quality and subjective quality (also called visual quality)) can be improved.

[0047] For example, the weighting factors corresponding to the first loss, the third loss, the L1 loss, the bit rate loss, the perceptual loss, and the edge loss may be the same or different, which is not limited in this application.

[0048] In relation to the first aspect or any one of the embodiments of the first aspect, the image processing network further includes a hyperprior encoding network and a hyperprior decoding network, and the hyperprior decoding network and the decoding network each include an upsampling layer. The period includes a first period and a second period. The first period is determined based on the number of upsampling layers in the decoding network. The second period is determined based on the first period and the number of upsampling layers in the hyperprior decoding network. The second period is greater than the first period.

[0049] In relation to the first aspect or any one of the embodiments of the first aspect, the step of dividing the first training image into M first image blocks and dividing the first predicted image into M second image blocks based on a periodicity includes the steps of: dividing the first training image into M first image blocks based on a first period and a second period, wherein the M first image blocks include M1 third image blocks and M2 fourth image blocks, wherein a size of the third image blocks is related to the first period and a size of the fourth image blocks is related to the second period, M1 and M2 are positive integers, and M1+M2=M; and dividing the first predicted image into M second image blocks based on the first period and the second period, wherein the M second image blocks include M1 fifth image blocks and M2 sixth image blocks, wherein a size of the fifth image blocks is related to the first period and a size of the sixth image blocks is related to the second period. When the first losses include fifth losses and sixth losses, determining the first losses based on the M first image blocks and the M second image blocks includes determining fifth losses based on M1 third image blocks and M1 fifth image blocks, and determining sixth losses based on M2 fourth image blocks and M2 sixth image blocks.

[0050]

[0016] Relating to the first aspect or any one of the embodiments of the first aspect, training the image processing network based on the first loss includes performing a weighting calculation on the fifth loss and the sixth loss to obtain a seventh loss, and training a pre-trained encoding network and a pre-trained decoding network based on the seventh loss.

[0051] When the image processing network includes an encoding network and a decoding network, and further includes a hyperprior encoding network and a hyperprior decoding network, the hyperprior decoding network causes a checkerboard effect with a longer period, so that in order to compensate for the checkerboard effect with a longer period, the image is divided into blocks with a longer period and then the loss is determined. In this way, the checkerboard effect with a longer period is eliminated to a certain extent, thereby further improving the image quality.

[0052] For example, the weighting coefficients corresponding to the fifth loss and the sixth loss may be the same or different, which is not limited in this application.

[0053] For example, the weighting factor corresponding to the fifth loss may be greater than the weighting factor corresponding to the sixth loss.

[0054] It should be understood that the period of the checkerboard effect may further include more periods different from the first period and the second period. For example, the period of the checkerboard effect may include k periods (the k periods may be the first period, the second period, ..., and the kth period, respectively, where k is an integer greater than 2). In this way, the first training image may be divided into M first image blocks based on the first period, the second period, ..., and the kth period. The M first image blocks may include M1 image blocks 11, M2 image blocks 12, ..., and Mk image blocks 1k. The size of image block 11 is related to the first period, the size of image block 12 is related to the second period, ..., the size of image block 1k is related to the kth period, and M1 + M2 + ... + Mk = M. The first predicted image may be divided into M second image blocks based on the first period, the second period, ..., and the kth period. The M second image blocks may include M1 image blocks 21, M2 image blocks 22, ..., and Mk image blocks 2k. The size of image block 21 is associated with the first period, the size of image block 22 is associated with the second period, ..., the size of image block 2k is associated with the kth period. Next, Loss 1 may be determined based on the M1 image blocks 11 and the M1 image blocks 21, Loss 2 may be determined based on the M2 image blocks 12 and the M2 image blocks 22, ..., and Loss sk may be determined based on the Mk image blocks 1k and the Mk image blocks 2k. Thereafter, the pre-trained encoding network and the pre-trained decoding network may be trained based on Loss 1, Loss 2, ..., and Loss sk.

[0055] According to a second aspect, an embodiment of the present application provides an encoding method. The method includes the steps of: obtaining an image to be encoded; inputting the image to be encoded into an encoding network to obtain a feature map output by the encoding network; processing the image to be encoded by the encoding network; and performing entropy encoding on the feature map to obtain a first bitstream. The encoding network is obtained by training using any one of the first aspect and the implementations of the first aspect. Correspondingly, a decoder decodes the first bitstream using a decoding network obtained by training using any one of the first aspect and the implementations of the first aspect. Furthermore, while it is guaranteed that the reconstructed image does not exhibit a checkerboard effect, the present application can encode the same image at a lower bitrate than the prior art. In addition, when the same image is encoded using the same bitrate (e.g., a medium bitrate), the encoding quality in the present application is higher than that in the prior art.

[0056] In accordance with the second aspect, the step of performing entropy coding on the feature map to obtain a first bitstream includes inputting the feature map into a hyperprior encoding network to obtain hyperprior features, processing the feature map through the hyperprior encoding network, inputting the hyperprior features into a hyperprior decoding network, processing the hyperprior features through the hyperprior decoding network, and then outputting a probability distribution. The hyperprior encoding network and the hyperprior decoding network are both obtained by training using the first aspect and any one of the embodiments of the first aspect. Hyperprior encoding networks and hyperprior decoding networks trained using prior art training methods result in a checkerboard effect with a larger period. In comparison, the present application can avoid the checkerboard effect with a larger period on the reconstructed image.

[0057]

[0013] In relation to the second aspect or any one of the embodiments of the second aspect, entropy coding is performed on the hyperprior features to obtain a second bitstream. Thus, after subsequently receiving the first bitstream and the second bitstream, to assist the decoder in performing decoding, the decoder may determine a probability distribution based on the hyperprior features obtained by decoding the second bitstream, and then decode and reconstruct the first bitstream based on the probability distribution to obtain a reconstructed image.

[0058] According to a third aspect, an embodiment of the present application provides a decoding method. The method includes the steps of: obtaining a first bitstream, the first bitstream being a bitstream of a feature map; then performing entropy decoding on the first bitstream to obtain a feature map; and inputting the feature map into a decoding network and processing the feature map by the decoding network to obtain a reconstructed image output by the decoding network. The decoding network is obtained by training using any one of the first aspect and the implementations of the first aspect. Correspondingly, the first bitstream is obtained by encoding performed by an encoder based on the encoding network obtained by training using any one of the first aspect and the implementations of the first aspect. Furthermore, when the bitrate of the first bitstream is lower than that of the prior art, a checkerboard effect does not appear in the reconstructed image in the present application. In addition, when the bitrate of the first bitstream is the same as that of the prior art (e.g., a medium bitrate), the quality of the reconstructed image in the present application is higher.

[0059] According to a third aspect, the method further includes a step of obtaining a second bitstream, the second bitstream being a bitstream of hyperprior features. The step of performing entropy decoding on the bitstream to obtain a feature map includes a step of performing entropy decoding on the second bitstream to obtain hyperprior features, a step of inputting the hyperprior features into a hyperprior decoding network and processing the hyperprior features by the hyperprior decoding network to obtain a probability distribution, and a step of performing entropy decoding on the first bitstream based on the probability distribution to obtain a feature map. In this way, the decoder can directly decode the bitstream to obtain the probability distribution without recalculation, thereby improving decoding efficiency. In addition, the hyperprior decoding network is obtained by training performed using any one of the first aspect and the embodiments of the first aspect. A hyperprior decoding network trained using a prior art training method results in a checkerboard effect with a larger period. In comparison, in the present application, the checkerboard effect with a larger period can be avoided on the reconstructed image.

[0060] According to a fourth aspect, an embodiment of the present application provides an electronic device including a memory and a processor. The memory is coupled to the processor. The memory stores program instructions. When the program instructions are executed by the processor, the electronic device is enabled to perform the training method according to the first aspect or any one of the possible implementations of the first aspect.

[0061] The fourth aspect and any one of the embodiments of the fourth aspect correspond to the first aspect and any one of the embodiments of the first aspect, respectively. For technical effects corresponding to the fourth aspect and any one of the embodiments of the fourth aspect, please refer to the technical effects corresponding to the first aspect and any one of the embodiments of the first aspect. Details will not be described again here.

[0062] According to a fifth aspect, an embodiment of the present application provides a chip including one or more interface circuits and one or more processors. The interface circuits are configured to receive signals from a memory of an electronic device and transmit the signals to the processor, the signals including computer instructions stored in the memory. When the processor executes the computer instructions, the electronic device is enabled to perform the training method according to the first aspect or any one of the possible implementations of the first aspect.

[0063] The fifth aspect and any one of the embodiments of the fifth aspect correspond to the first aspect and any one of the embodiments of the first aspect, respectively. For technical effects corresponding to the fifth aspect and any one of the embodiments of the fifth aspect, please refer to the technical effects corresponding to the first aspect and any one of the embodiments of the first aspect. Details will not be described again here.

[0064] According to a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium storing a computer program, which, when run on a computer or processor, enables the computer or processor to execute the training method according to the first aspect or any one of the possible implementations of the first aspect.

[0065] The sixth aspect and any one of the embodiments of the sixth aspect correspond to the first aspect and any one of the embodiments of the first aspect, respectively. For the technical effects corresponding to the sixth aspect and any one of the embodiments of the sixth aspect, please refer to the technical effects corresponding to the first aspect and any one of the embodiments of the first aspect. Details will not be described again here.

[0066] According to a seventh aspect, an embodiment of the present application provides a computer program product, the computer program product including a software program, which, when executed by a computer or processor, enables the computer or processor to perform the training method of the first aspect or any one of the possible implementations of the first aspect.

[0067] The seventh aspect and any one of the embodiments of the seventh aspect correspond to the first aspect and any one of the embodiments of the first aspect, respectively. For the technical effects corresponding to the seventh aspect and any one of the embodiments of the seventh aspect, please refer to the technical effects corresponding to the first aspect and any one of the embodiments of the first aspect. Details will not be described again here.

[0068] According to an eighth aspect, an embodiment of the present application provides a bitstream storage device. The device includes a receiver and at least one storage medium. The receiver is configured to receive a bitstream. The at least one storage medium is configured to store the bitstream. The bitstream is generated according to any one of the second aspect and the implementations of the second aspect.

[0069] The eighth aspect and any one of the embodiments of the eighth aspect correspond to the second aspect and any one of the embodiments of the second aspect, respectively. For the technical effects corresponding to the eighth aspect and any one of the embodiments of the eighth aspect, please refer to the technical effects corresponding to the second aspect and any one of the embodiments of the second aspect. Details will not be described again here.

[0070] According to a ninth aspect, an embodiment of the present application provides a bitstream transmission device. The device includes a transmitter and at least one storage medium. The at least one storage medium is configured to store a bitstream. The bitstream is generated according to any one of the second aspect and the implementations of the second aspect. The transmitter is configured to obtain the bitstream from the storage medium and transmit the bitstream to a terminal-side device using a transmission medium.

[0071] The ninth aspect and any one of the embodiments of the ninth aspect correspond to the second aspect and any one of the embodiments of the second aspect, respectively. For the technical effects corresponding to the ninth aspect and any one of the embodiments of the ninth aspect, please refer to the technical effects corresponding to the second aspect and any one of the embodiments of the second aspect. Details will not be described again here.

[0072] According to a tenth aspect, an embodiment of the present application provides a bitstream distribution system, including: at least one storage medium configured to store at least one bitstream, where the at least one bitstream is generated according to any one of the second aspect and the implementations of the second aspect; and a streaming media device configured to retrieve a target bitstream from the at least one storage medium and transmit the target bitstream to a terminal-side device, where the streaming media device includes a content server or a content distribution server.

[0073] The tenth aspect and any one of the embodiments of the tenth aspect correspond to the second aspect and any one of the embodiments of the second aspect, respectively. For the technical effects corresponding to the tenth aspect and any one of the embodiments of the tenth aspect, please refer to the technical effects corresponding to the second aspect and any one of the embodiments of the second aspect. Details will not be described again here. [Brief explanation of the drawings]

[0074] [Figure 1a] FIG. 1 is a diagram of an example of an artificial intelligence main framework. [Figure 1b] FIG. 1 is a diagram of an example of an application scenario. [Figure 1c] FIG. 1 is a diagram of an example of an application scenario. [Figure 1d] FIG. 1 is a diagram of an example of an application scenario. [Figure 1e] FIG. 1 is an example of a checkerboard effect. [Figure 2a] FIG. 1 is a diagram of an example of a training process. [Figure 2b] FIG. 1 is a diagram of an example of a deconvolution process. [Figure 3a] FIG. 1 is a diagram of an example end-to-end image compression framework. [Figure 3b] FIG. 1 is a diagram of an example end-to-end image compression framework. [Figure 3c(1)] FIG. 1 is a diagram of an example network structure. [Figure 3c(2)] FIG. 1 is a diagram of an example network structure. [Figure 4a] FIG. 1 is a diagram of an example of a training process. [Figure 4b] FIG. 1 is a diagram of an example chunking process. [Figure 4c] FIG. 10 is a diagram of an example process for generating a first feature block. [Figure 5] FIG. 1 is a diagram of an example of a training process. [Figure 6] FIG. 1 is a diagram of an example encoding process. [Figure 7] FIG. 1 is a diagram of an example of a decoding process. [Figure 8a]FIG. 10 is a diagram showing an example of an image quality comparison result. [Figure 8b] FIG. 10 is a diagram showing an example of an image quality comparison result. [Figure 8c] FIG. 10 is a diagram showing an example of an image quality comparison result. [Figure 9] 1 is a diagram of an example of the structure of the device. DETAILED DESCRIPTION OF THE INVENTION

[0075] The following clearly and completely describes the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. It is clear that the described embodiments are only a part, but not all, of the embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0076] The term "and / or" in this specification only describes an association relationship for describing associated objects, and represents that three relationships may exist. For example, A and / or B may represent the following three cases: only A exists, both A and B exist, and only B exists.

[0077] In the description and claims of the embodiments of the present application, the terms "first" and "second" are intended to distinguish between different objects, but do not indicate a specific order of the objects. For example, a first target object and a second target object are used to distinguish between different target objects, but are not used to describe a specific order of the target objects.

[0078] In the embodiments of the present application, words such as "example" and "for example" are used to indicate providing an example, illustration, or explanation. Any embodiment or design described in the embodiments of the present application as an "example" or "for example" is not described as being more preferred or having more advantages than another embodiment or design. Rather, the use of words such as "example" and "for example" is intended to present relative concepts in a concrete manner.

[0079] In describing the embodiments of this application, unless otherwise specified, "plurality" means two or more than two. For example, a plurality of processing units means two or more processing units, and a plurality of systems means two or more systems.

[0080] Figure 1a shows an example of an artificial intelligence main framework. The main framework describes the overall operation procedure of an artificial intelligence system and is applicable to the requirements of the general artificial intelligence field.

[0081] Below, we will explain the aforementioned artificial intelligence main framework from two aspects: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis).

[0082] The "intelligent information chain" reflects a series of processes from data acquisition to data processing. For example, the process may be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data goes through a refinement process of "data-information-knowledge-intelligence."

[0083] The "IT value chain" reflects the value brought to the information technology industry by artificial intelligence, from the underlying infrastructure and information of artificial intelligence (technical provision and processing implementation) to the industrial environment processes of the system.

[0084] (1) Infrastructure The infrastructure provides computing power support for the artificial intelligence system and communicates with the outside world, using a basic platform for this support. The infrastructure communicates with the outside world using sensors. Computing power is provided by smart chips (CPUs, NPUs, GPUs, ASICs, or hardware acceleration chips such as FPGAs). The basic platform includes related platform guarantees and support, such as a distributed computing framework and network, which may include cloud storage and computing, as well as interconnection and interworking networks. For example, sensors communicate with the outside world to acquire data, and the data is provided to intelligent chips in a distributed computing system provided by the basic platform for computing.

[0085] (2) Data The data in the upper layers of the infrastructure represent data sources in the field of artificial intelligence, including graphs, images, voice, and text, as well as data from the Internet of Things of traditional devices, service data from existing systems, and sensory data such as force, displacement, liquid level, temperature, and humidity.

[0086] (3) Data processing Data processing typically includes data training, machine learning, deep learning, search, inference, and decision-making.

[0087] Machine learning and deep learning can refer to performing symbolic and formal intelligent information modeling, extraction, preprocessing, training, etc. on data.

[0088] Inference is the process by which human intellectual reasoning is simulated in a computer or intelligent system, performing machine thinking and problem solving using formal information according to inference control policies. Typical functions are searching and matching.

[0089] Decision making is the process of making decisions after intelligent information is inferred, usually providing functions such as classification, ranking, and prediction.

[0090] (4) General abilities After the above-mentioned data processing is performed on the data, some general capabilities, such as algorithms or general systems, such as translation, text analysis, computer vision processing, voice recognition, image recognition, and texture mapping generation, may be further formed based on the data processing results.

[0091] (5) Intelligent products and industrial applications Smart products and industrial applications are the products and applications of artificial intelligence systems in various fields, and are a comprehensive solution package of artificial intelligence, which enables intelligent information decision-making and application. The application fields mainly include smart manufacturing, smart transportation, smart home, smart healthcare, smart security protection, autonomous driving, and smart terminals.

[0092] The image processing network in this application may be used to perform machine learning, deep learning, search, inference, decision-making, etc. The image processing network referred to in this application may include multiple types of neural networks, such as deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), residual networks, neural networks using transformer models, or other neural networks, which are not limited in this application.

[0093] The operation of each layer of a neural network is expressed by the formula

number

number

[0094] The goal of training a neural network is to eventually obtain a weight matrix (a weight matrix formed by a vector W across multiple layers) for all layers of the trained neural network. Therefore, the training process of a neural network is essentially a way to learn to control spatial transformations, and more specifically, to learn a weight matrix.

[0095] For example, according to the training method provided in this application, all image processing networks that can be used for image processing and that exhibit a checkerboard effect in the images obtained after image processing can be trained.

[0096] Figure 1b is a diagram of an example application scenario. For example, the image processing network in Figure 1b is an image super-resolution network, and the application scenario corresponding to Figure 1b is training an image super-resolution network. The image super-resolution network can be used for image super-resolution, i.e., to restore low-resolution images or videos to high-resolution images or videos.

[0097] Referring to FIG. 1b, for example, image 1 is input to the image super-resolution network, and the image super-resolution network processes image 1 to output image 2, where the resolution of image 2 is greater than that of image 1. Then, a loss can be determined based on image 1 and image 2, and then the image super-resolution network is trained based on the loss. The specific training process will be described later.

[0098] FIG. 1c is a diagram of an example application scenario. For example, the image processing network in FIG. 1c is an image restoration network, and the application scenario corresponding to FIG. 1c is to train the image restoration network. The image restoration network can be used for image restoration, i.e., to restore an image or video with blurred subregions to an image or video with clear details of the subregions.

[0099] Referring to Figure 1c, for example, image 1 (a partial region in image 1 is blurred) is input to the image restoration network, and image 2 is output after restoration performed by the image restoration network. Then, loss can be determined based on image 1 and image 2, and then the image restoration network is trained based on the loss. The specific training process will be described later.

[0100] Fig. 1d is a diagram of an example of an application scenario. The image processing network in Fig. 1d includes an encoding network and a decoding network, and the application scenario corresponding to Fig. 1d is to train the encoding network and the decoding network. An image / video can be encoded based on the encoding network, and then the encoded image / video is decoded based on the decoding network.

[0101] Referring to FIG. 1d, for example, image 1 (i.e., an original image) is input to the encoding network, which transforms image 1 and outputs feature map 1 to a quantization module. The quantization module may then quantize feature map 1 to obtain feature map 2. The quantization module may then input feature map 2 to an entropy estimation network, which performs entropy estimation and outputs entropy estimation information of feature points included in feature map 2 to an entropy encoding module and an entropy decoding module. The quantization module may input feature map 2 to an entropy encoding module, which may perform entropy encoding on the feature points included in feature map 2 based on the entropy estimation information of the feature points included in feature map 2 to obtain a bitstream, and then input the bitstream to an entropy decoding module. The entropy decoding module may then perform entropy decoding on the bitstream based on the entropy estimation information of all feature points included in feature map 2, and output feature map 2 to a decoding network. Subsequently, the decoding network may transform feature map 2 to obtain image 2 (i.e., reconstructed image). Then, Loss1 may be determined based on image 1 and image 2, and Loss2 may be determined based on the entropy estimation information determined by the entropy estimation network. The encoding network and the decoding network are trained based on Loss1 and Loss2. The specific training process will be described later.

[0102] The following uses Figure 1d as an example to explain the training process of the image processing network.

[0103] FIG. 1e is a diagram of an example of a checkerboard effect. For example, (1) in FIG. 1e is an original image, and (2) in FIG. 1e is a reconstructed image obtained by encoding and decoding the original image of (1) in FIG. 1e according to the encoding and decoding process of FIG. 1c. Referring to FIG. 1e, a checkerboard effect appears in the reconstructed image (the white box in (2) in FIG. 1e is one period of the checkerboard effect), and it can be seen that the checkerboard effect is periodic.

[0104] Based on this, in the process of training the image processing network in this application, the image before and after processing performed using the image processing network can be divided into multiple image blocks based on the period of the checkerboard effect. Then, a loss is calculated by comparing the difference between the image blocks before and after processing performed using the image processing network. The image processing network is trained based on the loss to compensate the image blocks after processing performed using the image processing network to different degrees, thereby eliminating the regularity of the checkerboard effect in the image processed using the image processing network. In this way, the visual quality of the image obtained by processing performed using the trained image processing network can be improved. The specific training process of the image processing network can be as follows: FIG. 2a is a diagram of an example of the training process.

[0105] S201: Obtain a first training image and a first predicted image, obtain a period of a checkerboard effect, and generate the first predicted image by performing image processing on the first training image based on an image processing network.

[0106] For example, a plurality of images may be acquired to be used to train the image processing network. For ease of explanation, the images used to train the image processing network may be referred to as first training images. In this application, an example in which the image processing network is trained using one first training image is used for explanation.

[0107] For example, a first training image may be input to an image processing network, which performs forward computation (i.e., image processing) to output a first predicted image.

[0108] For example, when the image processing network is an image super-resolution network, the image processing performed using the image processing network is image super-resolution, and the resolution of the first predicted image is higher than the resolution of the first training image. When the image processing network is an image restoration network, the image processing performed using the image processing network is image restoration, and the first training image is a partially blurred image. When the image processing network includes an encoding network and a decoding network, the image processing performed using the image processing network may include encoding and decoding. Specifically, the first training image may be encoded based on the encoding network, and then the encoding result of the first training image is decoded based on the decoding network. The first training image is the image to be encoded, and the first predicted image is a reconstructed image.

[0109] In a possible method, the first predicted image output by the image processing network has a checkerboard effect, and the period of the checkerboard effect can be determined by analyzing the first predicted image.

[0110] In a possible method, the image processing network includes an upsampling layer (e.g., the decoding network includes an upsampling layer). The image processing network performs an upsampling (e.g., deconvolution) operation in the image processing process, and this type of operation results in different calculation modes for adjacent pixels. Therefore, the final result is intensity / color differences between adjacent pixels, resulting in a periodic checkerboard effect. The following uses one-dimensional deconvolution as an example for explanation.

[0111] FIG. 2b is a diagram of an example of the deconvolution process.

[0112] Referring to Figure 2b, for example, assume that the inputs are a, b, c, d, and e (where the "0" in Figure 2b is not an input but is completed by a zero-padding operation in the deconvolution process), and the convolution kernel used for deconvolution is a 1*3 matrix (w1, w2, w3). Deconvolution is performed on the convolution kernel and the input, and the obtained outputs are x1, y1, x2, y2, x3, and y3. Here, x1, y1, x2, y2, x3, and y3 are calculated as follows: x1 = w2 × b, y1 = w1 × b + w3 × c, x2 = w2 × c, y2 = w1 × c + w3 × d, x3 = w2 × d, and y3 = w1 × d + w3 × e.

[0113] Referring to Figure 2b, because the calculation methods for x and y are all different in the calculation process, the modes (e.g., intensity / color) represented by x1, x2, and x3 are also different from the modes represented by y1, y2, and y3. As a result, a periodic checkerboard effect appears in the result. It should be understood that the same is true for two-dimensional deconvolution. Details will not be described here.

[0114] In addition, by examining the period of the checkerboard effect and the number of upsampling layers, it can be seen that the period of the checkerboard effect is related to the number of upsampling layers. Furthermore, in a possible manner, the period of the checkerboard effect can be determined based on the number of upsampling layers included in the image processing network.

[0115] For example, the period of the checkerboard effect is a two-dimensional period. Based on Figure 1e, it can be seen that the checkerboard effect is two-dimensional. Therefore, a two-dimensional period can indicate that the period of the checkerboard effect contains two-dimensional values, with the two dimensions corresponding to the length and width of the image.

[0116] In a possible method, the shape of the period of the checkerboard effect may be rectangular, and the size of the period of the checkerboard effect may be expressed as p*q, where p and q are positive integers, and the units of p and q are px (pixels), and p and q may be equal or unequal. This is not limited in the present application. The relationship between the number of upsampling layers included in the image processing network and the period of the checkerboard effect may be as follows: T checkboard =p*q=2 C *2 C

[0117] where T checkboard is the period of the checkerboard effect, C (C is a positive integer) is the number of upsampling layers, and p=q=2 C is.

[0118] It should be understood that p and q do not have to be equal, and this is not a limitation in this application.

[0119] It should be noted that the period of the checkerboard effect may alternatively be another shape (e.g., circular, triangular, elliptical, or irregular), and the details will not be described again here.

[0120] S202: Based on a period, divide the first training image into M first image blocks, and divide the first predicted image into M second image blocks, where both the size of the first image block and the size of the second image block are related to the period of the checkerboard effect, and M is an integer greater than 1.

[0121] For example, the number of periods of the checkerboard effect may be M.

[0122] For example, after the period of the checkerboard effect is determined, the first training image may be divided into M first image blocks based on the period of the checkerboard effect, and the first predicted image may be divided into M first image blocks based on the period of the checkerboard effect.

[0123] It should be noted that the sizes of the M second image blocks may be the same or different. This is not a limitation in the present application. The size of each second image block may be larger, equal, or smaller than the period of the checkerboard effect. For example, regardless of the relationship between the size of the second image block and the period of the checkerboard effect, it is only necessary to ensure that the second image block has only one complete or incomplete period of the checkerboard effect.

[0124] It should be noted that the sizes of the M first image blocks may be the same or different. This is not a limitation in the present application. The size of each first image block may be larger, equal, or smaller than the period of the checkerboard effect. For example, regardless of the relationship between the size of the first image block and the period of the checkerboard effect, it is only necessary to ensure that the area in the first predicted image at the same region position of the first image block has only one complete or incomplete period of the checkerboard effect.

[0125] For example, the sizes of the first image block and the second image block may be equal or unequal, which is not limited in this application.

[0126] S203: Determine a first loss based on the M first image blocks and the M second image blocks.

[0127] For example, the first loss may be determined by comparing the differences (or similarities) between M first image blocks and M second image blocks, and the first loss is used to compensate for a checkerboard effect in the image.

[0128] In one possible method, M first image blocks may be fused into one or more first feature blocks, and M second image blocks may be fused into one or more second feature blocks, where the number of first feature blocks and the number of second feature blocks are both less than M. Next, the difference (or similarity) between the first feature blocks and the second feature blocks is determined by comparing the first feature blocks and the second feature blocks. Thereafter, a first loss may be determined based on the difference (or similarity) between the first feature blocks and the second feature blocks.

[0129] S204: Train an image processing network based on the first loss.

[0130] For example, an image processing network may be trained (i.e., backpropagated) based on a first loss to adjust the network parameters of the image processing network.

[0131] Furthermore, the image processing network may be trained based on S201 to S204 using a plurality of first training images and a plurality of first predicted images until the image processing network satisfies a first preset condition. The first preset condition is a condition for stopping the training of the image processing network and may be set according to requirements. For example, a preset number of training attempts is reached or the loss becomes smaller than a preset loss. This is not a limitation of the present application.

[0132] Because the checkerboard effect is periodic, in this application, images before and after processing performed using an image processing network (the first training image is the image before processing, and the first predicted image is the image after processing) are divided into image blocks based on the period of the checkerboard effect. Then, a loss is calculated by comparing the differences between the image blocks before and after processing performed using the image processing network (i.e., by comparing M first image blocks with M second image blocks). The image processing network is trained based on the loss to effectively compensate for each image block (i.e., second image block) processed using the image processing network, thereby reducing the difference between each first image block and the corresponding second image block. Both the size of the first image block and the size of the second image block are related to the period of the checkerboard effect. As the difference between each first image block and the corresponding second image block becomes smaller, the checkerboard effect in each period is also eliminated to a certain extent. In this way, after the image processing network is trained based on the training method of the present application, the checkerboard effect of the image obtained by processing performed using the trained image processing network can be removed to a certain extent, thereby improving the visual quality of the image obtained by processing performed using the trained image processing network.

[0133] FIG. 3a is a diagram of an example end-to-end image compression framework.

[0134] 3a, for example, the end-to-end image compression framework includes an encoding network, a decoding network, a hyperprior encoding network, a hyperprior decoding network, an entropy estimation module, a quantization module (including a quantization module A1 and a quantization module A2), an entropy encoding module (including an entropy encoding module A1 and an entropy encoding module A2), and an entropy decoding module (including an entropy decoding module A1 and an entropy decoding module A2). The image processing network includes an encoding network, a decoding network, a hyperprior encoding network, and a hyperprior decoding network.

[0135] For example, the entropy estimation network of FIG. 1d may include the entropy estimation module, the hyperprior encoding network, and the hyperprior decoding network of FIG. 3a.

[0136] For example, the hyperprior encoding network and the hyperprior decoding network are configured to generate a probability distribution, and the entropy estimation module is configured to perform entropy estimation based on the probability distribution to generate the entropy estimation information.

[0137] 3b is a diagram of an example of an end-to-end image compression framework. In FIG. 3b, a discriminator network (also called a discriminator) is added based on FIG. 3a. In this case, the decoding network may be called a generator network (also called a generator), and the discriminator network and the decoding network may form a generative adversarial network (GAN network).

[0138] For example, an image processing network may first be pre-trained based on the end-to-end image compression framework shown in FIG. 3a. Specifically, an untrained encoding network, an untrained decoding network, an untrained hyperprior encoding network, and an untrained hyperprior decoding network are pre-trained together. Next, after the pre-training of the image processing network shown in FIG. 3a is completed, the image processing network is trained based on the end-to-end image compression framework of FIG. 3b based on FIG. 3a. Specifically, a pre-trained encoding network, a pre-trained decoding network, a pre-trained hyperprior encoding network, and a pre-trained hyperprior decoding network are trained together. Note that the discriminative network in FIG. 3b may be a trained discriminative network or an untrained discriminative network (in this case, the discriminative network may be trained first, and the image processing network is trained after the discriminative network training is completed). This is not a limitation of the present application.

[0139] By pre-training the image processing network, in the subsequent process of training the image processing network, the image processing network can converge faster and the coding quality of the trained image processing network will be higher.

[0140] It should be noted that the pre-training of the image processing network may be skipped, and the image processing network may be directly trained based on the end-to-end image compression framework shown in FIG. 3b. This is not a limitation in this application. This application is described using an example in which the image processing network is pre-trained and trained.

[0141] Figures 3c(1) and 3c(2) are diagrams of example network structures. In the embodiments of Figures 3c(1) and 3c(2), network structures of an encoding network, a decoding network, a hyperprior encoding network, a hyperprior decoding network, and an identification network are shown.

[0142] Referring to Figure 3c(1), for example, the encoding network may include convolutional layer A1, convolutional layer A2, convolutional layer A3, and convolutional layer A4. It should be understood that Figure 3c(1) is merely an example of the encoding network in this application. The encoding network in this application may include more or fewer convolutional layers than those in Figure 3c(1), or other network layers.

[0143] For example, the size of the convolution kernel of convolution layer A1, convolution layer A2, convolution layer A3, and convolution layer A4 may be 5*5, and the convolution step is 2. (The convolution kernel may also be called a convolution operator. The convolution operator may essentially be a weight matrix. The weight matrix is ​​usually pre-defined. Image processing is used as an example. Different weight matrices are used to extract different features in an image. For example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors in the image, and yet another weight matrix is ​​used to blur unnecessary noise in the image. The weight values ​​of these weight matrices need to be obtained by extensive training in practical applications. Each weight matrix formed using the weight values ​​obtained by training can be used to extract information from input data, allowing the network to perform correct predictions.) It should be understood that the size of the convolution kernel and the convolution step of convolution layer A1, convolution layer A2, convolution layer A3, and convolution layer A4 are not limited in this application. It should be noted that the convolution step of convolutional layer A1, convolutional layer A2, convolutional layer A3, and convolutional layer A4 is 2, which indicates that convolutional layer A1, convolutional layer A2, convolutional layer A3, and convolutional layer A4 perform downsampling operations simultaneously with the convolution operations.

[0144] Referring to Figure 3c(1), for example, the decoding network may include upsampling layer D1, upsampling layer D2, upsampling layer D3, and upsampling layer D4. It should be understood that Figure 3c(1) is only an example of the decoding network in this application. The decoding network in this application may include more or fewer upsampling layers than the upsampling layers in Figure 3c(1), or other network layers.

[0145] For example, the upsampling layer D1, the upsampling layer D2, the upsampling layer D3, and the upsampling layer D4 may each include a deconvolution unit. The size of the convolution kernel of the deconvolution units included in the upsampling layer D1, the upsampling layer D2, the upsampling layer D3, and the upsampling layer D4 may be 5*5, and the convolution step is 2. It should be understood that the size of the convolution kernel and the convolution step of the deconvolution units included in the upsampling layer D1, the upsampling layer D2, the upsampling layer D3, and the upsampling layer D4 are not limited in the present application.

[0146] Referring to Figure 3c(1), for example, a hyperprior coding network may include a convolutional layer B1, a convolutional layer B2, and a convolutional layer B3. It should be understood that Figure 3c(1) is merely an example of a hyperprior coding network in this application. The hyperprior coding network in this application may include more or fewer convolutional layers than those in Figure 3c(1), or other network layers.

[0147] For example, convolutional layer B1 and convolutional layer B2 may each include a convolutional unit and an activation unit (which may be a Leaky ReLU (Leaky Rectified Linear Unit)). The size of the convolutional kernel of the convolutional unit may be 5*5, and the convolution step may be 2. The size of the convolutional kernel of convolutional layer B3 may be 3*3, and the convolution step may be 1. It should be understood that the size of the convolutional kernel and the convolution step of the convolutional units of convolutional layer B1 and convolutional layer B2, and the size of the convolutional kernel and the convolution step of convolutional layer B3 are not limited in the present application. Note that the convolution step of convolutional layer B3 is 1, which indicates that convolutional layer B3 only performs a convolution operation and does not perform a downsampling operation.

[0148] Referring to Figure 3c(1), for example, a hyperprior decoding network may include an upsampling layer C1, an upsampling layer C2, and a convolutional layer C3. It should be understood that Figure 3c(1) is merely an example of a hyperprior decoding network in this application. The hyperprior decoding network in this application may include more or fewer upsampling layers than the upsampling layers in Figure 3c(1), or other network layers.

[0149] For example, the upsampling layer C1 and the upsampling layer C2 may each include a deconvolution unit and an activation unit (which may be a Leaky ReLU (Leaky Rectified Linear Unit)). The size of the convolution kernel of the deconvolution units included in the upsampling layer C1 and the upsampling layer C2 may be 5*5, and the convolution step is 2. The size of the convolution kernel of the convolution layer C3 may be 3*3, and the convolution step is 1. It should be understood that the size and convolution step of the convolution kernel of the deconvolution units included in the upsampling layer C1 and the upsampling layer C2, and the size and convolution step of the convolution kernel of the convolution layer C3 are not limited in the present application.

[0150] Referring to Figure 3c(2), for example, the identification network may include convolutional layer E1, convolutional layer E2, convolutional layer E3, convolutional layer E4, convolutional layer E5, convolutional layer E6, convolutional layer E7, convolutional layer E8, and convolutional layer E9. It should be understood that Figure 3c(2) is only an example of an identification network in the present application. The identification network in the present application may include more or fewer convolutional layers than those in Figure 3c(2), or other network layers.

[0151] For example, convolutional layer E1 may include a convolutional unit and an activation unit (which may be a Leaky ReLU (Leaky Regularized Linear Unit)). The size of the convolutional kernel of the convolutional unit of convolutional layer E1 may be 3*3, and the convolution step may be 1. For example, convolutional layer E2, convolutional layer E3, convolutional layer E4, convolutional layer E5, convolutional layer E6, convolutional layer E7, and convolutional layer E8 may include a convolutional unit, an activation unit (a Leaky ReLU (Leaky Regularized Linear Unit)), and a Batch Normalization (BN) unit, respectively. The size of the convolutional kernel of the convolutional units included in convolutional layer E2, convolutional layer E3, convolutional layer E4, convolutional layer E5, convolutional layer E6, convolutional layer E7, and convolutional layer E8 may be 3*3. The convolution step of the convolution units included in convolution layer E3, convolution layer E5, and convolution layer E7 may be 1. The convolution step of the convolution units included in convolution layer E2, convolution layer E4, convolution layer E6, and convolution layer E8 may be 2. The size of the convolution kernel of convolution layer E9 may be 3*3, and the convolution step may be 1. It should be understood that the size and convolution step of the convolution kernel of the deconvolution units included in convolution layer E1 to convolution layer E8, and the size and convolution step of the convolution kernel of convolution layer E9 are not limited in the present application.

[0152] Based on Figures 3a and 3b, the following describes the pre-training and training process of the image processing network.

[0153] FIG. 4a is a diagram of an example of the training process.

[0154] An image processing network can be pre-trained based on the framework of Fig. 3a, see S401-S403.

[0155] S401: Obtain a second training image and a second predicted image, where the second predicted image is obtained by encoding the second training image based on an untrained encoding network, and then decoding the encoding result of the second training image based on an untrained decoding network.

[0156] For example, a plurality of images may be acquired to be used for pre-training the image processing network. For ease of explanation, the images used for pre-training the image processing network may be referred to as second training images. In this application, an example in which the image processing network is pre-trained using one second training image is used for explanation.

[0157] Referring to FIG. 3a, for example, a second training image may be input to the encoding network, which transforms the second training image and outputs feature map 1 to the quantization module A1. The quantization module A1 may then quantize feature map 1 to obtain feature map 2. The quantization module A1 may then input feature map 2 to the hyperprior encoding network, which performs hyperprior coding on feature map 2 to output hyperprior features. The hyperprior features may then be input to the quantization module A2. The quantization module A2 quantizes the hyperprior features and inputs the quantized hyperprior features to the entropy coding module A2, which performs entropy coding on the quantized hyperprior features to obtain bitstream 2. The bitstream 2 may then be input to the entropy decoding module A2 for entropy decoding to obtain quantized hyperprior features, which are input to the hyperprior decoding network. The hyperprior decoding network may then perform hyperprior decoding on the quantized hyperprior features and output a probability distribution to the entropy estimation module. The entropy estimation module may perform entropy estimation based on the probability distribution and output entropy estimation information of the feature points included in feature map 2 to the entropy encoding module A1 and the entropy decoding module A1. The quantization module A1 may input feature map 2 to the entropy encoding module A1, which may perform entropy encoding on the feature points included in feature map 2 based on the entropy estimation information of the feature points included in feature map 2 to obtain bitstream 1, and then input bitstream 1 to the entropy decoding module A1. The entropy decoding module A1 may then perform entropy decoding on the bitstream based on the entropy estimation information of all feature points included in feature map 2 and output feature map 2 to the decoding network.Subsequently, the decoding network may transform feature map 2 to obtain a second predicted image (i.e., a reconstructed image).

[0158] Next, the image processing network may be pre-trained based on the second predicted image and the second training image, see S402 and S403.

[0159] S402: Determine a second loss based on the second predicted image and the second training image.

[0160] S403: Pre-train an untrained image processing network based on the second loss.

[0161] For example, after the second predicted image is determined, an untrained image processing network may be pre-trained based on the second predicted image and the second training image.

[0162] For example, the second loss may be determined based on the second predicted image and the second training image. The calculation formula for the second loss may be shown in Equation (1) below. L fg =L rate +α*L mse (1)

[0163] where L fg represents the second loss, and L rate represents the bit rate loss, and L mse represents the MSE (Mean Square Error) loss, and α is the mse is the weighting factor corresponding to

[0164] For example, L rate The calculation formula can be shown as the following formula (2).

number

[0165] where S i1is the entropy estimation information of the i1-th feature point in feature map 2, and H1 is the total number of feature points in feature map 2.

[0166] Furthermore, the entropy estimation information of each feature point in the feature map 2 estimated by the entropy estimation module can be obtained, and then the entropy estimation information of all feature points in the feature map 2 is calculated by subtracting the bitrate loss L rate are added according to equation (2) to obtain

[0167] For example, L mse The calculation formula can be shown as the following formula (3).

number

[0168] where H2 is the number of pixels contained in the second predicted image or the second training image, and Y 2i is the pixel at the i2-th position in the second predicted image (i.e., the pixel value of the pixel at the i2-th position in the second predicted image),

number

[0169] Furthermore, L mse can be obtained by calculation according to Equation (3), the pixels of the second training image, and the pixels of the second predicted image.

[0170] In this way, the bitrate loss L rate and the MSE loss L mse After determining the bit rate loss, we use the bit rate loss L to obtain the second loss. rate (i.e., "1") and the MSE loss L mse Based on the weighting factor (i.e., “α”) corresponding to rate and the MSE loss L mseA weighting calculation can be performed on

[0171] Next, the untrained image processing network may be pre-trained based on the second loss. In the above-described method, the untrained image processing network may be pre-trained using a plurality of second training images and a plurality of corresponding second predicted images until the image processing network satisfies a second preset condition. The second preset condition is a condition for stopping the pre-training of the image processing network, and may be set according to requirements. For example, a preset number of training times is reached or the loss becomes smaller than the preset loss. This is not a limitation of the present application.

[0172] MSE loss function L mse is used as a loss function to pre-train the image processing network, thereby improving the objective quality (e.g., PSNR) of an image obtained by image processing performed using the pre-trained image processing network. In this way, the image processing network is pre-trained based on the second loss, thereby improving the objective quality of an image obtained by image processing performed using the pre-trained image processing network. In addition, by pre-training the image processing network, the image processing network can converge more quickly and better in a subsequent training process.

[0173] Subsequently, an image processing network can be trained based on the framework of Figure 3b, see S404-S411.

[0174] S404: Obtain a first training image and a first predicted image, obtain a period of a checkerboard effect, and the first predicted image is generated by performing image processing on the first training image based on an image processing network.

[0175] For example, regarding S404, please refer to the description of S201, and the details will not be described again here.

[0176] For example, there may be multiple first training images and there may be multiple second training images.

[0177] In one possible method, there is a common set of a set containing a plurality of first training images and a set containing a plurality of second training images.

[0178] In a possible method, the set including the first plurality of training images and the set including the second plurality of training images do not have a common set, although this is not a limitation of the present application.

[0179] The following example is used for illustration, in which the decoding network in the image processing network contains C upsampling layers, and correspondingly, the period of the checkerboard effect is T checkboard =p*q=2 C *2 C is.

[0180] For example, if the decoding network in an image processing network contains two upsampling layers, the period of the checkerboard effect is 4*4.

[0181] S405: Divide the first training image into M first image blocks and divide the first predicted image into M second image blocks based on a period.

[0182] The following example is used for illustration, where the size of the first image block and the size of the second image block are both equal to the period of the checkerboard effect.

[0183] For example, the first training images are all 2 C *2 C The first predicted image may be divided into M first image blocks, all of whose sizes are 2 C *2 C The image may be divided into M second image blocks, where M is the number of image blocks.

[0184] Figure 4b is a diagram of an example of the chunking process. In Figure 4b, when the decoding network in the image processing network includes two upsampling layers, the period of the checkerboard effect is 4*4.

[0185] Referring to (1) in Figure 4b, for example, (1) in Figure 4b is a first training image, and the size of the first training image is 60*40. The first training image may be divided into 150 (i.e., M=150) first image blocks, each of which is 4*4.

[0186] Referring to (2) in Figure 4b, for example, (2) in Figure 4b is a first predicted image, and the size of the first predicted image is 60*40. The first training image can be divided into 150 (i.e., M=150) second image blocks, each of which is 4*4 in size.

[0187] S406: To obtain characteristic values ​​of corresponding positions in the first feature block, calculations are performed based on pixels of corresponding positions in the M first image blocks, and one characteristic value is correspondingly obtained for pixels of the same positions in the M first image blocks.

[0188] For example, a method for fusing M first image blocks into a first feature block may involve performing calculations based on pixels at corresponding positions in the M first image blocks to obtain feature values ​​at corresponding positions in the first feature blocks. In this way, information from the M first image blocks is aggregated into one or more first feature blocks, and information from the M second image blocks is aggregated into one or more second feature blocks. Next, a first loss is calculated by comparing the first feature block with the second feature block to more intentionally compensate for the periodic checkerboard effect and thereby achieve a better effect of eliminating the checkerboard effect.

[0189] In one possible method, all or part of the first image blocks may be fused. In this way, N first image blocks may be selected from the M first image blocks, and then feature values ​​at corresponding positions in the first feature blocks may be obtained by calculation based on pixels at corresponding positions in the N first image blocks, where N is a positive integer less than or equal to M. When the first feature blocks are obtained by calculation based on pixels at corresponding positions in some of the first image blocks and the second feature blocks are obtained by calculation based on pixels at corresponding positions in some of the second image blocks, less information is used to calculate the first loss, thereby improving the efficiency of calculating the first loss. When the first feature blocks are obtained by calculation based on pixels at corresponding positions in all of the first image blocks and the second feature blocks are obtained by calculation based on pixels at corresponding positions in all of the second image blocks, more comprehensive information is used to calculate the first loss, thereby improving the accuracy of the first loss.

[0190] In a possible method, all or some of the pixels in each of the N first image blocks may be fused. Furthermore, to obtain a characteristic value for a corresponding first position in the first feature block, calculation may be performed based on first target pixels in all of the N first image blocks at corresponding first positions. The number of first target pixels is less than or equal to the total number of pixels included in the first image block. The first positions may be set according to requirements, and the number of first target pixels may be set according to requirements. This is not limited in this application. When calculation is performed based on some pixels in the image block, less information is used to calculate the first loss, thereby improving the efficiency of calculating the first loss. When calculation is performed based on all pixels in the image block, more comprehensive information is used to calculate the first loss, thereby improving the accuracy of the first loss.

[0191] It should be noted that there may be one or more first feature blocks, which is not limited in this application. The following example is used for explanation, where N=M, the number of first feature blocks is 1, the number of first target pixels is equal to the total number of pixels contained in the first image block, and the first position is the position of all pixels in the first image block.

[0192] 4c is a diagram of an example of a process for generating a first feature block. In the embodiment of FIG. 4c, a process for fusing M first image blocks (the method for dividing the first training image into M first image blocks is shown in FIG. 4b) into one first feature block is described.

[0193] Referring to Figure 4c, for example, M first image blocks obtained by dividing the first training image can be called B1, B2, B3, ..., and BM, respectively. The 16 pixels included in B1 are e 11 , e 12 , e 13 , …, and e 116 The 16 pixels in B2 are called e 21 , e 22 , e 23 , …, and e 216 The 16 pixels in B3 are called e 31 , e 32 , e 33 , …, and e 316 The 16 pixels included in the BM are called E M1 , e M2 , e M3 , …, and e M16 It is called.

[0194] For example, a linear calculation may be performed based on pixels at corresponding positions in the M first image blocks to obtain feature values ​​at corresponding positions in the first feature block.

[0195] In a possible method, to obtain the characteristic values ​​of the corresponding positions in the first feature block, a linear calculation can be performed based on the pixels of the corresponding positions in the M first image blocks, with reference to the following equation (4):

number

[0196] where F i represents the feature value at the ith position in the first feature block, and sum is the summation function. 1i represents the pixel at the i-th position in the first image block, and e 2i represents the pixel at the ith position in the second first image block, and e 3i represents the pixel at the i-th position in the third first image block, ..., e Mi represents the pixel at the ith position in the Mth first image block.

[0197] An example of calculating the feature value F1 of the first location in the first feature block is used for illustration. For example, pixel e of B1 11 , pixel e of B2 21 , pixel e of B3 31 , …, and pixel e of BM M1 An average value of M feature values ​​for M positions in the first feature block may be calculated, and the obtained average value may be used as the feature value F1 for the first position in the first feature block. Similarly, M feature values ​​for M positions in the first feature block may be obtained.

[0198] In this way, the characteristic value of the first feature block is determined by calculating the pixel average value, which is simple to calculate and thus improves the efficiency of calculating the first loss.

[0199] It should be understood that Equation (4) is just an example of linear calculation, and in the present application, the linear calculation may alternatively be performed in another manner (e.g., linear weighting) based on pixels at corresponding positions in the M first image blocks to obtain feature values ​​at corresponding positions in the first feature blocks.

[0200] It should be noted that in the present application, the nonlinear calculation may alternatively be performed based on pixels at corresponding positions in the M first image blocks to obtain characteristic values ​​at corresponding positions in the first feature blocks, see, for example, Equation (5). F i =A1*e 1i 2 +A2*e 2i 2 +A3*e 3i 2 +…+AM*e Mi 2 (5)

[0201] A1 represents the weighting factor corresponding to the pixel at the i-th position in the first first image block (B1), A2 represents the weighting factor corresponding to the pixel at the i-th position in the second first image block (B2), A3 represents the weighting factor corresponding to the pixel at the i-th position in the third first image block (B3), ..., AM represents the weighting factor corresponding to the pixel at the i-th position in the M-th first image block (BM).

[0202] It should be noted that in the present application, calculations can be performed on pixels based on convolutional layers to determine the characteristic values ​​of feature blocks. For example, M first image blocks can be input to a convolutional layer (one or more layers) and a fully connected layer to obtain a first feature block output by the fully connected layer.

[0203] It should be understood that alternatively, a combination of different methods may be used, and this is not a limitation of this application.

[0204] S407: To obtain characteristic values ​​of corresponding positions in the second feature block, calculations are performed based on pixels of corresponding positions in the M second image blocks, and one characteristic value is correspondingly obtained for pixels of the same positions in the M second image blocks.

[0205] For example, regarding S407, please refer to the description of S408, and the details will not be described again here.

[0206] S408: Determine a first loss based on the first feature block and the second feature block.

[0207] In one possible method, the first loss may be determined based on the point-to-point loss between the first feature block and the second feature block.

[0208] For example, the point-to-point loss between a first feature block and a second feature block may be based on a pixel at a third position in the first feature block and a pixel at a third position in the second feature block.

[0209] In a possible method, the point-to-point loss may include the distance Ln. For example, the distance Ln is the distance L1, and the distance L1 may be calculated with reference to the following equation (6):

number

[0210] where L1 is the L1 distance and F i represents the feature value at the i-th position in the first feature block,

number

[0211] It should be understood that the Ln distance may further include the L2 distance and the L-Inf distance, etc. This is not limited in the present application. The calculation formulas for the L2 distance and the L-Inf distance are similar to the calculation formula for the L1 distance. The details will not be described again here.

[0212] For example, the point-to-point loss may be determined as the first loss, which may be shown in equation (7) below.

number

[0213] where L pc is the first loss.

[0214] In a possible method, the first loss may be determined based on a feature-based loss between the first feature block and the second feature block.

[0215] For example, the feature-based loss may include SSIM, MSSSIM, LPIPS, etc. (The calculation formulas of SSIM, MSSSIM, and LPIPS may refer to the description of the prior art, and will not be described in detail here.) This is not limited in the present application.

[0216] For example, the first feature block and the second feature block may be further input into a neural network (e.g., a convolutional network or a VGG (Visual Geometry Group Network) network), which outputs a first feature of the first feature block and a second feature of the second feature block. Then, the distance between the first feature and the second feature is calculated to obtain a feature-based loss.

[0217] For example, the inverse of the feature-based loss may be used as the first loss. For example, when the feature-based loss is SSIM, the relationship between the first loss and SSIM may be expressed by the following equation (8):

number

[0218] where L pc is the first loss, F is the first feature block,

number

[0219] For example, a method for training an image processing network based on the first loss may be shown in S409 and S410 below.

[0220] S409: Determine a third loss based on the classification results obtained by the classification network on the first training image and the first predicted image, where the third loss is a generative adversarial network (GAN) loss.

[0221] For example, the calculation formula for the GAN loss can be shown in the following equation (9). L GAN =ΣE(logD d )+ΣE[log(1-D(G(p)))] (9)

[0222] where E is the expected value and D d is the classification result of the classification network for the first training image, and D(G(p)) is the classification result of the classification network for the first predicted image. In the process of training the image processing network, E(logD d ) can be a constant item.

[0223] Furthermore, the third loss may be calculated according to equation (9) based on the classification result of the classification network on the first training image and the classification result of the classification network on the first predicted image.

[0224] S410: Determine a fourth loss, where the fourth loss includes at least one of the following: an L1 loss, a bitrate loss, a perceptual loss, and an edge loss.

[0225] For example, the final loss L used to train an image processing network fg can be calculated using equation (10) below: L fg =L rate +α*L1+β*(L percep +γL GAN )+δ*L edge +η*L pc (10)

[0226] where L rate represents the bit rate loss, L1 represents the L1 distance, and α is the weighting factor of L1. percep represents the perceptual loss, and L GAN represents the GAN loss, and β is the percep and L GAN is the weighting coefficient of δL edge represents the edge loss, and δ is L edge is the weighting factor of L pc represents the first loss, and η is L pc is the weighting coefficient.

[0227] The final loss L used to train the image processing network fg is L in Eq. (10). pc and L GAN and L1, L rate , L percep , and L edge It should be understood that the final loss L can be calculated based on any one or more of the following, which is not a limitation in this application. The following example is used for illustration, where the final loss L is used to train an image processing network. fg is calculated according to equation (10).

[0228] Bitrate Loss L rate Calculate the L1 distance and the first loss L pc Calculate the third loss L GAN Please refer to the above explanation for how to calculate , and the details will not be repeated here.

[0229] For example, L percep can be calculated based on the method of calculating LPIPS loss using the first training image and the first predicted image as calculation data. For details, please refer to the method of calculating LPIPS loss in the prior art, which will not be described in detail here.

[0230] For example, L edgemay be the L1 distance. The L1 distance may be calculated for the edge region, and an edge loss may be obtained. For example, an edge detector (which may also be referred to as an edge detection network) may be used to detect a first edge region of a first training image and a second edge region of a first predicted image. Then, the L1 distance between the first edge region and the second edge region may be calculated to obtain an edge loss.

[0231] S411: Train a pre-trained image processing network based on the first loss, the third loss, and the fourth loss.

[0232] For example, after the first loss, the third loss, and the fourth loss are obtained, a weighting calculation may be performed on the first loss, the third loss, and the fourth loss based on a weighting factor corresponding to the first loss (e.g., “η” in Equation (10)), a weighting factor corresponding to the third loss (e.g., “β*γ” in Equation (10)), and a weighting factor corresponding to the fourth loss (e.g., in Equation (10), the weighting factor corresponding to the bitrate loss is “1”, the weighting factor corresponding to the L1 loss is “α”, the weighting factor corresponding to the perceptual loss is “β”, and the weighting factor corresponding to the edge loss is “δ”) to obtain a final loss. Then, an image processing network is trained based on the final loss.

[0233] It should be understood that the image processing network may alternatively be trained based only on the first loss and the third loss, and compared to using only the first loss, the checkerboard effect can be better compensated for by referring to the first loss and the third loss to remove the checkerboard effect to a greater extent.

[0234] When an image processing network is trained based on the first loss, the third loss, and the fourth loss, the quality (including objective quality and subjective quality (also referred to as visual quality)) of the image processed by the trained image processing network can be improved.

[0235] For example, the upsampling layer of the decoding network causes a checkerboard effect, and the upsampling layer of the hyperprior decoding network also causes a checkerboard effect (which is weaker and has a longer period than the checkerboard effect caused by the decoding network). Furthermore, to further improve the visual quality of the reconstructed image, the period of the checkerboard effect caused by the upsampling layer of the decoding network (hereinafter referred to as the first period) and the period of the checkerboard effect caused by the upsampling layer of the hyperprior decoding network (hereinafter referred to as the second period) may be obtained. Next, based on the first period and the second period, the first training image may be divided into M first image blocks, and the first predicted image may be divided into M second image blocks. Then, the pre-trained image processing network (including the encoding network, the decoding network, the hyperprior encoding network, and the hyperprior decoding network) is trained based on the first loss determined based on the M first image blocks and the M second image blocks. The specific process may be as follows:

[0236] FIG. 5 is a diagram of an example of the training process.

[0237] S501: Obtain a second training image and a second predicted image, where the second predicted image is obtained by encoding the second training image based on an untrained encoding network, and then decoding the encoding result of the second training image based on an untrained decoding network.

[0238] S502: Determine a second loss based on the second predicted image and the second training image.

[0239] S503: Pre-train an untrained image processing network based on the second loss.

[0240] For example, for S501 to S503, please refer to the description of S401 to S403, and the details will not be described again here.

[0241] S504: Obtain a first training image and a first predicted image, and obtain a period of a checkerboard effect, wherein the first predicted image is generated by performing image processing on the first training image based on an image processing network, and the period of the checkerboard effect includes a first period and a second period.

[0242] For example, the first period is smaller than the second period.

[0243] For example, the first period may be determined based on the number of upsampling layers of the decoding network. The first period may be expressed as p1*q1, where p1 and q1 are positive integers, and the units of p1 and q1 are pixels (px), and p1 and q1 may be equal or unequal. This is not limited in the present application.

[0244] For example, if the number of upsampling layers in the decoding network is C, the first period is T1 checkboard =p1*q1=2 C *2 C is.

[0245] For example, the second period may be determined based on the first period and the number of upsampling layers of the hyperprior decoding network. The second period may be expressed as p2 * q2, where p2 and q2 are positive integers, and the units of p2 and q2 are pixels (px), and p2 and q2 may be equal or unequal. This is not limited in the present application.

[0246] In a possible way, the second period can be an integer multiple of the first period, i.e., T2 checkboard= p2 * q2 = (G * p1) * (G * q1), where G is an integer greater than 1. In a possible method, G is directly proportional to the number of upsampling layers of the hyperprior decoding network. Specifically, a larger number of upsampling layers of the hyperprior decoding network indicates a larger value of G, and a smaller number of upsampling layers of the hyperprior decoding network indicates a smaller value of G.

[0247] For example, G = 2. If the first period of the checkerboard effect is 16*16, then the second period of the checkerboard effect is 32*32.

[0248] It should be understood that the first period and the second period may also be determined based on an analysis of the first predicted image output by the image processing network, which is not a limitation of the present application.

[0249] S505: Divide the first training image into M first image blocks based on the first period and the second period, where the M first image blocks include M1 third image blocks and M2 fourth image blocks, and the size of the third image blocks is related to the first period and the size of the fourth image blocks is related to the second period.

[0250] For example, the first training image may be divided into M1 third image blocks based on the first period, and the first training image may be divided into M2 fourth image blocks based on the second period. For specific division methods, please refer to the above description of S405. Details will not be described again here. Here, M1 and M2 are positive integers, and M1+M2=M.

[0251] S506: Divide the first predicted image into M second image blocks based on the first period and the second period, where the M second image blocks include M1 fifth image blocks and M2 sixth image blocks, and the size of the fifth image blocks is related to the first period and the size of the sixth image blocks is related to the second period.

[0252] For example, the first predicted image may be divided into M1 fifth image blocks according to the first period, and the first predicted image may be divided into M2 sixth image blocks according to the second period.For specific division methods, please refer to the above description of S405.Details will not be described again here.

[0253] S507: Determine a fifth loss based on the M1 third image blocks and the M1 fifth image blocks.

[0254] For example, regarding S507, please refer to the explanation of S406 to S408, and the details will not be explained again here.

[0255] S508: Determine a sixth loss based on the M2 fourth image blocks and the M2 sixth image blocks.

[0256] For example, regarding S508, please refer to the description of S406 to S408, and the details will not be described again here.

[0257] S509: Determine a third loss based on the classification results obtained by the classification network on the first training image and the first predicted image, where the third loss is a generative adversarial network (GAN) loss.

[0258] S510: Determine a fourth loss, where the fourth loss includes at least one of the following: an L1 loss, a bitrate loss, a perceptual loss, and an edge loss.

[0259] S511: Train a pre-trained image processing network based on the fifth loss, the sixth loss, the third loss, and the fourth loss.

[0260] For example, the final loss L used to train an image processing network fg can be calculated using equation (11) below: L fg =Lrate +α*L1+β*(L percep +γL GAN )+δ*L edge +η(L1 pc +εL2 pc ) (11)

[0261] Here, L1 pc represents the fifth loss, and η represents the L1 pc is the weighting coefficient of L2 pc represents the sixth loss, and η*ε is L2 pc is the weighting coefficient.

[0262] In a possible way, η is greater than η*ε, i.e. the weighting factor corresponding to the fifth loss is greater than the weighting factor corresponding to the sixth loss.

[0263] Possibly, η is less than η*ε, i.e., the weighting factor corresponding to the fifth loss is less than the weighting factor corresponding to the sixth loss.

[0264] In a possible way, η is equal to η*ε, i.e. the weighting factor corresponding to the fifth loss is equal to the weighting factor corresponding to the sixth loss.

[0265] For details of S509 to S511, please refer to the above description of S409 to S411, and the details will not be described again here.

[0266] It should be understood that the period of the checkerboard effect may further include more periods different from the first period and the second period. For example, the period of the checkerboard effect may include k periods (the k periods may be the first period, the second period, ..., and the kth period, respectively, where k is an integer greater than 2). In this way, the first training image may be divided into M first image blocks based on the first period, the second period, ..., and the kth period. The M first image blocks may include M1 image blocks 11, M2 image blocks 12, ..., and Mk image blocks 1k. The size of image block 11 is related to the first period, the size of image block 12 is related to the second period, ..., the size of image block 1k is related to the kth period, and M1 + M2 + ... + Mk = M. The first predicted image may be divided into M second image blocks based on the first period, the second period, ..., and the kth period. The M second image blocks may include M1 image blocks 21, M2 image blocks 22, ..., and Mk image blocks 2k. The size of image block 21 is associated with the first period, the size of image block 22 is associated with the second period, ..., the size of image block 2k is associated with the kth period. Next, Loss 1 may be determined based on the M1 image blocks 11 and the M1 image blocks 21, Loss 2 may be determined based on the M2 image blocks 12 and the M2 image blocks 22, ..., and Loss sk may be determined based on the Mk image blocks 1k and the Mk image blocks 2k. Thereafter, the pre-trained encoding network and the pre-trained decoding network may be trained based on Loss 1, Loss 2, ..., and Loss sk.

[0267] The following describes the encoding and decoding process based on the image processing modules (including an encoding network, a decoding network, a hyperprior encoding network, and a hyperprior decoding network) obtained by training.

[0268] 6 is a diagram of an example of an encoding process. In the embodiment of FIG. 6, the encoding network, the hyperprior encoding network, and the hyperprior decoding network are obtained using the training method described above.

[0269] S601: An image to be encoded is obtained.

[0270] S602: An image to be encoded is input to a coding network, which processes the image to be encoded to obtain a feature map that is output by the coding network.

[0271] For example, a coding network processing an image to be coded may indicate that the coding network transforms the image to be coded.

[0272] S603: Perform entropy coding on the feature map to obtain a first bitstream.

[0273] Referring to FIG. 3a, for example, a target image to be encoded may be input to a coding network, which transforms the target image to be encoded and outputs feature map 1 to a quantization module A1. Next, the quantization module A1 may quantize feature map 1 to obtain feature map 2. Thereafter, the quantization module A1 may input feature map 2 to a hyperprior coding network, which performs hyperprior coding on feature map 2 to output hyperprior features. Next, the hyperprior features may be input to a quantization module A2. The quantization module A2 quantizes the hyperprior features and inputs the quantized hyperprior features to an entropy coding module A2, which performs entropy coding on the quantized hyperprior features to obtain a second bitstream (corresponding to bitstream 2 in FIG. 3a). The second bitstream may then be input to an entropy decoding module A2 for entropy decoding to obtain quantized hyperprior features, and the quantized hyperprior features may be input to a hyperprior decoding network. The hyperprior decoding network may then perform hyperprior decoding on the quantized hyperprior features and output a probability distribution to an entropy estimation module. The entropy estimation module may perform entropy estimation based on the probability distribution and output entropy estimation information of the feature points included in feature map 2 to an entropy coding module. The quantization module A1 may input feature map 2 to the entropy coding module A1, which may perform entropy coding on the feature points included in feature map 2 based on the entropy estimation information of the feature points included in feature map 2 to obtain a first bitstream (corresponding to bitstream 1 in FIG. 3a).

[0274] For example, after the hyperprior features output by the hyperprior coding network are obtained, entropy coding may be performed on the hyperprior features to obtain a second bitstream.

[0275] In a possible manner, the first bitstream and the second bitstream may be stored. In a possible manner, the first bitstream and the second bitstream may be transmitted to a decoder.

[0276] It should be noted that the first bitstream and the second bitstream may be packaged into one bitstream for storage / transmission, or of course, the first bitstream and the second bitstream may be stored / transmitted as two bitstreams, which is not a limitation in this application.

[0277] An embodiment of the present application further provides a bitstream distribution system, which includes: at least one storage medium configured to store at least one bitstream, where the at least one bitstream is generated based on the aforementioned encoding method; and a streaming media device configured to obtain a target bitstream from the at least one storage medium and send the target bitstream to a terminal-side device, where the streaming media device includes a content server or a content distribution server.

[0278] Figure 7 is a diagram of an example of a decoding process. The decoding process of Figure 7 corresponds to the encoding process of Figure 6. In the embodiment of Figure 7, the decoding network and the hyperprior decoding network are obtained using the training method described above.

[0279] S701: Obtain a first bitstream, where the first bitstream is a feature map bitstream.

[0280] S702: Perform entropy decoding on the first bitstream to obtain a feature map.

[0281] S703: Input the feature map into a decoding network, which processes the feature map to obtain a reconstructed image that is output by the decoding network.

[0282] Referring to FIG. 3a, for example, a decoder may receive a first bitstream and a second bitstream. The entropy decoding module A2 may first perform entropy decoding on the second bitstream to obtain quantized hyperprior features. The hyperprior features are then input to a hyperprior decoding network, which processes the quantized hyperprior features to obtain a probability distribution. The probability distribution may then be input to an entropy estimation module. The entropy estimation module determines entropy estimation information for feature points to be decoded in a feature map (i.e., feature map 2) based on the probability distribution, and inputs the entropy estimation information for feature points to be decoded in the feature map to the entropy decoding module A1. The entropy decoding module A1 may then perform entropy decoding on the first bitstream based on the entropy estimation information for feature points to be decoded in the feature map to obtain a feature map. The feature map may then be input to the decoding network. The decoding network transforms the feature maps to obtain a reconstructed image.

[0283] FIG. 8a is a diagram showing an example of the image quality comparison result.

[0284] Referring to (1) in Fig. 8a, (1) in Fig. 8a shows a reconstructed image obtained by encoding image 1 using an encoding network trained based on the prior art training method to obtain bitstream 1 (for the specific encoding process, please refer to the description of the embodiment in Fig. 6), and then decoding bitstream 1 using a decoding network trained based on the prior art training method (for the specific decoding process, please refer to the description of the embodiment in Fig. 7). The bitrate of bitstream 1 is 0.406 Bpp.

[0285] Referring to (2) in Fig. 8a, (2) in Fig. 8a shows a reconstructed image obtained by encoding image 1 using an encoding network trained based on the training method of the present application to obtain bitstream 2 (for the specific encoding process, please refer to the description of the embodiment in Fig. 6), and then decoding bitstream 2 using a decoding network trained based on the training method of the present application (for the specific decoding process, please refer to the description of the embodiment in Fig. 7). The bitrate of bitstream 2 is 0.289 Bpp.

[0286] When Fig. 8a (1) is compared with Fig. 8a (2), a checkerboard effect appears within the white circle in Fig. 8a (1), while no checkerboard effect appears within the white circle in Fig. 8a (2). It can be seen that, compared to the prior art, there are fewer bit rate points where the checkerboard effect appears in images obtained by processing performed using an image processing network trained according to the training method of the present application.

[0287] FIG. 8b is a diagram showing an example of the image quality comparison result.

[0288] Referring to (1) in Fig. 8b, (1) in Fig. 8b shows a reconstructed image obtained by encoding image 1 using an encoding network trained based on the prior art training method to obtain bitstream 1 (for the specific encoding process, please refer to the description of the embodiment in Fig. 6), and then decoding bitstream 1 using a decoding network trained based on the prior art training method (for the specific decoding process, please refer to the description of the embodiment in Fig. 7). The bitrate of bitstream 1 is 0.353 Bpp.

[0289] Referring to (2) in Figure 8b, (2) in Figure 8b shows a reconstructed image obtained by encoding image 1 using an encoding network trained based on the training method of the present application to obtain bitstream 2 (for the specific encoding process, please refer to the description of the embodiment in Figure 6), and then decoding bitstream 2 using a decoding network trained based on the training method of the present application (for the specific decoding process, please refer to the description of the embodiment in Figure 7). The bitrate of bitstream 2 is 0.351 Bpp.

[0290] The bit rates of bitstream 1 and bitstream 2 are approximately equal. However, when Fig. 8a (1) is compared with Fig. 8a (2), a checkerboard effect appears within the white circle in Fig. 8a (1), while no checkerboard effect appears within the white circle in Fig. 8a (2). It can be seen that, for relatively low bit rates, the checkerboard effect of an image obtained by processing performed using an image processing network trained according to the training method of the present application can be removed to some extent.

[0291] FIG. 8c is a diagram showing an example of the image quality comparison result.

[0292] Referring to (1) in Figure 8c, (1) in Figure 8c shows a reconstructed image obtained by encoding image 1 using an encoding network trained based on a conventional training method to obtain bitstream 1 (for the specific encoding process, please refer to the description of the embodiment in Figure 6), and then decoding bitstream 1 using a decoding network trained based on a conventional training method (for the specific decoding process, please refer to the description of the embodiment in Figure 7). The bitrate of bitstream 1 is a medium bitrate (for example, it may be 0.15 Bpp to 0.3 Bpp, and may be specifically set according to requirements).

[0293] Referring to (2) in Fig. 8c, (2) in Fig. 8c shows a reconstructed image obtained by encoding image 1 using an encoding network trained based on the training method of the present application to obtain bitstream 2 (for the specific encoding process, please refer to the description of the embodiment in Fig. 6), and then decoding bitstream 2 using a decoding network trained based on the training method of the present application (for the specific decoding process, please refer to the description of the embodiment in Fig. 7). The bitrate of bitstream 2 is a medium bitrate.

[0294] The bit rates of bitstream 1 and bitstream 2 are both medium. In (1) of FIG. 8c and (2) of FIG. 8c, the checkerboard effect is not apparent. However, when the area surrounded by the ellipse in (1) of FIG. 8c is compared with the area surrounded by the ellipse in (2) of FIG. 8c, the area surrounded by the ellipse in (2) of FIG. 8a has more details than the area surrounded by the ellipse in (1) of FIG. 8a. It can be seen that in the case of medium bit rates, images obtained by encoding and decoding performed using an image processing network trained according to the training method of the present application have higher quality.

[0295] 9 is a block diagram of an apparatus 900 according to an embodiment of the present application. The apparatus 900 may include a processor 901 and a transceiver / transceiver pins 902, and optionally further includes a memory 903.

[0296] The components of the device 900 are coupled to one another via a bus 904. In addition to a data bus, the bus 904 further includes a power bus, a control bus, and a status signal bus. However, for clarity of explanation, the various buses are referred to as the bus 904 in the figures.

[0297] Optionally, the memory 903 may be configured to store instructions in the aforementioned method embodiments. The processor 901 may be configured to execute the instructions in the memory 903 and to control the receive pin to receive a signal and the transmit pin to transmit a signal.

[0298] The apparatus 900 may be an electronic device or a chip of an electronic device in the method embodiments described above.

[0299] All relevant contents of the steps in the foregoing method embodiments can be cited in the functional descriptions of the corresponding functional modules, and the details will not be described again here.

[0300] One embodiment further provides a computer-readable storage medium that stores program instructions that, when run on an electronic device, enable the electronic device to execute the associated method steps described above to implement the method of the embodiment.

[0301] An embodiment further provides a computer program product, which, when running on a computer, enables the computer to execute the associated steps described above to implement the method of the embodiment.

[0302] In addition, an embodiment of the present application further provides an apparatus. The apparatus may specifically be a chip, a component, or a module. The apparatus may include a processor and a memory connected thereto. The memory is configured to store computer-executable instructions. When the apparatus operates, the processor may execute the computer-executable instructions stored in the memory to enable the chip to perform the method in the above-mentioned method embodiment.

[0303] The electronic device, computer-readable storage medium, computer program product, or chip provided in the embodiments is configured to execute the corresponding method provided above. Therefore, for the beneficial effects that can be achieved, please refer to the beneficial effects of the corresponding method provided above. Details will not be described here.

[0304] Based on the description of the above embodiments, those skilled in the art can understand that for the sake of convenience, the above division into functional modules is used as an example for explanation. In actual application, the above functions can be assigned to different functional modules and implemented based on requirements. In other words, the internal structure of the device is divided into different functional modules to implement all or part of the above-described functions.

[0305] In some embodiments provided in the present application, it should be understood that the disclosed devices and methods may be implemented in other ways. For example, the described device embodiments are merely examples. For example, the division into modules or units is merely a logical division of functions, and other divisions may be used in actual implementations. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not implemented. In addition, the shown or described mutual couplings or direct couplings or communication connections may be implemented using some interfaces. Indirect couplings or communication connections between devices or units may be implemented in electronic, mechanical, or other forms.

[0306] The units described as separate parts may or may not be physically separate, and the parts presented as units may be one or more physical units, located in one place, or distributed in different places. Some or all of the units may be selected according to actual requirements to achieve the objectives of the solutions of the embodiments.

[0307] In addition, the functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0308] Any content in the embodiments of the present application and any content in the same embodiment can be freely combined. Any combination of the aforementioned content is within the scope of the present application.

[0309] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, the integrated unit may be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application may essentially, or the portion contributing to the prior art, or all or part of the technical solutions may be implemented in the form of a software product. The software product is stored in a storage medium and includes several instructions for instructing a device (which may be a single-chip microcomputer or chip, etc.) or a processor to perform all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes any medium capable of storing program code, such as a USB flash drive, a removable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0310] The above describes the embodiments of the present application with reference to the accompanying drawings. However, the present application is not limited to the specific embodiments described above. The specific embodiments described above are merely examples, not limitations. Inspired by the present application, those skilled in the art may make further modifications without departing from the purpose of the present application and the scope of protection of the claims, and all modifications shall fall within the scope of protection of the present application.

[0311] The method or algorithm steps described in connection with the contents disclosed in this embodiment of the present application may be implemented by hardware or by a processor executing software instructions. The software instructions may include corresponding software modules. The software modules may be stored in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disk, removable hard disk, compact disk read-only memory (CD-ROM), or any other form of storage medium known in the art. For example, the storage medium is coupled to the processor such that the processor can read information from and write information to the storage medium. Of course, the storage medium may be a component of the processor. The processor and the storage medium may be located in an ASIC.

[0312] Those skilled in the art will recognize that the functions described in the embodiments of the present application in one or more of the foregoing examples may be implemented by hardware, software, firmware, or any combination thereof. When the functions are implemented by software, the functions may be stored on a computer-readable medium or transmitted as one or more instructions or code in a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, and communication media include any medium that allows a computer program to be transmitted from one place to another. Storage media may be any available medium accessible to a general-purpose or special-purpose computer.

[0313] The above describes the embodiments of the present application with reference to the accompanying drawings. However, the present application is not limited to the specific embodiments described above. The specific embodiments described above are merely examples, not limitations. Inspired by the present application, those skilled in the art may make further modifications without departing from the purpose of the present application and the scope of protection of the claims, and all modifications shall fall within the scope of protection of the present application. [Explanation of symbols]

[0314] 900 equipment 901 processor 902 Transceiver / Transceiver Pin 903 Memory 904 Bus

Claims

1. 1. A method for training an image processing network, the method comprising: obtaining a first training image and a first predicted image, and obtaining a period of a checkerboard effect, wherein the first predicted image is generated by performing image processing on the first training image based on the image processing network; Dividing the first training image into M first image blocks and the first predicted image into M second image blocks based on the period, where the size of the first image blocks and the size of the second image blocks are both related to the period, and M is an integer greater than 1; determining a first loss based on the M first image blocks and the M second image blocks; training the image processing network based on the first loss; A method comprising:

2. determining a first loss based on the M first image blocks and the M second image blocks; obtaining a first feature block based on the M first image blocks, wherein the characteristic value of the first feature block is obtained by calculation based on pixels at corresponding positions in the M first image blocks; obtaining a second feature block based on the M second image blocks, wherein the characteristic value of the second feature block is obtained by calculation based on pixels at corresponding positions in the M second image blocks; determining the first loss based on the first feature block and the second feature block; 2. The method of claim 1, comprising:

3. The step of obtaining a first feature block based on the M first image blocks includes: obtaining the first feature block based on N first image blocks among the M first image blocks, wherein the characteristic value of the first feature block is obtained by calculation based on pixels at corresponding positions in the N first image blocks; Including, The step of obtaining a second feature block based on the M second image blocks includes: obtaining the second feature block based on N second image blocks among the M second image blocks, wherein the characteristic value of the second feature block is obtained by calculation based on pixels at corresponding positions in the N second image blocks; 3. The method of claim 2, comprising:

4. The step of obtaining a first feature block based on the M first image blocks includes: performing calculations based on first target pixels at corresponding first positions in all of the M first image blocks to obtain characteristic values ​​at corresponding first positions in the first feature blocks, wherein the number of first target pixels is equal to or less than a total number of pixels included in the first image blocks, and one characteristic value is correspondingly obtained for first target pixels at the same first position in the M first image blocks; Including, The step of obtaining a second feature block based on the M second image blocks includes: performing calculations based on second target pixels at corresponding second positions in all of the M second image blocks to obtain characteristic values ​​at corresponding second positions in the second feature blocks, wherein the number of second target pixels is equal to or less than the total number of pixels included in the second image blocks, and one characteristic value is correspondingly obtained for the second target pixels at the same second positions in the M second image blocks; 3. The method of claim 2, comprising:

5. The step of obtaining a first feature block based on the M first image blocks includes: determining characteristic values ​​of corresponding positions in the first feature blocks based on average values ​​of pixels of corresponding positions in all of the M first image blocks, wherein one characteristic value is correspondingly obtained for the average value of pixels of the same positions in the M first image blocks; Including, The step of obtaining a second feature block based on the M second image blocks includes: determining characteristic values ​​of corresponding positions in the second feature blocks based on average values ​​of pixels at corresponding positions in all of the M second image blocks, wherein one characteristic value is correspondingly obtained for the average value of pixels at the same positions in the M second image blocks; 3. The method of claim 2, comprising:

6. determining the first loss based on the first feature block and the second feature block, determining the first loss based on a point-to-point loss between the first feature block and the second feature block; Including, The method of any one of claims 2 to 5, wherein the point-to-point loss comprises an L1 distance, an L2 distance, or an L-Inf distance.

7. determining the first loss based on the first feature block and the second feature block, determining the first loss based on a feature-based loss between the first feature block and the second feature block; Including, The method of claim 2 , wherein the feature-based loss comprises structural similarity SSIM, multi-scale structural similarity MSSSIM, or learned perceptual image patch similarity LPIPS.

8. The image processing network comprises an encoding network and a decoding network, and prior to the step of obtaining a first training image and a first predicted image, the method includes: obtaining a second training image and a second predicted image, wherein the second predicted image is obtained by encoding the second training image based on an untrained encoding network, and then decoding the encoding result of the second training image based on an untrained decoding network; determining a second loss based on the second predicted image and the second training image; pre-training the untrained encoding network and the untrained decoding network based on the second loss; 8. The method of claim 1, further comprising:

9. The first predicted image is obtained by encoding the first training image based on a pre-trained encoding network, and then decoding the encoding result of the first training image based on a pre-trained decoding network; training the image processing network based on the first loss determining a third loss based on a classification result obtained by a classification network on the first training image and the first predicted image, the third loss being a loss of a generative adversarial network (GAN), the GAN network comprising the classification network and the decoding network; training the pre-trained encoding network and the pre-trained decoding network based on the first loss and the third loss; 9. The method of claim 8, comprising:

10. training the image processing network based on the first loss determining a fourth loss, wherein the fourth loss comprises at least one of the following: an L1 loss, a bitrate loss, a perceptual loss, and an edge loss; further comprising The step of training the pre-trained encoding network and the pre-trained decoding network based on the first loss and the third loss includes: training the pre-trained encoding network and the pre-trained decoding network based on the first loss, the third loss, and the fourth loss; 10. The method of claim 9, comprising:

11. the image processing network further comprises a hyperprior encoding network and a hyperprior decoding network, the hyperprior decoding network and the decoding network each comprising an upsampling layer; the period includes a first period and a second period, the first period is determined based on the number of upsampling layers in the decoding network, the second period is determined based on the first period and the number of upsampling layers in the hyperprior decoding network, and the second period is greater than the first period; The method of claim 10.

12. The step of dividing the first training image into M first image blocks and dividing the first predicted image into M second image blocks based on the periodicity includes: Dividing the first training image into the M first image blocks based on the first period and the second period, the M first image blocks including M1 third image blocks and M2 fourth image blocks, a size of the third image blocks related to the first period, a size of the fourth image blocks related to the second period, M1 and M2 are positive integers, and M1 + M2 = M; Dividing the first predicted image into the M second image blocks based on the first period and the second period, the M second image blocks including M1 fifth image blocks and M2 sixth image blocks, the size of the fifth image blocks being related to the first period and the size of the sixth image blocks being related to the second period; Including, When the first losses include a fifth loss and a sixth loss, the step of determining the first losses based on the M first image blocks and the M second image blocks includes: determining the fifth loss based on the M1 third image blocks and the M1 fifth image blocks; determining the sixth loss based on the M2 fourth image blocks and the M2 sixth image blocks; 12. The method of claim 11, comprising:

13. training the image processing network based on the first loss performing a weighting calculation on the fifth loss and the sixth loss to obtain a seventh loss; training the pre-trained encoding network and the pre-trained decoding network based on the seventh loss; 13. The method of claim 12, comprising:

14. 1. A method of encoding, said method comprising: obtaining an image to be encoded; inputting the image to be coded into a coding network and processing the image to be coded by the coding network to obtain a feature map output by the coding network, the coding network being obtained by training using a method according to any one of claims 8 to 13; performing entropy coding on the feature map to obtain a first bitstream; 10. An encoding method comprising:

15. performing entropy coding on the feature map to obtain a first bitstream, inputting the feature map into a hyperprior coding network to obtain hyperprior features, and processing the feature map through the hyperprior coding network; inputting the hyperprior features into a hyperprior decoding network, processing the hyperprior features by the hyperprior decoding network, and then outputting a probability distribution, wherein both the hyperprior encoding network and the hyperprior decoding network are obtained by training using a method according to any one of claims 9 to 13; performing entropy coding on the feature map based on the probability distribution to obtain the first bitstream; 15. The method of claim 14, comprising:

16. The method comprises: performing entropy coding on the hyperprior features to obtain a second bitstream.

16. The method of claim 15, further comprising:

17. 1. A method of decoding, said method comprising: obtaining a first bitstream, the first bitstream being a feature map bitstream; performing entropy decoding on the first bitstream to obtain the feature map; inputting said feature map into said decoding network and processing said feature map by said decoding network to obtain a reconstructed image output by said decoding network, said decoding network being obtained by training using a method according to any one of claims 8 to 13; A decoding method comprising:

18. The method comprises: obtaining a second bitstream, the second bitstream being a bitstream of hyperprior features; further comprising performing entropy decoding on the bitstream to obtain the feature maps, performing entropy decoding on the second bitstream to obtain the hyperprior features; inputting the hyperprior features into a hyperprior decoding network and processing the hyperprior features by the hyperprior decoding network to obtain a probability distribution, the hyperprior decoding network being obtained by training using a method according to any one of claims 9 to 13; performing entropy decoding on the first bitstream based on the probability distribution to obtain the feature map; 18. The method of claim 17, comprising:

19. 1. An electronic device comprising: a memory and a processor, the memory coupled to the processor; 14. An electronic device, wherein the memory stores program instructions that, when executed by the processor, enable the electronic device to perform the training method of any one of claims 1 to 13.

20. 14. A chip comprising one or more interface circuits and one or more processors, the interface circuits configured to receive signals from a memory of an electronic device and to transmit the signals to the processor, the signals comprising computer instructions stored in the memory, and wherein, when the processor executes the computer instructions, the electronic device is enabled to perform the training method of any one of claims 1 to 13.

21. 14. A computer-readable storage medium storing a computer program, the computer program being capable of, when run on a computer or a processor, executing the training method of any one of claims 1 to 13.

22. 14. A computer program product, said computer program product comprising a software program, which when executed by a computer or processor performs the steps of the method of any one of claims 1 to 13.

23. 1. A bitstream storage device, the device comprising: a receiver; and at least one storage medium; the receiver is configured to receive a bitstream; the at least one storage medium is configured to store the bitstream; A bitstream storage device, wherein the bitstream is generated according to the encoding method of any one of claims 14 to 16.

24. A bitstream transmission device, said device comprising: a transmitter; and at least one storage medium; The at least one storage medium is configured to store a bitstream, the bitstream being generated according to the encoding method of any one of claims 14 to 16, The transmitter is configured to retrieve the bitstream from the storage medium and transmit the bitstream to a terminal-side device using a transmission medium.

25. 1. A bitstream distribution system, comprising: - at least one storage medium configured to store at least one bitstream, said at least one bitstream being generated according to the encoding method of any one of claims 14 to 16; a streaming media device configured to retrieve a target bitstream from the at least one storage medium and transmit the target bitstream to a terminal-side device, the streaming media device comprising a content server or a content distribution server; A bitstream distribution system comprising:

Images