Image processing method, device, equipment, computer storage medium and system

Abstract

The embodiments of the present application disclose an image processing method, apparatus, device, computer storage medium and system, the method comprising: obtaining a plurality of reconstructed blocks; wherein the plurality of reconstructed blocks are obtained by dividing a to-be-processed image into a plurality of image blocks through a preset encoding network model and a preset decoding network model; determining a central area of ​​at least one reconstructed block among the plurality of reconstructed blocks; performing quality enhancement on the central area of ​​the at least one reconstructed block to obtain at least one target reconstructed block; generating a reconstructed image according to the at least one target reconstructed block and the plurality of reconstructed blocks; and performing filtering on block boundaries in the reconstructed image to obtain a target image.

Description

Technical Field

[0001] The present application relates to the field of video coding and decoding technology, and in particular to an image processing method, device, equipment, computer storage medium and system. Background Art

[0002] In recent years, artificial neural networks have developed to the stage of deep learning. Deep learning is a branch of machine learning. It is an algorithm that attempts to use multiple processing layers containing complex structures or composed of multiple nonlinear transformations to perform high-level abstraction of data. Its powerful expression ability has made it widely used in computer vision and image processing, and it also has good performance in video and image processing.

[0003] At present, deep learning-based image encoding and decoding and image post-processing technologies often adopt the solution of inputting the entire image into the encoding and decoding network for processing at one time. However, as the image size increases and the encoding and decoding network deepens, this solution will lead to a significant increase in the running time and running memory requirements of the encoding and decoding. In addition, even if there are some block-based image encoding and decoding solutions, the structure of the existing solutions makes the encoding and decoding between blocks not completely independent, and it is impossible to achieve parallelization of encoding and decoding processing, and it is still impossible to reduce the running time and running memory requirements of the encoding and decoding. Summary of the invention

[0004] The present application proposes an image processing method, apparatus, device, computer storage medium and system, which can realize the parallelization of encoding and decoding processing, reduce the running time and running memory requirements of encoding and decoding, and also improve the peak signal-to-noise ratio of the reconstructed image.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides an image processing method, which is applied to an image processing device, and the method includes:

[0007] Acquire multiple reconstructed blocks; wherein the multiple reconstructed blocks are obtained by dividing the image to be processed into multiple image blocks through a preset encoding network model and a preset decoding network model;

[0008] determining a central area of ​​at least one reconstructed block among the plurality of reconstructed blocks;

[0009] Performing quality enhancement on a central area of ​​the at least one reconstructed block to obtain at least one target reconstructed block;

[0010] generating a reconstructed image according to the at least one target reconstructed block and the plurality of reconstructed blocks;

[0011] A filtering process is performed on the block boundaries in the reconstructed image to obtain a target image.

[0012] In a second aspect, an embodiment of the present application provides an image processing method, which is applied to a decoding device, and the method includes:

[0013] Receiving a code stream transmitted by an encoding device; wherein the code stream is obtained by dividing a plurality of image blocks into which an image to be processed is divided through a preset encoding network model;

[0014] Parsing the bitstream using a preset decoding network model to obtain a plurality of reconstructed blocks;

[0015] Using a preset post-processing network model, quality enhancement is performed on a central area of ​​at least one of the multiple reconstructed blocks to obtain at least one target reconstructed block;

[0016] generating a reconstructed image according to the at least one target reconstructed block and the plurality of reconstructed blocks;

[0017] The preset post-processing network model is used to filter the block boundaries in the reconstructed image to obtain a target image.

[0018] In a third aspect, an embodiment of the present application provides an image processing method, which is applied to an encoding device, and the method includes:

[0019] Get the image to be processed;

[0020] Dividing the image to be processed into blocks to obtain a plurality of image blocks; wherein the plurality of image blocks are equal in size and have no overlap;

[0021] Encoding the multiple image blocks using a preset encoding network model to generate a bitstream;

[0022] The code stream is transmitted to a decoding device.

[0023] In a fourth aspect, an embodiment of the present application provides an image processing device, the image processing device comprising: an acquisition unit, a determination unit, an enhancement unit and a processing unit; wherein:

[0024] The acquisition unit is configured to acquire a plurality of reconstructed blocks; wherein the plurality of reconstructed blocks are obtained by dividing a plurality of image blocks into which the image to be processed is divided through a preset encoding network model and a preset decoding network model;

[0025] The determining unit is configured to determine a central area of ​​at least one reconstructed block among the multiple reconstructed blocks;

[0026] The enhancement unit is configured to perform quality enhancement on a central area of ​​the at least one reconstructed block to obtain at least one target reconstructed block;

[0027] The processing unit is configured to generate a reconstructed image according to the at least one target reconstructed block and the multiple reconstructed blocks; and perform filtering processing on the block boundaries in the reconstructed image to obtain a target image.

[0028] In a fifth aspect, an embodiment of the present application provides an image processing device, the image processing device comprising: a first memory and a first processor; wherein:

[0029] The first memory is used to store executable instructions that can be run on the first processor;

[0030] The first processor is used to execute the method as described in the first aspect when running the executable instructions.

[0031] In a sixth aspect, an embodiment of the present application provides a decoding device, the decoding device comprising: a receiving unit, a decoding unit and a post-processing unit; wherein,

[0032] The receiving unit is configured to receive a code stream transmitted by an encoding device; wherein the code stream is obtained by dividing a plurality of image blocks into which an image to be processed is divided through a preset encoding network model;

[0033] The decoding unit is configured to parse the bitstream using a preset decoding network model to obtain a plurality of reconstructed blocks;

[0034] The post-processing unit is configured to use a preset post-processing network model to perform quality enhancement on a central area of ​​at least one of the multiple reconstructed blocks to obtain at least one target reconstructed block; and generate a reconstructed image based on the at least one target reconstructed block and the multiple reconstructed blocks, and use the preset post-processing network model to filter the block boundaries in the reconstructed image to obtain a target image.

[0035] In a seventh aspect, an embodiment of the present application provides a decoding device, the decoding device comprising: a second memory and a second processor; wherein:

[0036] The second memory is used to store executable instructions that can be run on the second processor;

[0037] The second processor is used to execute the method as described in the second aspect when running the executable instructions.

[0038] In an eighth aspect, an embodiment of the present application provides a coding device, the coding device comprising: an acquisition unit, a block unit, a coding unit and a sending unit; wherein,

[0039] The acquisition unit is configured to acquire the image to be processed;

[0040] The blocking unit is configured to block the image to be processed to obtain a plurality of image blocks; wherein the plurality of image blocks are equal in size and have no overlap;

[0041] The encoding unit is configured to encode the multiple image blocks using a preset encoding network model to generate a bit stream;

[0042] The sending unit is configured to transmit the code stream to a decoding device.

[0043] In a ninth aspect, an embodiment of the present application provides a coding device, the coding device comprising: a third memory and a third processor; wherein:

[0044] The third memory is used to store executable instructions that can be run on the third processor;

[0045] The third processor is used to execute the method as described in the third aspect when running the executable instructions.

[0046] In the tenth aspect, an embodiment of the present application provides a computer storage medium, which stores an image processing program. When the image processing program is executed by a first processor, it implements the method as described in the first aspect; when it is executed by a second processor, it implements the method as described in the second aspect; or when it is executed by a third processor, it implements the method as described in the third aspect.

[0047] In an eleventh aspect, an embodiment of the present application provides a video system, the video system comprising: an encoding device and a decoding device; wherein:

[0048] The encoding device is configured to obtain an image to be processed; and divide the image to be processed into blocks to obtain a plurality of image blocks; wherein the plurality of image blocks are of equal size and have no overlap; and encode the plurality of image blocks using a preset encoding network model to generate a code stream; and transmit the code stream to a decoding device;

[0049] The decoding device is configured to receive a code stream transmitted by the encoding device; and parse the code stream using a preset decoding network model to obtain multiple reconstructed blocks; and use a preset post-processing network model to enhance the quality of a central area of ​​at least one of the multiple reconstructed blocks to obtain at least one target reconstructed block; and generate a reconstructed image based on the at least one target reconstructed block and the multiple reconstructed blocks, and use the preset post-processing network model to filter the block boundaries in the reconstructed image to obtain a target image.

[0050] An image processing method, apparatus, device, computer storage medium and system provided by the embodiments of the present application obtain a plurality of reconstructed blocks; wherein the plurality of reconstructed blocks are obtained by dividing a to-be-processed image into a plurality of image blocks through a preset encoding network model and a preset decoding network model; determine the central area of ​​at least one reconstructed block among the plurality of reconstructed blocks; perform quality enhancement on the central area of ​​the at least one reconstructed block to obtain at least one target reconstructed block; generate a reconstructed image according to the at least one target reconstructed block and the plurality of reconstructed blocks; and perform filtering on the block boundaries in the reconstructed image to obtain a target image. In this way, for the multiple image blocks divided by the image to be processed, the blocks are completely independent of each other, and the preset encoding network model and the preset decoding network model can be used to implement multi-core parallel encoding and decoding processing; and since the image size input into the preset encoding network model and the preset decoding network model is reduced after block division, the running time and running memory requirements of the encoding and decoding can also be reduced; in addition, by enhancing the quality of the central area of ​​the reconstructed block, the peak signal-to-noise ratio of the reconstructed image can also be improved, and at the same time, the computational redundancy caused by the overlap with the area to be filtered is reduced; by filtering the block boundaries in the reconstructed image, the block effect at the block boundaries can be eliminated, and the peak signal-to-noise ratio of the reconstructed image is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of a block diagram of image processing provided for a related technical solution;

[0052] Figure 2 A flowchart of an image processing method provided in an embodiment of the present application;

[0053] Figure 3 A schematic diagram of the structure of a post-processing network model provided in an embodiment of the present application;

[0054] Figure 4 A schematic diagram of the structure of a residual block provided in an embodiment of the present application;

[0055] Figure 5 A structural schematic diagram of quality enhancement using a preset post-processing network model provided in an embodiment of the present application;

[0056] Figure 6 A schematic diagram of a structure for eliminating block effects by using a preset post-processing network model provided in an embodiment of the present application;

[0057] Figure 7 A flowchart of another image processing method provided in an embodiment of the present application;

[0058] Figure 8A schematic diagram of an end-to-end structure including a preset encoding network model and a preset decoding network model provided in an embodiment of the present application;

[0059] Fig. 9 A flowchart of another image processing method provided in an embodiment of the present application;

[0060] Fig.10 A detailed flowchart of an image processing method provided in an embodiment of the present application;

[0061] Fig.11A A schematic diagram of a reconstructed image with a block effect provided in an embodiment of the present application;

[0062] Fig. 11B A schematic diagram of a reconstructed image without a blocking effect provided in an embodiment of the present application;

[0063] Fig.12 A schematic diagram of comparing image rate-distortion curves before and after a preset post-processing network model provided in an embodiment of the present application;

[0064] Fig.13 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0065] Fig.14 A schematic diagram of the hardware structure of an image processing device provided in an embodiment of the present application;

[0066] Fig.15 A schematic diagram of the structure of a decoding device provided in an embodiment of the present application;

[0067] Fig.16 A schematic diagram of the hardware structure of a decoding device provided in an embodiment of the present application;

[0068] Fig.17 A schematic diagram of the structure of a coding device provided in an embodiment of the present application;

[0069] Fig.18 A schematic diagram of the hardware structure of an encoding device provided in an embodiment of the present application;

[0070] Fig.19 A schematic diagram of the composition structure of a video system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. It is understood that the specific embodiments described herein are only used to explain the related applications, rather than to limit the applications. It should also be noted that, for the convenience of description, only the parts related to the related applications are shown in the accompanying drawings.

[0072] Artificial Neural Networks (ANNs) can also be referred to as neural networks or connection models. It is an algorithmic mathematical model that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. This network relies on the complexity of the system and adjusts the interconnected relationships between a large number of internal nodes to achieve the purpose of processing information.

[0073] In recent years, artificial neural networks have developed to the stage of deep learning. Deep learning is a branch of machine learning. It is an algorithm that attempts to use multiple processing layers with complex structures or multiple nonlinear transformations to perform high-level abstraction of data. Its powerful expressive ability enables it to achieve the best results in various machine learning tasks, and its performance in video and image processing also exceeds current related technologies.

[0074] It should be understood that the autoencoder is an important part of deep learning. The neural network can continuously improve its accuracy by end-to-end training with a large number of data sets, and the autoencoder makes the input and output closer and closer by designing the encoding and decoding processes. This is an unsupervised learning process. Here, due to the excellent performance of deep learning in video and image processing, the basic idea of ​​Autoencoder in deep learning is combined for video and image compression, and a new encoding and decoding method is provided by deep learning. This is a good start for the future development of the video compression field, and is also beneficial to the future video compression direction. The new scheme based on neural network has better performance and prospects than the traditional scheme in the whole system.

[0075] It should also be noted that current image compression schemes generally cause image information loss, image quality degradation, and artifacts. In this case, post-processing is required after image compression, and post-processing refers to methods for designing quality enhancement and artifact elimination for compressed images to improve the visual effect of the image. Similarly, post-processing is also widely used in video compression. In this way, since deep learning has been widely used in computer vision and image processing in recent years, some current research works can also use deep learning for image or video compression post-processing, and have achieved certain results.

[0076] See also Figure 1 , which shows a block diagram of an image processing provided by a related technical solution. Figure 1As shown, it mainly includes: an encoder 11, a channel transmission module 12, a decoder 13 and a post-processing module 14. Among them, the decoder 13 and the post-processing module 14 can be integrated in the same device.

[0077] After the input image is obtained, the input image is encoded by the encoder 11 and a code stream is generated; the code stream is transmitted to the decoder 13 through the channel transmission module 12, and the decoder 13 parses the code stream to obtain a reconstructed image. At this time, the reconstructed image may have problems such as image information loss, image quality degradation, and block effects, so the reconstructed image needs to be processed by the post-processing module 14 to obtain the final target image.

[0078] However, in the current related technical solutions, the entire image is often input into the encoder and decoder at one time for processing. However, as the image size increases and the encoding network used by the encoder and the decoding network used by the decoder become deeper, this solution will lead to a significant increase in the running time of encoding and decoding and the running memory requirements, which is not conducive to practical applications. In addition, even if there are some block-based image encoding and decoding solutions, the existing solutions use a long short-term memory artificial neural network structure, which makes the encoding and decoding between blocks not completely independent, and it is impossible to achieve parallelization of encoding and decoding processing, and it is still impossible to reduce the running time and running memory requirements of encoding and decoding.

[0079] Based on this, the embodiment of the present application provides an image processing method, the basic idea of ​​which is: obtain an image to be processed, perform block processing on the image to be processed to obtain corresponding multiple image blocks, and then use a preset coding network model to encode the multiple image blocks to generate a code stream, and transmit the code stream to a decoding device. After receiving the code stream, the decoding device parses the code stream using a preset network model to obtain multiple reconstructed blocks, and then determines the central area of ​​at least one reconstructed block among the multiple reconstructed blocks; performs quality enhancement on the central area of ​​at least one reconstructed block to obtain at least one target reconstructed block; generates a reconstructed image based on at least one target reconstructed block and multiple reconstructed blocks; and performs filtering on the block boundaries in the reconstructed image to obtain a target image.

[0080] In this way, for the multiple image blocks divided by the image to be processed, the blocks are completely independent from each other, and the preset encoding network model and the preset decoding network model can be used to realize multi-core parallel encoding and decoding processing; and because the image size input to the preset encoding network model and the preset decoding network model is reduced after the block division, the running time and running memory requirements of the encoding and decoding can also be reduced. In addition, by enhancing the quality of the central area of ​​the reconstructed block, the peak signal-to-noise ratio of the reconstructed image can be improved, and at the same time, the computational redundancy caused by the overlap with the area to be filtered is reduced; by filtering the block boundary in the reconstructed image, the block effect at the block boundary can be eliminated, and the peak signal-to-noise ratio of the reconstructed image is further improved.

[0081] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0082] In one embodiment of the present application, see Figure 2 , which shows a schematic flow chart of an image processing method provided by an embodiment of the present application. Figure 2 As shown, the method may include:

[0083] S101: Acquire multiple reconstructed blocks; wherein the multiple reconstructed blocks are obtained by dividing a plurality of image blocks of an image to be processed through a preset encoding network model and a preset decoding network model.

[0084] It should be noted that the method is applied to an image processing device, or a device integrated with an image processing device, such as a decoding device. In addition, a specific embodiment of the image processing device is Figure 1 The post-processing module 14 is shown.

[0085] It should also be noted that the codec network includes a coding network model and a decoding network model, and the preset coding network model and the preset decoding network model are obtained by model training based on a neural network structure. Among them, the preset coding network model is used to encode multiple image blocks divided by the image to be processed to generate a bitstream, and the preset decoding network model is used to parse the bitstream to obtain multiple reconstructed blocks.

[0086] In addition, for the image blocks divided by the image to be processed, one image block can be a tile or block, and multiple image blocks are of equal size and have no overlap. Typically, the size of each image block can be 64*64, 128*128, 192*192, 256*256 or 512*512, etc., depending on the image size and the needs or requirements of the codec for the image block size. In an embodiment of the present application, the size of the image block is typically set to 128*128, but the embodiment of the present application does not specifically limit this.

[0087] In this way, for the multiple image blocks divided by the image to be processed, the blocks are completely independent of each other, and a preset encoding network model and a preset decoding network model can be used to perform parallel encoding and decoding processing, so that multiple reconstructed blocks can be obtained in parallel.

[0088] S102: Determine a central area of ​​at least one reconstructed block among the multiple reconstructed blocks.

[0089] S103: Perform quality enhancement on a central area of ​​the at least one reconstructed block to obtain at least one target reconstructed block.

[0090] It should be noted that after obtaining multiple reconstructed blocks, the central area of ​​at least one of the multiple reconstructed blocks can be determined. Here, if the size of each reconstructed block is 128*128, then the size of the central area of ​​the reconstructed block can be 112*112, 110*110, 100*100, etc. In the embodiment of the present application, the central area of ​​the reconstructed block with a size of 112*112 can usually be extracted as the central area of ​​each reconstructed block.

[0091] In this way, after obtaining the central area of ​​at least one reconstructed block, the central area of ​​the at least one reconstructed block may be quality enhanced to obtain at least one target reconstructed block. Specifically, in some embodiments, the quality enhancement of the central area of ​​the at least one reconstructed block to obtain at least one target reconstructed block may include:

[0092] Performing quality enhancement on a central region of the at least one reconstructed block to obtain at least one enhanced region;

[0093] The central area of ​​the at least one reconstructed block is replaced according to the at least one enhanced area to obtain the at least one target reconstructed block.

[0094] That is, by performing quality enhancement on the central area of ​​at least one reconstructed block, at least one enhanced area can be obtained; then the central area of ​​the at least one reconstructed block is replaced with the at least one enhanced area, thereby obtaining at least one target reconstructed block.

[0095] It should also be noted that, in some embodiments, the method may further include:

[0096] Build a preset post-processing network model.

[0097] Accordingly, for S103, the step of performing quality enhancement on the central area of ​​the at least one reconstructed block to obtain at least one target reconstructed block may include:

[0098] The preset post-processing network model is used to perform quality enhancement on a central area of ​​the at least one reconstructed block to obtain at least one target reconstructed block.

[0099] Here, the preset post-processing network model is also obtained by model training based on the neural network structure. In some embodiments, the construction of the preset post-processing network model may include:

[0100] Obtain multiple reconstruction training blocks;

[0101] A post-processing network model is constructed, and the post-processing network model is trained with the multiple reconstructed training blocks and the at least one training image to obtain the preset post-processing network model.

[0102] It should be noted that the multiple reconstructed training blocks can be obtained by dividing the multiple training blocks of at least one training image in the training set through the preset encoding network model and the preset decoding network model. Specifically, before model training, it is necessary to first obtain a training set, which includes at least one training image. Here, the selection of the training set has a great influence on the training of the entire neural network. In the embodiment of the present application, a neural network image codec (Neutral Image Codec, NIC) data set can be selected. The NIC data set is a development data set of the Institute of Electrical and Electronics Engineers (Institute of Electrical and Electronics Engineers, IEEE) standard test model NIC for image compression based on deep learning. In this data set, a training set may be included, and a verification set and a test set may also be included; wherein the image size in the training set may be 256*256, and the image size in the verification set and the test set may also correspond to 256*256; of course, in the present embodiment, the image size in the training set, the verification set, and the test set may also be set to other sizes, for example, 512*512 or other sizes, which may be determined according to the image size corresponding to the video to be encoded and decoded, and is not limited in any way in the present embodiment of the application.

[0103] It should also be noted that the post-processing network model adopts a neural network structure, which may be composed of a convolutional layer, an activation function, and a plurality of cascaded residual blocks for improving model performance. Figure 3 An example of the above post-processing network structure is given. Here, the activation function can be a linear rectifier function (Rectified Linear Unit, ReLU), also known as a rectified linear unit, which is a commonly used activation function in artificial neural networks, usually referring to a nonlinear function represented by a ramp function and its variants.

[0104] For example, taking the elimination of the block effect at the boundary of the reconstructed block as an example, Figure 3As shown, for the reconstructed block boundary pixels with obvious block effects, they can pass through the first convolution layer, multiple residual blocks (for example, 9 cascaded residual blocks), the second convolution layer and the third convolution layer in sequence, and then use the adder to superimpose the output of the third convolution layer and the input of the first convolution layer, so as to output the reconstructed block boundary pixels without obvious block effects. Among them, the first convolution layer and the second convolution layer include activation functions, and the third convolution layer does not include activation functions. In an optional embodiment, the first convolution layer and the second convolution layer are represented as k3n128+ReLU, indicating that the convolution kernel size of the first convolution layer and the second convolution layer is 3*3, the output feature number is 128, the step size is 1, and an activation function is included; and the third convolution layer is represented as k3n3, indicating that the convolution kernel size of the third convolution layer is 3*3, the output feature number is 3, the step size is 1, but the activation function is not included.

[0105] For each residual block, its network structure is as follows Figure 4 As shown. Figure 4 In Figure 3 The feature map output after the first convolution layer is used as input, and then passes through the fourth convolution layer and the fifth convolution layer in sequence, and then the output of the fifth convolution layer and the input of the fourth convolution layer are superimposed by the adder to obtain the output feature map. Among them, the fourth convolution layer includes an activation function, and the fifth convolution layer does not include an activation function. In an optional embodiment, the fourth convolution layer is represented as k3n128+ReLU, indicating that the convolution kernel size of the first convolution layer and the second convolution layer is 3*3, the output feature number is 128, the step size is 1, and an activation function is included; and the fifth convolution layer is represented as k3n128, indicating that the convolution kernel size of the fifth convolution layer is 3*3, the output feature number is 128, the step size is 1, but the activation function is not included.

[0106] In this way, after the post-processing network model is constructed, the post-processing network model can be trained using the training set and the preset algorithm. Specifically, in some embodiments, the post-processing network model is trained using the multiple reconstructed training blocks and the at least one training image to obtain the preset post-processing network model, which may include:

[0107] Based on the multiple reconstructed training blocks and the at least one training image, performing model training on the post-processing network model using a preset algorithm;

[0108] When the loss value corresponding to the cost function of the model training converges to a preset threshold, the post-processing network model obtained after training is determined as the preset post-processing network model.

[0109] It should be noted that in the training set, the training images can be of sizes such as 256*256 and 512*512. Here, a training image of size 256*256 is taken as an example. After it is divided into equal and non-overlapping 128*128 training blocks (it can also be training blocks of other sizes) and input into the preset encoding network model and the preset decoding network model, the obtained reconstructed blocks are reassembled into a 256*256 (the size of the original image) reconstructed training image with block effects. At this time, multiple reconstructed blocks can be directly used as the input of the post-processing network model, or the reconstructed training image with block effects can be used as the training input image of the post-processing network model; and the training images in the training set are used as the training target images of the post-processing network model, and then the cost function of the model training can be constructed according to the mean square error of the training input image and the training target image. Here, the cost function can be a rate-distortion cost function, and the distortion is the mean square error of the training input image and the training target image.

[0110] It should also be noted that the preset algorithm can be an adaptive moment estimation (Adaptive moment estimation, Adam) gradient optimization algorithm. The Adam gradient optimization algorithm is an extension of the stochastic gradient descent method, which is widely used in deep learning applications in computer vision and natural language processing. It works well and can quickly achieve good results. In this way, the post-processing network is trained using the Adam gradient optimization algorithm, and the network parameters of the preset encoding network model and the preset decoding network model are kept fixed during the training process, and only the post-processing network model is iteratively updated. After the loss value corresponding to its cost function reaches convergence and converges to a preset threshold, the post-processing network model obtained by training at this time is the preset post-processing network model. Here, the preset threshold is specifically set according to the actual situation, and the embodiments of the present application do not impose any restrictions.

[0111] Specifically, see Figure 5 , which shows a schematic diagram of a structure for quality enhancement using a preset post-processing network model provided by an embodiment of the present application. Figure 5 As shown, for the multiple reconstructed blocks obtained, the central areas of the multiple reconstructed blocks are first extracted; then the extracted central areas are input into a preset post-processing network model, and multiple enhanced areas are correspondingly output after passing through the preset post-processing network model; the central areas of the multiple reconstructed blocks are correspondingly replaced by the multiple enhanced areas, so that multiple target reconstructed blocks, i.e., reconstructed blocks with enhanced quality, can be obtained.

[0112] S104: Generate a reconstructed image according to the at least one target reconstructed block and the multiple reconstructed blocks.

[0113] It should be noted that after obtaining at least one target reconstruction block, a reconstructed image may be generated by stitching. Specifically, in some embodiments, generating a reconstructed image according to the at least one target reconstruction block and the plurality of reconstruction blocks may include:

[0114] At least one reconstructed block among the multiple reconstructed blocks is replaced with the at least one target reconstructed block, and the multiple reconstructed blocks obtained after the replacement are spliced ​​to generate the reconstructed image.

[0115] That is to say, if the quality of each of the multiple reconstructed blocks is enhanced, multiple target reconstructed blocks can be obtained; then the multiple target reconstructed blocks are used to replace all the original multiple reconstructed blocks, and the multiple new reconstructed blocks obtained after the replacement are spliced ​​to obtain a reconstructed image. Alternatively, if the quality of only some of the multiple reconstructed blocks is enhanced, partial target reconstructed blocks are obtained; then the partial target reconstructed blocks are used to replace the original partial reconstructed blocks in the multiple reconstructed blocks, and the multiple new reconstructed blocks obtained after the replacement are spliced ​​to obtain a reconstructed image. At this time, the reconstructed image obtained has obvious block effects.

[0116] S105: Perform filtering processing on the block boundaries in the reconstructed image to obtain a target image.

[0117] It should be noted that, since the reconstructed image has obvious block effects, the filtering process here specifically refers to eliminating the block effects at the block boundaries in the reconstructed image.

[0118] In one embodiment, the filtering process performed on the reconstructed image to eliminate the blocking effect at the block boundary may adopt any filtering algorithm for removing the blocking effect, which is not limited in this embodiment.

[0119] In some embodiments, filtering the reconstructed image to eliminate the block effect at the block boundary may be performed in other ways. Specifically, for S105, filtering the block boundary in the reconstructed image to obtain the target image may include:

[0120] The preset post-processing network model is used to filter the block boundaries in the reconstructed image to obtain a target image.

[0121] It should be noted that the preset post-processing network model here and the preset post-processing network model used for the aforementioned quality enhancement belong to the same post-processing network model, which can reduce the storage space required for storing network parameters in the image processing device or decoding device.

[0122] Specifically, in some embodiments, filtering the block boundaries in the reconstructed image based on the preset post-processing network model to obtain the target image may include:

[0123] determining at least one rectangular region in the reconstructed image that includes the block boundary;

[0124] Inputting the at least one rectangular area into the preset post-processing network model to obtain at least one processed rectangular area;

[0125] The target image is obtained by replacing a corresponding local area including the block boundary in the reconstructed image with the at least one processed rectangular area.

[0126] It should be noted that first, it is necessary to extract the rectangular area around the block boundary in the reconstructed image. For the horizontal boundary, the specific range of the rectangular area is, horizontal direction: the preset pixel on the left side of the block boundary to the preset pixel on the right side of the block boundary (wherein the preset pixel can be 16 pixels); vertical direction: the upper edge to the lower edge of the reconstructed block. For the vertical boundary, the specific range of the rectangular area is, vertical direction: the preset pixel on the upper side of the block boundary to the preset pixel on the lower side of the block boundary (wherein the preset pixel can be 16 pixels); horizontal direction: the left edge to the right edge of the reconstructed block. Here, the preset pixels can be 16 pixels, or 10 pixels, 8 pixels, etc. In addition, the different preset pixels result in different specific ranges of the extracted rectangular area. Generally, the preset pixels can be 16 pixels, but the embodiments of the present application do not impose any limitation.

[0127] For example, for the horizontal boundary, the specific range of the rectangular area is, horizontal direction: 16 pixels on the left side of the block boundary to 16 pixels on the right side of the block boundary; vertical direction: the upper edge to the lower edge of the reconstruction block; for the vertical boundary, the specific range of the rectangular area is, vertical direction: 16 pixels on the upper side of the block boundary to 16 pixels on the lower side of the block boundary; horizontal direction: the left edge to the right edge of the reconstruction block. If the reconstructed image is spliced ​​with 128*128 reconstruction blocks, the size of the rectangular area is 32*128; if the reconstructed image is spliced ​​with 192*192 reconstruction blocks, the size of the rectangular area is 32*192. Here, the unit of 32 is pixel, the unit of 128 is pixel, and the unit of 192 is also pixel; that is, the size of the reconstructed image or the reconstructed block is expressed in pixels.

[0128] In this way, after at least one rectangular region including a block boundary in the reconstructed image is determined by extraction, the at least one rectangular region can be input into a preset post-processing network model to obtain at least one processed rectangular region; then the at least one processed rectangular region is used to replace the corresponding local region including the block boundary in the reconstructed image to obtain a target image. At this time, the target image obtained can reduce the block effect.

[0129] In order to further eliminate the block effect, the boundary image distortion caused by the convolution operation of the preset post-processing network model on the boundary zero padding can also be eliminated. At this time, it is necessary to further crop the at least one processed rectangular area. In some embodiments, after obtaining the at least one processed rectangular area, the method may also include:

[0130] Clipping the at least one processed rectangular area to obtain at least one target rectangular area;

[0131] The target image is obtained by replacing a corresponding local area including the block boundary in the reconstructed image with the at least one target rectangular area.

[0132] It should be noted that, taking a 32*128 rectangular area as an example, for at least one processed rectangular area output by the preset post-processing network model, the edge pixel areas with a width of 8 pixels and a height of 128 pixels on the left and right sides can be discarded, and only a rectangular area with a center size of 16*128 can be retained; or, the edge pixel areas with a width of 4 pixels and a height of 128 pixels on the left and right sides can be discarded, and only a rectangular area with a center size of 24*128 can be retained; the rectangular area retained here is the at least one target rectangular area obtained by cropping; finally, the at least one target rectangular area is used to replace the corresponding local area including the block boundary in the reconstructed image, so as to obtain a target image without obvious block effect.

[0133] Specifically, see Figure 6 , which shows a schematic diagram of a structure for eliminating block effects using a preset post-processing network model provided by an embodiment of the present application. Figure 6 As shown, for a reconstructed image with obvious block effects, at least one rectangular area including a block boundary is first extracted. Figure 6 The grid-filled rectangular area shown represents the rectangular area of ​​the longitudinal boundary; at least one extracted rectangular area is input into a preset post-processing network model, and at least one processed rectangular area is output; then, by cropping, the edge pixel areas on the left and right sides of the rectangular area are discarded to obtain at least one target rectangular area; finally, by replacement, that is, using the at least one target rectangular area to replace the corresponding local area including the block boundary in the reconstructed image, a reconstructed image without obvious block effect can be obtained.

[0134] This embodiment provides an image processing method, by obtaining multiple reconstructed blocks; wherein the multiple reconstructed blocks are obtained by dividing the image to be processed into multiple image blocks through a preset coding network model and a preset decoding network model; determining the central area of ​​at least one reconstructed block among the multiple reconstructed blocks; performing quality enhancement on the central area of ​​the at least one reconstructed block to obtain at least one target reconstructed block; generating a reconstructed image according to the at least one target reconstructed block and the multiple reconstructed blocks; performing filtering on the block boundaries in the reconstructed image to obtain a target image. In this way, for the multiple image blocks divided into the image to be processed, the blocks are completely independent from each other, and the preset coding network model and the preset decoding network model can be used to implement multi-core parallel coding and decoding processing; and because the image size input to the preset coding network model and the preset decoding network model after block division is reduced, the running time and running memory requirements of the coding and decoding can also be reduced; in addition, by performing quality enhancement on the central area of ​​the reconstructed block, the peak signal-to-noise ratio of the reconstructed image can be improved, and the calculation redundancy caused by the overlap with the area to be filtered can be reduced; by performing filtering on the block boundaries in the reconstructed image, the block effect at the block boundary can be eliminated, and the peak signal-to-noise ratio of the reconstructed image can be further improved.

[0135] In another embodiment of the present application, see Figure 7 , which shows a flow chart of another image processing method provided by an embodiment of the present application. Figure 7 As shown, the method may include:

[0136] S601: Receive a code stream transmitted by an encoding device; wherein the code stream is obtained by dividing a to-be-processed image into a plurality of image blocks through a preset encoding network model.

[0137] S602: parse the bitstream using a preset decoding network model to obtain multiple reconstructed blocks.

[0138] It should be noted that the method is applied to a decoding device. After the encoding device generates a bitstream by compressing and encoding using a preset encoding network model, the bitstream can be transmitted to a decoding device, and the decoding device uses the preset decoding network model to parse the bitstream, thereby obtaining multiple reconstructed blocks. Here, after obtaining multiple reconstructed blocks, the aforementioned Figure 1 The method is used to obtain the final target image; the target image here not only realizes the quality enhancement of the central area of ​​the reconstructed block, but also eliminates the block effect at the block boundary in the reconstructed image.

[0139] It should also be noted that the preset encoding network model and the preset decoding network model are obtained by model training based on a neural network structure. The preset encoding network model is used to instruct the encoding device to encode multiple image blocks divided by the image to be processed to generate a bitstream, and the preset decoding network model is used to instruct the decoding device to parse the bitstream to obtain multiple reconstructed blocks.

[0140] Here, for the preset encoding network model and the preset decoding network model, in some embodiments, the method may further include:

[0141] Acquire a training set; wherein the training set includes at least one training image;

[0142] Constructing a coding network model and a decoding network model, and performing model training on the coding network model and the decoding network model based on the training set to obtain the preset coding network model and the preset decoding network model.

[0143] It should be noted that before model training, it is necessary to first obtain a training set, which includes at least one training image. Specifically, a standardized high-definition static image dataset, such as the NIC dataset, can be collected and organized; then, based on the NIC dataset, a training set for model training, as well as a test set and a cross-validation set for model testing and model validation can be obtained.

[0144] In addition, for the construction of encoding network model and decoding network model, it is necessary to establish a multi-layer deep neural network model, that is, an end-to-end encoding and decoding network structure. Figure 8 As shown, it shows a schematic diagram of an end-to-end network structure including a preset encoding network model and a preset decoding network model provided by an embodiment of the present application. Figure 8In the coding end, the coding network model structure is adopted, which may include a main encoder, a hyper prior encoder, a context model, a quantization module and an entropy encoder. Among them, for the input image, it can be divided into multiple image blocks. For each image block, the main encoder is used to transform the input original image into a feature map with 192 channels and row and column sizes of 1 / 16 of the original size. The role of the hyper prior codec and the context model is to estimate the probability distribution of pixels in the feature map according to the feature map, and provide the probability distribution to the entropy encoder. The entropy encoder here can use arithmetic coding, and it is lossless entropy coding compression. At the coding end, the feature map generated by the main encoder can be quantized by rounding to the nearest integer through the quantization module, and the entropy encoder uses the probability distribution provided by the hyper prior encoder, the hyper prior decoder and the context model to perform lossless entropy coding (such as arithmetic coding) on ​​the quantized feature map to form a code stream; and the compressed data generated by the hyper prior encoder uses a fixed probability distribution for probability calculation, and is added to the final code stream as additional information after passing through the entropy encoder. The decoding end adopts a decoding network model structure, which may include a main decoder, a super priori decoder, a context model, and an entropy decoder. The super priori decoder and the context model decode the probability distribution of pixels in the feature map through the additional information added in the aforementioned bitstream and provide it to the entropy decoder, which parses the feature map; the main decoder restores the feature map to a reconstructed block, and then splices the reconstructed block into a reconstructed image.

[0145] It is also important to note that in Figure 8 In the above, the main encoder is used to convert the pixel domain of the image into the feature domain, the super prior encoder is used to convert the feature domain into a probability distribution; the super prior decoder is used to convert the probability distribution into the feature domain, and the main decoder converts the feature domain into the pixel domain to reconstruct the reconstructed image. In addition, the probability distribution of the context model can be expressed as (μ, σ); where μ represents the mean and σ represents the variance.

[0146] In this way, after the encoding network model and the decoding network model are constructed, the training set and the preset algorithm can be used to perform model training on the encoding network model and the decoding network model. Specifically, in some embodiments, the model training of the encoding network model and the decoding network model based on the training set to obtain the preset encoding network model and the preset decoding network model may include:

[0147] Based on the training set, performing model training on the encoding network model and the decoding network model using a preset algorithm;

[0148] When the loss value corresponding to the cost function of the model training converges to a preset threshold, the encoding network model and the decoding network model obtained after training are determined as the preset encoding network model and the preset decoding network model.

[0149] It should be noted that the preset algorithm can be an Adam gradient optimization algorithm. The Adam gradient optimization algorithm is used to perform model training on the end-to-end network structure of the encoding network model and the decoding network model. Here, the cost function can be a rate-distortion cost function, and the distortion is the mean square error between the training image input to the network structure and the reconstructed image output by the network structure. Among them, the bit rate is estimated by calculating the amount of information contained in the pixel in the feature map using the probability distribution obtained by the super-prior encoder, the super-prior decoder and the context model. The encoding network model and the decoding network model are fully trained using the training set. After the loss value corresponding to the cost function reaches convergence and converges to the preset threshold, the encoding network model and the decoding network model are saved as the preset encoding network model and the preset decoding network model of the end-to-end network structure in the embodiment of the present application. Here, the preset threshold is specifically set according to the actual situation, and the embodiment of the present application does not impose any limitation.

[0150] Based on the preset encoding network model and the preset decoding network model, the post-processing network model is trained. Specifically, in some embodiments, the method may further include:

[0151] Acquire a plurality of reconstructed training blocks; wherein the plurality of reconstructed training blocks are obtained by dividing a plurality of training blocks of at least one training image in the training set through the preset encoding network model and the preset decoding network model;

[0152] A post-processing network model is constructed, and the post-processing network model is trained with the multiple reconstructed training blocks and the at least one training image to obtain the preset post-processing network model.

[0153] Furthermore, the training of the post-processing network model with the multiple reconstructed training blocks and the at least one training image to obtain the preset post-processing network model may include:

[0154] Based on the multiple reconstructed training blocks and the at least one training image, performing model training on the post-processing network model using a preset algorithm;

[0155] When the loss value corresponding to the cost function of the model training converges to a preset threshold, the post-processing network model obtained after training is determined as the preset post-processing network model.

[0156] It should be noted that the Adam gradient optimization algorithm can also be used for model training of the post-processing network model. For the training images in the training set, they can be divided into multiple training blocks of equal size and without overlap, and after inputting the preset encoding network model and the preset decoding network model, the obtained multiple reconstructed blocks are re-joined into a reconstructed training image with a block effect. At this time, the reconstructed training image with a block effect can be used as the training input image of the post-processing network model, and the training image in the training set can be used as the training target image of the post-processing network model; then the cost function of the model training can be constructed according to the mean square error of the training input image and the training target image. In the process of training the post-processing network using the Adam gradient optimization algorithm, the network parameters of the preset encoding network model and the preset decoding network model are kept fixed, and only the post-processing network model is iteratively updated. When the loss value corresponding to its cost function reaches convergence, the post-processing network model obtained by training at this time is the preset post-processing network model. Here, the preset threshold is specifically set according to the actual situation, and the embodiment of the present application does not impose any limitation.

[0157] S603: Use a preset post-processing network model to perform quality enhancement on a central area of ​​at least one of the multiple reconstructed blocks to obtain at least one target reconstructed block.

[0158] S604: Generate a reconstructed image according to the at least one target reconstructed block and the multiple reconstructed blocks.

[0159] S605: Utilize the preset post-processing network model to filter the block boundaries in the reconstructed image to obtain a target image.

[0160] It should be noted that after obtaining the preset post-processing network model, the preset post-processing network model can be used to enhance the quality of the central area of ​​at least one reconstructed block among multiple reconstructed blocks to obtain at least one target reconstructed block; and based on the at least one target reconstructed block, a reconstructed image can be spliced ​​to generate; since there is a block effect in the reconstructed image here, the same preset post-processing network model can be used to filter the block boundaries in the reconstructed image to obtain a target image with the block effect eliminated.

[0161] Specifically, in some embodiments, the filtering process of the block boundaries in the reconstructed image using the preset post-processing network model to obtain the target image may include:

[0162] determining at least one rectangular region in the reconstructed image that includes the block boundary;

[0163] Inputting the at least one rectangular area into the preset post-processing network model to obtain at least one processed rectangular area;

[0164] The target image is obtained by replacing a corresponding local area including the block boundary in the reconstructed image with the at least one processed rectangular area.

[0165] Furthermore, after obtaining at least one processed rectangular area, the method may further include:

[0166] Clipping the at least one processed rectangular area to obtain at least one target rectangular area;

[0167] The target image is obtained by replacing a corresponding local area including the block boundary in the reconstructed image with the at least one target rectangular area.

[0168] That is to say, after extracting and determining at least one rectangular area including a block boundary in the reconstructed image, the at least one rectangular area can be input into a preset post-processing network model to obtain at least one processed rectangular area; then the at least one processed rectangular area is used to replace the corresponding local area including the block boundary in the reconstructed image to obtain the target image. The target image obtained at this time can reduce the block effect. In order to further eliminate the block effect, the boundary image distortion caused by the convolution operation of the preset post-processing network model on the boundary 0 can also be eliminated. At this time, it is necessary to crop the at least one processed rectangular area, such as discarding the edge pixel areas with a width of 8 pixels and a height of 128 pixels on the left and right sides, and only retaining the central rectangular area with a size of 16*128 to obtain at least one target rectangular area; finally, the corresponding local area including the block boundary in the reconstructed image is replaced by the at least one target rectangular area, and a target image without obvious block effect can be obtained.

[0169] The present embodiment provides an image processing method, which receives a code stream transmitted by an encoding device; wherein the code stream is obtained by dividing a to-be-processed image into a plurality of image blocks through a preset encoding network model; the code stream is parsed using a preset decoding network model to obtain a plurality of reconstructed blocks; a preset post-processing network model is used to perform quality enhancement on a central area of ​​at least one of the plurality of reconstructed blocks to obtain at least one target reconstructed block; a reconstructed image is generated based on the at least one target reconstructed block and the plurality of reconstructed blocks; and the preset post-processing network model is used to perform filtering processing on block boundaries in the reconstructed image to obtain a target image. In this way, for the multiple image blocks divided by the image to be processed, the blocks are completely independent of each other, and the preset encoding network model and the preset decoding network model can be used to implement multi-core parallel encoding and decoding processing; and since the image size input into the preset encoding network model and the preset decoding network model is reduced after block division, the running time and running memory requirements of the encoding and decoding can also be reduced; in addition, by enhancing the quality of the central area of ​​the reconstructed block, the peak signal-to-noise ratio of the reconstructed image can also be improved, and at the same time, the computational redundancy caused by the overlap with the area to be filtered is reduced; by filtering the block boundaries in the reconstructed image, the block effect at the block boundaries can be eliminated, and the peak signal-to-noise ratio of the reconstructed image is further improved.

[0170] In another embodiment of the present application, see Fig. 9 , which shows a flow chart of another image processing method provided by an embodiment of the present application. Fig. 9 As shown, the method may include:

[0171] S801: Acquire an image to be processed.

[0172] S802: Divide the image to be processed into blocks to obtain a plurality of image blocks; wherein the plurality of image blocks are equal in size and have no overlap.

[0173] S803: Encode the multiple image blocks using a preset encoding network model to generate a bitstream.

[0174] S804: Transmit the code stream to a decoding device.

[0175] It should be noted that the method is applied to an encoding device. After the encoding device generates a bitstream by compressing and encoding using a preset encoding network model, the bitstream can be transmitted to a decoding device, which parses the bitstream using a preset decoding network model to obtain multiple reconstructed blocks.

[0176] It should also be noted that the preset encoding network model and the preset decoding network model are obtained by model training based on a neural network structure. The preset encoding network model is used to instruct the encoding device to encode multiple image blocks divided by the image to be processed to generate a bitstream, and the preset decoding network model is used for the decoding device to parse the bitstream to obtain multiple reconstructed blocks.

[0177] Here, for the preset encoding network model and the preset decoding network model, in some embodiments, the method may further include:

[0178] Acquire a training set; wherein the training set includes at least one training image;

[0179] A coding network model and a decoding network model are constructed, and model training is performed on the coding network model and the decoding network model based on the training set to obtain a preset coding network model and a preset decoding network model.

[0180] Furthermore, the performing model training on the encoding network model and the decoding network model based on the training set to obtain a preset encoding network model and a preset decoding network model may include:

[0181] Based on the training set, performing model training on the encoding network model and the decoding network model using a preset algorithm;

[0182] When the loss value corresponding to the cost function of the model training converges to a preset threshold, the encoding network model and the decoding network model obtained after training are determined as the preset encoding network model and the preset decoding network model.

[0183] It should be noted that for the construction of the encoding network model and the decoding network model, it is necessary to establish a multi-layer deep neural network model, that is, an end-to-end encoding and decoding network structure, as shown below: Figure 8 As shown. After constructing the encoding network model and the decoding network model, the Adam gradient optimization algorithm can be used to train the encoding network model and the decoding network model. Here, the cost function can be a rate-distortion cost function, and the distortion is the mean square error between the training image input by the network structure and the reconstructed image output by the network structure. The encoding network model and the decoding network model are fully trained using the training set. After the loss value corresponding to the cost function converges and converges to the preset threshold, the encoding network model and the decoding network model obtained by training at this time are the preset encoding network model and the preset decoding network model in the embodiment of the present application. Here, the preset threshold is specifically set according to the actual situation, and the embodiment of the present application does not impose any limitation.

[0184] This embodiment provides an image processing method, which includes obtaining an image to be processed; dividing the image to be processed into blocks to obtain multiple image blocks; wherein the multiple image blocks are equal in size and have no overlap; encoding the multiple image blocks using a preset coding network model to generate a code stream; and transmitting the code stream to a decoding device. In this way, for the multiple image blocks divided by the image to be processed, the blocks are completely independent from each other, and the preset coding network model and the preset decoding network model can be used to implement multi-core parallel coding and decoding processing; and because the image size input to the preset coding network model and the preset decoding network model is reduced after the block division, the running time of the coding and decoding and the running memory requirements are reduced.

[0185] In another embodiment of the present application, based on the same inventive concept as the above embodiment, see Fig.10 , which shows a detailed flow chart of an image processing method provided by an embodiment of the present application. Fig.10 As shown, the detailed process may include:

[0186] S901: Build a training set.

[0187] It should be noted that, for step S901, a suitable static image training set can be selected. Here, the selection of the training set has a great influence on the training of the entire neural network. In the embodiment of the present application, the NIC data set can be selected. The NIC data set is a development data set of the IEEE standard test model NIC for image compression based on deep learning. In this data set, a training set with an image size of 256*256 may be included, and a verification set and a test set with an image size of 256*256 may also be included.

[0188] S902: Establish a multi-layer deep neural network model, including an encoding network model, a decoding network model and a post-processing network model.

[0189] It should be noted that the end-to-end network structure of the encoding network model and the decoding network model is as follows Figure 8As shown. The encoding end adopts a coding network model structure, including a main encoder, a super a priori encoder and a context model. The function of the main encoder is to transform the input image into a feature map with 192 channels and row and column sizes of 1 / 16 of the original size. The function of the super a priori codec and the context model is to estimate the probability distribution of pixels in the feature map based on the feature map and provide it to the entropy encoder. At the encoding end, the compressed data generated by the super a priori encoder uses a fixed probability distribution for probability calculation, and is added to the final compressed bitstream as additional information after entropy coding. The decoding end adopts a decoding network model structure, including a main decoder, a super a priori decoder and a context model. The function of the super a priori decoder and the context model is to decode the probability distribution of pixels in the feature map through additional information and provide it to the entropy decoder. The function of the main decoder is to restore the feature map to a reconstructed image.

[0190] For the post-processing network model, Figure 3 As shown in , it can be composed of a convolutional layer, an activation function, and multiple cascaded residual blocks for improving model performance. The specific network structure inside the residual block is as follows Figure 4 In the figure, k3n128 represents a convolutional layer with a convolution kernel size of 3*3, an output feature number of 128, and a stride of 1; k33 represents a convolutional layer with a convolution kernel size of 3*3, an output feature number of 3, and a stride of 1.

[0191] S903: Perform model training on the encoding network model and the decoding network model using the training set and the preset algorithm to obtain a preset encoding network model and a preset decoding network model.

[0192] It should be noted that, for step S903, the Adam gradient optimization algorithm can be used to train the end-to-end encoding network model and decoding network model. Among them, the cost function is a rate-distortion cost function, and the distortion is the mean square error between the training image input to the network structure and the reconstructed image output by the network structure; the bit rate is estimated by calculating the amount of information contained in the pixel in the feature map using the probability distribution obtained by the super-prior encoder, super-prior decoder and context model. By fully training on the basis of the training set established in S901, when the loss value corresponding to its cost function reaches convergence, the trained encoding network model and decoding network model are saved and used as the end-to-end preset encoding network model and preset decoding network model.

[0193] S904: Based on the preset encoding network model and the preset decoding network model obtained through training, a post-processing network model is trained using a training set and a preset algorithm to obtain a preset post-processing network model.

[0194] It should be noted that the preset encoding network model and the preset decoding network model saved in step S903 are used to train the post-processing network model. The training image of size 256*256 in the training set described in step S901 is divided into equal and non-overlapping 128*128 training blocks and input into the preset encoding network model and the preset decoding network model, and then the obtained reconstructed blocks are reassembled into a 256*256 reconstructed training image with block effect. At this time, the reconstructed training image with block effect can be used as the training input image of the post-processing network model, and the training image without encoding compression in the training set can be used as the training target image of the post-processing network model; then the cost function of the model training can be constructed according to the mean square error of the training input image and the training target image, and the Adam gradient optimization algorithm is used to train the post-processing network. During the training process, the network parameters of the preset encoding network model and the preset decoding network model are kept fixed, and only the post-processing network model is iteratively updated. After the loss value corresponding to its cost function reaches convergence, the post-processing network model obtained by training at this time is the preset post-processing network model.

[0195] S905: Divide the image to be processed into 128*128 image blocks of equal size and without overlap and input the blocks into a preset coding network model to generate a code stream to be transmitted.

[0196] It should be noted that the image to be processed can be divided into multiple image blocks of equal size and without overlap, and these image blocks are input into a preset coding network model to generate a bit stream; specifically, the output data of the preset coding network model can be quantized and losslessly entropy encoded to output as compressed data.

[0197] That is to say, at the encoding end, the image to be processed is divided into non-overlapping image blocks of equal size of 128*128 and then input into the preset encoding network model, and each image block is independently encoded using the preset encoding network model to generate a feature map. The feature map is then quantized by rounding off, and the entropy encoder uses the probability distribution provided by the super-a priori encoder, super-a priori decoder and context model to perform lossless entropy encoding (such as arithmetic coding) on ​​the quantized feature map to form a bit stream, and is superimposed with the additional bit stream generated by the super-a priori encoder as the final compressed data, and then transmitted to the decoding end in the form of a bit stream.

[0198] S906: Parse the bitstream using a preset decoding network model to obtain a 128*128 reconstructed block.

[0199] It should be noted that the decoding end reconstructs the feature map of each block into a 128*128 reconstructed block through an entropy decoder and a preset decoding network model in a symmetrical manner with the encoding end.

[0200] S907: Use a preset post-processing network model to enhance the quality of the central area of ​​the reconstructed block, and splice to generate a reconstructed image.

[0201] It should be noted that, for step S907, the center enhancement processing is performed on the reconstructed block in step S906. Figure 5 The central area of ​​the reconstructed block with a size of 112*112 is extracted in the manner shown. The central area is input into the preset post-processing network model trained in step S904 to enhance the image quality of the central area of ​​the block; finally, the output of the preset post-processing network model is used to replace the central area corresponding to the reconstructed block in step S906; and a reconstructed image is generated by splicing.

[0202] S908: Perform local post-processing on the block boundaries in the reconstructed image using a preset post-processing network model to obtain a target image.

[0203] It should be noted that, for step S908, local post-processing is performed on the block boundaries of the reconstructed image in step S907. Figure 6 The rectangular area near the image boundary is extracted in the manner shown. The specific range of the rectangular area of ​​the horizontal boundary is, horizontally: 16 pixels on the left side of the block boundary to 16 pixels on the right side of the block boundary, and vertically: the upper edge of the block to the lower edge of the block. The specific range of the rectangular area of ​​the vertical boundary is, vertically: 16 pixels on the upper side of the block boundary to 16 pixels on the lower side of the block boundary. Horizontally: from the left edge of the block to the right edge of the block. For the reconstructed image spliced ​​by the reconstruction block of size 128*128 in step S906, the size of its rectangular area is 32*128 pixels. The pixels of the rectangular area are input into the preset post-processing network model trained in step S904, and the rectangular area of ​​the boundary with reduced block effect is output. In order to further eliminate the boundary image distortion caused by the convolution operation of the boundary filling 0 in the preset post-processing network model, the embodiment of the present application can further crop the rectangular area output by the preset post-processing network model, for example, discard the edge pixel areas with a width of 8 pixels and a height of 128 pixels on the left and right sides, and only retain the central rectangular area with a size of 16*128. Finally, the corresponding block boundary rectangular area in the original reconstructed image is replaced with the rectangular area of ​​16*128 in size, so that a reconstructed image without obvious block effect can be obtained.

[0204] Thus, the embodiment of the present application provides a block encoding and decoding scheme for static images. By dividing the input image into blocks and then encoding and decoding each image block independently, it is possible to realize multi-core parallel encoding and decoding processing of the image, thereby reducing the running time required for encoding and decoding the image and the running memory requirements of each core; in addition, by enhancing the center of the reconstructed block, its peak signal-to-noise ratio can be improved; by performing local post-processing on the boundary of the reconstructed image, the block effect at the boundary of the block can be reduced. The specific steps are as follows: (1) Selecting a suitable static image training set, training set and verification set; (2) Establishing an end-to-end network encoding network model, decoding network model and post-processing network model of the reconstructed image; (3) Training the encoding network model and decoding network model of the end-to-end network, and obtaining a preset encoding network model and a preset decoding network model after training; (4) Dividing the training images in the training set into 128*128 non-overlapping blocks and inputting them into the trained preset encoding network model and preset decoding network model, and then splicing the reconstructed blocks obtained by decoding into a reconstructed image as new training data to train the post-processing network model model, and obtain the preset post-processing network model after training; (5) the output data of the encoder after the preset encoding network model is quantized and lossless entropy encoded as compressed data, and transmitted to the decoder in the form of a bitstream; (6) the decoder restores the bitstream into a 128*128 reconstructed block through the preset decoding network model; (7) the central area of ​​the reconstructed block is enhanced by the preset post-processing network model, and it is spliced ​​into a reconstructed image; (8) the block boundary area in the reconstructed image is locally post-processed using the same preset post-processing network model as in step (7) to reduce the block effect at the boundary, and finally the target image is obtained.

[0205] In short, in an embodiment of the present application, the input image is processed in blocks based on the existing image encoding and decoding network structure, and the blocks are encoded and decoded independently to realize multi-core parallel processing of encoding and decoding, which can reduce the running time and single-core running memory requirements. In addition, a post-processing network model is used to enhance the central area of ​​the reconstructed block to improve the peak signal-to-noise ratio of the reconstructed image. At the same time, only the central area of ​​the block is post-processed, and the computational redundancy caused by the overlap with the area to be filtered is reduced. At the same time, the same preset post-processing network model as the quality enhancement processing is used for the rectangular area at the boundary of the block in the reconstructed image for local post-processing. In this way, the use of the same preset post-processing network model can reduce the storage space required for storing network parameters at the decoding end. Here, each rectangular area is completely independent during the processing process, and the parallelization of post-processing can be realized, thereby achieving the effect of reducing the running time and single-core memory requirements.

[0206] That is to say, the technical solution of the embodiment of the present application can realize multi-core parallel encoding and decoding of images, and reduce the running time and running memory requirements of single-core encoding and decoding. Since the encoding and decoding process based on the preset encoding network model and the preset decoding network model is independent of each other for the blocks divided into multiple image blocks, multi-core parallel encoding and decoding of images can be realized. In addition, since the image size input to the preset encoding network model and the preset decoding network model after block division is reduced, the running time and running memory requirements required for single-core encoding and decoding are also reduced. As shown in Table 1, in the Kodak test set with an original image size of 512*768, the running time of each core and the single-core running memory requirements are reduced to 1 / 20 of the non-block encoding and decoding processing. Among them, the unit of running time is second (Second, s), and the unit of running memory requirement is megabyte (MByte, MB).

[0207] Table 1

[0208] Non-block codec network parameters There are block encoding and decoding network parameters Run time(s) 10.85 0.5 Running memory requirement (MB) 5195 263

[0209] In addition, the technical solution of the embodiment of the present application adopts a preset post-processing network model to enhance the central area of ​​the reconstructed block, which can improve the peak signal-to-noise ratio of the reconstructed image. Moreover, the technical solution of the embodiment of the present application adopts a preset post-processing network model to eliminate the block effect in the reconstructed image, further improving the peak signal-to-noise ratio of the reconstructed image. Here, the technical solution of the embodiment of the present application adopts a preset post-processing network model to solve the block effect caused by the discontinuity of the block boundary due to the block encoding and decoding; Fig.11A and Fig. 11B As shown, Fig.11A There is obvious block effect in the reconstructed image. After post-processing, Fig. 11B It can be clearly seen that the blocking effect problem of the reconstructed image has been effectively solved. Fig.12 As shown in FIG. 2 , an example of a rate-distortion curve between bit rate and peak signal-to-noise ratio is shown; after post-processing by a preset post-processing network model, the peak signal-to-noise ratio can be increased by about 0.08 to 0.11 dB, and while realizing multi-core parallel post-processing of the reconstructed image blocks, a peak signal-to-noise ratio gain effect of 80% to 90% can be achieved for post-processing of the entire image. Here, as shown in Table 2, when the central area of ​​the reconstructed block is enhanced, the running time and running memory required for each core post-processing are 1 / 30 of the post-processing of the entire image; when the boundary area in the reconstructed image is deblocked, the running time and running memory required for each core post-processing are 1 / 90 of the post-processing of the entire image.

[0210] Table 2

[0211]

[0212] The present embodiment provides an image processing method, through which the specific implementation of the aforementioned embodiment is elaborated in detail. It can be seen that for the multiple image blocks divided by the image to be processed, the blocks are completely independent of each other, and the preset encoding network model and the preset decoding network model can be used to implement multi-core parallel encoding and decoding processing; and since the image size input to the preset encoding network model and the preset decoding network model is reduced after block division, the running time and running memory requirement of the encoding and decoding can also be reduced; in addition, by enhancing the quality of the central area of ​​the reconstructed block, the peak signal-to-noise ratio of the reconstructed image can also be improved, and at the same time, the computational redundancy caused by the overlap with the area to be filtered is reduced; by filtering the block boundaries in the reconstructed image, the block effect at the block boundaries can also be eliminated, and the peak signal-to-noise ratio of the reconstructed image is further improved.

[0213] In another embodiment of the present application, based on the same inventive concept as the above embodiment, see Fig.13 , which shows a schematic diagram of the composition structure of an image processing device 120 provided in an embodiment of the present application. Fig.13 As shown, the image processing device 120 may include: an acquisition unit 1201, a determination unit 1202, an enhancement unit 1203 and a processing unit 1204; wherein,

[0214] The acquisition unit 1201 is configured to acquire a plurality of reconstructed blocks; wherein the plurality of reconstructed blocks are obtained by dividing a plurality of image blocks into which the image to be processed is divided through a preset encoding network model and a preset decoding network model;

[0215] A determining unit 1202 is configured to determine a central area of ​​at least one reconstructed block among the multiple reconstructed blocks;

[0216] The enhancement unit 1203 is configured to perform quality enhancement on a central area of ​​the at least one reconstructed block to obtain at least one target reconstructed block;

[0217] The processing unit 1204 is configured to generate a reconstructed image according to the at least one target reconstructed block and the multiple reconstructed blocks; and perform filtering processing on the block boundaries in the reconstructed image to obtain a target image.

[0218] In some embodiments, the enhancement unit 1203 is specifically configured to perform quality enhancement on the central area of ​​the at least one reconstructed block to obtain at least one enhanced area; and replace the central area of ​​the at least one reconstructed block according to the at least one enhanced area to obtain the at least one target reconstructed block.

[0219] In some embodiments, the processing unit 1204 is specifically configured to replace at least one reconstructed block among the multiple reconstructed blocks with the at least one target reconstructed block, and splice the multiple reconstructed blocks obtained after the replacement to generate the reconstructed image.

[0220] In some embodiments, see Fig.13 , the image processing device 120 may further include a construction unit 1205 configured to construct a preset post-processing network model;

[0221] The enhancement unit 1203 is specifically configured to use the preset post-processing network model to perform quality enhancement on the central area of ​​the at least one reconstructed block to obtain at least one target reconstructed block;

[0222] The processing unit 1204 is specifically configured to use the preset post-processing network model to perform filtering processing on the block boundaries in the reconstructed image to obtain a target image.

[0223] In some embodiments, see Fig.13 , the image processing device 120 may further include a training unit 1206; wherein,

[0224] The acquisition unit 1201 is further configured to acquire a plurality of reconstructed training blocks; wherein the plurality of reconstructed training blocks are obtained by dividing a plurality of training blocks from at least one training image in the training set through the preset encoding network model and the preset decoding network model;

[0225] The construction unit 1205 is further configured to construct a post-processing network model;

[0226] The training unit 1206 is configured to train the post-processing network model with the multiple reconstructed training blocks and the at least one training image to obtain the preset post-processing network model.

[0227] Furthermore, the training unit 1206 is specifically configured to perform model training on the post-processing network model using a preset algorithm based on the multiple reconstructed training blocks and the at least one training image; when the loss value corresponding to the cost function of the model training converges to a preset threshold, the post-processing network model obtained after training is determined as the preset post-processing network model.

[0228] In some embodiments, the determining unit 1202 is further configured to determine at least one rectangular region including the block boundary in the reconstructed image;

[0229] The processing unit 1204 is specifically configured to input the at least one rectangular area into the preset post-processing network model to obtain at least one processed rectangular area; and use the at least one processed rectangular area to replace the corresponding local area including the block boundary in the reconstructed image to obtain the target image.

[0230] Furthermore, the processing unit 1204 is also configured to crop the at least one processed rectangular area to obtain at least one target rectangular area; and use the at least one target rectangular area to replace the corresponding local area including the block boundary in the reconstructed image to obtain the target image.

[0231] It is understandable that in the embodiments of the present application, a "unit" may be a part of a circuit, a part of a processor, a part of a program or software, etc., and of course, it may be a module, or it may be non-modular. Moreover, the components in the present embodiment may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional module.

[0232] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0233] Therefore, an embodiment of the present application provides a computer storage medium, which is applied to the image processing device 120. The computer storage medium stores an image processing program, and when the image processing program is executed by the first processor, the method described in any one of the above embodiments is implemented.

[0234] Based on the composition of the above-mentioned image processing device 120 and the computer storage medium, see Fig.14 , which shows a schematic diagram of the hardware structure of the image processing device 120 provided in an embodiment of the present application. Fig.14As shown, the image processing device 120 may include: a first communication interface 1301, a first memory 1302 and a first processor 1303; each component is coupled together via a first bus system 1304. It is understood that the first bus system 1304 is used to realize the connection and communication between these components. In addition to the data bus, the first bus system 1304 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Fig.14 In the figure, various buses are labeled as the first bus system 1304. Among them,

[0235] The first communication interface 1301 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;

[0236] A first memory 1302, used to store a computer program that can be run on the first processor 1303;

[0237] The first processor 1303 is configured to, when running the computer program, execute:

[0238] Acquire multiple reconstructed blocks; wherein the multiple reconstructed blocks are obtained by dividing the image to be processed into multiple image blocks through a preset encoding network model and a preset decoding network model;

[0239] determining a central area of ​​at least one reconstructed block among the plurality of reconstructed blocks;

[0240] Performing quality enhancement on a central area of ​​the at least one reconstructed block to obtain at least one target reconstructed block;

[0241] generating a reconstructed image according to the at least one target reconstructed block and the plurality of reconstructed blocks;

[0242] A filtering process is performed on the block boundaries in the reconstructed image to obtain a target image.

[0243] It can be understood that the first memory 1302 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM). The first memory 1302 of the system and method described in the present application is intended to include, but is not limited to, these and any other suitable types of memory.

[0244] The first processor 1303 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the first processor 1303. The above-mentioned first processor 1303 can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the first memory 1302, and the first processor 1303 reads the information in the first memory 1302 and completes the steps of the above method in combination with its hardware.

[0245] It is to be understood that these embodiments described in the present application can be implemented with hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processors (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable logic devices (Programmable Logic Device, PLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units or combinations thereof for performing functions described in the present application. For software implementation, the technology described in the present application can be implemented by a module (such as a process, function, etc.) that performs functions described in the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in a processor or outside a processor.

[0246] Optionally, as another embodiment, the first processor 1303 is further configured to execute any one of the methods described in the foregoing embodiments when running the computer program.

[0247] This embodiment provides an image processing device, which may include an acquisition unit, a determination unit, an enhancement unit, and a processing unit. In this way, for the multiple image blocks divided by the image to be processed, the blocks are completely independent from each other, and the preset encoding network model and the preset decoding network model can be used to implement multi-core parallel encoding and decoding processing; and because the image size input to the preset encoding network model and the preset decoding network model is reduced after block division, the running time and running memory requirements of the encoding and decoding can also be reduced; in addition, by enhancing the quality of the central area of ​​the reconstructed block, the peak signal-to-noise ratio of the reconstructed image can also be improved, and at the same time, the computational redundancy caused by the overlap with the area to be filtered is reduced; by filtering the block boundaries in the reconstructed image, the block effect at the block boundaries can also be eliminated, and the peak signal-to-noise ratio of the reconstructed image is further improved.

[0248] In another embodiment of the present application, based on the same inventive concept as the above embodiment, see Fig.15 , which shows a schematic diagram of the structure of a decoding device 140 provided in an embodiment of the present application. Fig.15 As shown, the decoding device 140 may include: a receiving unit 1401, a decoding unit 1402 and a post-processing unit 1403; wherein,

[0249] The receiving unit 1401 is configured to receive a code stream transmitted by an encoding device; wherein the code stream is obtained by dividing a plurality of image blocks into which an image to be processed is divided through a preset encoding network model;

[0250] The decoding unit 1402 is configured to parse the bitstream using a preset decoding network model to obtain a plurality of reconstructed blocks;

[0251] The post-processing unit 1403 is configured to use a preset post-processing network model to perform quality enhancement on a central area of ​​at least one of the multiple reconstructed blocks to obtain at least one target reconstructed block; and to generate a reconstructed image based on the at least one target reconstructed block and the multiple reconstructed blocks, and to use the preset post-processing network model to filter the block boundaries in the reconstructed image to obtain a target image.

[0252] In some embodiments, see Fig.15 , the decoding device 140 may further include an acquisition unit 1404, a construction unit 1405 and a training unit 1406; wherein,

[0253] The acquisition unit 1404 is configured to acquire a training set; wherein the training set includes at least one training image;

[0254] A construction unit 1405 is configured to construct an encoding network model and a decoding network model;

[0255] The training unit 1406 is configured to perform model training on the encoding network model and the decoding network model based on the training set to obtain the preset encoding network model and the preset decoding network model.

[0256] Furthermore, the training unit 1406 is specifically configured to perform model training on the encoding network model and the decoding network model based on the training set using a preset algorithm; and when the loss value corresponding to the cost function of the model training converges to a preset threshold, the encoding network model and the decoding network model obtained after training are determined as the preset encoding network model and the preset decoding network model.

[0257] In some embodiments, the acquisition unit 1404 is further configured to acquire a plurality of reconstructed training blocks; wherein the plurality of reconstructed training blocks are obtained by dividing a plurality of training blocks from at least one training image in the training set through the preset encoding network model and the preset decoding network model;

[0258] The construction unit 1405 is further configured to construct a post-processing network model;

[0259] The training unit 1406 is further configured to train the post-processing network model using the multiple reconstructed training blocks and the at least one training image to obtain the preset post-processing network model.

[0260] Furthermore, the training unit 1406 is specifically configured to perform model training on the post-processing network model using a preset algorithm based on the multiple reconstructed training blocks and the at least one training image; and when the loss value corresponding to the cost function of the model training converges to a preset threshold, the post-processing network model obtained after training is determined as the preset post-processing network model.

[0261] In some embodiments, the post-processing unit 1403 is specifically configured to determine at least one rectangular area in the reconstructed image that includes the block boundary; and input the at least one rectangular area into the preset post-processing network model to obtain at least one processed rectangular area; and use the at least one processed rectangular area to replace the corresponding local area in the reconstructed image that includes the block boundary to obtain the target image.

[0262] Furthermore, the post-processing unit 1403 is also configured to crop the at least one processed rectangular area to obtain at least one target rectangular area; and use the at least one target rectangular area to replace the corresponding local area including the block boundary in the reconstructed image to obtain the target image.

[0263] It can be understood that in this embodiment, a "unit" can be a part of a circuit, a part of a processor, a part of a program or software, etc., and of course it can also be a module, or it can be non-modular. Moreover, the components in this embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional module.

[0264] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, this embodiment provides a computer storage medium, which is applied to the decoding device 140, and the computer storage medium stores an image processing program. When the image processing program is executed by the second processor, the method described in any one of the above embodiments is implemented.

[0265] Based on the composition of the above decoding device 140 and the computer storage medium, see Fig.16 , which shows a schematic diagram of the hardware structure of the decoding device 140 provided in an embodiment of the present application. Fig.16 As shown, the decoding device 140 may include: a second communication interface 1501, a second memory 1502, and a second processor 1503; each component is coupled together via a second bus system 1504. It is understood that the second bus system 1504 is used to realize the connection and communication between these components. In addition to the data bus, the second bus system 1504 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig.16 In FIG. 1 , various buses are labeled as the second bus system 1504. Among them,

[0266] The second communication interface 1501 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;

[0267] The second memory 1502 is used to store a computer program that can be run on the second processor 1503;

[0268] The second processor 1503 is configured to execute, when running the computer program:

[0269] Receiving a code stream transmitted by an encoding device; wherein the code stream is obtained by dividing a plurality of image blocks into which an image to be processed is divided through a preset encoding network model;

[0270] Parsing the bitstream using a preset decoding network model to obtain a plurality of reconstructed blocks;

[0271] Using a preset post-processing network model, quality enhancement is performed on a central area of ​​at least one of the multiple reconstructed blocks to obtain at least one target reconstructed block;

[0272] generating a reconstructed image according to the at least one target reconstructed block and the plurality of reconstructed blocks;

[0273] The preset post-processing network model is used to filter the block boundaries in the reconstructed image to obtain a target image.

[0274] Optionally, as another embodiment, the second processor 1503 is further configured to execute any one of the methods described in the foregoing embodiments when running the computer program.

[0275] It can be understood that the hardware functions of the second memory 1502 and the first memory 1302 are similar, and the hardware functions of the second processor 1503 and the first processor 1303 are similar; they will not be described in detail here.

[0276] The present embodiment provides a decoding device, which may include a receiving unit, a decoding unit, and a post-processing unit. In this way, for the multiple image blocks divided by the image to be processed, the blocks are completely independent from each other, and the preset encoding network model and the preset decoding network model can be used to implement multi-core parallel encoding and decoding processing; and because the image size input to the preset encoding network model and the preset decoding network model is reduced after the block division, the running time and running memory requirements of the encoding and decoding can also be reduced; in addition, by enhancing the quality of the central area of ​​the reconstructed block, the peak signal-to-noise ratio of the reconstructed image can also be improved, and at the same time, the computational redundancy caused by the overlap with the area to be filtered is reduced; by filtering the block boundaries in the reconstructed image, the block effect at the block boundaries can also be eliminated, and the peak signal-to-noise ratio of the reconstructed image is further improved.

[0277] In another embodiment of the present application, based on the same inventive concept as the above embodiment, see Fig.17 , which shows a schematic diagram of the composition structure of an encoding device 160 provided in an embodiment of the present application. Fig.17 As shown, the encoding device 160 may include: an acquisition unit 1601, a block unit 1602, an encoding unit 1603 and a sending unit 1604; wherein,

[0278] An acquisition unit 1601 is configured to acquire an image to be processed;

[0279] The blocking unit 1602 is configured to block the image to be processed to obtain a plurality of image blocks; wherein the plurality of image blocks are equal in size and have no overlap;

[0280] The encoding unit 1603 is configured to encode the multiple image blocks using a preset encoding network model to generate a bit stream;

[0281] The sending unit 1604 is configured to transmit the code stream to a decoding device.

[0282] In some embodiments, see Fig.17 , the encoding device 160 may also include a construction unit 1605 and a training unit 1606; wherein,

[0283] The acquisition unit 1601 is further configured to acquire a training set; wherein the training set includes at least one training image;

[0284] A construction unit 1605 is configured to construct an encoding network model and a decoding network model;

[0285] The training unit 1606 is configured to perform model training on the encoding network model and the decoding network model based on the training set to obtain a preset encoding network model and a preset decoding network model; wherein the preset decoding network model is used to instruct the decoding device to parse the bitstream to obtain multiple reconstructed blocks.

[0286] Furthermore, the training unit 1606 is specifically configured to perform model training on the encoding network model and the decoding network model based on the training set using a preset algorithm; and when the loss value corresponding to the cost function of the model training converges to a preset threshold, the encoding network model and the decoding network model obtained after training are determined as the preset encoding network model and the preset decoding network model.

[0287] It can be understood that in this embodiment, a "unit" can be a part of a circuit, a part of a processor, a part of a program or software, etc., and of course it can also be a module, or it can be non-modular. Moreover, the components in this embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional module.

[0288] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, this embodiment provides a computer storage medium, which is applied to the encoding device 160, and the computer storage medium stores an image processing program. When the image processing program is executed by the third processor, the method described in any one of the above embodiments is implemented.

[0289] Based on the composition of the encoding device 160 and the computer storage medium, see Fig.18 , which shows a schematic diagram of the hardware structure of the encoding device 160 provided in an embodiment of the present application. Fig.18 As shown, the decoding device 140 may include: a third communication interface 1701, a third memory 1702 and a third processor 1703; each component is coupled together via a third bus system 1704. It is understandable that the third bus system 1704 is used to realize the connection and communication between these components. In addition to the data bus, the third bus system 1704 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Fig.18 In the figure, various buses are labeled as the third bus system 1704. Among them,

[0290] The third communication interface 1701 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;

[0291] The third memory 1702 is used to store a computer program that can be run on the third processor 1703;

[0292] The third processor 1703 is configured to, when running the computer program, execute:

[0293] Get the image to be processed;

[0294] Dividing the image to be processed into blocks to obtain a plurality of image blocks; wherein the plurality of image blocks are equal in size and have no overlap;

[0295] Encoding the multiple image blocks using a preset encoding network model to generate a bitstream;

[0296] The code stream is transmitted to a decoding device.

[0297] Optionally, as another embodiment, the third processor 1703 is further configured to execute any one of the methods described in the foregoing embodiments when running the computer program.

[0298] It can be understood that the hardware functions of the third memory 1702 and the first memory 1302 are similar, and the hardware functions of the third processor 1703 and the first processor 1303 are similar; they will not be described in detail here.

[0299] This embodiment provides a coding device, which may include an acquisition unit, a block division unit, a coding unit, and a sending unit. In this way, for the multiple image blocks divided by the image to be processed, the blocks are completely independent from each other, and the preset coding network model and the preset decoding network model can be used to implement multi-core parallel coding and decoding processing; and because the image size input to the preset coding network model and the preset decoding network model is reduced after the block division, the running time of the coding and decoding and the running memory requirements are reduced.

[0300] In another embodiment of the present application, based on the same inventive concept as the above embodiment, see Fig.19 , which shows a schematic diagram of the composition structure of a video system 180 provided in an embodiment of the present application. Fig.19 As shown, the video system 180 may include: the encoding device 160 described in the above embodiment and the decoding device 140 described in the above embodiment.

[0301] The encoding device 160 is configured to obtain an image to be processed; and divide the image to be processed into blocks to obtain a plurality of image blocks; wherein the plurality of image blocks are equal in size and have no overlap; and encode the plurality of image blocks using a preset encoding network model to generate a code stream; and transmit the code stream to the decoding device 140;

[0302] The decoding device 140 is configured to receive a code stream transmitted by the encoding device 160; and parse the code stream using a preset decoding network model to obtain multiple reconstructed blocks; and use a preset post-processing network model to enhance the quality of a central area of ​​at least one of the multiple reconstructed blocks to obtain at least one target reconstructed block; and generate a reconstructed image based on the at least one target reconstructed block and the multiple reconstructed blocks, and use the preset post-processing network model to filter the block boundaries in the reconstructed image to obtain a target image.

[0303] In an embodiment of the present application, for the multiple image blocks divided by the image to be processed, the blocks are completely independent of each other, and the preset encoding network model and the preset decoding network model can be used to implement multi-core parallel encoding and decoding processing; and since the image size input to the preset encoding network model and the preset decoding network model is reduced after block division, the running time and running memory requirements of the encoding and decoding can also be reduced; in addition, by enhancing the quality of the central area of ​​the reconstructed block, the peak signal-to-noise ratio of the reconstructed image can also be improved, and at the same time, the computational redundancy caused by the overlap with the area to be filtered is reduced; by filtering the block boundaries in the reconstructed image, the block effect at the block boundaries can be eliminated, and the peak signal-to-noise ratio of the reconstructed image is further improved.

[0304] It should be noted that, in this application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0305] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0306] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0307] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0308] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0309] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: Applied to an image processing device, the method comprises: Acquire multiple reconstructed blocks; wherein the multiple reconstructed blocks are obtained by dividing the image to be processed into multiple image blocks through a preset encoding network model and a preset decoding network model; determining a central area of ​​at least one reconstructed block among the plurality of reconstructed blocks; Performing quality enhancement on a central area of ​​the at least one reconstructed block to obtain at least one target reconstructed block; generating a reconstructed image according to the at least one target reconstructed block and the plurality of reconstructed blocks; A filtering process is performed on the block boundaries in the reconstructed image to obtain a target image.

2. The method according to claim 1, characterized in that The step of performing quality enhancement on the central area of ​​the at least one reconstructed block to obtain at least one target reconstructed block comprises: Performing quality enhancement on a central region of the at least one reconstructed block to obtain at least one enhanced region; The central area of ​​the at least one reconstructed block is replaced according to the at least one enhanced area to obtain the at least one target reconstructed block.

3. The method according to claim 1, characterized in that The step of generating a reconstructed image according to the at least one target reconstructed block and the plurality of reconstructed blocks comprises: At least one reconstructed block among the multiple reconstructed blocks is replaced with the at least one target reconstructed block, and the multiple reconstructed blocks obtained after the replacement are spliced ​​to generate the reconstructed image.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Build a preset post-processing network model; Accordingly, the step of performing quality enhancement on the central area of ​​the at least one reconstructed block to obtain at least one target reconstructed block includes: Using the preset post-processing network model, quality enhancement is performed on a central area of ​​the at least one reconstructed block to obtain at least one target reconstructed block; The filtering process is performed on the block boundaries in the reconstructed image to obtain the target image, including: The preset post-processing network model is used to filter the block boundaries in the reconstructed image to obtain a target image.

5. The method according to claim 4, characterized in that The step of constructing a preset post-processing network model includes: Acquire a plurality of reconstructed training blocks; wherein the plurality of reconstructed training blocks are obtained by dividing a plurality of training blocks from at least one training image in a training set through the preset encoding network model and the preset decoding network model; A post-processing network model is constructed, and the post-processing network model is trained with the multiple reconstructed training blocks and the at least one training image to obtain the preset post-processing network model.

6. The method according to claim 5, characterized in that The step of training the post-processing network model with the multiple reconstructed training blocks and the at least one training image to obtain the preset post-processing network model includes: Based on the multiple reconstructed training blocks and the at least one training image, performing model training on the post-processing network model using a preset algorithm; When the loss value corresponding to the cost function of the model training converges to a preset threshold, the post-processing network model obtained after training is determined as the preset post-processing network model.

7. The method according to claim 4, characterized in that The filtering process of the block boundaries in the reconstructed image by using the preset post-processing network model to obtain the target image includes: determining at least one rectangular region in the reconstructed image that includes the block boundary; Inputting the at least one rectangular area into the preset post-processing network model to obtain at least one processed rectangular area; The target image is obtained by replacing a corresponding local area including the block boundary in the reconstructed image with the at least one processed rectangular area.

8. The method according to claim 7, characterized in that After obtaining at least one processed rectangular area, the method further includes: Clipping the at least one processed rectangular area to obtain at least one target rectangular area; The target image is obtained by replacing a corresponding local area including the block boundary in the reconstructed image with the at least one target rectangular area.

9. An image processing method, characterized in that: Applied to a decoding device, the method comprises: Receiving a code stream transmitted by an encoding device; wherein the code stream is obtained by dividing a plurality of image blocks into which an image to be processed is divided through a preset encoding network model; Parsing the bitstream using a preset decoding network model to obtain a plurality of reconstructed blocks; Using a preset post-processing network model, quality enhancement is performed on a central area of ​​at least one of the multiple reconstructed blocks to obtain at least one target reconstructed block; generating a reconstructed image according to the at least one target reconstructed block and the plurality of reconstructed blocks; The preset post-processing network model is used to filter the block boundaries in the reconstructed image to obtain a target image.

10. The method according to claim 9, characterized in that The method further comprises: Acquire a training set; wherein the training set includes at least one training image; Constructing a coding network model and a decoding network model, and performing model training on the coding network model and the decoding network model based on the training set to obtain the preset coding network model and the preset decoding network model.

11. The method according to claim 10, characterized in that The performing model training on the encoding network model and the decoding network model based on the training set to obtain the preset encoding network model and the preset decoding network model includes: Based on the training set, performing model training on the encoding network model and the decoding network model using a preset algorithm; When the loss value corresponding to the cost function of the model training converges to a preset threshold, the encoding network model and the decoding network model obtained after training are determined as the preset encoding network model and the preset decoding network model.

12. The method according to claim 10, characterized in that The method further comprises: Acquire a plurality of reconstructed training blocks; wherein the plurality of reconstructed training blocks are obtained by dividing a plurality of training blocks of at least one training image in the training set through the preset encoding network model and the preset decoding network model; A post-processing network model is constructed, and the post-processing network model is trained with the multiple reconstructed training blocks and the at least one training image to obtain the preset post-processing network model.

13. The method according to claim 12, characterized in that The step of training the post-processing network model with the multiple reconstructed training blocks and the at least one training image to obtain the preset post-processing network model includes: Based on the multiple reconstructed training blocks and the at least one training image, performing model training on the post-processing network model using a preset algorithm; When the loss value corresponding to the cost function of the model training converges to a preset threshold, the post-processing network model obtained after training is determined as the preset post-processing network model.

14. The method according to any one of claims 9 to 13, characterized in that: The filtering process of the block boundaries in the reconstructed image by using the preset post-processing network model to obtain the target image includes: determining at least one rectangular region in the reconstructed image that includes the block boundary; Inputting the at least one rectangular area into the preset post-processing network model to obtain at least one processed rectangular area; The target image is obtained by replacing a corresponding local area including the block boundary in the reconstructed image with the at least one processed rectangular area.

15. The method according to claim 14, characterized in that After obtaining at least one processed rectangular area, the method further includes: Clipping the at least one processed rectangular area to obtain at least one target rectangular area; The target image is obtained by replacing a corresponding local area including the block boundary in the reconstructed image with the at least one target rectangular area.

16. An image processing method, characterized in that: Applied to an encoding device, the method comprises: Get the image to be processed; Dividing the image to be processed into blocks to obtain a plurality of image blocks; wherein the plurality of image blocks are equal in size and have no overlap; Encoding the multiple image blocks using a preset encoding network model to generate a bit stream; The code stream is transmitted to a decoding device, so that the decoding device executes the method according to any one of claims 9 to 15.

17. The method according to claim 16, characterized in that The method further comprises: Acquire a training set; wherein the training set includes at least one training image; Construct a coding network model and a decoding network model, and perform model training on the coding network model and the decoding network model based on the training set to obtain a preset coding network model and a preset decoding network model; wherein the preset decoding network model is used to instruct the decoding device to parse the bitstream to obtain multiple reconstructed blocks.

18. The method according to claim 17, characterized in that The performing model training on the encoding network model and the decoding network model based on the training set to obtain a preset encoding network model and a preset decoding network model includes: Based on the training set, performing model training on the encoding network model and the decoding network model using a preset algorithm; When the loss value corresponding to the cost function of the model training converges to a preset threshold, the encoding network model and the decoding network model obtained after training are determined as the preset encoding network model and the preset decoding network model.

19. An image processing device, characterized in that: The image processing device comprises: an acquisition unit, a determination unit, an enhancement unit and a processing unit; wherein, The acquisition unit is configured to acquire a plurality of reconstructed blocks; wherein the plurality of reconstructed blocks are obtained by dividing a plurality of image blocks into which the image to be processed is divided through a preset encoding network model and a preset decoding network model; The determining unit is configured to determine a central area of ​​at least one reconstructed block among the multiple reconstructed blocks; The enhancement unit is configured to perform quality enhancement on a central area of ​​the at least one reconstructed block to obtain at least one target reconstructed block; The processing unit is configured to generate a reconstructed image according to the at least one target reconstructed block and the multiple reconstructed blocks; and perform filtering processing on the block boundaries in the reconstructed image to obtain a target image.

20. An image processing device, characterized in that: The image processing device comprises: a first memory and a first processor; wherein, The first memory is used to store executable instructions that can be run on the first processor; The first processor is configured to execute the method according to any one of claims 1 to 8 when running the executable instructions.

21. A decoding device, characterized in that: The decoding device comprises: a receiving unit, a decoding unit and a post-processing unit; wherein, The receiving unit is configured to receive a code stream transmitted by an encoding device; wherein the code stream is obtained by dividing a plurality of image blocks into which an image to be processed is divided through a preset encoding network model; The decoding unit is configured to parse the bitstream using a preset decoding network model to obtain a plurality of reconstructed blocks; The post-processing unit is configured to use a preset post-processing network model to perform quality enhancement on a central area of ​​at least one of the multiple reconstructed blocks to obtain at least one target reconstructed block; and generate a reconstructed image based on the at least one target reconstructed block and the multiple reconstructed blocks, and use the preset post-processing network model to filter the block boundaries in the reconstructed image to obtain a target image.

22. A decoding device, characterized in that: The decoding device comprises: a second memory and a second processor; wherein, The second memory is used to store executable instructions that can be run on the second processor; The second processor is configured to execute the method according to any one of claims 9 to 15 when running the executable instructions.

23. A coding device, characterized in that: The encoding device comprises: an acquisition unit, a block unit, an encoding unit and a sending unit; wherein, The acquisition unit is configured to acquire the image to be processed; The blocking unit is configured to block the image to be processed to obtain a plurality of image blocks; wherein the plurality of image blocks are equal in size and have no overlap; The encoding unit is configured to encode the multiple image blocks using a preset encoding network model to generate a bit stream; The sending unit is configured to transmit the code stream to a decoding device so that the decoding device executes the method according to any one of claims 9 to 15.

24. An encoding device, characterized in that The encoding device comprises: a third memory and a third processor; wherein, The third memory is used to store executable instructions that can be run on the third processor; The third processor is configured to execute the method according to any one of claims 16 to 18 when running the executable instructions.

25. A computer storage medium, characterized in that The computer storage medium stores an image processing program, which implements the method according to any one of claims 1 to 8 when executed by a first processor, or implements the method according to any one of claims 9 to 15 when executed by a second processor, or implements the method according to any one of claims 16 to 18 when executed by a third processor.

26. A video system, characterized in that: The video system comprises: an encoding device and a decoding device; wherein, The encoding device is configured to obtain an image to be processed; and divide the image to be processed into blocks to obtain a plurality of image blocks; wherein the plurality of image blocks are of equal size and have no overlap; and encode the plurality of image blocks using a preset encoding network model to generate a code stream; and transmit the code stream to a decoding device; The decoding device is configured to receive a code stream transmitted by the encoding device; and parse the code stream using a preset decoding network model to obtain multiple reconstructed blocks; and use a preset post-processing network model to enhance the quality of a central area of ​​at least one of the multiple reconstructed blocks to obtain at least one target reconstructed block; and generate a reconstructed image based on the at least one target reconstructed block and the multiple reconstructed blocks, and use the preset post-processing network model to filter the block boundaries in the reconstructed image to obtain a target image.

Images