Image decoding method and device based on neural network, chip, storage medium and program product

Abstract

The invention discloses an image decoding method and device based on a neural network, a chip, a storage medium and a program product. The method comprises the following steps: acquiring a current input tensor, wherein the current input tensor is generated based on a quantization coefficient block in a compressed image; based on a preset decoding model and each target input tensor, respectively processing the current input tensor to respectively output reconstructed images corresponding to each target input tensor, the target input tensor being at least one of a plurality of preset input tensors; and outputting a reconstructed image set based on the reconstructed image corresponding to each target input tensor. The probability that the reconstructed image with less distortion exists in the reconstructed image set is high, so that the distortion probability of the reconstructed image is reduced. And a plurality of different reconstructed images can meet the differentiated image reconstruction requirements of the user, and the user experience is improved.

Description

Technical Field

[0001] The present application belongs to the field of image decoding technology, and in particular, relates to an image decoding method, an image decoding device, a chip, a computer-readable storage medium, and a computer program product. Background Art

[0002] In the JPEG (Joint Photographic Experts Group) image coding and decoding scheme, during the coding and compression process, the image is generally segmented first to obtain multiple image blocks, and then the image blocks are subjected to discrete cosine transform (DCT), and then normalized and quantized respectively. During the quantization process, effective image information is lost. The decoding and reconstruction process is the inverse process of coding and compression. The decoding and reconstruction cannot make up for the lost information, resulting in a high probability of distortion of the reconstructed image. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes an image decoding method based on a neural network, an image decoding device, a chip, a computer-readable storage medium and a computer program product, which can reduce the distortion probability of the reconstructed image.

[0004] In a first aspect, the present application provides an image decoding method, comprising obtaining a current input tensor, wherein the current input tensor is generated based on a quantization coefficient block in a compressed image; processing the current input tensor based on a preset decoding model and each target input tensor, respectively, to output a reconstructed image corresponding to each of the target input tensors, wherein the target input tensor is at least one of a plurality of preset input tensors; and outputting a reconstructed image set based on the reconstructed image corresponding to each of the target input tensors.

[0005] In some embodiments, the decoding model includes an N-layer network module, wherein the network module includes a first convolutional layer, a batch normalization layer, and a first activation layer, and the N-th network module is connected to a second convolutional layer and a second activation layer.

[0006] In some embodiments, the activation function of the first activation layer is a Relu activation function, and the activation function of the second activation layer is a Sigmoid activation function.

[0007] In some embodiments, the convolution kernel sizes of the first convolution layer and the second convolution layer are both 3*3.

[0008] In some embodiments, the processing of the current input tensors based on the preset decoding model and each target input tensor to output reconstructed images corresponding to each target input tensor includes:

[0009] Obtaining residual parameters of the second convolutional layer output when each of the target input tensors is input;

[0010] Based on the current input tensor, the preset matrix in the compressed image and the residual parameter, the reconstructed image corresponding to each target input tensor is output.

[0011] In some embodiments, the obtaining of the residual parameters output by the second convolutional layer when each of the target input tensors is input includes:

[0012] The target input tensor is concatenated with the input of each of the network modules so that the second convolutional layer outputs each of the target input tensors corresponding to the residual parameters.

[0013] In some embodiments, the target input tensor and the current input tensor have the same size.

[0014] In some embodiments, the method further includes obtaining the decoding model; the obtaining the decoding model includes:

[0015] Initializing the basic model for training based on a first loss function to generate an initial model, wherein the first loss function includes a first loss value, and the first loss value is determined based on an absolute value norm of an original image corresponding to the training image and a reconstructed image obtained through training;

[0016] Based on the second loss function, the initial decoding model is trained here to generate the decoding model;

[0017] Among them, the second loss function includes the second loss value, the third loss value and the fourth loss value, the second loss value is an adversarial loss, the third loss value is determined based on the pixel value of the reconstructed image output during the training process and the preset pixel range, and the fourth loss value is determined based on the original image corresponding to the training image and the absolute value norm of the reconstructed image obtained by training under each preset input tensor.

[0018] In some embodiments, the initial decoding model is trained based on the second loss function to generate the decoding model, including:

[0019] Obtaining the fourth loss value of the initial model during multiple iterations;

[0020] Determine a loss tensor that minimizes the fourth loss value; and

[0021] In a subsequent iteration process, the fourth loss value is calculated based on the loss tensor.

[0022] In some embodiments, the second loss function is as follows:

[0023] L=L Adv +λ Range L Range +λ Map L Map ; Where L is the total loss, L Adv is the second loss value, λ Range L Range is the third loss value, λ Map L Map is the fourth loss value.

[0024] 11. The image decoding method according to claim 10, characterized in that: Among them, k is determined based on the input tensor, To reconstruct the image, the clip function is used to limit The range of pixel values ​​in .

[0025] In some embodiments, L Map =min z ||ψ(x Q ,z)-x||1, where L Map In order to make ||ψ(x Q ,z)-x||1Minimum Z, min Z Indicates that for the variable Z, the expression that follows is to be minimized, X Q is the quantization parameter block of the training image, Z is the preset input tensor, X is the original image corresponding to the training image, ||ψ(x Q ,Z)-x||1 represents the absolute value norm of the reconstructed image and the original image obtained by decoding the training image under the preset input tensor.

[0026] In some embodiments, the method further includes randomly selecting a quality factor within a preset range; generating a normalized matrix based on the selected quality factor; and compressing the original image based on the normalized matrix to generate the training image.

[0027] In some embodiments, the normalized matrix is ​​generated based on the following formula:

[0028] M=50*Qbaseline / QF, where M is a normalization matrix, Qbaseline is a preset baseline quantization table, and QF is the quality factor.

[0029] In some embodiments, the preset range is [5, 49].

[0030] In some embodiments, the compressed image includes 8*8 quantization coefficient blocks, the size of the quantization coefficient blocks is m*n, and the current input tensor is an m*n*64 tensor obtained by arranging the 8*8 quantization coefficient blocks in sequence.

[0031] In a second aspect, the present application provides an image decoding device, which includes a model training module, a processing module, and an output module. The model training module is used to obtain a current input tensor, which is generated based on a quantization coefficient block in a compressed image; the processing module is used to process the current input tensor based on a preset decoding model and each target input tensor, so as to output a reconstructed image corresponding to each target input tensor, respectively, wherein the target input tensor is at least one of a plurality of preset input tensors; the output module is used to output a reconstructed image set based on the reconstructed image corresponding to each target input tensor.

[0032] In some embodiments, the decoding model includes an N-layer network module, the network module includes a first convolution layer, a batch normalization layer and a first activation layer, the N-th layer of the network module is connected to the second convolution layer and the second activation layer, and the processing module is specifically used to:

[0033] Obtaining residual parameters of the second convolutional layer output when each of the target input tensors is input;

[0034] Based on the current input tensor, the preset matrix in the compressed image and the residual parameter, the reconstructed image corresponding to each target input tensor is output.

[0035] In some embodiments, the processing module is further configured to:

[0036] The target input tensor is concatenated with the input of each of the network modules so that the second convolutional layer outputs each of the target input tensors corresponding to the residual parameters.

[0037] In some embodiments, the apparatus further comprises:

[0038] A model training module, used to obtain the decoding model;

[0039] The model training module is specifically used for:

[0040] Initializing the basic model for training based on a first loss function to generate an initial model, wherein the first loss function includes a first loss value, and the first loss value is determined based on an absolute value norm of an original image corresponding to the training image and a reconstructed image obtained through training;

[0041] Based on the second loss function, the initial decoding model is trained here to generate the decoding model;

[0042] Among them, the second loss function includes the second loss value, the third loss value and the fourth loss value, the second loss value is an adversarial loss, the third loss value is determined based on the pixel value of the reconstructed image output during the training process and the preset pixel range, and the fourth loss value is determined based on the original image corresponding to the training image and the absolute value norm of the reconstructed image obtained by training under each preset input tensor.

[0043] In some embodiments, the model training module is further used to:

[0044] Obtaining the fourth loss value of the initial model during multiple iterations;

[0045] Determine a loss tensor that minimizes the fourth loss value; and

[0046] In a subsequent iteration process, the fourth loss value is calculated based on the loss tensor.

[0047] In some embodiments, the apparatus further comprises a training image generation module: the training image generation module is used to:

[0048] A quality factor randomly selected within a preset range;

[0049] generating a normalized matrix based on the selected quality factor;

[0050] The original image is compressed based on the normalization matrix to generate the training image.

[0051] In a third aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned image decoding method when executed by a processor.

[0052] In a fourth aspect, the present application provides a chip, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned image decoding method when executing the program.

[0053] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the above-mentioned image decoding method when executed by a processor.

[0054] The image decoding method, image decoding device, chip, computer-readable storage medium and computer program product provided by the embodiments of the present application can adjust the decoding strategy of the reconstructed image through each target input tensor determined in multiple preset input tensors, so that the decoding model can decode and reconstruct the current compressed image based on different target input tensors, thereby obtaining an image set composed of different reconstructed images based on different decoding strategies. Compared with the conventional decoder decoding based on a fixed strategy to obtain a reconstructed image, the probability of distortion of the reconstructed image is relatively high, and the probability of the reconstructed image with less distortion in the reconstructed image set is relatively high, thereby reducing the distortion probability of the reconstructed image. In addition, multiple different reconstructed images can meet the differentiated image reconstruction needs of users and improve user experience.

[0055] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0057] Figure 1 It is a schematic diagram of the encoding principle of the image decoding method provided in an embodiment of the present application;

[0058] Figure 2 is a schematic diagram of an application scenario of the image decoding method provided in an embodiment of the present application;

[0059] Figure 3 It is a flowchart of an image decoding method provided by an embodiment of the present application;

[0060] Figure 4 It is a schematic diagram of the principle of the decoding model provided in the embodiment of the present application;

[0061] Figure 5 It is a flowchart of an image decoding method provided by an embodiment of the present application;

[0062] Figure 6 It is a flowchart of an image decoding method provided by an embodiment of the present application;

[0063] Figure 7 It is a flowchart of an image decoding method provided by an embodiment of the present application;

[0064] Figure 8 It is a flowchart of an image decoding method provided by an embodiment of the present application;

[0065] Fig. 9 It is a flowchart of an image decoding method provided by an embodiment of the present application;

[0066] Fig.10 is a flowchart of an image decoding method provided in an embodiment of the present application; and

[0067] Fig.11 It is a module schematic diagram of the image decoding device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0069] For ease of understanding, the technical background and application scenarios of this application are first introduced below:

[0070] See also Figure 1 JPEG (Joint Photographic Experts Group) is a widely used lossy compression standard, especially suitable for digital storage and transmission of color and grayscale images. This format reduces the file size by removing image information that is not easily perceived by the human eye, thereby achieving a high compression ratio.

[0071] The JPEG encoding compression process is as follows:

[0072] (1) Divide the image into 8*8 image blocks X of size m*n;

[0073] (2) DCT transformation: For each image block X, discrete cosine transform is performed to obtain XD = DCT(X);

[0074] (3) Normalization: The encoder divides each element in XD by a predefined matrix M∈Z 8×8

[0075] , XN=XD / M

[0076] (4) Quantization: Quantize the data of XN (e.g., round off) to obtain a quantization coefficient block XQ = round(XN)

[0077] (5) Storage: The individual quantized coefficient blocks {XQ} are stored together with M in the JPEG file using lossless processing, where the normalized matrix M contains the quantization intervals for each coefficient. These intervals are determined as a function of the scalar quantization factor QF parameter, but this function may vary in different JPEG implementations, so QF itself is ambiguous for extracting the image.

[0078] JPEG decoding and reconstruction is the inverse process of encoding compression, which will not be described in detail here.

[0079] See also Figure 2 , Figure 2 1 is an application scenario diagram of an image decoding method provided in an embodiment of the present application. The application scenario provided in the present application includes a terminal device 101 and a server 102, and the image decoding method provided in the present application can be executed by at least one of the terminal device 101 and the server 102.

[0080] Among them, the terminal devices may include but are not limited to: smart phones (such as Android phones, IOS phones, etc.), tablet computers, laptops, desktop computers, smart speakers, smart watches, portable personal computers, mobile Internet devices (Mobile Internet Devices, referred to as MID), intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, wearable devices, etc., and the embodiments of the present application do not limit this.

[0081] The terminal device may be integrated with a client, which may be a client having the function of displaying data information such as text, images, audio and video, including but not limited to an instant messaging client (such as a personal instant messaging client (i.e., social software) or an enterprise instant messaging client (i.e., collaborative office software), etc.), a social client (e.g., a content publishing platform client), a browser client, a cloud service control client, an entertainment client (e.g., a game client), a multimedia client (e.g., a video client), an information client (e.g., a news client), a shopping client, a car client, etc. The client may be an independent client or an embedded sub-client integrated in a client (e.g., a social client), which is not limited here.

[0082] The server can be an independent physical server or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, but the embodiments of the present application do not limit this.

[0083] It should be noted that Figure 2 The number of terminal devices and servers is only used as an example, and the number of terminal devices and servers can be more or less, which is not limited here. The terminal devices and servers can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0084] The image decoding method involved in this application can be implemented based on cloud technology.

[0085] Among them, cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data calculation, storage, processing and sharing.

[0086] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The backend services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the backend system for logical processing. Data of different levels will be processed separately. All kinds of industry data need strong system backing support, which can only be achieved through cloud computing.

[0087] The image decoding method of the present application can be implemented based on cloud computing. Cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides resources is called a "cloud". The resources in the "cloud" are infinitely expandable in the eyes of users, and can be obtained at any time, used on demand, expanded at any time, and paid for by use.

[0088] As a provider of basic cloud computing capabilities, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform) will be established, and various types of virtual resources will be deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices.

[0089] The image decoding method in the embodiment of the present application can be executed by a chip, and the chip can be installed in at least one of the server and the terminal device, that is, the method can be executed by the server or the terminal device alone, or by the server and the terminal device together. Therefore, the execution subject of each step will not be repeatedly described below.

[0090] Based on the above technical background and the introduction of related scenarios, an embodiment of the present application provides an image decoding method, which is described in detail below:

[0091] See also Figure 3An image decoding method provided in an embodiment of the present application is implemented by steps 011 to 013, which are described in detail below.

[0092] Step 011: Get the current input tensor, which is generated based on the quantization coefficient block in the compressed image.

[0093] Among them, a tensor can be a one-dimensional vector, a two-dimensional matrix, or a higher-dimensional data structure. A tensor can be regarded as an array defined in multiple dimensions, and its elements in any dimension can be operated according to a linear relationship.

[0094] After obtaining the compressed image, the decoding end can extract each quantized coefficient block from the compressed image to generate the current input tensor for subsequent decoding.

[0095] Optionally, the compressed image includes 8*8 quantization coefficient blocks, the size of the quantization coefficient blocks is m*n, and the current input tensor is an m*n*64 tensor obtained by arranging the 8*8 quantization coefficient blocks in sequence, where m and n are positive integers.

[0096] Step 012: Based on the preset decoding model and each target input tensor, the current input tensor is processed respectively to output the reconstructed image corresponding to each target input tensor respectively, and the target input tensor is at least one of multiple preset input tensors.

[0097] Among them, the decoding model can be a neural network model (such as a generative adversarial network model), which can be obtained through advance training.

[0098] The target input tensor may be one or more tensors selected from a plurality of preset input tensors.

[0099] Among them, the preset input tensor is a preset tensor used to restore different image details lost in the reconstructed image. When using different preset input tensors, the decoding model will perform decoding and reconstruction based on different decoding strategies, so that the reconstructed images after decoding and reconstruction are different.

[0100] Specifically, when generating a reconstructed image, the decoding model will first select one of the target input tensors, and then use the target input tensor and the current input tensor as inputs to the decoding model, thereby generating a reconstructed image corresponding to each target input tensor. Since each target input tensor corresponds to a decoding strategy, different reconstructed images can be generated.

[0101] According to the distortion of compressed images in different situations, different decoding strategies are used to achieve decoding reconstruction, and the probability of obtaining a reconstructed image without distortion or with a low degree of distortion is higher.

[0102] Optionally, the decoding model 10 is a generative adversarial network model (GANs), which includes a generator and a discriminator. The network structures of the generator and the discriminator are the same. After the training is completed, the decoding model 10 only needs to use the generator.

[0103] Optionally, a generative adversarial network model is conditioned

[0104] See also Figure 4 The generator includes an N-layer network module 11, the network module 11 includes a first convolutional layer, a batch normalization layer 112 and a first activation layer 113, and the N-th first convolutional layer 111 is connected to the second convolutional layer 12 and the second activation layer 13.

[0105] As a result, the receptive field of this network architecture is expanded 8 times in each axis compared to operating in the pixel domain, so it can utilize information at a larger scale.

[0106] Among them, the convolution layer is used to perform convolution operations. Batch Normalization Layer 112 (Batch Normalization Layer, referred to as BN layer) is a regularization technology widely used in deep learning networks. Its main purpose is to accelerate the network training process and improve the stability and generalization ability of the model. The activation function layer is a key component in the neural network. It is located at the output end of the neuron. Its main purpose is to introduce nonlinearity so that the neural network can learn and express complex nonlinear relationships. Without the activation function, the multi-layer neural network will degenerate into a linear model, and even if there are more hidden layers, it will not be able to learn nonlinear patterns. The activation function layer transforms the output value of the neuron, increases the expression ability of the network, and enables the network to fit a wider range of data distributions.

[0107] Optionally, N may be 10, that is, both the generator and the discriminator use 10 first convolutional layers 111, and all convolutional layers use 320 output channels except the last second convolutional layer 12. It can be understood that the more the first convolutional layers 111 have, the stronger the ability to express complex features. Setting 10 first convolutional layers 111 can improve the decoding accuracy of the decoding model 10 and enhance the decoding effect.

[0108] Optionally, the activation function of the first activation layer 113 is a Relu activation function, and the activation function of the second activation layer 13 is a Sigmoid activation function. In this way, the output of the decoding model 10 is ensured to be within a valid range by the offset Sigmoid function.

[0109] Optionally, the convolution kernel sizes of the first convolution layer and the second convolution layer 12 are both 3*3.

[0110] See also Figure 4 and Figure 5 Optionally, step 012 includes:

[0111] Step 0121: Obtain the residual parameters output by the second convolutional layer 12 when each target input tensor is input;

[0112] Step 0122: Based on the current input tensor, the preset matrix in the compressed image and the residual parameters, output the reconstructed image corresponding to each target input tensor.

[0113] Specifically, based on the JPEG encoding compression process mentioned in the technical background, it can be known that during decoding and reconstruction, the quantization parameter block XQ needs to be converted into SN first, and then converted into XD, and finally each image block X is obtained through an inverse DCT transform to generate a reconstructed image.

[0114] Among them, the current input tensor is composed of various quantization parameter blocks XQ, and the main image detail loss of coding compression lies in the quantization process of converting XN to XQ. Therefore, to achieve decoding reconstruction, it is only necessary to predict the residual parameter Δ required to reconstruct XQ to XN, that is, XN=XQ+Δ.

[0115] Therefore, the decoding model 10 can first obtain each target input tensor, and the current input tensor is processed by the multi-layer first convolution layer 111, and finally the second convolution layer 12 outputs the residual parameter Δ. At this time, based on the residual parameter Δ and the current input tensor, XN can be obtained, and XN=XD / M, where M is the matrix used in the normalization process, which can be extracted from the compressed image. When XN is determined, XD can be quickly determined, and then XD is inversely DCT-transformed to obtain each image block X, thereby generating a reconstructed image.

[0116] Each target input tensor can obtain a corresponding residual parameter Δ, so decoding and reconstruction can obtain multiple reconstructed images, that is, the reconstructed image corresponding to each target input tensor.

[0117] See also Figure 6 Optionally, step 0121 includes:

[0118] Step 01211: Concatenate the target input tensor with the input of each first convolutional layer 111 so that the second convolutional layer 12 outputs the residual parameters corresponding to each target input tensor.

[0119] Specifically, see Figure 4When the target input tensor is input, the target input tensor Z is concatenated with the input of each network model. That is to say, the input of the first convolutional layer 111 includes the current input tensor and the target input tensor Z. In addition to the output of the previous convolutional layer 111, each first convolutional layer 111 also includes the target input tensor Z.

[0120] In this way, the target input tensor is used to influence the input of each first convolutional layer 111, so that the decoding model 10 can generate different residual parameters based on different generation strategies for the same compressed image to achieve reconstruction of different decoded images. Moreover, by splicing the target input tensor at the input of each first convolutional layer 111, the training direction can be adjusted, which can speed up the convergence of the model compared to random model training.

[0121] Optionally, the target input tensor and the current input tensor have the same size. In this way, the target input tensor and the current input tensor can be normally spliced, and the target input tensor and the input of each first convolutional layer 111 can be normally spliced.

[0122] Optionally, the number of target input tensors can be set based on user needs. For example, if the preset input tensors are 9, but the user does not want to decode too many images each time, the number of target input tensors can be customized to 3. In this case, 3 can be selected from multiple preset input tensors as target input tensors.

[0123] Optionally, when the multiple reconstructed images generated based on the set multiple target input tensors do not meet the user's needs, the user can choose to redefine the target input tensor, thereby selecting multiple target input tensors from the unselected tensors of the multiple preset input tensors, and regenerating the corresponding reconstructed images based on the redetermined target input tensor. In this way, while reducing the amount of calculation for generating a reconstructed image set each time and meeting the user's customized needs, a reconstructed image that meets the user's needs is generated as much as possible.

[0124] Step 013: Based on the reconstructed images corresponding to each target input tensor, output a reconstructed image set.

[0125] Specifically, the decoding model eventually combines the reconstructed images corresponding to each target input tensor into a reconstructed image set. The user can directly view the reconstructed image set and select one or more reconstructed images that best meet the needs from the reconstructed image set.

[0126] The image decoding method of the embodiment of the present application can adjust the decoding strategy of the reconstructed image through each target input tensor determined in multiple preset input tensors, so that the decoding model can decode and reconstruct the current compressed image based on different target input tensors, thereby obtaining an image set composed of different reconstructed images based on different decoding strategies. Compared with the conventional decoder decoding based on a fixed strategy to obtain a reconstructed image, the probability of distortion of the reconstructed image is higher, and the probability of the reconstructed image with less distortion in the reconstructed image set is higher, thereby reducing the distortion probability of the reconstructed image. In addition, multiple different reconstructed images can meet the differentiated image reconstruction needs of users and improve user experience.

[0127] See also Figure 7 and Figure 8 In some embodiments, the image decoding method further includes step 013, which is described in detail below.

[0128] Step 014: Obtain the decoding model.

[0129] Optionally, step 014 includes:

[0130] Step 0141: Based on a first loss function, the basic model is initialized and trained to generate an initial model, wherein the first loss function includes a first loss value, and the first loss value is determined based on the absolute value norm of the original image corresponding to the training image and the reconstructed image obtained through training;

[0131] Step 0142: Based on the second loss function, the initial decoding model is trained to generate a decoding model;

[0132] Among them, the basic model and the initial model are generative adversarial training models. During the training process, both the generator and the discriminator need to be used at the same time.

[0133] Among them, the second loss function includes a second loss value, a third loss value and a fourth loss value. The second loss value is an adversarial loss. The third loss value is determined based on the pixel value of the reconstructed image output during the training process and a preset pixel range. The fourth loss value is determined based on the minimum value of the absolute value norm of the original image corresponding to the training image and the reconstructed image obtained by training under each preset input tensor.

[0134] Among them, the absolute value norm is also called Manhattan distance or L1 norm, which is the sum of the absolute values ​​of vector elements in mathematics.

[0135] Among them, the adversarial loss is the Wasserstein GAN loss. GAN (Generative Adversarial Network) loss consists of two parts: the loss of the generator and the loss of the discriminator. The Wasserstein GAN loss is based on the GAN loss and introduces the Wasserstein distance (also known as Earth Mover's Distance (EMD), which is a method of measuring the distance between two probability distributions) as a new loss function. The Wasserstein GAN loss encourages the reconstructed image to follow the statistical properties of its natural image counterpart.

[0136] Specifically, in order to ensure the decoding performance of the decoding model, the decoding model needs to be trained.

[0137] The training process is divided into two steps:

[0138] (1) Initialization step: The initialization step is used to roughly train the basic model. At this time, the basic model will be initialized and trained based on the first loss function to generate an initial model. The first loss function only includes a first loss value that focuses on the difference between the reconstructed image and the original image. The initialization step only needs to train until the first loss value is minimized. After the initialization step, the initial model of preliminary training can be obtained.

[0139] (2) Training step: In the training step, the loss function of the initial model is modified to a second loss function. The second loss function not only focuses on adversarial loss and encourages the reconstructed image to follow the statistical characteristics of its natural image counterpart, but also focuses on the abnormal loss of pixel values ​​of the reconstructed image, as well as the fourth loss value related to the preset input tensor, the original image and the reconstructed image, to ensure that the absolute value norm between the reconstructed image and the original image after training of each preset input tensor is small, and to ensure the decoding and reconstruction effect of each preset input tensor. In this way, the decoding model can be obtained by training with the second loss function of the training step, specifically, the trained generator is used as the decoding model.

[0140] Optionally, the second loss function is as follows:

[0141] L=L Adv +λ Range L Range +λ Map L Map ; Where L is the total loss, L Adv is the second loss value, λ Range L Range is the third loss value, λ Map L Map is the fourth loss value.

[0142] Optionally, Among them, k is determined based on the input tensor, To reconstruct the image, the clip function is used to limit The range of pixel values ​​in .

[0143] Among them, [16, 235] is the valid range of brightness values ​​in the Y-Cb-Cr color space.

[0144] This helps prevent the model from diverging by penalizing pixel values ​​outside the range [16, 235].

[0145] Optionally, L Map =min Z ||ψ(x Q ,Z)-x||1, where L Map In order to make ||ψ(x Q ,Z)-x||1Minimum Z,min Z Indicates that for the variable Z, the expression that follows is to be minimized, X Q is the quantization parameter block of the training image, Z is the preset input tensor, X is the original image corresponding to the training image, ||ψ(x Q ,Z)-x||1 represents the absolute value norm of the reconstructed image and the original image obtained by decoding the training image under the preset input tensor.

[0146] L Map The main role of is to prevent the decoding model from ignoring its preset input tensor Z, and to ensure that our network can use the preset input tensor to generate each real natural image corresponding to the training image of the training set.

[0147] See also Fig. 9 Optionally, step 0142 includes:

[0148] Step 01421: Obtain the fourth loss value of the initial model during multiple iterations;

[0149] Step 01422: Determine the loss tensor that minimizes the fourth loss value; and

[0150] Step 01423: In a subsequent iteration process, a fourth loss value is calculated based on the loss tensor.

[0151] Specifically, during training, we first obtain the fourth loss value corresponding to multiple (such as 10, 15, etc.) selection generations (i.e., model parameter iterations), and then based on the L of multiple iterations map To calculate the loss tensor Z*, Z* = argmin{L map}, that is, determine the preset input tensor that minimizes the fourth loss value of multiple iterations as the loss tensor. Then, use this fixed Z* to minimize the loss in the second loss function, that is, first randomly determine one from multiple preset input tensors Z as input, and after 10 generations, replace Z of the fourth loss value with Z*.

[0152] Optionally, during the training of the decoding model, the Adam optimizer is used to implement iterations.

[0153] The Adam (Adaptive Moment Estimation) optimizer is an efficient gradient descent algorithm in deep learning and machine learning. It combines the adaptive learning rate advantage of the Adagrad algorithm and the use of the exponentially decaying average of the squared gradient by the RMSprop algorithm. It aims to provide better training results, especially when dealing with complex models with a large number of parameters.

[0154] See also Fig.10 In some embodiments, the image decoding method further includes:

[0155] Step 015: a quality factor randomly selected within a preset range;

[0156] Step 016: Generate a normalized matrix based on the selected quality factor;

[0157] Step 017: Compress the original image based on the normalized matrix to generate a training image.

[0158] Specifically, during the training process, in order to ignore the influence of the quality factor, a quality factor can be randomly selected within a preset range, and then a normalized matrix is ​​generated. Finally, the original image is encoded and compressed based on the normalized matrix to obtain training images corresponding to different quality factors.

[0159] Optionally, the normalized matrix is ​​generated based on the following formula:

[0160] M=50*Qbaseline / QF, where M is a normalization matrix, Qbaseline is a preset baseline quantization table, and QF is a quality factor.

[0161] Optionally, the preset range is [5, 49]. By setting a larger preset range, the influence of different quality factors on decoding reconstruction can be avoided as much as possible.

[0162] According to the method described in the above embodiment, the present application embodiment also provides an image decoding device 300, which is used to perform the steps in the above image decoding method. Fig.11 , Fig.113 is a schematic diagram of the structure of an image decoding device 300 provided in an embodiment of the present application. The image decoding device 300 includes an acquisition module 301, a processing module 302 and an output module 303, wherein:

[0163] An acquisition module 301 is used to acquire a current input tensor, where the current input tensor is generated based on a quantization coefficient block in a compressed image;

[0164] A processing module 302 is used to process the current input tensor based on a preset decoding model and each target input tensor, so as to output a reconstructed image corresponding to each target input tensor, wherein the target input tensor is at least one of a plurality of preset input tensors;

[0165] The output module 303 is used to output a reconstructed image set based on the reconstructed images corresponding to each target input tensor.

[0166] In some embodiments, the processing module 302 is specifically configured to:

[0167] Get the residual parameters of the second convolutional layer output when each target input tensor is input;

[0168] Based on the current input tensor, the preset matrix in the compressed image and the residual parameters, the reconstructed image corresponding to each target input tensor is output.

[0169] In some embodiments, the processing module 302 is further configured to:

[0170] The target input tensor is concatenated with the input of each network module so that the second convolutional layer outputs the residual parameters corresponding to each target input tensor.

[0171] In some embodiments, the image decoding apparatus 300 further includes:

[0172] A model training module 304 is used to obtain a decoding model;

[0173] The model training module 304 is specifically used for:

[0174] Initializing the basic model for training based on a first loss function to generate an initial model, the first loss function includes a first loss value, and the first loss value is determined based on an absolute value norm of an original image corresponding to the training image and a reconstructed image obtained through training;

[0175] Based on the second loss function, the initial decoding model is trained here to generate a decoding model;

[0176] Among them, the second loss function includes a second loss value, a third loss value and a fourth loss value. The second loss value is an adversarial loss. The third loss value is determined based on the pixel value of the reconstructed image output during the training process and a preset pixel range. The fourth loss value is determined based on the original image corresponding to the training image and the absolute value norm of the reconstructed image obtained by training under each preset input tensor.

[0177] In some embodiments, the model training module 304 is further used to:

[0178] Get the fourth loss value of the initial model at multiple iterations;

[0179] Determine the loss tensor that minimizes the fourth loss value; and

[0180] In subsequent iterations, a fourth loss value is calculated based on the loss tensor.

[0181] In some embodiments, the image decoding apparatus 300 further includes a training image generation module 305: the training image generation module 305 is used to:

[0182] A quality factor randomly selected within a preset range;

[0183] generating a normalized matrix based on the selected quality factor;

[0184] The original image is compressed based on the normalized matrix to generate the training image.

[0185] It should be noted that the specific details of each module unit in the above-mentioned image decoding device 300 have been described in detail in the embodiment of the above-mentioned image decoding method, and will not be repeated here.

[0186] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0187] In some embodiments, the image decoding device in the embodiments of the present application can be implemented in hardware, such as a chip, or a component in a chip, such as an integrated circuit or a chip; the image decoding device can also be implemented in software, such as as an application installed in a chip.

[0188] In some embodiments, the chip of the embodiment of the present application can execute each process of the embodiment of the above-mentioned image decoding method and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0189] Among them, chips, also known as integrated circuits (ICs), microcircuits or microchips, are the core components of modern electronic devices. They are made of semiconductor materials (usually silicon) and integrate millions or even billions of microelectronic components (such as transistors, resistors, capacitors, etc.) through a series of complex manufacturing processes to form an electronic circuit with specific functions. These components and their interconnections are manufactured on the same small silicon wafer. They are small in size, light in weight, but powerful in function. They are the basis of various electronic products such as computers, smart phones, home appliances, automobiles, medical equipment, etc.

[0190] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the embodiment of the above-mentioned image decoding method are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0191] The processor may be the chip in the above embodiment. The computer readable storage medium may be a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0192] Computer readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory 504 and mass storage device 507 can be collectively referred to as memory.

[0193] The embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned image decoding method when executed by a processor. The processor may be a chip in the above-mentioned embodiment. When the computer program is executed by the processor, each process of the embodiment of the above-mentioned image decoding method is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0194] It is understandable that in the specific implementation of this application, data related to user identity or characteristics is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0195] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A neural network-based image decoding method, characterized in that: include: Obtaining a current input tensor, wherein the current input tensor is generated based on a quantized coefficient block in a compressed image; Based on a preset decoding model and each target input tensor, the current input tensor is processed respectively to output a reconstructed image corresponding to each target input tensor respectively, wherein the target input tensor is at least one of a plurality of preset input tensors; Based on the reconstructed images corresponding to the target input tensors, a reconstructed image set is output.

2. The image decoding method according to claim 1, characterized in that: The decoding model includes N layers of network modules, each of which includes a first convolutional layer, a batch normalization layer, and a first activation layer, and the Nth layer of the network module is connected to a second convolutional layer and a second activation layer.

3. The image decoding method according to claim 2, characterized in that: The activation function of the first activation layer is the Relu activation function, and the activation function of the second activation layer is the Sigmoid activation function.

4. The image decoding method according to claim 2, characterized in that: The convolution kernel sizes of the first convolution layer and the second convolution layer are both 3*3.

5. The image decoding method according to any one of claims 2 to 4, characterized in that: The processing of the current input tensors based on the preset decoding model and each target input tensor to output reconstructed images corresponding to each target input tensor includes: Obtaining residual parameters of the second convolutional layer output when each of the target input tensors is input; Based on the current input tensor, the preset matrix in the compressed image and the residual parameter, the reconstructed image corresponding to each target input tensor is output.

6. The image decoding method according to claim 5, characterized in that: The obtaining of the residual parameters output by the second convolutional layer when each of the target input tensors is input includes: The target input tensor is concatenated with the input of each of the network modules so that the second convolutional layer outputs each of the target input tensors corresponding to the residual parameters.

7. The image decoding method according to claim 5 or 6, characterized in that: The target input tensor has the same size as the current input tensor.

8. The image decoding method according to any one of claims 1 to 7, characterized in that: Also includes: Acquire the decoding model; The obtaining of the decoding model comprises: Initializing the basic model for training based on a first loss function to generate an initial model, wherein the first loss function includes a first loss value, and the first loss value is determined based on an absolute value norm of an original image corresponding to the training image and a reconstructed image obtained through training; Based on the second loss function, the initial decoding model is trained here to generate the decoding model; Among them, the second loss function includes the second loss value, the third loss value and the fourth loss value, the second loss value is an adversarial loss, the third loss value is determined based on the pixel value of the reconstructed image output during the training process and the preset pixel range, and the fourth loss value is determined based on the original image corresponding to the training image and the absolute value norm of the reconstructed image obtained by training under each preset input tensor.

9. The image decoding method according to claim 8, characterized in that: The initial decoding model is trained based on the second loss function to generate the decoding model, including: Obtaining the fourth loss value of the initial model during multiple iterations; Determine a loss tensor that minimizes the fourth loss value; and In a subsequent iteration process, the fourth loss value is calculated based on the loss tensor.

10. The image decoding method according to claim 8, characterized in that: The second loss function is as follows: L=L Adv +λ Range L Range +λ Map L Map ; Where L is the total loss, L Adv is the second loss value, λ Range L Range is the third loss value, λ Map L Map is the fourth loss value.

11. The image decoding method according to claim 10, characterized in that: Among them, k is determined based on the input tensor, To reconstruct the image, the clip function is used to limit The range of pixel values ​​in .

12. The image decoding method according to claim 10, characterized in that: L Map =min z ||ψ(x Q ,z)-x||1, where L Map In order to make ||ψ(x Q ,z)-x||1Minimum Z, min Z Indicates that for the variable Z, the expression that follows is to be minimized, X Q is the quantization parameter block of the training image, Z is the preset input tensor, X is the original image corresponding to the training image, ||ψ(x Q ,Z)-x||1 represents the absolute value norm of the reconstructed image and the original image obtained by decoding the training image under the preset input tensor.

13. The image decoding method according to claim 8, characterized in that: Also includes: A quality factor randomly selected within a preset range; generating a normalized matrix based on the selected quality factor; The original image is compressed based on the normalization matrix to generate the training image.

14. The image decoding method according to claim 13, characterized in that: The normalized matrix is ​​generated based on the following formula: M=50*Qbaseline / QF, where M is a normalization matrix, Qbaseline is a preset baseline quantization table, and QF is the quality factor.

15. The image decoding method according to claim 13, characterized in that: The preset range is [5, 49].

16. The image decoding method according to claim 1, characterized in that: The compressed image includes 8*8 quantization coefficient blocks, the size of the quantization coefficient blocks is m*n, and the current input tensor is an m*n*64 tensor obtained by arranging the 8*8 quantization coefficient blocks in sequence.

17. An image decoding device, characterized in that: include: An acquisition module, used for acquiring a current input tensor, where the current input tensor is generated based on a quantization coefficient block in a compressed image; A processing module, configured to process the current input tensor based on a preset decoding model and each target input tensor, so as to output a reconstructed image corresponding to each target input tensor, wherein the target input tensor is at least one of a plurality of preset input tensors; An output module is used to output a reconstructed image set based on the reconstructed images corresponding to each of the target input tensors.

18. The image decoding device according to claim 17, characterized in that: The decoding model includes N layers of network modules, the network modules include a first convolution layer, a batch normalization layer and a first activation layer, the Nth layer of the network module is connected to a second convolution layer and a second activation layer, and the processing module is specifically used for: Obtaining residual parameters of the second convolutional layer output when each of the target input tensors is input; Based on the current input tensor, the preset matrix in the compressed image and the residual parameter, the reconstructed image corresponding to each target input tensor is output.

19. The image decoding device according to claim 18, characterized in that: The processing module is further specifically used for: The target input tensor is concatenated with the input of each of the network modules so that the second convolutional layer outputs each of the target input tensors corresponding to the residual parameters.

20. The image decoding device according to any one of claims 17 to 19, characterized in that: The device also includes: A model training module, used to obtain the decoding model; The model training module is specifically used for: Initializing the basic model for training based on a first loss function to generate an initial model, wherein the first loss function includes a first loss value, and the first loss value is determined based on an absolute value norm of an original image corresponding to the training image and a reconstructed image obtained through training; Based on the second loss function, the initial decoding model is trained here to generate the decoding model; Among them, the second loss function includes the second loss value, the third loss value and the fourth loss value, the second loss value is an adversarial loss, the third loss value is determined based on the pixel value of the reconstructed image output during the training process and the preset pixel range, and the fourth loss value is determined based on the original image corresponding to the training image and the absolute value norm of the reconstructed image obtained by training under each preset input tensor.

21. The image decoding device according to claim 20, characterized in that: The model training module is also specifically used for: Obtaining the fourth loss value of the initial model during multiple iterations; Determine a loss tensor that minimizes the fourth loss value; and In a subsequent iteration process, the fourth loss value is calculated based on the loss tensor.

22. The image decoding device according to claim 20, characterized in that: The device further comprises a training image generation module: the training image generation module is used to: A quality factor randomly selected within a preset range; generating a normalized matrix based on the selected quality factor; The original image is compressed based on the normalization matrix to generate the training image.

23. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image decoding method according to any one of claims 1 to 16 is implemented.

24. A chip, characterized in that: The chip is used to implement the image decoding method as described in any one of claims 1-16.

25. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, implements the image decoding method according to any one of claims 1 to 16.

Images