Image encoding device, probability model generation device, and image compression system

Abstract

To provide an image encoding method and apparatus, and an image compression system, capable of reducing time for decoding.SOLUTION: An image encoding device includes: a first feature extraction unit configured to perform feature extraction on an input image to obtain feature maps of N channels; a weighting unit configured to assign a weight to a feature map of each channel; and a second feature extraction unit configured to perform dimension reduction processing on the feature maps processed by the weighting unit and obtain and output feature maps of M channels, where M is smaller than N.EFFECT: Hence, a corresponding importance may be obtained by multiplying different feature maps by a weight and dimension reduction processing is performed by the second feature extraction unit on the feature maps processed by the weighting unit.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention relates to the technical fields of image compression and deep learning. [Background technology]

[0002] In recent years, deep learning has dominated the field of computer vision, and deep learning has become an important technology for image research, whether in image recognition or super-resolution reconstruction, but its capabilities are not limited to these tasks. Currently, deep learning techniques have been introduced into the field of image compression, where they have gradually shown great potential and become a hot research field. (For example, see Patent Document 1 (International Publication No. 2016 / 127271) and Patent Document 2 (European Patent Application Publication No. 3432263).)

[0003] It should be noted that the above description of the technical background is for the purpose of making the technical solutions of the present invention clear and complete, and is described for the purpose of making those skilled in the art understand. These technical solutions are merely described as part of the background technology of the present invention, and are not well known to those skilled in the art. Summary of the Invention [Problem to be solved by the invention]

[0004] Embodiments of the present invention provide an image compression method that reduces decoding time by reducing the bottleneck of latent variables, and reduces codestream requirements by using an independent entropy model to achieve accurate prediction of probability distributions. [Means for solving the problem]

[0005] In a first aspect of an embodiment of the present invention, there is provided an image encoding device including: a first feature extraction unit that performs feature extraction on an input image to obtain feature maps for N channels; a weighting unit that assigns weights to the feature maps of each channel; and a second feature extraction unit that performs dimensionality reduction processing on the feature maps processed by the weighting unit to obtain and output feature maps for M channels, where M is smaller than N.

[0006] In a second aspect of an embodiment of the present invention, there is provided a probability model generation device including: a hyper decoder that performs hyper decoding on a code stream received from a hyper encoder to obtain auxiliary information; a context model processing unit that receives an output of an encoder as an input and obtains a content-based prediction; and an entropy model processing unit that combines the output of the context model processing unit and the output of the hyper decoder to obtain a predicted probability model and provides the predicted probability model to the encoder and the decoder, wherein the context model processing unit includes a first processing unit that obtains an average value portion of a prediction result of the content-based prediction based on the output of the encoder; and a second processing unit that obtains a variance portion of a prediction result of the content-based prediction based on the output of the encoder, and the entropy model processing unit includes a third processing unit that combines the average value portion of the prediction result obtained by the first processing unit with auxiliary information output by the hyper decoder to obtain an average value portion of the predicted probability model; and a fourth processing unit that combines the variance portion of the prediction result obtained by the second processing unit with the auxiliary information output by the hyper decoder to obtain the variance portion of the predicted probability model.

[0007] In a third aspect of an embodiment of the present invention, there is provided an image compression system including: an image encoding device that downsamples an input image and converts the input image into a latent representation; a probability model generation device that predicts a probability distribution of the latent representation and obtains a probability model of the latent representation; and an image decoding device that upsamples the latent representation obtained by decoding based on the probability model and maps the latent representation back to the input image, wherein the image encoding device includes the device described in the first aspect above, or the probability model generation device includes the device described in the second aspect above.

[0008] In a fourth aspect of an embodiment of the present invention, there is provided an image encoding method including the steps of: performing feature extraction on an input image to obtain feature maps for N channels; assigning weights to the feature maps of each channel; and performing a dimensionality reduction process on the feature maps of the N channels to which the weights have been assigned, to obtain and output feature maps for M channels, where M is smaller than N.

[0009] In a fifth aspect of an embodiment of the present invention, a method for generating a probability model is provided, comprising the steps of: using a hyper-decoder to decode a code stream received from an encoder to obtain auxiliary information; using a context model to input an output of the encoder and obtain a content-based prediction; and using an entropy model to combine the output of the context model and the output of the hyper-decoder to obtain a predicted probability model and provide it to the encoder and decoder, wherein the entropy model combines an average part of the prediction result obtained by the context model with the auxiliary information output by the hyper-decoder to obtain an average part of the probability model, and combines a variance part of the prediction result obtained by the context model with the auxiliary information output by the hyper-decoder to obtain a variance part of the probability model.

[0010] In another aspect of an embodiment of the present invention, there is provided a computer readable program which, when executed in an image processing device, causes the image processing device to perform a method according to the fourth and / or fifth aspects described above.

[0011] In another aspect of an embodiment of the present invention, there is provided a storage medium having a computer readable program stored thereon, the computer readable program causing an image processing device to perform a method according to the fourth and / or fifth aspects above.

[0012] The advantageous effects of the embodiments of the present invention are as follows: According to at least one aspect of the embodiments of the present invention, in image compression, the bottleneck of latent variables can be reduced to shorten the decoding time, i.e., the weighting unit multiplies different feature maps with one weight to obtain corresponding importance, and the second feature extraction unit performs dimensional reduction on the feature maps processed by the weighting unit to shorten the decoding time. Also, according to at least one aspect of the embodiments of the present invention, the code stream requirement can be reduced by using independent entropy models to realize accurate prediction of probability distribution, i.e., two independent context model processing units and entropy model processing units obtain two parameters mu and sigma of the probability model, so that the code stream required for encoding can be reduced by a more accurate entropy model.

[0013] Certain embodiments of the invention are disclosed in detail, as set forth in the following description and in the drawings, which show how the principles of the invention may be employed, but are not intended to be limited in scope, and all such modifications, alterations, and equivalents are within the spirit and content of the appended claims.

[0014] Features described and / or shown in one embodiment may be used in the same or similar manner in one or more other embodiments, may be combined with features in the other embodiments, or may substitute for features in the other embodiments.

[0015] It should be noted that the term "comprising" when used in the present text means the presence of a feature, element, step or component and does not exclude the presence or addition of one or more other features, elements, steps or components. [Brief description of the drawings]

[0016] Elements and features illustrated in any drawing or embodiment of an embodiment of the invention may be combined with elements and features shown in one or more other drawings or embodiments. In the drawings, like reference numerals represent corresponding components in multiple drawings and may represent corresponding components used in multiple aspects.

[0017] The drawings included herein are for understanding the embodiments of the present invention, constitute a part of this specification, and are for illustrating the embodiments of the present invention, and together with the description of the text, explain the principles of the present invention. Note that the drawings described herein are merely for illustrating the embodiments of the present invention, and those skilled in the art can easily obtain other drawings based on these drawings. [Figure 1] 1 is a schematic diagram of an image compression system according to a first embodiment. [Diagram 2] FIG. 11 is a schematic diagram of an image encoding device according to a second embodiment. [Diagram 3] 3 is a schematic diagram of a network structure of one embodiment of a first feature extraction unit of the image encoding device shown in FIG. 2. [Figure 4] 3 is a schematic diagram of a weighting unit of the image encoding device shown in FIG. 2. [Diagram 5] FIG. 5 is a schematic diagram of a network structure corresponding to the weighting unit shown in FIG. 4. [Figure 6] FIG. 11 is a schematic diagram of a probabilistic model generating device according to a third embodiment. [Figure 7] FIG. 2 is a schematic diagram of a network structure of an embodiment of an image compression system according to the first embodiment. [Figure 8] FIG. 13 is a schematic diagram of an image encoding method according to a fourth embodiment. [Figure 9] FIG. 13 is a schematic diagram of a probabilistic model generation method according to a fifth embodiment. [Figure 10] FIG. 13 is a schematic diagram of an image processing device according to a sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] The above and other features of the present invention will become apparent from the drawings and the following description. The specification and drawings disclose specific embodiments of the present invention, i.e., some embodiments according to the principles of the present invention. It is to be understood that the present invention is not limited to the described embodiments, and the present invention includes all modifications, variations, and equivalents within the scope of the appended claims.

[0019] In the embodiments of the present invention, the terms "first" and "second" are used to distinguish different elements by name and do not imply the spatial arrangement or temporal order of these elements, and the elements are not limited to these terms. The term "and / or" includes any one or more of the listed terms and combinations thereof. The terms "including," "including," and "having" refer to the presence of stated features, elements, components or members but do not exclude the presence or addition of one or more other features, elements, components or members.

[0020] In the present embodiment, the singular forms "a", "the", "the" and the like include the plural, meaning "a kind" or "a class", and are not limited to "one". Additionally, the term "said" includes both the singular and the plural, unless the context clearly dictates otherwise. Additionally, the term "according to" means "at least in part depending on" and the term "based on" means "at least in part based on", unless the context clearly dictates otherwise.

[0021] Hereinafter, various aspects of the embodiment of the present invention will be described with reference to the drawings. These aspects are merely illustrative and do not limit the present invention.

[0022] <Example 1> An embodiment of the present invention provides an image compression system. FIG. 1 is a schematic diagram of an image compression system of the embodiment 1. As shown in FIG. 1, the image compression system 100 of the embodiment of the present invention includes an image encoding device 101, a probability model generating device 102, and an image decoding device 103. The image encoding device 101 downsamples an input image and converts the input image into a latent representation. The probability model generating device 102 predicts a probability distribution of the latent representation and obtains a probability model of the latent representation. The image decoding device 103 upsamples the latent representation obtained by decoding based on the probability model, and maps the latent representation back to the input image.

[0023] In an embodiment of the present invention, as shown in Fig. 1, the image compression system 100 may further include an arithmetic encoder 104 and an arithmetic decoder 105. The arithmetic encoder 104 encodes the output of the image encoding device 101 based on the probability model generated by the probability model generating device 102. The arithmetic decoder 105 decodes the received codestream based on the probability model generated by the probability model generating device 102, and provides it to the image decoding device 103.

[0024] In an embodiment of the present invention, an image encoding device 101 converts an input image (which in this embodiment of the present invention are the pixels of the input image) into a latent representation that can reduce the dimensional space (i.e., has reduced dimensions), an image decoding device 103 maps the latent representation back to the pixels using an approximate inverse function, and a probabilistic model generating device 102 predicts a probability distribution of the latent representation using an entropy model to obtain a probabilistic model of the latent representation.

[0025] FIG. 2 is a schematic diagram of an image encoding device 101 according to a second embodiment of the present invention. As shown in FIG. 2, the image encoding device 101 according to the embodiment of the present invention includes a first feature extraction unit 201, a weighting unit 202, and a second feature extraction unit 203. The first feature extraction unit 201 performs feature extraction on an input image to obtain feature maps of N channels. The weighting unit 202 assigns weights to the feature maps of each channel. The second feature extraction unit 203 performs dimensionality reduction processing on the feature maps processed by the weighting unit 202 (i.e., each weighted feature map), and obtains and outputs feature maps of M channels. Here, M is smaller than N.

[0026] In an embodiment of the present invention, the first feature extraction unit 201 may perform feature extraction on the input image using multiple convolution layers (the convolution layers may be referred to as filters). FIG. 3 is a schematic diagram of a network structure of one embodiment of the first feature extraction unit 201. As shown in FIG. 3, in this example, feature extraction is performed on the input image by multiple convolution layers and one connection layer to obtain feature maps of N channels. FIG. 3 shows only one example, and the embodiment of the present invention is not limited to the network structure of the first feature extraction unit 201, and for example, more convolution layers may be added or the number of convolution layers may be reduced to perform feature extraction on the input image.

[0027] In an embodiment of the present invention, the weighting unit 202 may use one weighting layer to assign weights to the feature maps of each of the N channels so as to enhance useful features and suppress less useful features.

[0028] In the embodiment of the present invention, the second feature extraction unit 203 may perform a dimensionality reduction process on the feature maps of the N channels processed by the weighting unit 202 through one convolution layer to obtain feature maps of M channels. The convolution layer may be an M×1×1 convolution layer, where M is the number of channels and 1×1 is the kernel of the convolution layer (also called the convolution kernel). The convolution layer can achieve the purpose of dimensionality reduction on the feature maps of the N channels. In addition, the operating principle of the dimensionality reduction process may refer to the prior art, and the description thereof will be omitted here.

[0029] In the embodiment of the present invention, since the entropy model is very important for image compression, as part of the input of the entropy model, the context model can effectively improve the prediction accuracy by using the previous pixel information of the current pixel. However, since the context model is an autoregressive network, it is necessary to encode the latent representation for each pixel, and when the bottleneck of the latent representation becomes large, the encoding time will increase significantly. The embodiment of the present invention adds one weighting layer (the weighting layer may be regarded as the last layer selection of the encoder part) to assign weights to different channels, effectively enhancing useful features and suppressing less useful features, and reducing the number of feature maps from N to M using one convolutional layer, thereby shortening the encoding time.

[0030] 4 is a schematic diagram of one embodiment of the weighting unit 202 of the present invention. As shown in FIG. 4, the weighting unit 202 includes a pooling unit 401, a third feature extraction unit 402, a fourth feature extraction unit 403, and a first calculation unit 404.

[0031] The pooling unit 401 calculates the average value of the feature map of each channel of the input N channels, and obtains the statistical characteristics of the feature map of each channel. The pooling unit 401 may use one global average pooling layer to perform pooling processing on the input feature map. The operation principle of the global average pooling layer may refer to the prior art, and the description thereof will be omitted here.

[0032] The third feature extraction unit 402 performs a dimensionality reduction process on the feature map processed by the pooling unit 401 to obtain feature maps of M channels. The third feature extraction unit 402 may be realized by one convolution layer, which may be an M×1×1 convolution layer, where M is the number of channels and 1×1 is the convolution kernel (kernel) of the convolution layer. The operation principle of the convolution layer may refer to the prior art, and the description thereof will be omitted here.

[0033] The fourth feature extraction unit 403 performs dimension increase processing on the feature map of the M channels to obtain the feature map of the N channels. The fourth feature extraction unit 403 may also be realized by one convolution layer, and the convolution layer may be an N×1×1 convolution layer, where N is the number of channels and 1×1 is the convolution kernel (kernel) of the convolution layer. The operation principle of the convolution layer may refer to the prior art, and the description thereof will be omitted here.

[0034] The first calculation unit 404 multiplies the feature map of the N channels extracted by the fourth feature extraction unit 403 by the feature map of the N channels input (i.e., the feature map of the N channels from the encoder) to obtain the feature map of the N channels after weighting processing, and outputs it to the second feature extraction unit 203. The first calculation unit 404 may be realized by a scale function. The operating principle of the scale function may refer to the prior art, and the description thereof will be omitted here.

[0035] The weighting unit 202 in the embodiment of the present invention provides weights to the last layer of the encoder part to selectively enhance useful features and suppress less useful features. First, one global average pooling layer is used to generate each channel statistical characteristics, and then two convolution layers are used to increase or decrease the number of channels to better learn the nonlinear interactions between channels. Furthermore, since the number of feature maps needs to be reduced from N to M, the embodiment of the present invention uses the two convolution layers to change the number of channels from M to N to obtain more appropriate weights.

[0036] In the embodiment of the present invention, as shown in FIG. 4, the weighting unit 202 may further include a second calculation unit 405, a third calculation unit 406, and a fourth calculation unit 407. The second calculation unit 405 is located before the pooling unit 401, and calculates the absolute value of the input feature map of N channels (the feature map of N channels from the encoder), and outputs it to the pooling unit 401. The second calculation unit 405 may be realized by one abs function, and the explanation of the operation principle is omitted. The third calculation unit 406 is located between the third feature extraction unit 402 and the fourth feature extraction unit 403, and performs activation calculation on the feature map of M channels from the third feature extraction unit 402. The third calculation unit 406 may be realized by one relu function, and the explanation of the operation principle is omitted. The fourth calculation unit 407 is located between the fourth feature extraction unit 403 and the first calculation unit 404, and limits the feature map of N channels from the fourth feature extraction unit 403 to a range of 0 to 1. The fourth calculation unit 407 may be realized by one sigmoid function, and the explanation of the operation principle thereof will be omitted.

[0037] FIG. 5 is a schematic diagram of a network structure of one embodiment of the weighting unit 202 of the embodiment of the present invention. As shown in FIG. 5, the coupling layer 501 corresponds to the last layer of the encoder, and its output is the feature map of N channels as shown in FIG. 3. The abs 502 corresponds to the second calculation unit 405 of FIG. 4, and obtains the absolute value of the feature map of the N channels. The global pooling layer 503 corresponds to the pooling unit 401 of FIG. 4, and performs a pooling process on the feature map of N channels output by the abs. The convolution layer 504 corresponds to the third feature extraction unit 402 of FIG. 4, and performs a dimensionality reduction process on the feature map of N channels output by the global pooling layer 503 to obtain a feature map of M channels. The Relu 505 corresponds to the third calculation unit 406 of FIG. 4, and performs an activation operation on the feature map of the M channels. The convolution layer 506 corresponds to the fourth feature extraction unit 403 in Fig. 4, and performs a dimensionality increase process on the feature map of the M channels output by Relu to obtain a feature map of the N channels. The sigmoid 507 corresponds to the fourth calculation unit 407 in Fig. 4, and limits the feature map of the N channels to a range of 0 to 1. The scale 508 corresponds to the first calculation unit 404 in Fig. 4, and performs a multiplication process on the feature map of the N channels output by the combination layer and the feature map of the N channels output by sigmoid to obtain and output a feature map of the N channels.

[0038] The network structure of the weighting layer shown in Figure 5 is merely an example, and in the embodiment of the present invention, the weighting layer may have other network structures, for example, a convolution layer may be added, or the global pooling layer may be deleted, as long as weights can be assigned to the input feature maps of N channels.

[0039] By adding this weighting layer to the image coder and assigning weights to different channels, useful features can be strengthened and less useful features suppressed.

[0040] 6 is a schematic diagram of a probability model generating device 102 according to an embodiment of the present invention. As shown in FIG. 6, the probability model generating device 102 according to an embodiment of the present invention includes a hyper-decoder 601, a context model processing unit 602, and an entropy model processing unit 603. The hyper-decoder 601 performs decoding on the code stream received from the hyper-encoder to obtain auxiliary information. The context model processing unit 602 receives the output of the encoder and obtains a content-based prediction. The entropy model processing unit 603 combines the output of the context model processing unit 602 with the output of the hyper-decoder 601 to obtain a predicted probability model and provide it to the encoder and the decoder.

[0041] In an embodiment of the present invention, as shown in FIG. 6, the context model processing unit 602 includes a first processing unit and a second processing unit. The first processing unit obtains an average part of the prediction result based on the output of the encoder. The second processing unit obtains a variance part of the prediction result based on the output of the encoder. The entropy model processing unit 603 includes a third processing unit and a fourth processing unit. The third processing unit combines the average part of the prediction result obtained by the first processing unit with the auxiliary information output by the hyperdecoder 601 to obtain an average part of the probability model. The fourth processing unit combines the variance part of the prediction result obtained by the second processing unit with the auxiliary information output by the hyperdecoder 601 to obtain the variance part of the probability model.

[0042] In an embodiment of the present invention, the entropy model processing unit 603 predicts a probability model of a latent expression, combines a context model (latent autoregressive model) with a hypernetwork (hyperencoder and hyperdecoder), corrects the context-based prediction information with useful information learned by the hypernetwork, and generates the mean and scale parameters (variance) of a conditional Gaussian entropy model (above probability model). Different from the prior art, the embodiment of the present invention combines the mean part of the context model with the output of the hyperdecoder to obtain the mean part of the entropy model, and combines the variance part of the context model with the output of the hyperdecoder to obtain the variance part of the entropy model. By respectively obtaining the mean part and variance part of the entropy model, the latent distribution can be analyzed more accurately.

[0043] In an embodiment of the present invention, as shown in Fig. 6, the probability model generating device 102 of the embodiment of the present invention may further include a calculation unit 604. The calculation unit 604 obtains the absolute value of the variance part of the prediction result obtained by the second processing unit, provides it to the fourth processing unit, and obtains the absolute value of the output of the hyper decoder 601, provides it to the fourth processing unit. The calculation unit 604 may be realized by an absolute value function Abs. Since the value of sigma mainly represents the variance of the data, adding a layer having an absolute value function before the entropy model for generating sigma can better represent sigma.

[0044] In an embodiment of the present invention, as shown in FIG. 6, the probability model generating device 102 of the embodiment of the present invention may further include a quantizer 608, a hyperencoder 605, an arithmetic encoder 606, and an arithmetic decoder 607. The quantizer 608 performs a quantization process on the output from the encoder to quantize the latent representation from the encoder and generate a discrete value vector. The hyperencoder 605 further encodes the output of the quantizer 608. The arithmetic encoder 606 arithmetically encodes the output of the hyperencoder 605 to generate and output a code stream. The arithmetic decoder 607 decodes the received code stream and outputs it to the hyperdecoder 601. The operation principles of the quantizer 608, the hyperencoder 605, the arithmetic encoder 606, and the arithmetic decoder 607 may refer to the prior art, and the description thereof will be omitted here.

[0045] FIG. 7 is a schematic diagram of a network structure of one embodiment of an image compression system according to an embodiment of the present invention. As shown in FIG. 7, the image compression system includes an image encoding device 71, an image decoding device 72, and a probability model generating device 73. The image encoding device 71 corresponds to the image encoding device 101 of FIG. 2, and enhances useful features by adding a weighting layer 711, suppresses less useful features, and reduces the number of pixels by reducing the number of feature maps (number of channels) from N to M using a convolutional layer 712. The probability model generating device 73 corresponds to the probability model generating device 102 of FIG. 6. Here, the mu part 731 of the entropy model combines the mu part 732 of the context model and the output of the hyperdecoder 733 to generate the mu part of the probability model, and the sigma part 734 of the entropy model combines the sigma part 735 of the context model and the output of the hyperdecoder 733 to generate the sigma part of the probability model. Additionally, abs 736 is added before the sigma portion 734 of the entropy model, which takes the absolute value of the sigma portion of the context model and the output of the hyper-decoder to better represent the sigma portion of the entropy model.

[0046] In an embodiment of the present invention, as shown in Fig. 7, the image compression system may further include an arithmetic encoder (AE) 74 and an arithmetic decoder (AD) 75. The arithmetic encoder 74 encodes the output of the image encoding device 71 based on the probability model generated by the probability model generating device 73. The arithmetic decoder 75 decodes the received code stream based on the probability model generated by the probability model generating device 73, and provides the decoded code stream to the image decoding device 72. Also, as shown in Fig. 7, the image compression system may further include a quantizer 76. The quantizer 76 quantizes the latent representation of the image encoding device 71 by performing a quantization process on the output from the image encoding device 71, generates a discrete vector, and provides the discrete vector to the arithmetic encoder 74 and the context models 732 and 735.

[0047] In an embodiment of the present invention, the image decoding device 72 uses four convolution layers to perform inverse mapping on the input feature map to obtain an output image. The embodiment of the present invention is not limited thereto, and for example, the image decoding device 72 may use more or fewer convolution layers to perform inverse mapping on the input feature map, and the specific content may refer to the prior art, and the description thereof will be omitted here.

[0048] The image compression system according to the embodiment of the present invention uses the image encoding device according to the embodiment of the present invention, and the weighting unit multiplies different feature maps by one weight to obtain corresponding importance, and the second feature extraction unit performs dimensional reduction on the feature maps processed by the weighting unit, thereby shortening the decoding time, thereby reducing the bottleneck of latent variables and shortening the decoding time. Also, the image compression system according to the embodiment of the present invention uses the probability model generation device according to the embodiment of the present invention, and two independent context model processing units and entropy model processing units obtain two parameters mu and sigma of the probability model, thereby reducing the code stream required for encoding with a more accurate entropy model, and the independent entropy models are used to accurately predict the probability distribution, thereby reducing the code stream requirement.

[0049] <Example 2> An embodiment of the present invention provides an image encoding device. Figure 2 is a schematic diagram of the image encoding device of the embodiment of the present invention, Figure 3 is a schematic diagram of a network structure of one embodiment of the first feature extraction unit 201 of the image encoding device of the embodiment of the present invention, Figure 4 is a schematic diagram of the weighting unit 202 of the image encoding device of the embodiment of the present invention, Figure 5 is a schematic diagram of a network structure of one embodiment of the weighting unit 202 shown in Figure 4, and Figure 7 shows the image encoding device of the embodiment of the present invention. Since the image encoding device has already been described in detail in the first embodiment, the contents thereof are incorporated herein and the description thereof is omitted.

[0050] According to the image encoding device of the embodiment of the present invention, the bottleneck of the latent variables is reduced, thereby shortening the decoding time.

[0051] <Example 3> An embodiment of the present invention provides a probability model generation device. Figure 6 is a schematic diagram of the probability model generation device of an embodiment of the present invention, and Figure 7 shows the probability model generation device of an embodiment of the present invention. Since the probability model generation device has already been described in detail in the first embodiment, the contents of the description are incorporated herein and the description is omitted.

[0052] The probability model generating apparatus according to the embodiment of the present invention can reduce codestream requirements by accurately predicting probability distributions using independent entropy models.

[0053] <Example 4> The embodiment of the present invention provides an image coding method. The problem solving principle of the method is the same as that of the method of embodiment 2, which has already been described in the embodiment, so that the specific implementation of the method may refer to the implementation of the device of embodiment 1 and embodiment 2, and the description of the similar content will be omitted.

[0054] 8 is a schematic diagram of an image encoding method according to an embodiment of the present invention. As shown in FIG. 8, the image encoding method includes the following steps:

[0055] 801: Feature extraction is performed on an input image to obtain feature maps of N channels.

[0056] 802: Assign weights to the feature maps of each channel.

[0057] 803: Perform a dimensionality reduction process on the feature maps of the N channels to which the weights have been assigned, and obtain and output feature maps of M channels, where M is less than N.

[0058] In the embodiment of the present invention, the implementation of each operation in FIG. 8 may refer to the implementation of each unit in FIG. 2 in the first embodiment, and the description thereof will be omitted here.

[0059] In operation 802, the following processing may occur.

[0060] A global average pooling layer is used to average the feature maps of each of the N channels from the encoder to obtain the statistical properties of the feature maps of each channel.

[0061] Using an M×1×1 convolutional layer, dimensionality reduction is performed on the feature maps of the N channels to obtain feature maps of M channels.

[0062] An N×1×1 convolutional layer is used to perform dimensionality increase processing on the feature maps of the M channels to obtain feature maps of N channels.

[0063] The N channel feature maps from the encoder are multiplied by the N channel feature maps from the N×1×1 convolution device to obtain and output weighted N channel feature maps.

[0064] In the embodiment of the present invention, the implementation of operation 802 may refer to the implementation of FIG. 4 and FIG. 5 in the first embodiment, and the description thereof will be omitted here.

[0065] In an embodiment of the present invention, before using the global average pooling layer to calculate the average value of the feature maps of each of the N channels from the encoder, an abs function may be used to calculate the absolute value of the feature maps of the N channels from the encoder, and the operation principle of the abs function will not be described here.

[0066] In an embodiment of the present invention, before performing dimensionality increase processing on the feature maps of the M channels using an N×1×1 convolutional layer, activation calculations may be performed on the feature maps of the M channels using a relu function, and the operating principle of the relu function will not be described here.

[0067] In an embodiment of the present invention, before multiplying the N-channel feature maps from the encoder by the N-channel feature maps from the N×1×1 convolution device, a sigmoid function may be used to restrict the N-channel feature maps to a range of 0 to 1, and the operating principle of the sigmoid function will not be described here.

[0068] According to the image coding method of the embodiment of the present invention, the bottleneck of the latent variables is reduced, thereby shortening the decoding time.

[0069] <Example 5> The embodiment of the present invention provides a probabilistic model generation method, the problem solving principle of which is the same as that of the method of embodiment 3, and has already been described in embodiment 1, so the specific implementation of the method may refer to the implementation of the device of embodiment 1 and embodiment 3, and the description of the similar content will be omitted.

[0070] 9 is a schematic diagram of a method for generating a probability model according to an embodiment of the present invention. As shown in FIG. 9, the method for generating a probability model includes the following steps:

[0071] 901: Using a hyperdecoder, decode the codestream received from the encoder to obtain side information.

[0072] 902: Taking the output of the encoder as input using a context model, obtain a content-based prediction.

[0073] 903: Combine the output of the context model and the output of the hyper-decoder using an entropy model to obtain a predicted probability model, and provide it to the encoder and decoder.

[0074] In an embodiment of the present invention, the entropy model combines the mu portion of the context model with the output of the hyper-decoder to obtain the mu portion of the probability model, and combines the sigma portion of the context model with the output of the hyper-decoder to obtain the sigma portion of the probability model.

[0075] In an embodiment of the present invention, prior to operation 901, a hyperencoder may be used to further encode the output of the encoder, an arithmetic encoder may be used to arithmetically encode the output of the hyperencoder to generate and output a codestream, and an arithmetic decoder may be used to decode the received codestream and provide it to the hyperdecoder.

[0076] In an embodiment of the present invention, prior to operation 903, the absolute value of the sigma part of the context model and the absolute value of the output of the hyper-decoder may be calculated using an absolute value function abs and provided to the entropy model. That is, the entropy model may combine the absolute value of the sigma part of the context model and the absolute value of the output of the hyper-decoder to obtain the sigma part of the probability model.

[0077] According to the probability model generation method of the embodiment of the present invention, the codestream requirements can be reduced by accurately predicting the probability distribution using an independent entropy model.

[0078] <Example 6> An embodiment of the present invention provides an image processing device, which includes the image encoding device of the first and second embodiments or the probability model generating device of the first and third embodiments, or includes both the image encoding device of the first and second embodiments and the probability model generating device of the first and third embodiments, or includes the probability model generating device of the first and third embodiments and the image decoding device of the first embodiment.

[0079] In the first to third embodiments, the image encoding device, the probability model generating device, and the image decoding device have already been described in detail, so the contents of those devices will be incorporated herein and the description thereof will be omitted.

[0080] Fig. 10 is a schematic diagram of an image processing device according to an embodiment of the present invention. As shown in Fig. 10, the image processing device 1000 may include a central processing unit (CPU) 1001 and a storage device 1002, and the storage device 1002 is connected to the central processing unit 1001. The storage device 1002 may store various data and information processing programs, and executes the programs under the control of the central processing unit 1001.

[0081] In one aspect, the functions of the image encoding device and / or the probability model generating device and / or the image decoding device may be integrated into a central processing unit 1001. Here, the central processing unit 1001 may be configured to realize the method described in the fourth embodiment and / or the fifth embodiment.

[0082] In another aspect, the image encoding device and / or the probability model generation device and / or the image decoding device may each be arranged with a central processing unit 1001, for example, the image encoding device and / or the probability model generation device and / or the image decoding device may be chips connected to the central processing unit 1001 and configured to realize the functions of the image encoding device and / or the probability model generation device and / or the image decoding device under the control of the central processing unit 1001.

[0083] Also, as shown in Fig. 10, the image processing device may further include an input / output (I / O) device 1003 and a display device 1004. Here, the functions of the above-mentioned components are similar to those of the conventional technology, and the description thereof will be omitted here. Note that the image processing device does not need to include all of the components shown in Fig. 10. Also, the image processing device may include components not shown in Fig. 10, and may refer to the conventional technology.

[0084] An embodiment of the present invention provides a computer-readable program that, when executed in an image processing device, causes the image processing device to execute the method according to the fourth embodiment and / or the fifth embodiment.

[0085] An embodiment of the present invention further provides a storage medium storing a computer readable program for causing an image processing device to execute the method according to the fourth and / or fifth embodiment.

[0086] The above-mentioned apparatus and method of the present invention may be realized by hardware or a combination of hardware and software. The present invention relates to a computer-readable program, which, when executed by a logic unit, causes the logic unit to realize the above-mentioned apparatus or components, or to realize the above-mentioned various methods or steps. The present invention relates to a storage medium for storing the above-mentioned program, such as a hard disk, a magnetic disk, an optical disk, a DVD, a flash memory, etc.

[0087] The methods / apparatuses described with reference to the embodiments of the present invention may be implemented in hardware, software modules executed by a processor, or a combination of both. For example, one or more of the functional block diagrams shown in Figures 2 and 6, or one or more combinations of the functional block diagrams, may correspond to each software module in a computer program flow or each hardware module. These software modules may correspond to each step shown in Figures 8 and 9, respectively. These hardware modules may be realized by implementing these software modules in hardware, for example, using a field programmable gate array (FPGA).

[0088] The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium known to those skilled in the art. The storage medium may be connected to the processor so that the processor reads information from the storage medium and writes information to the storage medium, or the storage medium may be a component of the processor. The processor and the storage medium are located in an ASIC. The software module may be stored in the memory of the mobile terminal, or in a memory card inserted in the mobile terminal. For example, if the device (e.g., a mobile terminal) uses a MEGA-SIM card or a large capacity flash memory device with a relatively large capacity, the software module may be stored in the MEGA-SIM card or the large capacity flash memory device.

[0089] One or more of the functional blocks and / or one or more combinations of functional blocks depicted in the figures may be implemented in a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or any suitable combination thereof to perform the functions described herein. One or more of the functional blocks and / or one or more combinations of functional blocks depicted in the figures may be implemented in, for example, a combination of computing devices, such as a combination of a DSP and a microprocessor, a combination of multiple microprocessors, one or more microprocessors in combination with a DSP communication, or any other configuration.

[0090] Although the present invention has been described above with reference to specific embodiments, the above description is merely illustrative and does not limit the scope of protection of the present invention. Various modifications and changes may be made to the present invention without departing from the spirit and principles of the present invention, and these modifications and changes also fall within the scope of the present invention.

Claims

1. An image encoding device, comprising: a first feature extraction unit that performs feature extraction on an input image to obtain feature maps of N channels; a weighting unit that assigns weights to the feature maps of each channel; a second feature extraction unit that performs a dimensionality reduction process on the feature map weighted by the weighting unit to obtain and output feature maps for M channels, where M is smaller than N; The weighting unit is a pooling unit that calculates an average value of the feature maps of each of the N input channels to obtain a feature map of the N channels that indicates a statistical characteristic of the feature map of each of the N channels; a third feature extraction unit that performs a dimensionality reduction process on the feature map acquired by the pooling unit to acquire feature maps of M channels; a fourth feature extraction unit that performs a dimensionality increase process on the feature maps of the M channels to obtain feature maps of N channels; a first calculation unit that multiplies the feature map of the N channels extracted by the fourth feature extraction unit by the feature map of the N channels input to obtain a feature map of the N channels after weighting processing, and outputs the feature map to the second feature extraction unit.

2. The weighting unit is a second calculation unit located before the pooling unit, calculating absolute values ​​of the feature maps of the N channels inputted, and outputting the absolute values ​​to the pooling unit; a third calculation unit located between the third feature extraction unit and the fourth feature extraction unit, the third calculation unit performing an activation calculation on the feature maps of the M channels; 2. The image encoding device according to claim 1, further comprising: a fourth calculation unit located between the fourth feature extraction unit and the first calculation unit, the fourth calculation unit limiting the feature maps of the N channels to a range of 0 to 1.

3. 1. An image compression system, comprising: An image encoding device according to claim 1 or 2; a probability model generating device that predicts a probability distribution of a feature map of the M channels output by the image encoding device and obtains a probability model of the feature map; an arithmetic encoder that encodes an output of the image encoding device based on the probability model generated by the probability model generation device to generate and output a code stream; an arithmetic decoder that decodes the code stream received from the arithmetic encoder based on the probability model generated by the probability model generating device, and obtains and outputs a feature map; and an image decoding device that performs inverse mapping on the feature map provided by the arithmetic decoder to obtain an output image.

4. a quantizer that performs a quantization process on an output of the image encoding device to generate a discrete value vector; The system according to claim 3 , wherein the arithmetic encoder encodes an output of the quantizer based on the probability model generated by the probability model generating device, and generates and outputs the code stream.

Images