Image processing method, device, computer equipment, and computer program

Abstract

An embodiment of the present invention provides an image processing method and a related device. The method includes the steps of: obtaining a bitstream formed after an image is encoded; obtaining decoding instruction information, the decoding instruction information being used to indicate a decoding simplification operation to be performed in a decoding neural network, the decoding simplification operation being used to perform a grouping operation on convolution operations performed in at least one convolution layer included in the decoding neural network; and executing the decoding simplification operation according to the decoding instruction information to call the decoding neural network to perform a decoding process on the bitstream to reconstruct the image. According to the embodiment of the present invention, it is possible to reduce the decoding complexity on the decoding side and improve the decoding efficiency.

Description

[Technical Field]

[0001] This application claims priority from a Chinese patent application bearing application number 2023106009704, filed on May 24, 2023, and entitled "Image Processing Method and Related Apparatus," the entire contents of which are incorporated herein by reference.

[0002] The present invention relates to the field of image processing technology, and in particular to an image processing method, apparatus, computer device and computer-readable storage medium. [Background technology]

[0003] In the neural network model of end-to-end image compression, the encoding side (encoding end) maps the original image to latent variables through an analysis transformation sub-network, and writes the latent variables into a bitstream using an entropy encoder. The decoding side (decoding end) decodes the latent variables from the bitstream using an entropy decoder, and then inputs the latent variables into a synthesis transformation sub-network to obtain a reconstructed image. However, current image encoding / decoding schemes have problems such as high decoding complexity, low decoding efficiency, and lack of support for mobile terminal devices. Summary of the Invention

[0004] The embodiments of the present invention provide an image processing method and related device that can reduce the decoding complexity on the decoding side and improve the decoding efficiency.

[0005] In one aspect of an embodiment of the present invention, there is provided an image processing method, comprising the steps of: acquiring a bitstream formed after an image has been encoded; acquiring decoding instruction information, wherein the decoding instruction information is used to indicate a decoding simplification operation to be performed in a decoding neural network, and the decoding simplification operation is used to perform a grouping operation on convolution operations performed in at least one convolution layer included in the decoding neural network; and performing the decoding simplification operation in accordance with the decoding instruction information to call the decoding neural network to perform a decoding process on the bitstream in order to reconstruct the image.

[0006] In one aspect of an embodiment of the present invention, there is provided an image processing method including the steps of: obtaining an image to be processed; obtaining encoding instruction information, wherein the encoding instruction information is used to indicate an encoding simplification operation to be performed in an encoding neural network, and the encoding simplification operation is used to perform a grouping operation on convolution operations performed in at least one convolution layer included in the encoding neural network; and calling the encoding neural network according to the encoding instruction information to perform the encoding simplification operation to perform an encoding process on the image, and obtaining a bitstream.

[0007] In one aspect of an embodiment of the present invention, there is provided an image processing device including an acquisition unit and a processing unit, wherein the acquisition unit acquires a bit stream formed after an image has been encoded, the acquisition unit acquires decoding instruction information, the decoding instruction information is used to indicate a decoding simplification operation to be performed in a decoding neural network, the decoding simplification operation is used to perform a grouping process on convolution operations performed in at least one convolution layer included in the decoding neural network, and the processing unit performs the decoding simplification operation to call the decoding neural network in accordance with the decoding instruction information to perform a decoding process on the bit stream in order to reconstruct the image.

[0008] In one aspect of an embodiment of the present invention, there is provided an image processing device including an acquisition unit and a processing unit, wherein the acquisition unit acquires an image to be processed, the acquisition unit acquires encoding instruction information, the encoding instruction information is used to indicate an encoding simplification operation to be performed in an encoding neural network, the encoding simplification operation is used to perform a grouping process on convolution operations performed in at least one convolution layer included in the encoding neural network, and the processing unit performs the encoding simplification operation to call the encoding neural network to perform an encoding process on the image according to the encoding instruction information, and obtains a bitstream.

[0009] In one aspect of an embodiment of the present invention, a computer device is provided that includes a processor that executes a computer program and a computer-readable storage medium that stores the computer program, the computer program performing the image processing method described above when executed by the processor.

[0010] In one aspect of an embodiment of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, the computer program performing the above-described image processing method when executed by a processor.

[0011] In one aspect of an embodiment of the present invention, there is provided a computer program product including a computer program or computer instructions, which, when executed by a processor, implements the image processing method described above.

[0012] In an embodiment of the present invention, a bitstream formed after an image has been encoded is obtained, and decoding instruction information is obtained, which is used to indicate a simplified decoding operation to be performed in a decoding neural network, and the decoding neural network is called according to the decoding instruction information to perform a decoding process on the bitstream to reconstruct the image. When decoding the image, the decoding instruction information simplifies the operation when performing decoding in the decoding neural network on the decoding side, thereby reducing the complexity of decoding on the decoding side and improving the decoding efficiency. [Brief explanation of the drawings]

[0013] In order to more clearly describe the technology according to the embodiments of the present invention, the following briefly introduces drawings necessary for describing the embodiments, which merely show some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative work. [Figure 1A] FIG. 1 is a block diagram of a deep learning-based end-to-end image coding model according to one exemplary embodiment of the present invention. [Figure 1B] FIG. 2 is a block diagram of a deep learning-based end-to-end image coding model according to another exemplary embodiment of the present invention. [Figure 2A]1A and 1B are schematic diagrams illustrating a comparison of network structures for processing chrominance and luma components of a synthesis transformation sub-network and a lightened synthesis transformation sub-network, respectively, according to one exemplary embodiment of the present invention. [Figure 2B] 1 is a schematic diagram of a contrast between the structure of a residual block and a lightened residual block according to one exemplary embodiment of the present invention; [Figure 2C] 1 is a schematic diagram of a contrasting network structure for processing chrominance and luma components, respectively, of a hyper-decoding sub-network and a light-weight hyper-decoding sub-network according to one exemplary embodiment of the present invention; FIG. [Figure 2D] 1 is a schematic diagram of a contrast between network structures for processing chrominance and luma components of a hyper-scale decoding sub-network and a light-weight hyper-decoding sub-network, respectively, according to one exemplary embodiment of the present invention; FIG. [Figure 2E] FIG. 1 is a schematic diagram of a comparison of a normal convolution operation and a grouped convolution operation according to one exemplary embodiment of the present invention. [Figure 2F] FIG. 2 is a schematic diagram of channel reconstruction according to one exemplary embodiment of the present invention. [Figure 3] 1 is a schematic diagram of the architecture of an image processing system according to one exemplary embodiment of the present invention; [Figure 4] 1 is a flowchart of an image processing method according to one exemplary embodiment of the present invention. [Figure 5] 1 is a flowchart of an image processing method according to one exemplary embodiment of the present invention. [Figure 6] 1 is a schematic diagram of a configuration of an image processing device according to one exemplary embodiment of the present invention. [Figure 7] 1 is a schematic diagram of a configuration of an image processing device according to one exemplary embodiment of the present invention. [Figure 8] 1 is a schematic diagram of a computer device configuration according to one exemplary embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0014] The following clearly and completely describes the technology according to the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. It should be noted that the described embodiments are only a part of the embodiments of the present invention, and are not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative labor are within the protection scope of the present invention.

[0015] First, technical terms related to the present invention will be explained.

[0016] 1. Images An image may refer to an image to be coded. The image may be any one frame or multiple frames in a video. Here, the pixel format of the image may be RGB or YUV. If the pixel format of the image is RGB, the image may include an R (Red) component, a G (Green) component, and a B (Blue) component. If the pixel format of the image is YUV, the image may include a luma (Y) component and a chromaticity (UV) component.

[0017] II. Deep learning-based end-to-end image coding model An embodiment of the present invention relates to an end-to-end image coding model based on deep learning. For example, the end-to-end image coding model based on deep learning may be applied to a JPEG (Joint Photographic Experts Group) AI (Artificial Intelligence) platform. For example, the end-to-end image coding model based on deep learning may be a VM (Verification Model) model. FIG. 1A is a block diagram of an end-to-end image coding model based on deep learning according to an exemplary embodiment of the present invention. As shown in FIG. 1A, the end-to-end image coding model based on deep learning adopts a transform / encode structure. The end-to-end image coding model based on deep learning mainly includes an analysis transform net, a context model net, a hyper encoding net, a prediction fusion net, a hyper decoding net, a hyper scale decoding net, and a synthesis transform net. The following describes each of the above sub-networks in detail.

[0018] (a) Analysis and transformation subnetwork The analysis and transformation sub-network performs a nonlinear transformation on the image x to be coded, obtaining the transformation result y (also called latent variable) of image x. For example, if image x has luminance component x, Y and chromaticity component x UV , the analysis transform sub-network is Y and chromaticity component x UV are nonlinearly transformed, and the luminance component x Y The conversion result y corresponding to Y and chromaticity component x UV The conversion result y corresponding to UVmay be obtained.

[0019] (b) Context model subnetwork The context model subnetwork calculates the transformation value of image x. (outside 1) JPEG2026508263000002.jpg5118 is processed as a context. The output data of the context model sub-network is input to the prediction fusion sub-network. The transformation value of image x (outside 2) JPEG2026508263000003.jpg4118 is obtained by convolving the predicted value u of image x output by the prediction fusion sub-network with the data output by the inverse gain unit (invG unitRes: inverse gain unit residual). The input data of the inverse gain unit is the image residual of image x. (Outside 3) JPEG2026508263000004.jpg4118. Image residual (outside 4) JPEG2026508263000005.jpg5118 may be obtained by quantizing data output by a Gain unit residual (G unitRes). Specifically, the data type output by the Gain unit residual is a floating-point number. The quantization process involves performing an integer rounding process on the data output by the Gain unit residual to generate an integer-type image residual. (outside 5) This includes data type conversion processing (integerization processing) to obtain JPEG2026508263000006.jpg5118. The input data of the residual gain unit is the residual r obtained by calculating the difference between the transformation result y obtained for image x by the analysis transformation sub-network and the predicted value u output from the prediction fusion sub-network.

[0020] Image x has luminance component x Y and chromaticity component x UVIf it contains, the residual r of the luminance component Y is the luminance component x Y The converted value of y Y and the predicted value u of the luminance component output from the prediction fusion sub-network. Y The residual r of the chromaticity component may be obtained by subtracting UV is the chromaticity component x UV The converted value of y UV and the predicted value u of the chromaticity component output from the prediction fusion sub-network. UV It may be obtained by subtracting r Y and r UV quantized to integer (outside 6) JPEG2026508263000007.jpg5118 and (outer 7) Convert to JPEG2026508263000008.jpg5118 to generate a bitstream.

[0021] (c) The predictive fusion sub-network predicts image x and obtains a predicted value u of image x. The input data of the predictive fusion sub-network are the output data of the context model sub-network and the output data of the hyper-decoding sub-network. For example, the predictive fusion sub-network predicts the chrominance and luma components of an image and obtains a predicted value u of the luma component. Y and the predicted value of the chromaticity component u UV Obtain the predicted value u including

[0022] (d) The hyper-coding sub-network processes the transformation result y of the image x, then quantizes the output data of the hyper-coding sub-network to obtain the quantized result of the image, and losslessly encodes the quantized result to obtain a bitstream. The quantization process includes rounding. In this case, quantizing the output data of the hyper-coding sub-network to obtain the quantized result of the image may include rounding the output data of the hyper-coding sub-network to obtain the quantized result of the image.

[0023] (e) The input data of the hyper-decoding sub-network is the hyper-parameters obtained based on the bitstream. (outside 8) JPEG2026508263000009.jpg5118, and the output data of the hyper-decoding sub-network is the input data of the prediction fusion sub-network.

[0024] (f) The hyperscale decoding subnetwork is (outer 9) Based on JPEG2026508263000010.jpg5118, the mean value is 0 and the variance is (Outside 10) Gaussian distribution, JPEG2026508263000011.jpg5118 (Outside 11) The synthesis transform sub-network determines the transform value obtained by decoding the bitstream. (Outside 12) JPEG2026508263000013.jpg5118 is subjected to synthesis conversion processing, and the image (Outside 13) For example, the synthesis transform subnetwork reconstructs the transform value obtained by decoding the bitstream. (Outside 14) Transformed value of the luminance component in JPEG2026508263000015.jpg5118 (Outside 15) JPEG2026508263000016.jpg5118 is subjected to synthesis transformation processing, and the resulting transformed value is obtained by decoding the bitstream. (Outside 16) Chrominance component conversion values ​​in JPEG2026508263000017.jpg5118 (Outside 17) Luminance component for reconstructing the image by performing synthesis transformation on JPEG2026508263000018.jpg5118 (outside 18) JPEG2026508263000019.jpg6120 and chromaticity components (Outside 19) Get JPEG2026508263000020.jpg6120 and reconstruct the image (outside 20) Get JPEG2026508263000021.jpg5120.

[0025] In addition, the deep learning-based end-to-end image coding model may be a lightweight VM model. Figure 1B is a block diagram of a deep learning-based end-to-end image coding model according to another exemplary embodiment of the present invention. As shown in Figure 1B, the deep learning-based end-to-end image coding model mainly includes a lightweight analysis transform net, a hyper encoding net, a lightweight hyper decoding net, a lightweight hyper scale decoding net, and a lightweight synthesis transform net.

[0026] The light-weight analysis transformation subnetwork may have the same functionality as the analysis transformation subnetwork, but the complexity of the light-weight analysis transformation subnetwork is lower than the complexity of the analysis transformation subnetwork. The light-weight hyper-decoding subnetwork may have the same functionality as the hyper-decoding subnetwork, but the complexity of the light-weight hyper-decoding subnetwork is lower than the complexity of the hyper-decoding subnetwork. The light-weight hyperscale decoding subnetwork may have the same functionality as the hyperscale decoding subnetwork, but the complexity of the light-weight hyperscale decoding subnetwork is lower than the complexity of the hyperscale decoding subnetwork. The light-weight synthesis transformation subnetwork may have the same functionality as the synthesis transformation subnetwork, but the complexity of the light-weight synthesis transformation subnetwork is lower than the complexity of the synthesis transformation subnetwork.

[0027] Next, the network structures of the sub-networks corresponding to FIG. 1A and FIG. 1B are compared.

[0028] (1) Comparison of the network structure of the lightweight synthesis transformation subnetwork and the network structure of the lightweight synthesis transformation subnetwork Here, the light-weight synthesis transformation sub-network may synthesize and transform the transformation value of the luminance component of the image and the transformation value of the chrominance component of the image. The light-weight synthesis transformation sub-network may synthesize and transform the transformation value of the luminance component of the image and the transformation value of the chrominance component of the image. Figure 2A is a schematic diagram comparing the network structures for processing the chrominance component and the luma component of the synthesis transformation sub-network and the light-weight synthesis transformation sub-network, respectively, according to one exemplary embodiment of the present invention.

[0029] (a) Comparison of the network configuration of the Synthesis Transform Net_Y when processing the luminance component and the network configuration of the Lightweight Synthesis Transform Net_Y when processing the luminance component. As shown in Figure 2A, the synthesis transformation sub-network for processing the luminance component mainly includes two residual blocks (ResBlock), four convolutional layers, four crop layers (Crop), three residual nonlinear units (ResAU), and an attention module (residual non-local attention block: RNAB). Here, the four convolution layers are, from top to bottom, DConv128×3×3S2 (DConv represents deconvolution, the number of channels is 128, the convolution kernel size is 3×3, and S2 represents the convolution stride is 2), DConv128×3×3S2 (DConv represents deconvolution, the number of channels is 128, the convolution kernel size is 3×3, and S2 represents the convolution stride is 2), DConv128×3×3S2 (DConv represents deconvolution, the number of channels is 128, the convolution kernel size is 3×3, and S2 represents the convolution stride is 2), and DConv1×3×3S2 (DConv represents deconvolution, the number of channels is 1, the convolution kernel size is 3×3, and S2 represents the convolution stride is 2).

[0030] As shown in 22 of FIG. 2A, the light-weight synthesis transform sub-network for processing the luma component mainly includes a light-weight residual block (LightResBlock), four convolution layers, three crop layers (Crop), three residual nonlinear units (ResAU), and one pixel reconstruction (pixelshuffleS4). Here, the four convolution layers are, from top to bottom, DConv96×4×4S2 (DConv represents deconvolution, the number of channels is 96, the convolution kernel size is 4×4, and S2 represents the convolution stride is 2), DConv64×4×4S2 (DConv represents deconvolution, the number of channels is 64, the convolution kernel size is 4×4, and S2 represents the convolution stride is 2), Conv64×3×3S1 (Conv represents convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 represents the convolution stride is 1), and Conv16×1×1S1 (Conv represents convolution, the number of channels is 16, the convolution kernel size is 1×1, and S1 represents the convolution stride is 1).

[0031] As can be seen by comparing 21 and 22 in Figure 2A, when processing the luminance component, the network structure of the lightweight synthesis transformation subnetwork requires only one lightweight residual block and does not require an RNAB, while the synthesis transformation subnetwork requires two residual blocks but requires the involvement of an RNAB. In addition, the network parameters in the convolutional layers of the lightweight synthesis transformation subnetwork and the synthesis transformation subnetwork were also changed. As can be seen from this, the network structure of the lightweight synthesis transformation subnetwork is less complex than the synthesis transformation subnetwork.

[0032] (b) Comparison of the network configuration of the synthesis transform subnetwork (Synthesis Transform Net_UV) when processing chromaticity components and the network configuration of the light-weighted synthesis transform subnetwork (Light Weighted Synthesis Transform Net_UV) when processing chromaticity components. As shown in Figure 2A, the synthesis transformation sub-network for processing the chrominance component mainly includes two residual blocks (ResBlock), four convolutional layers, four crop layers (Crop), three residual nonlinear units (ResAU), and an attention module (RNAB). Here, the four convolution layers are, from top to bottom, DConv64×3×3S2 (Dconv represents deconvolution, the number of channels is 64, the convolution kernel size is 3×3, and S2 represents the convolution stride is 2), DConv64×3×3S2 (DConv represents deconvolution, the number of channels is 64, the convolution kernel size is 3×3, and S2 represents the convolution stride is 2), DConv64×3×3S2 (DConv represents deconvolution, the number of channels is 64, the convolution kernel size is 3×3, and S2 represents the convolution stride is 2), and DConv2×3×3S2 (DConv represents deconvolution, the number of channels is 2, the convolution kernel size is 3×3, and S2 represents the convolution stride is 2).

[0033] As shown in FIG. 2A , the light-weight synthesis transform sub-network for processing the chrominance components mainly includes a light-weight residual block (LightResBlock), four convolution layers, three crop layers (Crop), three residual nonlinear units (ResAU), and one pixel reconstruction (pixelshuffleS4). Here, the four convolution layers are, from top to bottom, DConv64×4×4S2 (Dconv stands for deconvolution, the number of channels is 64, the convolution kernel size is 4×4, and S2 stands for convolution stride 2), DConv64×4×4S2 (Dconv stands for deconvolution, the number of channels is 64, the convolution kernel size is 4×4, and S2 stands for convolution stride 2), Conv64×3×3S1 (Conv stands for convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 stands for convolution stride 1), and Conv32×1×1S1 (Conv stands for convolution, the number of channels is 32, the convolution kernel size is 1×1, and S1 stands for convolution stride 1).

[0034] As can be seen by comparing 23 and 24 in Figure 2A, when processing the chromaticity component, the network structure of the lightweight synthesis transformation subnetwork requires only one lightweight residual block and does not require an RNAB, whereas the synthesis transformation subnetwork requires two residual blocks but requires the involvement of an RNAB. In addition, the network parameters in the convolutional layers of the lightweight synthesis transformation subnetwork and the synthesis transformation subnetwork were also changed. As can be seen from this, the network structure of the lightweight synthesis transformation subnetwork is less complex than the synthesis transformation subnetwork.

[0035] Similarly, the structure of the residual block and the structure of the lightened residual block are different. FIG. 2B is a schematic diagram comparing the structures of the residual block and the lightened residual block according to an exemplary embodiment of the present invention. In FIG. 2B , the residual block may include an input layer (e.g., input image x), two convolution layers (Conv3×3S1G1 and Conv3×3S1G1, respectively, where the number of channels is 3×3, S1 represents a step size of 1, and G1 represents the number of groups of convolution operations of 1), and an activation function (Relu). The lightened residual block includes an input layer (e.g., input image x), one convolution layer (Conv3×3S1G1), and an activation function (Relu). As can be seen, the structural complexity of the lightened residual block is lower than that of the residual block.

[0036] (2) Comparison of the network structure of the HyperDecoder and the Lightweight HyperDecoder FIG. 2C is a schematic diagram of a contrasting network structure for processing chrominance and luma components, respectively, of a hyper-decoding sub-network and a light-weight hyper-decoding sub-network according to one exemplary embodiment of the present invention.

[0037] (a) Comparison of the network structure of the hyper-decoding sub-network (Hyper Decoder _Y) when processing the luminance component and the network structure of the light-weighted hyper-decoding sub-network (Light Weighted Hyper Decoder _Y) when processing the luminance component. As shown in 25 of Figure 2C, the hyper-decoding sub-network for processing the luminance component mainly includes five convolution layers, two crop layers (Crop), and three activation functions (LeakyRelu). Here, the five convolution layers are, from top to bottom, Conv128x3x3S1 (Conv represents convolution, the number of channels is 128, the size of the convolution kernel is 3x3, and S1 represents the convolution stride is 1), DConv128x3x3S2 (DConv represents deconvolution, the number of channels is 128, the size of the convolution kernel is 3x3, and S2 represents the convolution stride is 2), and Conv128x3x3S1 (Conv represents convolution, the number of channels is 128, the size of the convolution kernel is 3x3, and S2 represents the convolution stride is 2). is 128, the convolution kernel size is 3x3, and S1 indicates the convolution stride is 1), DConv192x3x3S2 (DConv represents deconvolution, the number of channels is 192, the convolution kernel size is 3x3, and S2 indicates the convolution stride is 2), and Conv128x3x3S1 (Conv represents convolution, the number of channels is 128, the convolution kernel size is 3x3, and S1 indicates the convolution stride is 1).

[0038] As shown in 26 of Figure 2C, the lightweight hyper-decoding sub-network for processing the luma component mainly includes five convolution layers, two crop layers (Crop), and three activation functions (LeakyRelu). Here, the five convolution layers are, from top to bottom, Conv128x1x1S1 (Conv represents convolution, the number of channels is 128, the size of the convolution kernel is 1x1, and S1 represents the convolution stride is 1), DConv128x4x4S2 (DConv represents deconvolution, the number of channels is 128, the size of the convolution kernel is 4x4, and S2 represents the convolution stride is 2), and Conv128x3x3S1 (Conv represents convolution, the number of channels is 1x1, and S2 represents the convolution stride is 2). The number of channels is 128, the size of the convolution kernel is 3x3, and S1 indicates that the convolution stride is 1), DConv128x4x4S2 (DConv represents deconvolution, the number of channels is 128, the size of the convolution kernel is 4x4, and S2 indicates that the convolution stride is 2), and Conv128x3x3S1 (Conv represents convolution, the number of channels is 128, the size of the convolution kernel is 3x3, and S1 indicates that the convolution stride is 1).

[0039] (b) Comparison of the network configuration of the hyper-decoder sub-network (Hyper Decoder _UV) when processing the luminance component and the network configuration of the light-weighted hyper-decoder sub-network (Light Weighted Hyper Decoder _UV) when processing the luminance component. As shown in 27 of Figure 2C, the hyper-decoding sub-network for processing the luminance component mainly includes five convolution layers, two crop layers (Crop), and three activation functions (LeakyRelu). Here, the five convolution layers are, from top to bottom, Conv64×3×3S1 (Conv represents convolution, the number of channels is 64, the size of the convolution kernel is 3×3, and S1 represents the convolution stride is 1), DConv64×3×3S2 (Dconv represents deconvolution, the number of channels is 64, the size of the convolution kernel is 3×3, and S2 represents the convolution stride is 2), and Conv64×3×3S1 (Conv represents convolution, the number of channels is 64, the size of the convolution kernel is 3×3, and S2 represents the convolution stride is 2). is 64, the convolution kernel size is 3x3, and S1 indicates the convolution stride is 1), DConv96x3x3S2 (DConv is deconvolution, the number of channels is 96, the convolution kernel size is 3x3, and S2 indicates the convolution stride is 2), and Conv64x3x3S1 (Conv is convolution, the number of channels is 64, the convolution kernel size is 3x3, and S1 indicates the convolution stride is 1).

[0040] As shown in 28 of Figure 2C, the lightweight hyper-decoding sub-network for processing chrominance components mainly includes five convolution layers, two crop layers (Crop), and three activation functions (LeakyRelu). Here, the five convolution layers are, from top to bottom, Conv64×1×1S1 (Conv represents convolution, the number of channels is 64, the size of the convolution kernel is 1×1, and S1 represents the convolution stride is 1), DConv64×4×4S2 (DConv represents deconvolution, the number of channels is 64, the size of the convolution kernel is 4×4, and S2 represents the convolution stride is 2), and Conv64×3×3S1 (Conv represents convolution, the number of channels is 1×1, and S2 represents the convolution stride is 2). is 64, the convolution kernel size is 3x3, and S1 indicates the convolution stride is 1), DConv64x4x4S2 (DConv represents deconvolution, the number of channels is 64, the convolution kernel size is 4x4, and S2 indicates the convolution stride is 2), and Conv64x3x3S1 (Conv represents convolution, the number of channels is 64, the convolution kernel size is 3x3, and S1 indicates the convolution stride is 1).

[0041] As can be seen from the comparison between 25 and 26 in Figure 2C and between 27 and 28 in Figure 2C, when processing the luma or chroma components, the network structure of the hyper-decoding sub-network and the network structure of the light-weight hyper-decoding sub-network are slightly different in the convolutional layers.

[0042] (3) Comparison of the network structure of the Hyper Scale Decoder and the network structure of the Lightweight Hyper Scale Decoder FIG. 2D is a schematic diagram of a contrasting network structure for processing chrominance and luma components of a hyper-scale decoding sub-network and a light-weight hyper-decoding sub-network, respectively, according to one exemplary embodiment of the present invention.

[0043] (a) Comparison of the network structure of the hyperscale decoding subnetwork (Hyper Scale Decoder _Y) when processing the luminance component and the network structure of the light-weighted hyperscale decoding subnetwork (Light Weighted Hyper Scale Decoder _Y) when processing the luminance component. As shown in Figure 2D, the hyperscale decoding sub-network for processing the luma component mainly includes six convolution layers, two crop layers (Crop), and five activation functions (LeakyRelu). Here, the six convolution layers are, from top to bottom, DConv128x5x5S2 (DConv represents deconvolution, the number of channels is 128, the size of the convolution kernel is 5x5, and S2 represents the convolution stride is 2), DConv192x5x5S2 (DConv represents deconvolution, the number of channels is 192, the size of the convolution kernel is 5x5, and S2 represents the convolution stride is 2), and Conv256x3x3S1 (Conv represents convolution, the number of channels is 256, the size of the convolution kernel is 3x3, and S1 represents the convolution stride). Conv212×3×3S1 (Conv stands for convolution, the number of channels is 212, the convolution kernel size is 3×3, and S1 stands for the convolution stride is 1), Conv170×3×3S1 (Conv stands for convolution, the number of channels is 170, the convolution kernel size is 3×3, and S1 stands for the convolution stride is 1), and Conv128×3×3S1 (Conv stands for convolution, the number of channels is 128, the convolution kernel size is 3×3, and S1 stands for the convolution stride is 1).

[0044] As shown in 30 of FIG. 2D, the lightweight hyper-decoding sub-network for processing the luma component mainly includes four convolution layers, two crop layers (Crop), and three activation functions (LeakyRelu). Here, the four convolution layers are, from top to bottom, DConv128×4×4S2 (DConv represents deconvolution, the number of channels is 128, the convolution kernel size is 4×4, and S2 represents the convolution stride is 2), Conv128×3×3S1 (Conv represents convolution, the number of channels is 128, the convolution kernel size is 3×3, and S1 represents the convolution stride is 1), DConv128×4×4S2 (DConv represents deconvolution, the number of channels is 128, the convolution kernel size is 4×4, and S2 represents the convolution stride is 2), and Conv128×3×3S1 (Conv represents convolution, the number of channels is 128, the convolution kernel size is 3×3, and S1 represents the convolution stride is 1).

[0045] (b) Comparison of the network configuration of the hyperscale decoding subnetwork (Hyper Scale Decoder _UV) when processing chromaticity components and the network configuration of the lightweight hyperscale decoding subnetwork (Light weighted Hyper Scale Decoder _UV) when processing chromaticity components. As shown in 31 of Figure 2D, the hyperscale decoding sub-network for processing chrominance components mainly includes six convolution layers, two crop layers (Crop), and five activation functions (LeakyRelu). Here, the six convolution layers are, from top to bottom, DConv64x5x5S2 (DConv represents deconvolution, the number of channels is 64, the size of the convolution kernel is 5x5, and S2 represents the convolution stride is 2), DConv96x5x5S2 (DConv represents deconvolution, the number of channels is 96, the size of the convolution kernel is 5x5, and S2 represents the convolution stride is 2), and Conv128x3x3S1 (Conv represents convolution, the number of channels is 128, the size of the convolution kernel is 3x3, and S1 represents the convolution stride). Conv106×3×3S1 (Conv stands for convolution, the number of channels is 106, the convolution kernel size is 3×3, and S1 stands for the convolution stride is 1), Conv85×3×3S1 (Conv stands for convolution, the number of channels is 85, the convolution kernel size is 3×3, and S1 stands for the convolution stride is 1), and Conv64×3×3S1 (Conv stands for convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 stands for the convolution stride is 1).

[0046] As shown in 32 of FIG. 2D, the lightweight hyper-decoding sub-network for processing the chrominance component mainly includes four convolution layers, two crop layers (Crop), and three activation functions (LeakyRelu). Here, the four convolution layers are, from top to bottom, DConv64×4×4S2 (DConv represents deconvolution, the number of channels is 64, the convolution kernel size is 4×4, and S2 represents the convolution stride is 2), Conv64×3×3S1 (Conv represents convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 represents the convolution stride is 1), DConv64×4×4S2 (DConv represents deconvolution, the number of channels is 64, the convolution kernel size is 4×4, and S2 represents the convolution stride is 2), and Conv64×3×3S1 (Conv represents convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 represents the convolution stride is 1).

[0047] As can be seen from the comparison between 29 and 30 and between 31 and 32 in Figure 2D, the network structure of the hyper-decoding sub-network when processing luma and chroma components is more complex than the network structure of the light-weight hyper-decoding sub-network, and the convolutional layers of the hyper-decoding sub-network when processing luma and chroma components are different from the convolutional layers of the light-weight hyper-decoding sub-network (e.g., the number of channels is different).

[0048] 3. Regular convolution (also called convolution) and grouped convolution Normal convolution refers to the convolution of feature maps of all input images collectively in a convolution layer. When performing normal convolution, each output channel is connected (interacts) with each input channel, with dense connections between the channels. The output channel is the output feature map after convolution, and the input channel is the input feature map. FIG. 2E is a schematic diagram comparing normal convolution and grouped convolution according to an exemplary embodiment of the present invention. For example, as shown in (1) of FIG. 2E, all upper-layer output channels (i.e., output channels a to h) are connected to lower-layer input channels (input channels 1 to 8). For example, output channel a is connected to input channels 1 to 8, respectively, and output channel b is connected to input channels 1 to 8, respectively, and so on. In this case, performing normal convolution operations inevitably leads to large computational complexity. As can be seen from the deep learning-based end-to-end encoding and decoding model, the whole model involves more convolutional layers. Applying normal convolution to the convolutional layers will increase the computational complexity of the whole image encoding and decoding process, and reduce the efficiency of encoding and decoding.

[0049] Grouped convolution refers to grouping feature maps of an input image by channel in a convolutional layer to obtain multiple groups, and then performing a convolution operation on each of the multiple groups to obtain convolution results for each group, e.g., feature maps for the channels included in each group. The convolutional results of multiple groups may be merged. For example, the input channels and output channels may be divided into multiple groups, and the output channels of each group are connected only to the input channels in the corresponding group and are unrelated to the channels in other groups. For example, as shown in (2) of FIG. 2E, a convolutional layer includes eight input channels (e.g., input channel 1 to input channel 8) and eight output channels (output channel a to output channel h). Next, the eight input channels and eight output channels may be divided into four groups, each of which includes two input channels and two output channels. For example, group 1 includes input channel 1, input channel 2, output channel a, and output channel b, group 2 includes input channel 3, input channel 4, output channel c, and output channel d, group 3 includes input channel 5, input channel 6, output channel e, and output channel f, and group 4 includes input channel 7, input channel 8, output channel g, and output channel h. Thus, in group 4 above, the two output channels in each group are connected to only two input channels in the corresponding group.

[0050] In this way, the grouped convolution can reduce the computational complexity to a certain extent. Therefore, in the embodiments of the present invention, the grouped convolution is introduced into the deep learning-based end-to-end encoding and decoding model, thereby reducing the computational complexity of the entire encoding and decoding process and improving the efficiency of encoding and decoding.

[0051] 4. Channel reconstruction Channel reconstruction can realize feature interactions between channels. After performing grouped convolution processing, channel reconstruction can realize feature map interactions between groups, effectively improving the accuracy of feature extraction during subsequent processing. In embodiments of the present invention, channel reconstruction may refer to rearranging feature maps corresponding to channels obtained after grouped convolution to realize information exchange between different groups. FIG. 2F is a schematic diagram of channel reconstruction according to an exemplary embodiment of the present invention. For example, as shown in FIG. 2F(a), channels are divided into three groups (illustrated in different colors in FIG. 2F(a) for illustrative purposes). Each group is input to a first convolutional layer to perform grouped convolution processing, and grouped convolutional features corresponding to each group are obtained. Then, the grouped convolutional features corresponding to each group are input to a second convolutional layer to perform grouped convolution processing. In FIG. 2F(a), channel reconstruction is not performed, so the grouped convolutional features between different groups cannot communicate, which may reduce the feature extraction ability of the model.

[0052] To solve this problem, we consider rearranging the grouped convolutional features output by the grouped convolution operation using channel reconstruction. As shown in FIG. 2F(b), multiple channels are divided into three groups (shown in different colors in FIG. 2F(b) for illustrative purposes). Each group is input to the first convolutional layer to perform grouped convolution processing, obtaining grouped convolutional features corresponding to each group. The grouped convolutional features corresponding to each group are then rearranged to obtain channel reconstruction results. For example, the grouped convolutional features corresponding to the first group include the convolutional features of channel 1, channel 2, and channel 3. The grouped convolutional features corresponding to the second group include the convolutional features of channel 4, channel 5, and channel 6. The grouped convolutional features corresponding to the third group include the convolutional features of channel 7, channel 8, and channel 9. The grouped convolutional features corresponding to the three groups are sorted to obtain channel sorting results, which are convolutional features of channel 1, convolutional features of channel 4, convolutional features of channel 7, convolutional features of channel 2, convolutional features of channel 5, convolutional features of channel 8, convolutional features of channel 3, convolutional features of channel 6, and convolutional features of channel 9. The channel reconstruction results are then input to the next convolutional layer (the second convolutional layer) for processing.

[0053] In this way, the channel reconstruction process realizes feature exchange between groups, and can better guarantee that the input of the next grouping convolution is from a different group, thereby improving the feature extraction ability of the model. Furthermore, since channel reconstruction simply rearranges the feature maps corresponding to channels, using channel reconstruction can reduce the introduction of additional computational operations.

[0054] An embodiment of the present invention provides an image processing scheme. The image processing scheme may be applied to an end-to-end image coding model based on deep learning. The image processing scheme can simplify the end-to-end image coding model based on deep learning. The image processing scheme provides decoding instruction information at a decoding side, where the decoding instruction information indicates a simplified decoding operation to be performed in a decoding neural network. Then, to reconstruct an image, the decoding neural network is invoked according to the decoding instruction information to perform a decoding process on the bitstream. When decoding an image, the decoding instruction information simplifies the operation when performing decoding in the decoding neural network on the decoding side, thereby reducing the decoding complexity on the decoding side and improving the decoding efficiency. Furthermore, the image processing scheme can be well supported by a mobile terminal device (i.e., a decoding device).

[0055] An image processing system according to an embodiment of the present invention will be described below.

[0056] 3 is a schematic diagram of the architecture of an image processing system according to an exemplary embodiment of the present invention. The image processing system may include an encoding device 301 and a decoding device 302. The encoding device 301 is located on the encoding side, and the decoding device 302 is located on the decoding side. The encoding device 301 may be a terminal or a server. The decoding device 302 may be a terminal or a server. A communication connection may be established between the encoding device 301 and the decoding device 302. Here, the terminal may be, but is not limited to, a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, an in-vehicle terminal, a smart TV, etc. The terminal may be an independent physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server providing fundamental cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, a CDN (Content Delivery Network), and a big data and artificial intelligence platform.

[0057] (1) Encoding device 301 The encoding device 301 may acquire an image to be processed, and the image may be captured by an imaging device or generated by a computer. The imaging device may be a hardware component provided in the encoding device 301. For example, the imaging device may be a general camera, a stereo camera, a light field camera, or the like provided in a terminal. The imaging device may also be a hardware component connected to the encoding device 301. For example, the imaging device may be a camera connected to a server.

[0058] The encoding device 301 may then invoke an encoding neural network to perform an encoding process on the acquired image to obtain a bitstream corresponding to the image. The encoding neural network may include an analysis and transformation sub-network such as that shown in Figure 1A, or may include a reduced analysis and transformation sub-network as shown in Figure 1B.

[0059] In one embodiment, the encoding neural network includes the analysis-transformation sub-network shown in Figure 1A. An image includes chrominance and luminance components. The encoding device 301 encodes the chrominance component x Y and the luminance component x UV The analysis and transformation sub-network shown in Figure 1A performs nonlinear transformation on each of these components, and the resulting luminance component is y Y and the chromaticity component conversion result y UV Next, the encoding device 301 may obtain the transformation result y Y The residual r of the luminance component is calculated by subtracting the predicted value of the luminance component output from the prediction fusion sub-network from Y and convert the chromaticity component y UV The residual r of the chromaticity component is calculated by subtracting the predicted value of the chromaticity component output from the prediction fusion sub-network from UV Next, the encoding device 301 obtains the residual r Y and the residual r of the chromaticity component UV and the residual of the integer luminance component (outside 21) JPEG2026508263000022.jpg6120 and the residual of integer chromaticity components (outside 22) Convert to JPEG2026508263000023.jpg5120 to form a bitstream.

[0060] In another aspect, when encoding an image, encoding instruction information may be configured in advance. The encoding instruction information is used to indicate an encoding simplification operation to be performed in the encoding neural network. Then, the encoding device 301 calls the encoding neural network according to the encoding instruction information to perform an encoding process on the image and obtain a bitstream. For example, it is assumed as an example that the encoding neural network includes the analysis and transformation sub-network shown in FIG. 1A. The encoding instruction information may be used to indicate an encoding simplification operation to be performed in the analysis and transformation sub-network. In this case, the encoding device 301 calls the analysis and transformation sub-network according to the encoding instruction information to perform an encoding simplification operation on the image and obtain a bitstream.

[0061] By performing the encoding simplification operation in the encoding neural network, the complexity of the encoding side can be reduced to a certain extent, and the encoding efficiency can be improved. The specific embodiment of calling the encoding neural network according to the encoding instruction information to perform the image encoding process and obtain the bitstream will be described in the following examples, so the description thereof will be omitted here.

[0062] After obtaining the bitstream, the encoding device 301 may transmit the bitstream to the decoding device 302, causing the decoding device 302 to perform a decoding process on the bitstream to reconstruct an image.

[0063] (2) Decryption device 302 The decoding device 302 receives the bitstream transmitted by the encoding device 301 and obtains decoding instruction information. The decoding instruction information is used to indicate a simplified decoding operation to be performed in the decoding neural network. The decoding device 302 calls the decoding neural network according to the decoding instruction information to decode the bitstream and reconstruct an image. Specifically, if the image includes a luminance component and a chrominance component, the decoding device 302 decodes the bitstream and obtains the residual of the luminance component. (outside 23) JPEG2026508263000024.jpg6120 and residual chromaticity component (outside 24) JPEG2026508263000025.jpg6120 and residuals (Outside 25) JPEG2026508263000026.jpg5120 may be obtained. Next, the decoding device 302 may obtain the residual of the luminance component. (outside 26) JPEG2026508263000027.jpg5120 and the predicted value u of the luminance component output from the prediction fusion sub-network Y and are superimposed to obtain the converted value of the luminance component. (outside 27) Get JPEG2026508263000028.jpg5120 and calculate the residual of the chromaticity component. (outside 28) JPEG2026508263000029.jpg6120 and the predicted value u of the chromaticity component output from the prediction fusion sub-network UV and are superimposed to obtain the converted value of the chromaticity component. (outside 29) You can get JPEG2026508263000030.jpg6120. Then, call the decoding neural network to get the transformed value of the luminance component. (Outside 30) JPEG2026508263000031.jpg6120 and converted values ​​of chromaticity components (Outside 31) Perform a decoding simplification operation on JPEG2026508263000032.jpg6120 to reconstruct the image.

[0064] In one aspect, the decoding simplification operation may include a grouped convolution operation. The decoding instruction information is used to indicate that the grouped convolution operation is to be performed in the decoding neural network. In this aspect, invoking the decoding neural network to perform the decoding simplification operation on the luma component transform value and the chroma component transform value to reconstruct an image may be invoking the decoding neural network in accordance with the decoding instruction information to perform the grouped convolution operation on the luma component transform value and the chroma component transform value, respectively, to reconstruct an image. The luma component transform value and the chroma component transform value are included in, for example, a feature map of the image.

[0065] In another aspect, the decoding simplification operation may include a grouped convolution operation and a channel reconstruction operation. The decoding instruction information is used to indicate that the decoding neural network should perform the grouped convolution operation and the channel reconstruction operation. In this aspect, invoking the decoding neural network to perform the decoding simplification operation on the transform values ​​of the luma component and the transform values ​​of the chroma component and reconstructing an image may be the decoding device 302 instructing the decoding neural network, according to the decoding instruction information, to perform a grouped convolution operation on the transform values ​​of the luma component to obtain a grouped convolution result for the luma component, to invoke the decoding neural network to perform a grouped convolution operation on the transform values ​​of the chroma component to obtain a grouped convolution result for the chroma component, to perform a channel reconstruction operation on the grouped convolution result for the luma component, and to perform a channel reconstruction operation on the grouped convolution result for the chroma component to obtain a channel reconstruction result for the luma component and a channel reconstruction result for the chroma component. The channel reconstruction result for the luma component and the channel reconstruction result for the chroma component are used to reconstruct the image.

[0066] Here, the decoding neural network may include a synthesis transformation subnetwork, a hyper-decoding subnetwork, a hyper-scale decoding subnetwork, etc., as shown in Fig. 1A. Alternatively, the decoding neural network may include a light-weight synthesis transformation subnetwork, a light-weight hyper-decoding subnetwork, a light-weight hyper-scale decoding subnetwork, etc., as shown in Fig. 1B. The present invention is not limited thereto.

[0067] In an embodiment of the present invention, encoding instruction information may be configured on the encoding device 301 side, but not on the decoding device 302 side, thereby simplifying the calculation complexity on the encoding side using the encoding instruction information and improving the encoding efficiency. Alternatively, decoding instruction information may be configured on the decoding device 302 side, but not on the encoding device 301 side, thereby simplifying the decoding complexity on the decoding side using the decoding instruction information and improving the decoding efficiency. Note that by configuring encoding instruction information on the encoding device 301 side, and configuring decoding instruction information on the decoding device 302 side, the complexity of the image encoding / decoding process may be reduced, improving the encoding / decoding efficiency.

[0068] Next, an image processing method according to an embodiment of the present invention will be described.

[0069] 4 is a flowchart of an image processing method according to an exemplary embodiment of the present invention. The image processing method may be performed by the decoding device 302. The image processing method may include the following steps S401 to S403.

[0070] Step S401: Obtain a bitstream formed after an image has been encoded.

[0071] Step S402: Obtain decoding instruction information. The decoding instruction information is used to indicate a decoding simplification operation to be performed in the decoding neural network. The decoding simplification operation is used to perform a grouping process on the convolution operations to be performed in at least one convolution layer included in the decoding neural network. Grouping the convolution operations to be performed in the convolution layer is called a grouped convolution operation.

[0072] Here, the decoding neural network may be at least one of a hyper-decoding subnetwork, a hyper-scale decoding subnetwork, and a synthesis transform subnetwork. The hyper-decoding subnetwork may be the hyper-decoding subnetwork shown in FIG. 1A or the light-weight hyper-decoding subnetwork shown in FIG. 1B. The hyper-scale decoding subnetwork may be the hyper-scale decoding subnetwork shown in FIG. 1A or the light-weight hyper-scale decoding subnetwork shown in FIG. 1B. The synthesis transform subnetwork may be the synthesis transform subnetwork shown in FIG. 1A or the light-weight synthesis transform subnetwork shown in FIG. 1B. The decoding simplification operation includes a grouping convolution operation. Alternatively, the decoding simplification operation includes a grouping convolution operation and a channel reconstruction operation.

[0073] (1) The decoding simplification operation includes a grouping convolution operation.

[0074] The decoding instruction information includes decoding grouping convolution instruction information. The decoding neural network includes at least one convolution layer, and the number of the at least one convolution layer is N, where N is a positive integer. The decoding grouping convolution instruction information is used to indicate performing a grouping convolution operation in at least one convolution layer of the decoding neural network. The grouping convolution operation includes: grouping feature maps of an image by channel in the at least one convolution layer to obtain multiple groups; and performing a convolution operation within each group of the multiple groups to obtain grouped convolution features for each group, where the grouped convolution features for each group include convolution features of channels in the group. The output channels of each group of the multiple groups interact only with the input channels in the group and are not related to channels in other groups of the multiple groups. By indicating that a grouped convolution operation is performed in a convolution layer of a decoding neural network, the convolution operations related to the convolution layer in the decoding neural network can be grouped, and the large computational complexity associated with performing the convolution operation on all input features simultaneously can be avoided, and the computational complexity can be reduced, thereby reducing the decoding complexity and improving the decoding efficiency.

[0075] The decoding grouping convolution instruction information may be set in the network structure of the convolution layer of the decoding neural network. For example, the decoding neural network is the synthesis transformation sub-network of FIG. 1A, and the decoding instruction information is used to indicate that the grouping convolution operation is performed in the convolution layer (DConv128×3×3S2). In this case, the decoding grouping convolution instruction information may be set in the network structure of the convolution layer. For example, the convolution layer may be represented as DConv128×3×3S2G2, where G2 represents that the number of groups performing the grouping convolution operation is 2.

[0076] In addition, the decoding grouping convolution instruction information is used to further indicate the number of groups of the grouping convolution operation performed in the convolution layer of the decoding neural network. The number of groups may be set according to experience. For example, the number of groups may be set to 2, 4, 8, etc. For example, if experience shows that a number of groups of 4 can reduce decoding complexity during decoding and ensure decoding performance, the number of groups used by the decoding grouping convolution instruction information to indicate the grouping convolution operation is 4.

[0077] Here, the number of groups of the grouped convolution operation indicated by the decoding grouped convolution instruction information may support dynamic adjustment according to changes in network parameters of the convolution layer of the decoding neural network, where the network parameters include at least one of the number of channels, feature data of the feature map of the input convolution layer, and rate points when encoding / decoding an image.

[0078] (a) The network parameters include the number of channels. When the number of channels in a convolutional layer that performs a grouping convolution operation in a decoding neural network changes, the number of groups in the grouping convolution operation needs to be dynamically adjusted. For example, if the number of channels in a convolutional layer that performs a grouping convolution operation changes from 128 to 64, the number of groups in the grouping convolution operation may be adjusted from 8 to 4. (b) The network parameters include feature data of the feature map of the input convolutional layer. For example, if the feature data input to the feature map of the convolutional layer of the decoding neural network includes the chrominance and luma components of an image, the number of groups for performing the grouping convolution operation on the chrominance components of the convolutional layer of the decoding neural network differs from the number of groups for performing the grouping convolution operation on the luma component of the convolutional layer. (c) The network parameters include the rate points (also called code rate points) when encoding / decoding is performed. When a new rate point is used for encoding / decoding a different image, the number of groups in the grouping convolution operation needs to be dynamically adjusted. If the image sources are different, the encoding / decoding process of different images will correspond to different rate points, for example, when performing the encoding / decoding process of different images, an image from a computer-generated animation image and an image from a screen content will correspond to different rate points.

[0079] In one embodiment, the decoding grouping convolution instruction information is used to further indicate n convolution layers that perform the grouping convolution operation among N convolution layers, where n is a positive integer and is less than or equal to N. For example, the decoding neural network includes the synthesis transformation subnetwork of FIG. 1A. For example, the synthesis transformation subnetwork of FIG. 1A includes four (i.e., N=4) convolution layers, and the decoding grouping convolution instruction information may be used to indicate two (n=2) convolution layers (i.e., DConv128×3×3S2 and DConv128×3×3S2) that perform the grouping convolution operation among the N convolution layers. In other words, the grouping convolution operation needs to be performed in the convolution layer DConv128×3×3S2 and the convolution layer DConv128×3×3S2.

[0080] The decoding grouped convolution indication information is also used to indicate the number of groups for which the grouped convolution operation is to be performed in each of the n convolution layers. Here, the number of groups for which the grouped convolution operation is to be performed in different convolution layers among the n convolution layers may all be the same. Alternatively, the number of groups for which the grouped convolution operation is to be performed in different convolution layers among the n convolution layers may be partially the same. Alternatively, the number of groups for which the grouped convolution operation is to be performed in different convolution layers among the n convolution layers may all be different. For example, the number of groups for which the grouped convolution operation is to be performed in one convolution layer may be four, and the number of groups for which the grouped convolution operation is to be performed in another convolution layer may be two.

[0081] Here, the selection rule for the n convolutional layers may be at least one of the following: (a) Select n convolutional layers to perform the grouping convolutional operation from the N convolutional layers in descending order of the distance between the convolutional layer and the output layer. For example, if the decoding neural network is the light-weight synthesis transformation subnetwork shown in FIG. 1B and the distance between the convolutional layer Conv64×3×3S1 and the output layer is greater than the distance between the convolutional layer Conv16×1×1S1 and the output layer, the convolutional layer Conv64×3×3S1 may be selected as the convolutional layer to perform the grouping convolutional operation. (b) Select n convolutional layers to perform the grouping convolutional operation from the N convolutional layers in descending order of the complexity of the network structure. For example, if the decoding neural network includes convolutional layer 1, convolutional layer 2, and convolutional layer 3, and the complexity of the network structure of convolutional layer 1 is greater than that of convolutional layer 2, which in turn is greater than that of convolutional layer 3, convolutional layer 1 and convolutional layer 2 may be selected as the convolutional layers that perform the grouping convolution operation.

[0082] When selecting a convolutional layer to perform a grouped convolutional operation, the grouped convolutional operation is performed on a convolutional layer with a relatively high complexity, thereby reducing the calculation complexity of the convolutional layer and the decoding complexity. Also, when selecting a convolutional layer to perform a grouped convolutional operation, the convolutional layer to perform the grouped convolutional operation is determined taking into account the complexity and the decoding performance of the decoding neural network, thereby reducing the complexity and ensuring the decoding performance.

[0083] (2) The decoding simplification operation includes a channel reconstruction operation and a grouping convolution operation. The channel reconstruction operation realizes information exchange between groups, and the channel reconstruction does not require the introduction of additional calculation operations, so it can effectively ensure the image feature extraction ability of the decoding neural network, effectively guarantee the decoding performance, and reduce the decoding complexity.

[0084] Here, the decoding instruction information includes decoding grouping convolution instruction information and decoding channel reconstruction instruction information. The decoding grouping convolution instruction information is used to indicate that a grouping convolution operation is to be performed in a convolution layer of a decoding neural network. The decoding grouping convolution instruction information may refer to (1) above, and a description thereof will be omitted here. The decoding channel reconstruction instruction information is used to indicate that a channel reconstruction operation is to be performed in a convolution layer of a decoding neural network. The decoding channel reconstruction instruction information may be set in the network structure of the convolution layer of the decoding neural network.

[0085] The convolutional layer that performs the decoding channel reconstruction operation is a convolutional layer that performs a group convolutional operation, where the number of convolutional layers of the decoding neural network is N, and the N convolutional layers include n convolutional layers that perform a group convolutional operation, where N and n are both positive integers and n is less than or equal to N, the decoding channel reconstruction indication information is used to further indicate t convolutional layers that perform the channel reconstruction operation among the n convolutional layers, where t is a positive integer and t is less than or equal to n.

[0086] In one embodiment, the decoding channel reconstruction indication information is used to further indicate the number of channels on which a channel reconstruction operation is performed. When the number of convolutional layers on which a channel reconstruction operation is performed is t, the decoding channel reconstruction indication information is used to further indicate the number of channels on which each of the t convolutional layers on which a channel reconstruction operation is performed may be performed. Here, the number of channels on which each of the t convolutional layers on which a channel reconstruction operation is performed may all be the same. Alternatively, the number of channels on which each of the t convolutional layers on which a channel reconstruction operation is performed may be partially the same. For example, the number of channels on which a channel reconstruction operation is performed in convolutional layer 1 is 4, the number of channels on which a channel reconstruction operation is performed in convolutional layer 2 is 4, and the number of channels on which a channel reconstruction operation is performed in convolutional layer 3 is 6. Alternatively, the number of channels on which a channel reconstruction operation is performed in each of the t convolutional layers on which a channel reconstruction operation is performed may all be different.

[0087] Here, the number of channels for the channel reconstruction operation indicated by the decoding channel reconstruction instruction information may be dynamically adjusted according to changes in network parameters of the convolutional layer of the decoding neural network, where the network parameters include at least one of the number of channels, feature data of the input feature map, and rate points when encoding / decoding is performed.

[0088] (a) The network parameters include the number of channels. The network parameters need to dynamically adjust the number of channels for the channel reconstruction operation when the number of channels for the convolutional layer performing the channel reconstruction operation changes. For example, if the number of channels for the convolutional layer performing the grouped convolution operation is changed from 128 to 64, the number of channels for the channel reconstruction operation may be adjusted from 128 to 64. (b) The network parameters include feature data of the feature map of the input convolutional layer. When the feature data of the feature map input to the convolutional layer of the decoding neural network includes chrominance and luma components of an image, the number of channels when the convolutional layer of the decoding neural network performs the channel reconstruction operation on the chrominance components is different from the number of channels when the convolutional layer performs the channel reconstruction operation on the luma component. (c) The network parameters include the rate point at which encoding / decoding is performed. The network parameters need to dynamically adjust the number of channels for the channel reconstruction operation when different rate points are to be used for encoding / decoding different images.

[0089] Here, if the convolutional layer includes P channels, the decoding channel reconstruction indication information is used to further indicate p channels among the P channels on which the channel reconstruction operation is to be performed. In other words, the decoding channel reconstruction indication information indicates the number of channels on which the channel reconstruction operation is to be performed and is used to indicate which of the P channels needs to undergo the channel reconstruction operation. In one aspect, if each channel corresponds to a channel identifier, for example, a channel index, the decoding channel reconstruction indication information may indicate the index value of the channel on which the channel reconstruction operation is to be performed. The channel number and channel index are parameters of a channel pixel. For example, the convolutional layer includes 64 channels, and the 64 channels correspond to index values ​​of 0 to 63, respectively. The decoding channel reconstruction indication information is used to indicate that the number of channels on which the channel reconstruction operation is to be performed is 32, and the 32 channels (p=32) on which the channel reconstruction operation is to be performed are 0 to 3, 10 to 13, 20 to 23, and 30 to 33.

[0090] The selection rule for the p channels may include at least one of the following: (a) p channels to perform the channel operation are selected from the P channels according to a rule that reduces memory resource consumption. For example, P=64, and the 64 channels are stored in different memory locations. When performing channel reconstruction, the memory consumption varies when retrieving the 64 channels from the memory locations of the 64 channels. Therefore, 32 channels to perform the channel reconstruction may be selected from the 64 channels in ascending order of memory consumption. (b) The P channels correspond to multiple groups of the grouped convolution operation, for example, the number of groups is 4, and P=64. The 64 channels correspond to four groups of the grouped convolution operation, each group including 16 channels. In this case, p channels to perform the channel reconstruction operation may be selected from the P channels according to the data communication requirements between each group. For example, if the data communication demand is to achieve characteristics between each group that can be obtained from all other groups, all 64 channels may be selected as channels to perform the channel reconstruction operation.

[0091] In one embodiment, the decoding neural network includes at least one of a hyper-decoding subnetwork, a hyper-scale decoding subnetwork, and a synthesis transformation subnetwork. Each subnetwork includes a convolutional layer. The decoding simplification operation includes a grouped convolutional operation. When the decoding instruction information indicates that all convolutional layers of each subnetwork perform a grouped convolutional operation, the number of groups that perform the grouped convolutional operation in the convolutional layers of each subnetwork indicated by the decoding instruction information may all be the same, or the number of groups that perform the grouped convolutional operation in the convolutional layers of each subnetwork indicated by the decoding instruction information may be partially the same, or the number of groups that perform the grouped convolutional operation in the convolutional layers of each subnetwork indicated by the decoding instruction information may all be different.

[0092] Furthermore, the decoding simplification operation includes a grouping convolution operation and a channel reconstruction operation. When the decoding instruction information indicates a channel reconstruction operation in the convolution layer of each subnetwork, the number of channels in the channel reconstruction operation in the convolution layer of each subnetwork indicated by the decoding instruction information may all be the same, or the number of channels in the channel reconstruction operation in the convolution layer of each subnetwork indicated by the decoding instruction information may be partially the same and partially different, or the number of channels in the channel reconstruction operation in the convolution layer of each subnetwork indicated by the decoding instruction information may all be different (for example, the number of channels in the convolution layer of the hyper-decoding subnet indicated by the decoding instruction information is 4, and the number of channels in the composite transform subnetwork is 32). The present invention is not limited thereto.

[0093] Step S403: Perform a decoding simplification operation to call a decoding neural network according to the decoding instruction information to perform a decoding process on the bitstream to reconstruct an image.

[0094] When the decoding neural network includes at least one convolutional layer, the decoding simplified operation includes a grouped convolutional operation, and the decoding instruction information indicates that the grouped convolutional operation is to be performed in the convolutional layer of the decoding neural network, the step of performing the decoding simplified operation to call the decoding neural network in accordance with the decoding instruction information to perform a decoding process on the bitstream to reconstruct an image may include the steps of calling the decoding neural network in accordance with the decoding instruction information to decode the bitstream and obtain transformed values ​​of the image, calling at least one convolutional layer of the decoding neural network to perform a grouped convolutional operation on the transformed values ​​of the image and obtain grouped convolution results corresponding to the transformed values ​​of the image, and decoding the grouped convolution results to reconstruct the image. Specifically, the decoding instruction information includes decoding grouped convolution instruction information, which indicates the number of groups of the grouped convolutional operation. The decoding device may perform a grouping process on the transformation values ​​of the image based on the number of groups to obtain one or more groups, and then perform a convolution operation on each group to obtain a grouped convolution result corresponding to the transformation values ​​of the image.

[0095] If the decoding simplification operation includes a grouped convolution operation and a channel reconstruction operation, and the decoding instruction information indicates that the grouped convolution operation and the channel reconstruction operation are to be performed in a convolution layer of the decoding neural network, the step of invoking the decoding neural network in accordance with the decoding instruction information to decode the bit stream and reconstruct an image may include: invoking the decoding neural network in accordance with the decoding instruction information to decode the bit stream and obtain transformed values ​​of the image; invoking at least one convolution layer of the decoding neural network to perform a grouped convolution operation on the transformed values ​​of the image and obtain a grouped convolution result corresponding to the transformed values ​​of the image; performing a channel reconstruction operation on the grouped convolution result, obtaining the channel reconstruction result, and performing a decoding process on the channel reconstruction processing result, to reconstruct the image.

[0096] Here, the grouped convolution result includes convolution features of each channel among the multiple channels, and the decoding instruction information includes decoding channel reconstruction instruction information, which may be used to further indicate the number of channels for the channel reconstruction operation. Performing a channel reconstruction operation on the grouped convolution result to obtain the channel reconstruction result may include sorting the grouped convolution result based on the number of channels indicated by the decoding channel reconstruction instruction information to obtain the channel reconstruction result.

[0097] In an embodiment of the present invention, a decoding side provides decoding instruction information, which indicates a simplified decoding operation to be performed in a decoding neural network. Then, to reconstruct an image, the decoding neural network is called according to the decoding instruction information to perform a decoding process on the bitstream. When decoding an image, the decoding instruction information simplifies the operation when the decoding neural network performs decoding on the decoding side, thereby reducing the decoding complexity on the decoding side, improving the decoding efficiency, and achieving better support for mobile terminal devices.

[0098] 5 is a flowchart of an image processing method according to an exemplary embodiment of the present invention. The image processing method may be performed by the encoding device 301. The image processing method may include the following steps S501 to S503.

[0099] Step S501: Obtain an image to be processed.

[0100] Step S502: Obtain encoding instruction information. The encoding instruction information is used to indicate an encoding simplification operation to be performed in the encoding neural network. The encoding simplification operation is used to perform a grouping process on the convolution operations performed in at least one convolution layer included in the encoding neural network. The process of grouping the convolution operations performed in the convolution layer is called a grouped convolution operation.

[0101] Here, the coding neural network may include at least one of an analysis-transformation sub-network and a hyper-coding sub-network. The coding simplification operation may include a grouping convolution operation, or may include a grouping convolution operation and a channel reconstruction operation.

[0102] (1) The encoding simplification operation includes a grouping convolution operation.

[0103] The encoding instruction information includes encoding grouped convolution instruction information. The encoding neural network includes at least one convolutional layer. The encoding grouped convolution instruction information is used to instruct the execution of a grouped convolution operation in the convolutional layer of the encoding neural network. Here, the encoding grouped convolution instruction information is provided in the network structure of the convolutional layer of the encoding neural network. The encoding grouped convolution operation includes: grouping feature maps of an image by channel in the at least one convolutional layer to obtain multiple groups; and performing an encoding grouped convolution operation in each group of the multiple groups to obtain grouped convolutional features for each group, where the grouped convolutional features for each group include convolutional features of the channels in the group. The output channels of each group of the multiple groups interact only with the input channels in that group and are not related to channels in other groups of the multiple groups.

[0104] The encoding grouped convolution instruction information is also used to indicate the number of groups of the grouped convolution operation performed in the convolution layer of the encoding neural network. The number of groups may be dynamically adjusted according to changes in network parameters of the convolution layer of the encoding neural network. Here, the network parameters may include at least one of the number of channels, feature data of a feature map input to the convolution layer, and a rate point when encoding / decoding is performed.

[0105] (a) The network parameters include the number of channels. When the number of channels in the convolutional layer of the encoding neural network changes, the number of groups in the grouping convolution operation needs to be dynamically adjusted. For example, when the number of channels in the convolutional layer of the encoding neural network changes from 128 to 64, the number of groups in the grouping convolution operation may be adjusted from 8 to 4. (b) The network parameters include feature data of a feature map input to the convolutional layer of the encoding neural network. When the feature data of the feature map input to the convolutional layer of the encoding neural network includes chrominance and luma components of an image, the number of groups when the convolutional layer performs grouping convolution on the chrominance components is different from the number of groups when the convolutional layer performs grouping convolution on the luma component. (c) The network parameters include a rate point when encoding / decoding is performed. When a new rate point is used for encoding / decoding a different image, the number of groups in the grouping convolution operation needs to be dynamically adjusted. When the image source is different, different rate points need to be associated with the encoding / decoding of different images. When different rate points are used for encoding / decoding processes of different images, the number of groups in the grouped convolution operation needs to be dynamically adjusted.

[0106] The number of convolutional layers in the encoding neural network is M, where M is a positive integer. The encoding grouped convolution indication information is used to further indicate m convolutional layers among the m convolutional layers that perform grouped convolutional operations, where m is a positive integer and is less than or equal to M. Here, the selection rule for the m convolutional layers may be at least one of the following: (a) Select the m convolutional layers that perform the grouped convolutional operations in descending order of distance from the output layer; or (b) Select the m convolutional layers that perform the encoding grouped convolutional operations in descending order of network structure complexity. The selection rule for the m convolutional layers is similar to the selection rule for the n convolutional layers described above, except that the selection rule for the m convolutional layers takes into account encoding considerations, which will not be described here.

[0107] The coding grouped convolution indication information is also used to indicate the number of groups for which the grouped convolution operation is to be performed in each of the m convolution layers. The numbers of groups for which the grouped convolution operation is to be performed in different convolution layers among the m convolution layers may all be the same. Alternatively, the numbers of groups for which the grouped convolution operation is to be performed in different convolution layers among the m convolution layers may be partially the same. Alternatively, the numbers of groups for which the grouped convolution operation is to be performed in different convolution layers among the m convolution layers may all be different.

[0108] (2) The encoding simplification operation includes a grouping convolution operation and a channel reconstruction operation.

[0109] The encoding instruction information may include encoding grouped convolution instruction information and encoding channel reconstruction instruction information. The encoding grouped convolution instruction information may refer to the relevant description in (1) above, and further description is omitted here. The encoding channel reconstruction instruction information is used to indicate that a channel reconstruction operation is to be performed in a convolution layer of a coding neural network, and the encoding channel reconstruction instruction information is set in the network structure of the convolution layer of the coding neural network. The channel reconstruction operation includes sorting the grouped convolution features of each group obtained by the grouped convolution operation to obtain the sorted grouped convolution features.

[0110] In addition, the encoding channel reconstruction indication information is used to further indicate the number of channels of the channel reconstruction operation performed in the convolution layer of the encoding neural network, and the number of channels may be dynamically adjusted according to changes in network parameters of the convolution layer of the encoding neural network.

[0111] Here, the convolutional layer includes I channels. The coded channel reconstruction indication information is used to further indicate O channels among the I channels that need to perform a channel reconstruction operation, where O is a positive integer and is less than or equal to I. In other words, the coded channel reconstruction indication information is used to indicate the number of channels that need to perform a channel reconstruction operation and which channels among the I channels need to perform a channel reconstruction operation. In one aspect, if each channel corresponds to a channel index value, the coded channel reconstruction indication information may indicate the index value of the channel on which the channel reconstruction operation is to be performed.

[0112] The selection rule for the O channels includes at least one of selecting O channels on which a channel operation needs to be performed from the I channels according to a rule that reduces memory resource consumption, or selecting O channels on which a channel reconstruction operation needs to be performed from the I channels according to data communication requirements between the O channels, where the O channels respectively correspond to multiple groups of the grouped convolution operation. Here, the selection rule for the O channels is similar to the selection rule for the p channels above, except that the selection rule for the O channels is considered from the viewpoint of coding, and therefore description thereof will be omitted here.

[0113] When the number of convolution layers of the encoding neural network is M, and the M convolution layers include m convolution layers that need to perform a grouped convolution operation, where M and m are both positive integers and m is less than or equal to M, the encoding channel reconstruction indication information is used to further indicate q convolution layers among the m convolution layers that need to perform a channel reconstruction operation, where q is a positive integer and q is less than or equal to m.

[0114] In addition, the coding channel reconstruction indication information is also used to indicate the number of channels performing the channel reconstruction operation in each of the q convolutional layers, where the number of channels performing the channel reconstruction operation in different convolutional layers among the q convolutional layers may be the same, or the number of channels performing the channel reconstruction operation in different convolutional layers among the q convolutional layers may be the same and different, or the number of channels performing the channel reconstruction operation in different convolutional layers among the q convolutional layers may all be different.

[0115] Here, the coding neural network includes at least one of an analytic transformation subnetwork and a hyper-coding subnetwork. When the coding simplification operation includes a grouping convolution operation, and the coding instruction information indicates that all convolutional layers of each subnetwork need to perform the grouping convolution operation, the number of groups that perform the grouping convolution operation in the convolutional layers of each subnetwork indicated by the coding instruction information may all be the same, or the number of groups that perform the grouping convolution operation in the convolutional layers of each subnetwork indicated by the coding instruction information may be the same for some and different for others, or the number of groups that perform the grouping convolution operation in the convolutional layers of each subnetwork indicated by the coding instruction information may all be different (for example, the coding instruction information indicates that the number of groups that perform the grouping convolution operation in the convolutional layer of the hyper-coding subnetwork is 4 and the number of groups that perform the grouping convolution operation in the convolutional layer of the analytic transformation subnetwork is 8).

[0116] When the encoding simplification operation includes a grouping convolution operation and a channel reconstruction operation, when the encoding instruction information indicates that a channel reconstruction operation is to be performed in the convolutional layer of each sub-network, the number of channels that perform the channel reconstruction operation in the convolutional layer of the sub-network indicated by the encoding instruction information may all be the same, or the number of channels that perform the channel reconstruction operation in the convolutional layer of the sub-network indicated by the encoding instruction information may be the same in part and different in others, or the number of channels that perform the channel reconstruction operation in the convolutional layer of the sub-network indicated by the encoding instruction information may all be different (for example, the encoding instruction information indicates that the number of channels that perform the channel reconstruction operation in the convolutional layer of the hyper-coding subnetwork is 16, and the number of channels that perform the channel reconstruction operation in the convolutional layer of the analytic transformation subnetwork is 32).

[0117] Step S503: According to the encoding instruction information, call the encoding neural network to perform the encoding simplification operation to perform the encoding process on the image, and obtain a bitstream.

[0118] When the encoding neural network includes a convolutional layer, the encoding simplification operation includes a grouped convolutional operation, and the encoding instruction information is used to indicate performing the grouped convolutional operation in the convolutional layer of the encoding neural network, the step of invoking the encoding neural network in accordance with the encoding instruction information to perform an encoding process on the image and obtain a bitstream includes invoking the encoding neural network to perform the grouped convolutional operation on the image and obtain a grouped convolution result, and then performing an encoding process (e.g., quantization) on the grouped convolution result to obtain a bitstream of the image.

[0119] When the encoding neural network includes a convolutional layer, the encoding reduction operation includes a grouped convolution operation and a channel reconstruction operation, and the encoding instruction information is used to indicate performing the grouped convolution operation and the channel reconstruction operation in the convolutional layer of the encoding neural network, the step of invoking the encoding neural network in accordance with the encoding instruction information to perform an encoding process on the image and obtain a bitstream includes: invoking the encoding neural network to perform a grouped convolution operation on the image, obtaining a grouped convolution result, performing a channel reconstruction operation on the grouped convolution result, obtaining a channel reconstruction result of the image, and performing an encoding process (e.g., quantization) on the channel reconstruction result, to obtain a bitstream of the image.

[0120] In one embodiment, the encoding neural network includes at least one convolutional layer. The encoding simplification operation includes a grouped convolution operation. The encoding instruction information includes encoding grouped convolution instruction information, which indicates performing a grouped convolution operation in at least one convolutional layer of the encoding neural network. The grouped convolution operation includes grouping feature maps of an image by channel in the at least one convolutional layer to obtain multiple groups, and performing a convolution operation within each group of the multiple groups to obtain grouped convolutional features for each group, where the grouped convolutional features for each group include convolutional features of channels in the group. The output channels of each group of the multiple groups interact only with the input channels in the group and are not related to channels in other groups of the multiple groups.

[0121] In one embodiment, the encoding grouped convolution indication information is used to further indicate the number of groups of the grouped convolution operation performed in the convolution layer of the encoding neural network.

[0122] In one embodiment, the number of groups of the grouped convolution operation indicated by the encoding grouped convolution indication information supports dynamic adjustment in response to changes in network parameters of the convolution layer of the encoding neural network.

[0123] In one embodiment, the number of the at least one convolutional layer is M, where M is a positive integer, and the encoding grouped convolution indication information is used to further indicate m convolutional layers that perform the grouped convolution operation among the M convolutional layers, where m is a positive integer and m is less than or equal to M.

[0124] In one embodiment, the selection rule for the m convolutional layers includes at least one of: selecting m convolutional layers that perform the grouped convolutional operation from the M convolutional layers in descending order of distance between the M convolutional layers and the output layer; or selecting m convolutional layers that perform the grouped convolutional operation from the M convolutional layers in descending order of complexity of the network structure.

[0125] In one embodiment, the encoding grouped convolution indication information is used to further indicate the number of groups for which the grouped convolution operation is to be performed in each of the m convolution layers, and the numbers of groups for which the grouped convolution operation is to be performed in different convolution layers among the m convolution layers are the same or different.

[0126] In one embodiment, the encoding simplification operation further includes a channel reconstruction operation, and the encoding instruction information includes encoding channel reconstruction instruction information, which is used to indicate performing the channel reconstruction operation in at least one convolution layer of the encoding neural network. The channel reconstruction operation includes sorting grouped convolutional features of each group obtained by the grouped convolution operation to obtain sorted grouped convolutional features.

[0127] In one embodiment, the encoding channel reconstruction indication information is set in the network structure of the convolution layer of the encoding neural network.

[0128] In one embodiment, the coding channel reconstruction indication information is used to further indicate the channel number and / or channel identifier of the channel reconstruction operation performed in the convolutional layer of the coding neural network.

[0129] In one embodiment, the number of channels for the channel reconstruction operation indicated by the encoding channel reconstruction indication information supports dynamic adjustment in response to changes in network parameters of the convolutional layer of the encoding neural network.

[0130] In one embodiment, one convolutional layer of the at least one convolutional layer includes I channels, and the coded channel reconstruction indication information is used to further indicate O channels among the I channels on which to perform the channel reconstruction operation, where O is a positive integer and O is less than or equal to I.

[0131] In one embodiment, the selection rule for the O channels includes at least one of selecting O channels from the I channels to perform a channel operation according to a rule that reduces memory resource consumption, or selecting O channels from the I channels to perform a channel reconstruction operation according to data communication requirements between each group, where the O channels respectively correspond to multiple groups of a grouped convolution operation.

[0132] In one embodiment, the number of convolutional layers is M, and the M convolutional layers include m convolutional layers that perform grouped convolution, where M and m are both positive integers and m is less than or equal to M, and the coded channel reconstruction indication information is used to indicate q convolutional layers among the m convolutional layers that perform channel reconstruction operations, where q is a positive integer and q is less than or equal to m.

[0133] In one embodiment, the coding grouping convolution indication information is used to further indicate the number of channels for performing the channel reconstruction operation in each of the q convolution layers, and the number of channels for performing the channel reconstruction operation in different of the q convolution layers may be the same or different.

[0134] In one embodiment, the coding neural network includes at least one of an analysis-transformation subnetwork and a hyper-coding subnetwork. The coding simplification operation includes a grouping convolution operation, and when the coding instruction information indicates that all convolutional layers of each subnetwork perform the grouping convolution operation, the number of groups for which the grouping convolution operation is performed in the convolutional layers of each subnetwork indicated by the coding instruction information is the same or different. The coding simplification operation further includes a channel reconstruction operation, and when the coding instruction information indicates that the convolutional layers of each subnetwork perform the channel reconstruction operation, the number of channels for which the channel reconstruction operation is performed in the convolutional layers of each subnetwork indicated by the coding instruction information is the same or different.

[0135] In an embodiment of the present invention, an image to be processed is acquired, and encoding instruction information is provided on the encoding device side, which indicates a simplified encoding operation to be performed in the encoding neural network.The encoding instruction information is then used to call the encoding neural network to perform an encoding process on the image and reconstruct the image.When encoding the image, the instructions in the encoding instruction information can simplify the operations performed by the encoding neural network on the encoding side, thereby effectively reducing the complexity of the encoding on the encoding side and improving the encoding efficiency.

[0136] An image processing apparatus according to an embodiment of the present invention will be described below.

[0137] 6 is a schematic diagram of the configuration of an image processing device according to one exemplary embodiment of the present invention. The image processing device may be a computer program (including program code) in a computing device, for example, application software in a computing device. The information processing device may be used to perform some or all of the steps in the method embodiment shown in FIG. 4. As shown in FIG. 6, the image processing device includes the following components:

[0138] The acquisition unit 601 acquires a bitstream formed after an image has been encoded.

[0139] The acquisition unit 601 acquires decoding instruction information, which is used to indicate a decoding simplification operation to be performed in a decoding neural network, and the decoding simplification operation is used to perform a grouping process on convolution operations to be performed in at least one convolution layer included in the decoding neural network.

[0140] The processing unit 602 performs a decoding simplification operation to invoke a decoding neural network to perform decoding processing on the bitstream according to the decoding instruction information to reconstruct an image.

[0141] In one embodiment, the decoding simplification operation includes a grouped convolution operation, and the decoding instruction information includes decoding grouped convolution instruction information, which is used to indicate performing a grouped convolution operation in at least one convolution layer of the decoding neural network. The grouped convolution operation includes: grouping feature maps of an image by channel in the at least one convolution layer to obtain multiple groups; and performing a convolution operation within each group of the multiple groups to obtain grouped convolution features for each group, where the grouped convolution features for each group include convolution features of channels in the group. The output channels of each group of the multiple groups interact only with the input channels in the group and are not related to channels in other groups of the multiple groups.

[0142] In one embodiment, the decoding grouping convolution instruction information is set in the network structure of the convolution layer of the decoding neural network.

[0143] In one embodiment, the decoding grouped convolution indication information is used to further indicate the number of groups in the grouped convolution operation.

[0144] In one embodiment, the number of groups of the grouped convolution operation indicated by the decoding grouped convolution indication information supports dynamic adjustment according to changes in network parameters of the convolution layer of the decoding neural network.

[0145] In one embodiment, the number of at least one convolutional layer of the decoding neural network is N, where N is a positive integer, and the decoding grouping convolution indication information is used to further indicate n convolutional layers among the N convolutional layers that perform the grouping convolution operation, where n is a positive integer and n is less than or equal to N.

[0146] In one embodiment, the selection rule for the n convolutional layers includes at least one of: selecting n convolutional layers that perform the grouped convolutional operation from the N convolutional layers in descending order of distance between the N convolutional layers and the output layer; or selecting n convolutional layers that perform the grouped convolutional operation from the N convolutional layers in descending order of complexity of the network structure.

[0147] In one embodiment, the decoding grouped convolution indication information is used to further indicate the number of groups for performing the grouped convolution operation in each convolution layer among the n convolution layers, and the numbers of groups for performing the grouped convolution operation in different convolution layers among the n convolution layers are the same or different.

[0148] In one embodiment, the decoding simplification operation further includes a channel reconstruction operation, and the decoding instruction information includes decoding channel reconstruction instruction information, which is used to indicate performing the channel reconstruction operation in at least one convolution layer of the decoding neural network. The channel reconstruction operation includes sorting grouped convolutional features of each group obtained by the grouped convolution operation to obtain sorted grouped convolutional features.

[0149] In one embodiment, the decoding channel reconstruction indication information is set in the network structure of the convolution layer of the decoding neural network.

[0150] In one embodiment, the decoded channel reconfiguration indication information is used to further indicate the channel number and / or channel identifier of the channel reconfiguration operation.

[0151] In one embodiment, the number of channels for the channel reconstruction operation indicated by the decoding channel reconstruction indication information supports dynamic adjustment in response to changes in network parameters of the convolutional layer of the decoding neural network.

[0152] In one embodiment, one convolutional layer of the at least one convolutional layer includes P channels, and the decoding channel reconstruction indication information is used to further indicate p channels among the P channels on which to perform the channel reconstruction operation, where p is a positive integer and p is less than or equal to P.

[0153] In one embodiment, the selection rule for the p channels includes at least one of selecting p channels from the P channels to perform a channel operation according to a rule that reduces memory resource consumption, or selecting p channels from the P channels to perform a channel reconstruction operation according to data communication requirements between each group, where the P channels respectively correspond to multiple groups of the grouped convolution operation.

[0154] In one embodiment, the number of at least one convolutional layer of the decoding neural network is N, and the N convolutional layers include n convolutional layers that perform a grouped convolution operation, where N and n are both positive integers and n is less than or equal to N. The decoding channel reconstruction indication information is used to further indicate t convolutional layers that perform a channel reconstruction operation among the n convolutional layers, where t is a positive integer and t is less than or equal to n.

[0155] In one embodiment, the decoding channel reconstruction indication information is used to further indicate the number of channels on which the channel reconstruction operation is performed in each of the t convolutional layers, and the number of channels on which the channel reconstruction operation is performed in different of the t convolutional layers is the same or different.

[0156] In one embodiment, the decoding neural network includes at least one of a hyper-decoding subnetwork, a hyper-scale decoding subnetwork, and a synthesis transformation subnetwork, each subnetwork including a convolutional layer. The decoding simplification operation includes a grouping convolutional operation, and when the decoding instruction information indicates that all convolutional layers of each subnetwork perform the grouping convolutional operation, the number of groups for which the grouping convolutional operation is performed in the convolutional layer of each subnetwork indicated by the decoding instruction information is the same or different. The decoding simplification operation further includes a channel reconstruction operation, and when the decoding instruction information indicates that the convolutional layer of each subnetwork performs a channel reconstruction operation, the number of channels for which the channel reconstruction operation is performed in the convolutional layer of each subnetwork indicated by the decoding instruction information is the same or different.

[0157] In one embodiment, the decoding neural network includes a convolution layer, and the decoding simplification operation includes a grouped convolution operation and a channel reconstruction operation. The processing unit 602 calls the decoding neural network according to the decoding instruction information to perform a decoding process on the bitstream to obtain transformed values ​​of an image, calls at least one convolution layer to perform a grouped convolution operation on the transformed values ​​of the image to obtain grouped convolution results corresponding to the transformed values ​​of the image, the transformed values ​​of the image being included in a feature map of the image, performs a channel reconstruction operation on the grouped convolution results, obtains the channel reconstruction result, and performs a decoding process on the channel reconstruction result to reconstruct the image.

[0158] In an embodiment of the present invention, a bitstream formed after an image is encoded is obtained, and decoding instruction information is obtained, which is used to indicate a simplified decoding operation to be performed in a decoding neural network, and the decoding neural network is called according to the decoding instruction information to perform a decoding process on the bitstream to reconstruct the image. When decoding the image, the instructions in the decoding instruction information can simplify the operation when performing decoding in the decoding neural network on the decoding side, thereby reducing the complexity of decoding on the decoding side and improving the decoding efficiency.

[0159] 7 is a schematic diagram of the configuration of an image processing device according to one exemplary embodiment of the present invention. The image processing device may be a computer program (including program code) in a computing device, for example, application software in a computing device. The information processing device may be used to perform some or all of the steps in the method embodiment shown in FIG. 4. As shown in FIG. 7, the image processing device includes the following components:

[0160] The acquisition unit 701 acquires an image to be processed.

[0161] The acquisition unit 701 acquires encoding instruction information, which is used to indicate an encoding simplification operation to be performed in an encoding neural network, and the encoding simplification operation is used to perform a grouping process on a convolution operation to be performed in at least one convolution layer included in the encoding neural network.

[0162] The processing unit 702 calls the encoding neural network according to the encoding instruction information, performs an encoding simplification operation to perform encoding processing on the image, and obtains a bitstream.

[0163] In one embodiment, the encoding simplification operation includes a grouped convolution operation. The encoding instruction information includes encoding grouped convolution instruction information, which indicates performing a grouped convolution operation in at least one convolution layer of the encoding neural network. The grouped convolution operation includes grouping feature maps of an image by channel in the at least one convolution layer to obtain multiple groups, and performing a convolution operation within each group of the multiple groups to obtain grouped convolution features for each group, where the grouped convolution features for each group include convolution features of channels in the group. The output channels of each group of the multiple groups interact only with the input channels in the group and are not related to channels in other groups of the multiple groups.

[0164] In one embodiment, the encoding grouped convolution indication information is used to further indicate the number of groups of the grouped convolution operation performed in the convolution layer of the encoding neural network.

[0165] In one embodiment, the number of groups of the grouped convolution operation indicated by the encoding grouped convolution indication information supports dynamic adjustment in response to changes in network parameters of the convolution layer of the encoding neural network.

[0166] In one embodiment, the number of the at least one convolutional layer is M, where M is a positive integer, and the encoding grouped convolution indication information is used to further indicate m convolutional layers that perform the grouped convolution operation among the M convolutional layers, where m is a positive integer and m is less than or equal to M.

[0167] In one embodiment, the selection rule for the m convolutional layers includes at least one of: selecting m convolutional layers that perform the grouped convolutional operation from the M convolutional layers in descending order of distance between the M convolutional layers and the output layer; or selecting m convolutional layers that perform the grouped convolutional operation from the M convolutional layers in descending order of complexity of the network structure.

[0168] In one embodiment, the encoding grouped convolution indication information is used to further indicate the number of groups for which the grouped convolution operation is to be performed in each of the m convolution layers, and the numbers of groups for which the grouped convolution operation is to be performed in different convolution layers among the m convolution layers are the same or different.

[0169] In one embodiment, the encoding simplification operation further includes a channel reconstruction operation, and the encoding instruction information includes encoding channel reconstruction instruction information, which is used to indicate performing the channel reconstruction operation in at least one convolution layer of the encoding neural network. The channel reconstruction operation includes sorting grouped convolutional features of each group obtained by the grouped convolution operation to obtain sorted grouped convolutional features.

[0170] In one embodiment, the encoding channel reconstruction indication information is set in the network structure of the convolution layer of the encoding neural network.

[0171] In one embodiment, the coding channel reconstruction indication information is used to further indicate the channel number and / or channel identifier of the channel reconstruction operation performed in the convolutional layer of the coding neural network.

[0172] In one embodiment, the number of channels for the channel reconstruction operation indicated by the encoding channel reconstruction indication information supports dynamic adjustment in response to changes in network parameters of the convolutional layer of the encoding neural network.

[0173] In one embodiment, one convolutional layer of the at least one convolutional layer includes I channels, and the coded channel reconstruction indication information is used to further indicate O channels among the I channels on which to perform the channel reconstruction operation, where O is a positive integer and O is less than or equal to I.

[0174] In one embodiment, the selection rule for the O channels includes at least one of selecting O channels from the I channels to perform a channel operation according to a rule that reduces memory resource consumption, or selecting O channels from the I channels to perform a channel reconstruction operation according to data communication requirements between each group, where the O channels respectively correspond to multiple groups of a grouped convolution operation.

[0175] In one embodiment, the number of convolutional layers is M, and the M convolutional layers include m convolutional layers that perform grouped convolution, where M and m are both positive integers and m is less than or equal to M, and the coded channel reconstruction indication information is used to indicate q convolutional layers among the m convolutional layers that perform channel reconstruction operations, where q is a positive integer and q is less than or equal to m.

[0176] In one embodiment, the coding grouping convolution indication information is used to further indicate the number of channels for performing the channel reconstruction operation in each of the q convolution layers, and the number of channels for performing the channel reconstruction operation in different of the q convolution layers may be the same or different.

[0177] In one embodiment, the coding neural network includes at least one of an analysis-transformation subnetwork and a hyper-coding subnetwork. The coding simplification operation includes a grouping convolution operation, and when the coding instruction information indicates that all convolutional layers of each subnetwork perform the grouping convolution operation, the number of groups for which the grouping convolution operation is performed in the convolutional layers of each subnetwork indicated by the coding instruction information is the same or different. The coding simplification operation further includes a channel reconstruction operation, and when the coding instruction information indicates that the convolutional layers of each subnetwork perform the channel reconstruction operation, the number of channels for which the channel reconstruction operation is performed in the convolutional layers of each subnetwork indicated by the coding instruction information is the same or different.

[0178] In an embodiment of the present invention, an image to be processed is acquired, and encoding instruction information is provided on the encoding device side, which indicates a simplified encoding operation to be performed in the encoding neural network.The encoding instruction information is then used to call the encoding neural network to perform an encoding process on the image and reconstruct the image.When encoding the image, the instructions in the encoding instruction information can simplify the operations performed by the encoding neural network on the encoding side, thereby effectively reducing the complexity of the encoding on the encoding side and improving the encoding efficiency.

[0179] The following describes a computer device according to an embodiment of the present invention.

[0180] 8 is a schematic diagram of a computer device according to one exemplary embodiment of the present invention. The computer device may include a processor 801, an input unit 802, an output unit 803, and a memory 804. The processor 801, the input unit 802, the output unit 803, and the memory 804 are connected via a bus. The memory 804 is used to store a computer program including program instructions. The processor 801 is used to execute the program instructions stored in the memory 804.

[0181] If the computer device is the decoding device described above, the processor 801 performs the following operations by executing program instructions in the memory 804:

[0182] A bitstream formed after the image has been encoded is obtained.

[0183] Decoding instruction information is obtained, the decoding instruction information is used to indicate a decoding simplification operation to be performed in the decoding neural network, and the decoding simplification operation is used to perform a grouping process on the convolution operations to be performed in at least one convolution layer included in the decoding neural network.

[0184] To reconstruct an image, a decoding simplification operation is performed to invoke a decoding neural network to perform a decoding process on the bitstream according to the decoding instruction information.

[0185] In one embodiment, the decoding simplification operation includes a grouped convolution operation, and the decoding instruction information includes decoding grouped convolution instruction information, which is used to indicate performing a grouped convolution operation in at least one convolution layer of the decoding neural network. The grouped convolution operation includes: grouping feature maps of an image by channel in the at least one convolution layer to obtain multiple groups; and performing a convolution operation within each group of the multiple groups to obtain grouped convolution features for each group, where the grouped convolution features for each group include convolution features of channels in the group. The output channels of each group of the multiple groups interact only with the input channels in the group and are not related to channels in other groups of the multiple groups.

[0186] In one embodiment, the decoding grouping convolution instruction information is set in the network structure of the convolution layer of the decoding neural network.

[0187] In one embodiment, the decoding grouped convolution indication information is used to further indicate the number of groups in the grouped convolution operation.

[0188] In one embodiment, the number of groups of the grouped convolution operation indicated by the decoding grouped convolution indication information supports dynamic adjustment according to changes in network parameters of the convolution layer of the decoding neural network.

[0189] In one embodiment, the number of at least one convolutional layer of the decoding neural network is N, where N is a positive integer, and the decoding grouping convolution indication information is used to further indicate n convolutional layers among the N convolutional layers that perform the grouping convolution operation, where n is a positive integer and n is less than or equal to N.

[0190] In one embodiment, the selection rule for the n convolutional layers includes at least one of: selecting n convolutional layers that perform the grouped convolutional operation from the N convolutional layers in descending order of distance between the N convolutional layers and the output layer; or selecting n convolutional layers that perform the grouped convolutional operation from the N convolutional layers in descending order of complexity of the network structure.

[0191] In one embodiment, the decoding grouped convolution indication information is used to further indicate the number of groups for performing the grouped convolution operation in each convolution layer among the n convolution layers, and the numbers of groups for performing the grouped convolution operation in different convolution layers among the n convolution layers are the same or different.

[0192] In one embodiment, the decoding simplification operation further includes a channel reconstruction operation, and the decoding instruction information includes decoding channel reconstruction instruction information, which is used to indicate performing the channel reconstruction operation in at least one convolution layer of the decoding neural network. The channel reconstruction operation includes sorting grouped convolutional features of each group obtained by the grouped convolution operation to obtain sorted grouped convolutional features.

[0193] In one embodiment, the decoding channel reconstruction indication information is set in the network structure of the convolution layer of the decoding neural network.

[0194] In one embodiment, the decoded channel reconfiguration indication information is used to further indicate the channel number and / or channel identifier of the channel reconfiguration operation.

[0195] In one embodiment, the number of channels for the channel reconstruction operation indicated by the decoding channel reconstruction indication information supports dynamic adjustment in response to changes in network parameters of the convolutional layer of the decoding neural network.

[0196] In one embodiment, one convolutional layer of the at least one convolutional layer includes P channels, and the decoding channel reconstruction indication information is used to further indicate p channels among the P channels on which to perform the channel reconstruction operation, where p is a positive integer and p is less than or equal to P.

[0197] In one embodiment, the selection rule for the p channels includes at least one of selecting p channels from the P channels to perform a channel operation according to a rule that reduces memory resource consumption, or selecting p channels from the P channels to perform a channel reconstruction operation according to data communication requirements between each group, where the P channels respectively correspond to multiple groups of the grouped convolution operation.

[0198] In one embodiment, the number of at least one convolutional layer of the decoding neural network is N, and the N convolutional layers include n convolutional layers that perform a grouped convolution operation, where N and n are both positive integers and n is less than or equal to N. The decoding channel reconstruction indication information is used to further indicate t convolutional layers that perform a channel reconstruction operation among the n convolutional layers, where t is a positive integer and t is less than or equal to n.

[0199] In one embodiment, the decoding channel reconstruction indication information is used to further indicate the number of channels on which the channel reconstruction operation is performed in each of the t convolutional layers, and the number of channels on which the channel reconstruction operation is performed in different of the t convolutional layers is the same or different.

[0200] In one embodiment, the decoding neural network includes at least one of a hyper-decoding subnetwork, a hyper-scale decoding subnetwork, and a synthesis transformation subnetwork, each subnetwork including a convolutional layer. The decoding simplification operation includes a grouping convolutional operation, and when the decoding instruction information indicates that all convolutional layers of each subnetwork perform the grouping convolutional operation, the number of groups for which the grouping convolutional operation is performed in the convolutional layer of each subnetwork indicated by the decoding instruction information is the same or different. The decoding simplification operation further includes a channel reconstruction operation, and when the decoding instruction information indicates that the convolutional layer of each subnetwork performs a channel reconstruction operation, the number of channels for which the channel reconstruction operation is performed in the convolutional layer of each subnetwork indicated by the decoding instruction information is the same or different.

[0201] In one embodiment, the decoding neural network includes at least one convolution layer, and the decoding simplification operation includes a grouping convolution operation and a channel reconstruction operation. The processor 801 may perform the following steps when performing the decoding simplification operation to call the decoding neural network to perform a decoding process on the bitstream according to the decoding instruction information to reconstruct an image:

[0202] A decoding neural network is called according to the decoding instruction information to perform decoding processing on the bitstream and obtain the transformation value of the image.

[0203] At least one convolution layer is invoked to perform a grouped convolution operation on the transformed values ​​of the image, and a grouped convolution result corresponding to the transformed values ​​of the image is obtained, and the transformed values ​​of the image are included in a feature map of the image.

[0204] A channel reconstruction operation is performed on the grouped convolution result to obtain a channel reconstruction result.

[0205] A decoding process is performed on the channel reconstruction result to reconstruct the image.

[0206] In an embodiment of the present invention, a bitstream formed after an image is encoded is obtained, and decoding instruction information is obtained, which is used to indicate a simplified decoding operation to be performed in a decoding neural network, and the decoding neural network is called according to the decoding instruction information to perform a decoding process on the bitstream to reconstruct the image. When decoding the image, the instructions in the decoding instruction information can simplify the operation when performing decoding in the decoding neural network on the decoding side, thereby reducing the complexity of decoding on the decoding side and improving the decoding efficiency.

[0207] Preferably, when the computer device is the coding device described above, the processor 801 performs the following operations by executing program instructions in the memory 804:

[0208] Acquire the image to be processed.

[0209] Obtain encoding instruction information, which is used to indicate an encoding simplification operation to be performed in the encoding neural network, and the encoding simplification operation is used to perform a grouping process on the convolution operations to be performed in at least one convolution layer included in the encoding neural network.

[0210] According to the encoding instruction information, the encoding neural network is called to perform an encoding simplification operation to perform an encoding process on the image, and a bitstream is obtained.

[0211] In an embodiment of the present invention, an image to be processed is acquired, and encoding instruction information is provided on the encoding device side, which indicates a simplified encoding operation to be performed in the encoding neural network.The encoding instruction information is then used to call the encoding neural network to perform an encoding process on the image and reconstruct the image.When encoding the image, the instructions in the encoding instruction information can simplify the operations performed by the encoding neural network on the encoding side, thereby effectively reducing the complexity of the encoding on the encoding side and improving the encoding efficiency.

[0212] In addition, an embodiment of the present invention further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, which includes program instructions. When a processor executes the program instructions, the processor can perform the method according to the embodiment corresponding to FIGS. 4 and 5. Here, a description thereof is omitted. For technical details not disclosed in the embodiment of the computer-readable storage medium of the present application, reference may be made to the description of the embodiment of the method of the present invention. For example, the program instructions may be deployed in one computer device, or may be executed by multiple computer devices located at one location, or may be executed by multiple computer devices distributed at multiple locations and interconnected via a communication network.

[0213] In one aspect of the present application, a computer program product is provided, including a computer program stored in a computer-readable storage medium. A processor of the computer reads the computer program from the computer-readable storage medium and executes the computer program, so that the computer can execute the method according to the embodiment corresponding to Figures 4 and 5. The description thereof is omitted here.

[0214] Those skilled in the art will understand that implementing all or part of the method flow of the above embodiments can be performed by a computer program that gives instructions to related hardware, and the program can include the flow such as the above method embodiments when executed. The above storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), etc.

[0215] The above disclosures are merely preferred embodiments of the present invention, and the scope of the present invention is not limited to these embodiments, and equivalent modifications according to the claims of this application are included in the protection scope of the present invention.

Claims

1. 1. An image processing method, comprising: obtaining a bitstream formed after the image has been encoded; obtaining decoding instruction information, the decoding instruction information being used to indicate a decoding simplification operation to be performed in a decoding neural network, the decoding simplification operation being used to perform a grouping process on convolution operations to be performed in at least one convolution layer included in the decoding neural network; and performing the decoding simplification operation to invoke the decoding neural network to perform a decoding process on the bitstream in accordance with the decoding instruction information to reconstruct the image.

2. the decoding simplification operation includes a grouped convolution operation; the decoding instruction information includes decoding group convolution instruction information; the decoding grouped convolution instruction information is used to indicate that a grouped convolution operation is to be performed in at least one convolution layer of the decoding neural network; The grouping convolution operation comprises: Grouping the feature maps of the image by channel in the at least one convolutional layer to obtain a plurality of groups; 2. The method of claim 1, comprising: performing a convolution operation within each group of the plurality of groups to obtain a grouped convolution feature for each group, wherein the grouped convolution feature for each group includes convolution features of channels in the group.

3. 3. The method of claim 2, wherein the output channels of each group of the plurality of groups interact only with the input channels in that group and are unrelated to channels in other groups of the plurality of groups.

4. The method of claim 2 , wherein the decoding grouped convolution indication information is used to further indicate the number of groups of the grouped convolution operation.

5. The method of claim 4 , wherein the number of groups of the grouped convolution operation indicated by the decoding grouped convolution indication information supports dynamic adjustment in response to changes in network parameters of a convolution layer of the decoding neural network.

6. The number of the at least one convolutional layer of the decoding neural network is N, where N is a positive integer; 3. The method of claim 2, wherein the decoding grouped convolution indication information is used to further indicate n convolution layers among N convolution layers that perform a grouped convolution operation, where n is a positive integer and n is less than or equal to N.

7. The selection rule for the n convolution layers is: Selecting n convolution layers to perform a grouping convolution operation from the N convolution layers in descending order of distance between the N convolution layers and an output layer; or 7. The method of claim 6, further comprising at least one of selecting n convolutional layers that perform grouped convolutional operations from the N convolutional layers in order of increasing complexity of a network structure.

8. the decoding grouped convolution indication information is used to further indicate the number of groups for performing a grouped convolution operation in each of the n convolution layers; The method of claim 6 , wherein the number of groups for performing the grouped convolution operation in different convolution layers among the n convolution layers is the same or different.

9. the decoding simplification operation further comprises a channel reconstruction operation; the decoding instruction information includes decoding channel reconstruction instruction information; The decoding channel reconstruction indication information is used to indicate that a channel reconstruction operation is to be performed in at least one convolutional layer of the decoding neural network; The channel reconfiguration operation includes: The method of claim 2 , further comprising: sorting the grouped convolution features of each group obtained by the grouped convolution operation to obtain sorted grouped convolution features.

10. The method according to claim 9 , wherein the decoding grouping convolution instruction information and / or the decoding channel reconstruction instruction information are set in a network structure of a convolution layer of the decoding neural network.

11. The method of claim 9 , wherein the decoded channel reconfiguration indication information is used to further indicate a channel number and / or a channel identifier of the channel reconfiguration operation.

12. The method of claim 11 , wherein the number of channels of the channel reconstruction operation indicated by the decoding channel reconstruction indication information supports dynamic adjustment according to changes in network parameters of a convolutional layer of the decoding neural network.

13. one convolutional layer of the at least one convolutional layer includes P channels; 10. The method of claim 9, wherein the decoding channel reconstruction indication information is used to further indicate p channels among the P channels on which to perform a channel reconstruction operation, where p is a positive integer and p is less than or equal to P.

14. The selection rule for the p channels is selecting p channels from the P channels to perform channel operations according to a rule that reduces memory resource consumption; or 14. The method of claim 13, wherein the P channels respectively correspond to a plurality of groups of the grouped convolution operation, and the method includes at least one of selecting p channels from the P channels on which to perform a channel reconstruction operation according to a data communication requirement between each group.

15. The number of the at least one convolutional layer of the decoding neural network is N, and the N convolutional layers include n convolutional layers that perform grouped convolution operations, where N and n are both positive integers, and n is less than or equal to N; 10. The method of claim 9, wherein the decoding channel reconstruction indication information is used to further indicate t convolutional layers among n convolutional layers that perform a channel reconstruction operation, where t is a positive integer and t is less than or equal to n.

16. the decoding channel reconstruction indication information is used to further indicate the number of channels on which a channel reconstruction operation is performed in each of the t convolutional layers; and The method of claim 15 , wherein the number of channels for which channel reconstruction operations are performed in different convolutional layers among the t convolutional layers is the same or different.

17. the decoding neural network includes at least one of a hyper-decoding sub-network, a hyper-scale decoding sub-network, and a synthesis transformation sub-network, each sub-network including a convolutional layer; the decoding simplification operation includes a grouped convolution operation, and when the decoding instruction information indicates that all convolution layers of each sub-network perform the grouped convolution operation, the number of groups that perform the grouped convolution operation in the convolution layers of each sub-network indicated by the decoding instruction information is the same or different; 2. The method of claim 1 , wherein the decoding simplification operation further includes a channel reconstruction operation, and when the decoding instruction information indicates that a convolutional layer of each sub-network performs a channel reconstruction operation, the number of channels for which the channel reconstruction operation is performed in the convolutional layer of each sub-network indicated by the decoding instruction information is the same or different.

18. performing the decoding simplification operation to call the decoding neural network to perform a decoding process on the bitstream in accordance with the decoding instruction information to reconstruct the image, calling the decoding neural network according to the decoding instruction information to perform a decoding process on the bitstream and obtain a transformation value of the image; Invoking the at least one convolution layer to perform a grouped convolution operation on a transformation value of the image to obtain a grouped convolution result corresponding to the transformation value of the image, wherein the transformation value of the image is included in a feature map of the image; performing a channel reconstruction operation on the grouped convolution result to obtain a channel reconstruction result; and performing a decoding process on the channel reconstruction result to reconstruct the image.

19. 1. An image processing method, comprising: acquiring an image to be processed; obtaining encoding instruction information, the encoding instruction information being used to indicate an encoding simplification operation to be performed in an encoding neural network, the encoding simplification operation being used to perform a grouping process on convolution operations to be performed in at least one convolution layer included in the encoding neural network; and calling the encoding neural network according to the encoding instruction information to perform the encoding simplification operation to perform an encoding process on the image, thereby obtaining a bitstream.

20. An image processing device including an acquisition unit and a processing unit, the acquisition unit acquires a bitstream formed after an image has been encoded; the acquisition unit acquires decoding instruction information, the decoding instruction information is used to indicate a decoding simplification operation to be executed in a decoding neural network, and the decoding simplification operation is used to perform a grouping process on convolution operations executed in at least one convolution layer included in the decoding neural network; The processing unit performs the decoding simplification operation to invoke the decoding neural network to perform a decoding process on the bitstream in accordance with the decoding instruction information to reconstruct the image.

21. An image processing device including an acquisition unit and a processing unit, the acquisition unit acquires an image to be processed; the acquiring unit acquires encoding instruction information, the encoding instruction information is used to indicate an encoding simplification operation to be executed in an encoding neural network, and the encoding simplification operation is used to perform a grouping process on convolution operations executed in at least one convolution layer included in the encoding neural network; The processing unit performs the encoding simplification operation to call the encoding neural network according to the encoding instruction information to perform an encoding process on the image and obtain a bitstream.

22. a processor for executing a computer program; and a computer readable storage medium having stored thereon a computer program, the computer program being adapted to perform the image processing method of any one of claims 1 to 19 when executed by the processor.

23. A computer program causing a computer to execute the image processing method according to any one of claims 1 to 19.

Images