Image processing methods and apparatuses, computer device and storage medium
By using decoding instructions information, packet convolution and channel reorganization operations in the decoding neural network, the problem of high decoding complexity in the prior art is solved, and efficient image decoding on mobile devices is realized.
Patent Information
- Application Number
- PCT/CN2024/094474
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-24
- Filing Date
- 2024-05-21
- Publication Date
- 2024-11-28
AI Technical Summary
In the existing end-to-end image compression neural network model, the decoding complexity and low decoding efficiency are high, and it cannot be effectively supported on mobile devices.
By providing decoding instructions, operations in decoding neural networks are simplified, packet convolution and channel recombination operations are adopted to reduce decoding complexity and improve decoding efficiency.
Effectively reduce the complexity of the decoding end, improve the decoding efficiency, and enable image processing to be efficiently supported on mobile devices.
Smart Images

Figure CN2024094474_28112024_PF_FP_ABST
Abstract
Description
Image processing method, device, computer equipment and storage medium
[0001] This application claims priority to the Chinese patent application with application number 2023106009704 filed on May 24, 2023, and invention name “An image processing method and related equipment”. Technical Field
[0002] The present application relates to the field of image processing technology, and in particular to an image processing method, an image processing apparatus, a computer device, and a computer-readable storage medium. Background Art
[0003] In an end-to-end image compression neural network model, the encoder maps the original image into latent variables using an analysis transform sub-network, and writes these latent variables into the bitstream using an entropy encoder. The decoder decodes the latent variables from the bitstream using an entropy decoder, and then inputs these latent variables into a synthesis transform sub-network to reconstruct the image. However, current image encoding and decoding solutions suffer from high decoding complexity, low decoding efficiency, and poor support for mobile devices.
[0004] Summary of the Invention
[0005] The embodiments of the present application provide an image processing method and related devices, which can reduce the decoding complexity of the decoding end and improve the decoding efficiency.
[0006] In one aspect, an embodiment of the present application provides an image processing method, the method comprising:
[0007] Obtaining a code stream formed after the image is encoded;
[0008] Obtaining decoding indication information, where the decoding indication information is used to indicate a decoding simplification operation to be performed in the decoding neural network, where the decoding simplification operation is used to group convolution operations performed by at least one convolutional layer included in the decoding neural network;
[0009] According to the decoding instruction information, the decoding neural network is called to perform decoding simplification operations to decode the code stream to reconstruct the image.
[0010] In one aspect, an embodiment of the present application provides an image processing method, the method comprising:
[0011] Get the image to be processed;
[0012] Obtaining encoding indication information, where the encoding indication information is used to indicate an encoding simplification operation to be performed in the encoding neural network, where the encoding simplification operation is used to group convolution operations performed by at least one convolutional layer included in the encoding neural network;
[0013] According to the encoding instruction information, the encoding neural network is called to perform encoding simplification operations to encode the image and obtain a code stream.
[0014] In one aspect, an embodiment of the present application provides an image processing device, comprising:
[0015] An acquisition unit, used for acquiring a code stream formed after the image is coded;
[0016] The acquisition unit is further configured to acquire decoding indication information, the decoding indication information being used to indicate a decoding simplification operation to be performed in the decoding neural network, the decoding simplification operation being used to group convolution operations performed by at least one convolutional layer included in the decoding neural network;
[0017] The processing unit is further configured to call a decoding neural network according to the decoding instruction information to perform a decoding simplification operation to decode the code stream to reconstruct the image.
[0018] In one aspect, an embodiment of the present application provides an image processing device, comprising:
[0019] an acquisition unit, configured to acquire an image to be processed;
[0020] The acquiring unit is further configured to acquire encoding indication information, the encoding indication information being used to indicate an encoding simplification operation to be performed in the encoding neural network, the encoding simplification operation being used to group convolution operations performed by at least one convolutional layer included in the encoding neural network;
[0021] The processing unit is used to call the encoding neural network according to the encoding instruction information to perform encoding simplification operations on the image to obtain a code stream.
[0022] In one aspect, an embodiment of the present application provides a computer device, comprising:
[0023] a processor suitable for executing a computer program;
[0024] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image processing method described above is implemented.
[0025] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is loaded by a processor and executes the above-mentioned image processing method.
[0026] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program or computer instructions, and when the computer program or computer instructions are executed by a processor, the above-mentioned image processing method is implemented.
[0027] In an embodiment of the present application, a code stream formed after image encoding is obtained, along with decoding instruction information, which indicates a decoding simplification operation to be performed in a decoding neural network. The decoding neural network is then invoked to decode the code stream according to the decoding instruction information to reconstruct the image. During the image decoding process, the decoding instruction information simplifies the decoding operations performed by the decoding neural network at the decoding end, thereby effectively reducing decoding complexity and improving decoding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0029] FIG1A is a structural diagram of an end-to-end image coding model based on deep learning provided by an exemplary embodiment of the present application;
[0030] FIG1B is a structural diagram of an end-to-end image coding model based on deep learning provided by another exemplary embodiment of the present application;
[0031] FIG2A is a schematic diagram showing a network structure comparison of a composition transformation sub-network and a lightweight composition transformation sub-network when processing chrominance components and luminance components, respectively, provided by an exemplary embodiment of the present application;
[0032] FIG2B is a schematic diagram showing a structural comparison of a residual block and a lightweight residual block provided by an exemplary embodiment of the present application;
[0033] FIG2C is a schematic diagram showing a comparison of the network structures of a super decoding sub-network and a lightweight super decoding sub-network when processing chrominance components and luminance components, respectively, according to an exemplary embodiment of the present application;
[0034] FIG2D is a schematic diagram showing a network structure comparison of a super-scale decoding sub-network and a lightweight super-scale decoding sub-network when processing chrominance components and luminance components, respectively, provided by an exemplary embodiment of the present application;
[0035] FIG2E is a schematic diagram comparing a conventional convolution operation and a grouped convolution operation provided by an exemplary embodiment of the present application;
[0036] FIG2F is a schematic diagram of a channel reorganization provided by an exemplary embodiment of the present application;
[0037] FIG3 is an architecture diagram of an image processing system provided by an exemplary embodiment of the present application;
[0038] FIG4 is a flow chart of an image processing method provided by an exemplary embodiment of the present application;
[0039] FIG5 is a flow chart of an image processing method provided by another exemplary embodiment of the present application;
[0040] FIG6 is a schematic structural diagram of an image processing device provided by an exemplary embodiment of the present application;
[0041] FIG7 is a schematic structural diagram of an image processing device provided by another exemplary embodiment of the present application;
[0042] FIG8 is a schematic structural diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0043] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0044] First, the relevant technical terms involved in this application are introduced.
[0045] 1. Image
[0046] An image can refer to an image to be encoded. The image can be any frame or multiple frames in a video. The image's pixel format can be RGB or YUV. When the image's pixel format is RGB, it includes an R (Red) component, a G (Green) component, and a B (Blue) component. When the image's pixel format is YUV, it includes a luminance (Y) component and a chrominance (UV) component.
[0047] 2. End-to-end image coding model based on deep learning
[0048] The embodiment of the present application relates to an end-to-end image coding model based on deep learning. Schematically, the end-to-end image coding model based on deep learning can be applied to the JPEG (Joint Photographic Experts Group) AI (Artificial Intelligence) platform. Schematically, the end-to-end image coding model based on deep learning can be a VM model (Verification Model). As shown in Figure 1A, a structural diagram of an end-to-end image coding model based on deep learning is provided for an exemplary embodiment of the present application. The end-to-end image coding model based on deep learning adopts a transform-coding structure. The end-to-end image coding model based on deep learning mainly includes: an analysis transform subnetwork (Analysis Transform Net), a context model subnetwork (Context Model Net), a hyper encoding subnetwork (Hyper Encoder Net), a prediction fusion subnetwork (Prediction Fusion Net), a hyper decoding subnetwork (Hyper Decoder Net), a hyper scale decoding subnetwork (Hyper Scale Decoder Net), and a synthesis transform subnetwork (Synthesis Transform Net). The above subnetworks are introduced in detail below:
[0049] ①Analysis transformation sub-network
[0050] The analysis transformation subnetwork is used to perform nonlinear transformation on the image x to be encoded, and obtain the transformation result y (or called latent variable) of the image x. Y and the chrominance component x UV When the analysis transformation sub-network can analyze the brightness component x Y and the chrominance component x UV Perform nonlinear transformation to obtain the brightness component x Y The corresponding transformation result y Y and the chrominance component x UV The corresponding transformation result y UV .
[0051] ②Context model sub-network
[0052] The context model subnetwork is used to transform the image x Perform context processing. The output data of the context model sub-network is used as the input of the prediction fusion sub-network. The transformed value of image x It is obtained by superimposing the predicted value u of the image x output by the prediction fusion sub-network and the data output by the residual inverse gain unit (invG unitRes, inverse Gain unit residual). The input data of the residual inverse gain unit is the image residual of the image x. The image residual The image residual can be obtained by quantizing the data output by the residual gain unit (G unitRes). Specifically, the data type output by the residual gain unit is a floating point type, and the quantization process includes a data type conversion process (i.e., rounding process), and the data output by the residual gain unit is rounded to obtain an integer image residual. The input data of the residual gain unit is the residual r obtained by performing a difference operation on the transformation result y obtained by the analysis transformation subnetwork of the image x and the predicted value u output by the prediction fusion subnetwork.
[0053] When the image x includes a luminance component x Y and the chrominance component x UV When the residual r of the brightness component Y Can be the brightness component x Y The transformed value y Y The predicted value u of the brightness component output by the prediction fusion sub-network Y The residual r of the chrominance component obtained by performing the difference operation UV Can be the chrominance component x UV The transformed value y UV The predicted value u of the chrominance component output by the prediction fusion sub-network UV Then, we can calculate r Y and r UV Quantize it and convert it into an integer and Perform operations to generate code streams.
[0054] ③ The prediction fusion sub-network is used to predict the image x and obtain the predicted value u of the image x. The input data of the prediction fusion sub-network is the output data of the context model sub-network and the output data of the super decoding sub-network. Schematically, the prediction fusion sub-network is used to predict the chrominance component and the luminance component of the image and obtain the predicted value u, including the predicted value u of the luminance component. Y , and the predicted value u of the chrominance component UV .
[0055] ③ The super-coding sub-network processes the transformation result y of image x, then quantizes the output data of the super-coding sub-network to obtain a quantized image result, and performs lossless encoding on the quantized result to obtain a bitstream. Quantization includes rounding. In this case, quantizing the output data of the super-coding sub-network to obtain a quantized image result may include rounding the output data of the super-coding sub-network to obtain a quantized image result.
[0056] ④The input data of the super decoding sub-network is the hyperparameter obtained based on the bit stream The output data of the super decoding sub-network is used as the input data of the prediction fusion sub-network.
[0057] ⑤Super-scale decoding sub-network is used based on hyperparameters Determine the mean to be 0 and the variance to be Gaussian distribution The synthesis transform subnetwork is used to transform the transform values obtained from the decoded bitstream Perform synthetic transformation processing to reconstruct the image Schematically, the synthesis transform subnetwork is used to transform the transform values obtained from the decoded bitstream The transformed value of the brightness component in Perform synthetic transformation processing and transform values obtained from code stream decoding The transformed value of the chrominance component in Perform synthetic transformation processing to obtain the brightness component of the reconstructed image and chrominance components Then we get the reconstructed image
[0058] In addition, the end-to-end image coding model based on deep learning can also be a lightweight VM model. As shown in Figure 1B, a structural diagram of an end-to-end image coding model based on deep learning is provided for another exemplary embodiment of the present application. As shown in Figure 1B, the end-to-end image coding model based on deep learning mainly includes a lightweight analysis transform subnetwork (Light weighted Analysis Transform Net), a hyper encoding subnetwork (Hyper Encoder Net), a lightweight hyper decoding subnetwork (Light weighted Hyper Decoder Net), a lightweight hyper scale decomposition subnetwork (Light weighted Hyper Scale Decoder Net), and a lightweight synthesis transform subnetwork (Light weighted Synthesis Transform Net).
[0059] The lightweight analysis transform subnetwork can have the same functionality as the analysis transform subnetwork, but the complexity of the lightweight analysis transform subnetwork is lower than that of the analysis transform subnetwork. The lightweight super-decoding subnetwork can have the same functionality as the super-decoding subnetwork, but the complexity of the lightweight super-decoding subnetwork is lower than that of the super-decoding subnetwork. The lightweight super-scale decoding subnetwork can have the same functionality as the super-scale decoding subnetwork, but the complexity of the lightweight super-scale decoding subnetwork is lower than that of the super-scale decoding subnetwork. The lightweight synthesis transform subnetwork can have the same functionality as the synthesis transform subnetwork, but the complexity of the lightweight synthesis transform subnetwork is lower than that of the synthesis transform subnetwork.
[0060] Next, the network structures of the sub-networks corresponding to FIG. 1A and FIG. 1B are compared as examples.
[0061] (1) Comparison of the network structure of the synthetic transformation subnetwork and the network structure of the lightweight synthetic transformation subnetwork.
[0062] The composite transform subnetwork can be used to perform composite transform processing on the transform values of the luma component and the transform values of the chroma components of an image. The lightweight composite transform subnetwork can be used to perform composite transform processing on the transform values of the luma component and the transform values of the chroma components of an image. Please refer to Figure 2A, which is a schematic diagram comparing the network structures of a composite transform subnetwork and a lightweight composite transform subnetwork, respectively, when processing chroma components and luma components, provided in an exemplary embodiment of the present application.
[0063] ① Comparison of the network structure of the synthesis transform subnetwork (Synthesis Transform Net_Y) when processing the luminance component and the network structure of the lightweight synthesis transform subnetwork (Light weighted Synthesis Transform Net_Y) when processing the luminance component.
[0064] As shown in 21 in Figure 2A, the synthetic transformation subnetwork when processing the luminance component mainly includes two residual blocks (ResBlock), four convolutional layers, four cropping layers (Crop), three residual nonlinear units (ResAU), and an attention module (residual non-local attention block, RNAB). Among them, the four convolutional layers are from top to bottom: DConv128×3×3S2 (DConv means deconvolution, the number of channels is 128, the convolution kernel size is 3×3, and S2 means the convolution step is 2), DConv128×3×3S2 (DConv means deconvolution, the number of channels is 128, the convolution kernel size is 3×3, and S2 means the convolution step is 2), DConv128×3×3S2 (DConv means deconvolution, the number of channels is 128, the convolution kernel size is 3×3, and S2 means the convolution step is 2), and DConv1×3×3S2 (DConv means deconvolution, the number of channels is 1, the convolution kernel size is 3×3, and S2 means the convolution step is 2).
[0065] As shown in 22 in Figure 2A, the lightweight synthesis transformation subnetwork when processing the brightness component mainly includes a lightweight residual block (LightResBlock), four convolutional layers, three cropping layers (Crop), three residual nonlinear units (ResAU) and 1 pixel reorganization (pixelshuffleS4). Among them, the four convolutional layers are from top to bottom: DConv96×4×4S2 (DConv means deconvolution, the number of channels is 96, the convolution kernel size is 4×4, and S2 means the convolution step is 2), DConv64×4×4S2 (DConv means deconvolution, the number of channels is 64, the convolution kernel size is 4×4, and S2 means the convolution step is 2), Conv64×3×3S1 (Conv means convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 means the convolution step is 1), and Conv16×1×1S1 (Conv means convolution, the number of channels is 16, the convolution kernel size is 1×1, and S1 means the convolution step is 1).
[0066] Comparing 21 and 22 in Figure 2A, we can see that when processing the luminance component, the lightweight synthesis transformer subnetwork's network structure requires only one lightweight residual block and no RNAB, while the synthesis transformer subnetwork requires two residual blocks but does require RNAB. Furthermore, the network parameters in the convolutional layers of the lightweight synthesis transformer subnetwork and the synthesis transformer subnetwork have also changed. This shows that the network structure of the lightweight synthesis transformer subnetwork is less complex than that of the synthesis transformer subnetwork.
[0067] ② Comparison of the network structure of the synthesis transform subnetwork (Synthesis Transform Net_UV) when processing chroma components and the network structure of the lightweight synthesis transform subnetwork (Light weighted Synthesis Transform Net_UV) when processing chroma components.
[0068] As shown in 23 in Figure 2A, the synthetic transformation subnetwork when processing the chrominance component mainly includes two residual blocks (ResBlock), four convolutional layers, four cropping layers (Crop), three residual nonlinear units (ResAU), and an attention module (RNAB). Among them, the four convolutional layers are from top to bottom: DConv64×3×3S2 (Dconv means deconvolution, the number of channels is 64, the convolution kernel size is 3×3, and S2 means the convolution step is 2), DConv64×3×3S2 (DConv means deconvolution, the number of channels is 64, the convolution kernel size is 3×3, and S2 means the convolution step is 2), DConv64×3×3S2 (DConv means deconvolution, the number of channels is 64, the convolution kernel size is 3×3, and S2 means the convolution step is 2), and DConv2×3×3S2 (DConv means deconvolution, the number of channels is 2, the convolution kernel size is 3×3, and S2 means the convolution step is 2).
[0069] As shown in 24 in Figure 2A, the lightweight synthesis transformation subnetwork when processing the chrominance component mainly includes a lightweight residual block (LightResBlock), four convolutional layers, three cropping layers (Crop), three residual nonlinear units (ResAU) and 1 pixel reorganization (pixelshuffleS4). Among them, the four convolutional layers are from top to bottom: DConv64×4×4S2 (Dconv means deconvolution, the number of channels is 64, the convolution kernel size is 4×4, and S2 means the convolution step is 2), DConv64×4×4S2 (Dconv means deconvolution, the number of channels is 64, the convolution kernel size is 4×4, and S2 means the convolution step is 2), Conv64×3×3S1 (Conv means convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 means the convolution step is 1), and Conv32×1×1S1 (Conv means convolution, the number of channels is 32, the convolution kernel size is 1×1, and S1 means the convolution step is 1).
[0070] Comparing 23 and 24 in Figure 2A, we can see that when processing chrominance components, the lightweight synthesis transformer subnetwork's network structure requires only one lightweight residual block and no RNAB, while the synthesis transformer subnetwork requires two residual blocks but does require RNAB. Furthermore, the network parameters in the convolutional layers of the lightweight synthesis transformer subnetwork and the synthesis transformer subnetwork have also changed. This shows that the network structure of the lightweight synthesis transformer subnetwork is less complex than that of the synthesis transformer subnetwork.
[0071] There are also differences between the structure of the above-mentioned residual block and the structure of the lightweight residual block. Please refer to Figure 2B, which is a schematic diagram of the structural comparison of a residual block and a lightweight residual block provided for an exemplary embodiment of the present application. In Figure 2B, the residual block may include an input layer (such as an input image x), two convolutional layers (Conv3×3S1G1 and Conv3×3S1G1, respectively, where the number of channels is 3×3, S1 indicates a step size of 1, and G1 indicates the number of groups of the convolution operation is 1), and an activation function (Relu). The lightweight residual block includes an input layer (such as an input image x), a convolutional layer (Conv3×3S1G1), and an activation function (Relu). It can be seen that the structural complexity of the lightweight residual block is lower than the complexity of the residual block.
[0072] (2) Comparison of the network structure of the hyperdecoder subnetwork (HyperDecoder) and the lightweight hyperdecoder subnetwork (Light weighted HyperDecoder).
[0073] Please refer to Figure 2C, which is a schematic diagram comparing the network structures of a super decoding subnetwork and a lightweight super decoding subnetwork provided by an exemplary embodiment of the present application when processing chrominance components and luminance components respectively.
[0074] ① Comparison of the network structure of the super decoding subnetwork (Hyper Decoder_Y) when processing the luminance component and the network structure of the lightweight super decoding subnetwork (Light weighted Hyper Decoder_Y) when processing the luminance component.
[0075] As shown in 25 in Figure 2C, the super decoding subnetwork when processing the luminance component mainly includes five convolutional layers, two cropping layers (Crop), and three activation functions (LeakyRelu). Among them, the five convolutional layers are from top to bottom: Conv128×3×3S1 (Conv means convolution, the number of channels is 128, the convolution kernel size is 3×3, and S1 means the convolution step is 1), DConv128×3×3S2 (DConv means deconvolution, the number of channels is 128, the convolution kernel size is 3×3, and S2 means the convolution step is 2), Conv128×3×3S1 (Conv means convolution, the number of channels is 128, the convolution kernel size is 3×3, and S1 means the convolution step is 1), DConv192×3×3S2 (DConv means deconvolution, the number of channels is 192, the convolution kernel size is 3×3, and S2 means the convolution step is 2), and Conv128×3×3S1 (Conv means convolution, the number of channels is 128, the convolution kernel size is 3×3, and S1 means the convolution step is 1).
[0076] As shown in 26 in Figure 2C, the lightweight super-decoding sub-network when processing the luminance component mainly includes five convolutional layers, two cropping layers (Crop), and three activation functions (LeakyRelu). Among them, the five convolutional layers are from top to bottom: Conv128×1×1S1 (Conv means convolution, the number of channels is 128, the convolution kernel size is 1×1, and S1 means the convolution step is 1), DConv128×4×4S2 (DConv means deconvolution, the number of channels is 128, the convolution kernel size is 4×4, and S2 means the convolution step is 2), Conv128×3×3S1 (Conv means convolution, the number of channels is 128, the convolution kernel size is 3×3, and S1 means the convolution step is 1), DConv128×4×4S2 (DConv means deconvolution, the number of channels is 128, the convolution kernel size is 4×4, and S2 means the convolution step is 2), and Conv128×3×3S1 (Conv means convolution, the number of channels is 128, the convolution kernel size is 3×3, and S1 means the convolution step is 1).
[0077] ② Comparison of the network structure of the hyper decoding sub-network (Hyper Decoder_UV) when processing chroma components and the network structure of the lightweight hyper decoding sub-network (Light weighted Hyper Decoder_UV) when processing chroma components.
[0078] As shown in 27 in Figure 2C, the super decoding subnetwork when processing the chrominance component mainly includes five convolutional layers, two cropping layers (Crop), and three activation functions (LeakyRelu). Among them, the five convolutional layers are from top to bottom: Conv64×3×3S1 (Conv means convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 means the convolution step is 1), DConv64×3×3S2 (Dconv means deconvolution, the number of channels is 64, the convolution kernel size is 3×3, and S2 means the convolution step is 2), Conv64×3×3S1 (Conv means convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 means the convolution step is 1), DConv96×3×3S2 (DConv means deconvolution, the number of channels is 96, the convolution kernel size is 3×3, and S2 means the convolution step is 2), and Conv64×3×3S1 (Conv means convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 means the convolution step is 1).
[0079] As shown in 28 in Figure 2C, the lightweight super-decoding sub-network when processing the chrominance component mainly includes five convolutional layers, two cropping layers (Crop), and three activation functions (LeakyRelu). Among them, the five convolutional layers are from top to bottom: Conv64×1×1S1 (Conv means convolution, the number of channels is 64, the convolution kernel size is 1×1, and S1 means the convolution step is 1), DConv64×4×4S2 (DConv means deconvolution, the number of channels is 64, the convolution kernel size is 4×4, and S2 means the convolution step is 2), Conv64×3×3S1 (Conv means convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 means the convolution step is 1), DConv64×4×4S2 (DConv means deconvolution, the number of channels is 64, the convolution kernel size is 4×4, and S2 means the convolution step is 2), Conv64×3×3S1 (Conv means convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 means the convolution step is 1).
[0080] From the comparison between 25 and 26 in Figure 2C, and 27 and 28, when processing the luminance component or the chrominance component, there are certain differences between the network structure of the super-decoding sub-network and the network structure of the lightweight super-decoding sub-network at the convolutional layer.
[0081] (3) Comparison of the network structure of the Hyper Scale Decoder and the Light Weighted Hyper Scale Decoder.
[0082] Please refer to Figure 2D, which is a schematic diagram comparing the network structures of a super-scale decoding sub-network and a lightweight super-scale decoding sub-network provided by an exemplary embodiment of the present application when processing chrominance components and luminance components respectively.
[0083] ① Comparison of the network structure of the hyper-scale decoding sub-network (Hyper Scale Decoder_Y) when processing the luminance component and the network structure of the lightweight hyper-scale decoding sub-network (Light weighted Hyper Scale Decoder_Y) when processing the luminance component.
[0084] As shown in 29 in Figure 2D, the super-scale decoding subnetwork for processing the luminance component mainly includes six convolutional layers, two cropping layers (Crop), and five activation functions (LeakyRelu). Among them, the six convolutional layers are, from top to bottom: DCnv128×5×5S2 (DConv represents deconvolution, the number of channels is 128, the convolution kernel size is 5×5, and S2 represents the convolution step size is 2), DCnv192×5×5S2 (Dconv represents deconvolution, the number of channels is 192, the convolution kernel size is 5×5, and S2 represents the convolution step size is 2), Conv256×3×3S1 (Conv represents convolution, the number of channels is 256, the convolution kernel size is 3×3, and S2 represents the convolution step size is 2), and Conv256×3×3S1 (Conv represents convolution, the number of channels is 256, the convolution kernel size is 3×3, and S2 represents the convolution step size is 2). 1 means the convolution step is 1), Conv212×3×3S1 (Conv means convolution, the number of channels is 212, the convolution kernel size is 3×3, S1 means the convolution step is 1), Conv170×3×3S1 (Conv means convolution, the number of channels is 170, the convolution kernel size is 3×3, S1 means the convolution step is 1), Conv128×3×3S1 (Conv means convolution, the number of channels is 128, the convolution kernel size is 3×3, S1 means the convolution step is 1).
[0085] As shown in 30 in Figure 2D, the lightweight super-decoding subnetwork for processing the luminance component mainly includes four convolutional layers, two cropping layers (Crop), and three activation functions (LeakyRelu). Among them, the four convolutional layers are, from top to bottom, DConv128×4×4S2 (DConv represents deconvolution, the number of channels is 128, the convolution kernel size is 4×4, and S2 represents the convolution stride is 2), Conv128×3×3S1 (Conv represents convolution, the number of channels is 128, the convolution kernel size is 3×3, and S1 represents the convolution stride is 1), DConv128×4×4S2 (DConv represents deconvolution, the number of channels is 128, the convolution kernel size is 4×4, and S2 represents the convolution stride is 2), and Conv128×3×3S1 (Conv represents convolution, the number of channels is 128, the convolution kernel size is 3×3, and S1 represents the convolution stride is 1).
[0086] ② Comparison of the network structure of the hyper-scale decoding sub-network (Hyper Scale Decoder_UV) when processing chroma components and the network structure of the lightweight hyper-scale decoding sub-network (Light weighted Hyper Scale Decoder_UV) when processing chroma components.
[0087] As shown in 31 in Figure 2D, the super-scale decoding subnetwork for processing chrominance components mainly includes six convolutional layers, two cropping layers (Crop), and five activation functions (LeakyRelu). Among them, the six convolutional layers are, from top to bottom: DConv64×5×5S2 (DConv represents deconvolution, the number of channels is 64, the convolution kernel size is 5×5, and S2 represents the convolution step size is 2), DConv96×5×5S2 (DConv represents deconvolution, the number of channels is 96, the convolution kernel size is 5×5, and S2 represents the convolution step size is 2), Conv128×3×3S1 (Conv represents convolution, the number of channels is 128, the convolution kernel size is 3×3, and S2 represents the convolution step size is 2), and Conv128×3×3S1 (Conv represents convolution, the number of channels is 128, the convolution kernel size is 3×3, and S2 represents the convolution step size is 2). 1 means the convolution step is 1), Conv106×3×3S1 (Conv means convolution, the number of channels is 106, the convolution kernel size is 3×3, S1 means the convolution step is 1), Conv85×3×3S1 (Conv means convolution, the number of channels is 85, the convolution kernel size is 3×3, S1 means the convolution step is 1), Conv64×3×3S1 (Conv means convolution, the number of channels is 64, the convolution kernel size is 3×3, S1 means the convolution step is 1).
[0088] As shown in 32 in Figure 2D, the lightweight super-decoding subnetwork for processing chrominance components mainly includes four convolutional layers, two cropping layers (Crop), and three activation functions (LeakyRelu). Among them, the four convolutional layers are, from top to bottom: DConv64×4×4S2 (DConv represents deconvolution, the number of channels is 64, the convolution kernel size is 4×4, and S2 represents the convolution stride is 2), Conv64×3×3S1 (Conv represents convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 represents the convolution stride is 1), DConv64×4×4S2 (DConv represents deconvolution, the number of channels is 64, the convolution kernel size is 4×4, and S2 represents the convolution stride is 2), and Conv64×3×3S1 (Conv represents convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 represents the convolution stride is 1).
[0089] From the comparison between 29 and 30, and 31 and 32 in Figure 2D, it can be seen that when processing the luminance component and the chrominance component, the network structure of the super-scale decoding sub-network is more complex than the network structure of the lightweight super-scale decoding sub-network, and when processing the luminance component and the chrominance component, there are certain differences between the convolutional layers of the super-scale decoding sub-network and the convolutional layers of the lightweight super-scale decoding sub-network (such as the different number of channels).
[0090] 3. Conventional convolution (or convolution) and grouped convolution
[0091] Conventional convolution refers to the convolution of the feature maps of all input images in the convolution layer. When performing a conventional convolution operation, each output channel (channel) is connected (interacted) with each input channel, and there is a dense connection between the channels. The output channel is the output feature map after the convolution, and the input channel is the input feature map. Schematically, Figure 2E is a comparative diagram of a conventional convolution operation and a grouped convolution operation provided by an exemplary embodiment of the present application. As shown in (1) in Figure 2E, all upper-layer output channels (i.e., output channel a-output channel h) are connected to lower-layer input channels (i.e., input channel 1-input channel 8). For example, output channel a is connected to input channel 1-input channel 8 respectively, output channel b is connected to input channel 1-input channel 8 respectively, and so on. Then, when performing conventional convolution operations, it will inevitably bring about greater computational complexity. It can be seen from the end-to-end encoding and decoding model based on deep learning that the entire model involves many convolutional layers. Therefore, using conventional convolution in the convolutional layer will cause the computational complexity of the entire image encoding and decoding process to be higher, thereby reducing the encoding and decoding efficiency.
[0092] Grouped convolution refers to grouping the feature maps of the input image according to channels in the convolution layer to obtain multiple groups. Then, a convolution operation is performed within each of the multiple groups to obtain the convolution results of each group, for example, the feature maps of the channels included in each group. The convolution results of multiple groups can be spliced. Schematically, the input channels and output channels can be divided into multiple groups (groups), and the output channels of each group are only connected to the input channels in the corresponding group, and are independent of the channels in other groups in the multiple groups. As shown in (2) in Figure 2E, the convolution layer includes 8 input channels (such as input channel 1-input channel 8) and 8 output channels (output channel a-output channel h). Then, the 8 input channels and 8 output channels can be divided into 4 groups, each of which contains 2 input channels and 2 output channels. For example, group 1 includes input channel 1, input channel 2, output channel a, and output channel b, and group 2 includes input channel 3, input channel 4, output channel c, and output channel d. Group 3 includes input channels 5, 6, e, and f; group 4 includes input channels 7, 8, g, and h. As can be seen, in each of the four groups, the two output channels in each group are only connected to the two input channels in the corresponding group.
[0093] It can be seen that group convolution can reduce the computational complexity to a certain extent. Therefore, in an embodiment of the present application, group convolution can be introduced into an end-to-end encoding and decoding model based on deep learning, thereby reducing the computational complexity of the entire encoding and decoding process and improving encoding and decoding efficiency.
[0094] 4. Channel Reorganization
[0095] Channel reorganization can be used to achieve feature exchange between channels. After group convolution processing, channel reorganization can be used to achieve feature map exchange between groups, thereby effectively ensuring that the accuracy of feature extraction can be improved during subsequent processing. In an embodiment of the present application, channel reorganization may refer to: reordering the feature maps corresponding to the channels obtained after group convolution to achieve information flow between different groups. For example, as shown in Figure 2F, a schematic diagram of a channel reorganization provided by an exemplary embodiment of the present application is shown. As shown in (a) in Figure 2F, the channels are divided into three groups (in Figure 2F (a), different colors are used for schematic illustration). Each group is input to the first convolution layer for group convolution processing to obtain the group convolution features corresponding to each group. The group convolution features corresponding to each group are then input to the second convolution layer for group convolution processing. In (a) in Figure 2F, since channel reorganization is not performed, the group convolution features between different groups cannot communicate, which may reduce the feature extraction capability of the model.
[0096] To solve this problem, we consider using channel reorganization to reorder the grouped convolution features output by the grouped convolution operation. As shown in (b) in Figure 2F, multiple channels are divided into three groups (illustrated by different colors in Figure 2F(b)). Each group is input into the first convolution layer for grouped convolution processing to obtain the grouped convolution features corresponding to each group. Then, the grouped convolution features corresponding to each group are reordered to obtain the channel reorganization result. For example, the grouped convolution features corresponding to the first group include the convolution features of channel 1, the convolution features of channel 2, and the convolution features of channel 3; the grouped convolution features corresponding to the second group include the convolution features of channel 4, the convolution features of channel 5, and the convolution features of channel 6; and the grouped convolution features corresponding to the third group include the convolution features of channel 7, the convolution features of channel 8, and the convolution features of channel 9. The grouped convolution features corresponding to the three groups are reordered to obtain the channel reordering result. The channel rearrangement results are: convolution features of channel 1, convolution features of channel 4, convolution features of channel 7, convolution features of channel 2, convolution features of channel 5, convolution features of channel 8, convolution features of channel 3, convolution features of channel 6, and convolution features of channel 9. The channel rearrangement results are then input into the next convolution layer (the second convolution layer) for processing.
[0097] It can be seen that after channel reorganization, feature exchange between groups is realized, which can better ensure that the input of the next group convolution comes from different groups, and can improve the feature extraction ability of the model. Channel reorganization only reorders the feature maps corresponding to the channels, so the use of channel reorganization reduces the introduction of additional computing operations.
[0098] An embodiment of the present application provides an image processing solution, which can be applied to an end-to-end image coding model based on deep learning, and the image processing solution can simplify the end-to-end image coding model based on deep learning. The image processing solution is as follows: decoding indication information is provided at the decoding end, and the decoding indication information is used to indicate the decoding simplification operation to be performed in the decoding neural network; then the decoding neural network is called according to the decoding indication information to perform decoding processing on the code stream to reconstruct the image. In the process of decoding the image, the operation of decoding in the decoding neural network of the decoding end can be simplified through the indication of the decoding indication information, thereby effectively reducing the decoding complexity of the decoding end, improving the decoding efficiency, and at the same time being better supported by the mobile device (i.e., the decoding device).
[0099] Next, the image processing system provided in the embodiments of the present application is described.
[0100] Please refer to Figure 3, which is a structural diagram of an image processing system provided by an exemplary embodiment of the present application. The image processing system may include an encoding device 301 and a decoding device 302. The encoding device 301 is located at the encoding end, and the decoding device 302 is located at the decoding end. The encoding device 301 can be a terminal or a server. The decoding device 302 can be a terminal or a server. A communication connection can be established between the encoding device 301 and the decoding device 302. Among them, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a car terminal, a smart TV, etc., but is not limited to these. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.
[0101] (1) For the encoding device 301
[0102] The encoding device 301 can obtain an image to be processed, which can be captured by a camera or generated by a computer. The camera can be a hardware component provided in the encoding device 301. For example, the camera can be a common camera, a stereo camera, a light field camera, etc. provided in the terminal. The camera can also refer to a hardware device connected to the encoding device 301, such as a camera connected to a server.
[0103] Then, the encoding device 301 can call the encoding neural network to encode the acquired image to obtain a code stream corresponding to the image. The encoding neural network can include the analysis and transformation subnetwork shown in Figure 1A, etc., or the encoding neural network can include the lightweight analysis and transformation subnetwork shown in Figure 1B.
[0104] In one implementation, the encoding neural network may include the analysis and transformation subnetwork shown in FIG1A. The image includes a chrominance component and a luminance component. The encoding device 301 may convert the chrominance component x Y and the brightness component x UV The nonlinear transformation processing is performed through the analysis transformation sub-network as shown in Figure 1A to obtain the transformation result y of the brightness component Y And the transformation result y of the chrominance component UV Then, the encoding device 301 converts the transformation result y of the luminance component into Y The difference operation is performed with the predicted value of the brightness component output by the prediction fusion sub-network to obtain the residual r of the brightness component Y, the transformation result y of the chromaticity component UV The difference operation is performed on the predicted value of the chrominance component output by the prediction fusion sub-network to obtain the residual r of the chrominance component UV Next, the encoding device 301 performs the residual r Y and the residual r of the chrominance component UV Quantize and convert the residual of the luminance component into an integer and the residual of the integer chrominance component Form a code stream.
[0105] In another implementation, when encoding an image, encoding indication information may be pre-configured. The encoding indication information is used to indicate the encoding simplification operation to be performed in the encoding neural network. Then, the encoding device 301 calls the encoding neural network according to the encoding indication information to perform encoding processing on the image to obtain a code stream. Schematically, the encoding neural network may include the analysis transformation subnetwork shown in Figure 1A as an example. The encoding indication information may be used to indicate the encoding simplification operation to be performed in the analysis transformation subnetwork. In this case, the encoding device 301 calls the analysis transformation subnetwork according to the encoding indication information to perform the encoding simplification operation on the image to obtain a code stream.
[0106] By performing encoding simplification operations within the encoding neural network, encoding complexity can be reduced to a certain extent, improving encoding efficiency. The specific implementation of encoding the image using the encoding neural network according to the encoding instructions to generate the bitstream will be described in subsequent embodiments and will not be repeated here.
[0107] After obtaining the code stream, the encoding device 301 may transmit the code stream to the decoding device 302 so that the decoding device 302 decodes the code stream to reconstruct the image.
[0108] (2) Decoding device 302
[0109] The decoding device 302 receives the code stream transmitted by the encoding device 301 and obtains decoding instruction information. The decoding instruction information is used to indicate the decoding simplification operation to be performed in the decoding neural network. Based on the decoding instruction information, the decoding device 302 calls the decoding neural network to decode the code stream to reconstruct the image. Specifically, when the image includes luminance components and chrominance components, the decoding device 302 can decode the code stream to obtain the residual Including the residual of the luminance component and the residual of the chrominance component The decoding device 302 then converts the residual of the luminance component And the predicted value u of the brightness component output by the prediction fusion sub-network Y Superimpose to obtain the transformed value of the brightness component And the residual of the chrominance component And the predicted value u of the chrominance component output by the prediction fusion sub-network UV Superposition is performed to obtain the transformed value of the chrominance component Then call the decoding neural network to transform the brightness component and the transformed values of the chrominance components A decoding simplification operation is performed to reconstruct the image.
[0110] As an implementation, the decoding simplification operation may include a grouped convolution operation, and the decoding indication information is used to indicate that the grouped convolution operation should be performed in the decoding neural network. In this implementation, calling the decoding neural network to perform the decoding simplification operation on the transformed values of the luminance component and the transformed values of the chrominance components to reconstruct the image may be: the decoding device 302 calls the decoding neural network according to the decoding indication information to perform the grouped convolution operation on the transformed values of the luminance component and the transformed values of the chrominance components to reconstruct the image. The transformed values of the luminance component and the transformed values of the chrominance components are, for example, included in a feature map of the image.
[0111] As another implementation, the decoding simplification operation may include a grouped convolution operation and a channel reorganization operation, and the decoding indication information is used to indicate that the grouped convolution operation and the channel reorganization operation need to be performed in the decoding neural network. In this implementation, calling the decoding neural network to perform the decoding simplification operation on the transformed values of the luminance component and the transformed values of the chrominance components to reconstruct the image can be: the decoding device 302 calls the decoding neural network according to the decoding indication information to perform a grouped convolution operation on the transformed values of the luminance component to obtain a grouped convolution result of the luminance component; calling the decoding neural network to perform a grouped convolution operation on the transformed values of the chrominance components to obtain a grouped convolution result of the chrominance components, and performing a channel reorganization operation on the grouped convolution result of the luminance component and the grouped convolution result of the chrominance components to obtain a channel reorganization result of the luminance component and a channel reorganization result of the chrominance components; the channel reorganization result of the luminance component and the channel reorganization result of the chrominance components are used to reconstruct the image.
[0112] The decoding neural network may include a synthetic transform subnetwork, a super decoding subnetwork, an ultra-scale decoding subnetwork, and the like as shown in FIG1A . Alternatively, the decoding neural network may include a lightweight synthetic transform subnetwork, a lightweight super decoding subnetwork, a lightweight ultra-scale decoding subnetwork, and the like as shown in FIG1B . This application does not limit this.
[0113] In an embodiment of the present application, encoding indication information can be configured on the encoding device 301 side, while decoding indication information is not configured on the decoding device 302 side. The encoding indication information simplifies the computational complexity of the encoding end, thereby improving encoding efficiency. Decoding indication information can also be set on the decoding device 302 side, while encoding indication information is not configured on the encoding device 301 side. The decoding indication information simplifies the decoding complexity of the decoding end, thereby improving decoding efficiency. Of course, decoding indication information can be configured on the encoding device 301 side and decoding indication information can be configured on the decoding device 302 side, thereby reducing the complexity of the image encoding and decoding process and improving encoding and decoding efficiency.
[0114] Next, an image processing method provided in an embodiment of the present application is described.
[0115] Please refer to Figure 4, which is a flowchart of an image processing method provided by an exemplary embodiment of the present application. The image processing method can be executed by the decoding device 302. The image processing method can include the following steps S401-S403.
[0116] Step S401: Obtain a code stream formed after the image is encoded.
[0117] Step S402: Obtain decoding indication information, where the decoding indication information is used to indicate a decoding simplification operation to be performed in the decoding neural network. The decoding simplification operation is used to group convolution operations performed by at least one convolutional layer included in the decoding neural network. Grouping convolution operations performed by a convolutional layer is called a grouped convolution operation.
[0118] The decoding neural network may be at least one of a super decoding subnetwork, an ultra-scale decoding subnetwork, and a synthetic transform subnetwork. The super decoding subnetwork may be a super decoding subnetwork as shown in FIG1A or a lightweight super decoding subnetwork as shown in FIG1B. The ultra-scale decoding subnetwork may be an ultra-scale decoding subnetwork as shown in FIG1A or a lightweight ultra-scale decoding subnetwork as shown in FIG1B. The synthetic transform subnetwork may be a synthetic transform subnetwork as shown in FIG1A or a lightweight synthetic transform subnetwork as shown in FIG1B. The decoding simplification operation includes a grouped convolution operation. Alternatively, the decoding simplification operation includes a grouped convolution operation and a channel reassembly operation.
[0119] (1) The decoding simplification operation includes a group convolution operation.
[0120] The decoding instruction information includes decoding group convolution instruction information. The decoding neural network includes at least one convolutional layer, and the number of the at least one convolutional layer can be N, where N is a positive integer. The decoding group convolution instruction information is used to instruct the execution of a group convolution operation in at least one convolutional layer of the decoding neural network. The group convolution operation includes: grouping the feature map of the image by channel in the at least one convolutional layer to obtain multiple groups; performing a convolution operation within each of the multiple groups to obtain group convolution features for each group, wherein the group convolution features of each group include the convolution features of the channels in the group. The output channels of each group in the multiple groups only interact with the input channels within the group and are independent of the channels in other groups in the multiple groups. By instructing the execution of the group convolution operation in the convolutional layer of the decoding neural network, the convolution operations involved in the convolutional layer of the decoding neural network can be grouped, avoiding the high computational complexity caused by performing convolution operations on all input features simultaneously, reducing computational complexity, and thereby reducing decoding complexity and improving decoding efficiency.
[0121] The decoding group convolution indication information can be set in the network structure of the convolution layer of the decoding neural network. Schematically, the decoding neural network is the synthetic transformation subnetwork in Figure 1A, and the decoding indication information is used to indicate that a group convolution operation is performed in the convolution layer (DConv128×3×3S2). In this case, the decoding group convolution indication information can be set in the network structure of the convolution layer. Schematically, the convolution layer can be expressed as DConv128×3×3S2G2, where G2 indicates that the number of groups performing the group convolution operation is 2.
[0122] Furthermore, the decoding grouped convolution indication information is also used to indicate the number of groups of the grouped convolution operation performed in the convolution layer of the decoding neural network. The number of groups can be set based on experience. For example, the number of groups can be set to 2, 4, 8, etc. For example, based on experience, when the number of groups is 4, the decoding complexity during decoding can be reduced while ensuring decoding performance. In this case, the decoding grouped convolution indication information is used to indicate that the number of groups of the grouped convolution operation is 4.
[0123] The number of groups in the grouped convolution operation indicated by the decoding grouped convolution indication information can support dynamic changes as network parameters of the convolution layer of the decoding neural network change. The network parameters include at least one of the following: the number of channels, feature data of the feature map of the input convolution layer, and the bit rate point during image encoding and decoding.
[0124] ① Network parameters include the number of channels. When the number of channels in the convolutional layer performing grouped convolution in the decoding neural network changes, the number of groups in the grouped convolution operation needs to be dynamically adjusted. For example, if the number of channels in the convolutional layer performing grouped convolution changes from 128 to 64, the number of groups in the grouped convolution operation can be adjusted from 8 to 4. ② Network parameters include the feature data of the feature map input to the convolutional layer. For example, if the feature data of the feature map input to the decoding neural network's convolutional layer includes both chrominance and luminance components of an image, the number of groups in the convolutional layer performing grouped convolution on the chrominance component will differ from the number of groups in the convolutional layer performing grouped convolution on the luminance component. ③ Network parameters include the bitrate used during encoding and decoding. When a new bitrate is used for encoding and decoding different images, the number of groups in the grouped convolution operation needs to be dynamically adjusted. When the sources of images differ, the encoding and decoding of these images will correspond to different bitrates. For example, an image derived from a computer-generated animation and another image derived from screen content will correspond to different bitrates during encoding and decoding.
[0125] In one embodiment, the decoding grouped convolution indication information is also used to indicate n convolutional layers among the N convolutional layers that need to perform grouped convolution operations, where n is a positive integer and n is less than or equal to N. Schematically, the decoding neural network includes a synthetic transformation subnetwork as shown in Figure 1A. The synthetic transformation subnetwork as shown in Figure 1A includes four (i.e., N=4) convolutional layers, and the decoding grouped convolution indication information can be used to indicate 2 (n=2) convolutional layers (i.e., DConv128×3×3S2, DConv128×3×3S2) among the N convolutional layers that need to perform grouped convolution operations. That is, grouped convolution operations need to be performed in the convolution layer DConv128×3×3S2 and the convolution layer DConv128×3×3S2.
[0126] Furthermore, the decoding grouped convolution indication information is also used to indicate the number of groups for grouped convolution operations performed by each convolution layer in the n convolution layers. The number of groups for grouped convolution operations performed by different convolution layers in the n convolution layers may all be the same. Alternatively, the number of groups for grouped convolution operations performed by different convolution layers in the n convolution layers may be partially the same. Alternatively, the number of groups for grouped convolution operations performed by different convolution layers in the n convolution layers may all be different. Schematically, the number of groups for grouped convolution operations performed by one convolution layer may be 4, and the number of groups for grouped convolution operations performed by another convolution layer may be 2.
[0127] Among them, the selection rules of the n convolutional layers may include at least one of the following: ① Selecting n convolutional layers that need to perform grouped convolution operations from the N convolutional layers in descending order of the distance between the convolutional layer and the output layer. For example, when the decoding neural network is the lightweight synthetic transformation subnetwork shown in Figure 1B, the distance between the convolutional layer Conv64×3×3S1 and the output layer is greater than the distance between the convolutional layer Conv16×1×1S1 and the output layer, then the convolutional layer Conv64×3×3S1 can be selected as the convolutional layer that needs to perform grouped convolution operations. ② Selecting n convolutional layers that need to perform grouped convolution operations from the N convolutional layers in descending order of the complexity of the network structure. Schematically, the decoding neural network includes convolution layer 1, convolution layer 2 and convolution layer 3. The complexity of the network structure of convolution layer 1 is greater than the complexity of the network structure of convolution layer 2, and the complexity of the network structure of convolution layer 2 is greater than the complexity of the network structure of convolution layer 3. Then convolution layer 1 and convolution layer 2 can be selected as the convolution layers that need to perform group convolution operations.
[0128] When selecting a convolutional layer to perform grouped convolution operations, it is possible to consider performing grouped convolution operations on convolutional layers with higher complexity, thereby reducing the computational complexity of the convolutional layer and, in turn, reducing decoding complexity. Furthermore, when selecting a convolutional layer to perform grouped convolution operations, the complexity and decoding performance of the decoding neural network can be considered to determine the convolutional layer to perform grouped convolution operations, thereby reducing complexity while ensuring decoding performance.
[0129] (2) Decoding simplification operations include channel reorganization and group convolution. Channel reorganization enables information exchange between groups, and channel reorganization does not require additional computational operations. While effectively ensuring the decoding neural network's ability to extract image features, it also effectively guarantees decoding performance and reduces decoding complexity.
[0130] The decoding indication information includes decoding group convolution indication information and decoding channel reorganization indication information. The decoding group convolution indication information is used to indicate that a group convolution operation is performed in the convolution layer of the decoding neural network. The decoding group convolution indication information can be referred to in (1) above and will not be described again here. The decoding channel reorganization indication information is used to indicate that a channel reorganization operation is performed in the convolution layer of the decoding neural network. The decoding channel reorganization indication information can be set in the network structure of the convolution layer of the decoding neural network.
[0131] The convolutional layer that performs the channel reorganization operation is the convolutional layer that performs the grouped convolution operation. When the number of convolutional layers in the decoding neural network is N, and the N convolutional layers include n convolutional layers that need to perform the grouped convolution operation, and N and n are both positive integers and n is less than or equal to N, the decoding channel reorganization indication information is further used to indicate t convolutional layers among the n convolutional layers that need to perform the channel reorganization operation, where t is a positive integer and t is less than or equal to n.
[0132] In one embodiment, the decoding channel reorganization indication information is also used to indicate the number of channels for the channel reorganization operation. When the number of convolutional layers that need to perform the channel reorganization operation is t, the decoding channel reorganization indication information is also used to indicate the number of channels for each convolutional layer in the t convolutional layers that need to perform the channel reorganization operation. Among them, the number of channels for each convolutional layer in the t convolutional layers that need to perform the channel reorganization operation can be all the same. Alternatively, the number of channels for each convolutional layer in the t convolutional layers that need to perform the channel reorganization operation can be partially the same. For example, the number of channels for convolutional layer 1 that need to perform the channel reorganization operation is 4, the number of channels for convolutional layer 2 that need to perform the channel reorganization operation is 4, and the number of channels for convolutional layer 3 that need to perform the channel reorganization operation is 6. Alternatively, the number of channels for each convolutional layer in the t convolutional layers that need to perform the channel reorganization operation can be all different.
[0133] The number of channels indicated by the decoding channel reorganization indication information for the channel reorganization operation can be dynamically adjusted as network parameters of the convolutional layer of the decoding neural network change. The network parameters include at least one of the following: the number of channels, feature data of the input feature map, and the bit rate point during encoding and decoding.
[0134] ① Network parameters include the number of channels. When the number of channels of the convolutional layer performing the channel reorganization operation changes, the number of channels for the channel reorganization operation needs to be dynamically adjusted. For example, when the number of channels of the convolutional layer performing the grouped convolution operation changes from 128 to 64, the number of channels for the channel reorganization operation can be adjusted from 128 to 64. ② Network parameters include the feature data of the feature map input to the convolutional layer. For example, when the feature data of the feature map input to the decoding neural network includes the chrominance and luminance components of the image, the number of channels of the convolutional layer of the decoding neural network when performing the channel reorganization operation on the chrominance component is different from the number of channels of the convolutional layer when performing the channel reorganization operation on the luminance component. ③ Network parameters include the bitrate point during encoding and decoding. When encoding and decoding different images correspond to different bitrate points, the number of channels for the channel reorganization operation needs to be dynamically adjusted.
[0135] When the convolution layer includes P channels, the decoded channel reorganization indication information is further used to indicate the p channels among the P channels that need to perform the channel reorganization operation. In other words, the decoded channel reorganization indication information is used to indicate the number of channels for the channel reorganization operation and to indicate which channels among the P channels need to perform the channel reorganization operation. In one implementation, when each channel corresponds to a channel identifier, such as a channel index, the index value of the channel for performing the channel reorganization operation can be indicated in the decoded channel reorganization indication information. The number of channels and the channel index are both parameters of the channel pixel. Schematically, the convolution layer includes 64 channels, and the index values corresponding to the 64 channels are 0 to 63, respectively. The decoded channel reorganization indication information is used to indicate that the number of channels for the channel reorganization operation is 32, and the 32 (p=32) channels that need to perform the channel reorganization operation are 0-3, 10-13, 20-23, and 30-33.
[0136] The selection rules for the p channels may include at least one of the following: ① Selecting the p channels to perform the channel reorganization operation from the P channels according to the rule of reducing memory resource consumption. Schematically, P = 64, and the 64 channels will be stored in different memory locations. When performing the channel reorganization operation, the memory resource consumption when obtaining the 64 channels from the memory locations of the 64 channels is different. Therefore, 32 channels to perform the channel reorganization operation can be selected from the 64 channels in order of increasing memory resource consumption. ② The P channels correspond to multiple groups of the grouped convolution operation. For example, if the number of groups is 4, P = 64. The 64 channels correspond to 4 groups of the grouped convolution operation, and each group includes 16 channels. In this way, according to the data communication requirements between each group, the p channels to perform the channel reorganization operation can be selected from the P channels. For example, if the data communication requirement is to ensure that the features between each group can come from other groups, then all 64 channels can be selected as channels to perform the channel reorganization operation.
[0137] In one embodiment, the decoding neural network includes at least one of the following: a super decoding subnetwork, a super-scale decoding subnetwork, and a synthetic transformation subnetwork. Each subnetwork includes a convolutional layer. The decoding simplification operation includes a grouped convolution operation. If the decoding instruction information indicates that the convolutional layers of each subnetwork need to perform a grouped convolution operation, the number of groups for the convolutional layers of each subnetwork indicated by the decoding instruction information to perform the grouped convolution operation can be all the same, the number of groups for the convolutional layers of each subnetwork indicated by the decoding instruction information to perform the grouped convolution operation can be partially the same, or the number of groups for the convolutional layers of each subnetwork indicated by the decoding instruction information to perform the grouped convolution operation can be all different.
[0138] Furthermore, the decoding simplification operation includes a grouped convolution operation and a channel reorganization operation. If the decoding indication information indicates that the convolution layer of each sub-network performs a channel reorganization operation, the number of channels of the convolution layer of each sub-network indicated by the decoding indication information for the channel reorganization operation may all be the same; or, the number of channels of the convolution layer of each sub-network indicated by the decoding indication information for the channel reorganization operation may be partially the same and partially different; or, the number of channels of the convolution layer of each sub-network indicated by the decoding indication information for the channel reorganization operation may all be different (such as the decoding indication information is used to indicate that the number of channels of the convolution layer of the super-decoding sub-network for the channel reorganization operation is 4, and the number of channels of the synthetic transformation sub-network for the channel reorganization operation is 32). This application does not impose any restrictions on this.
[0139] Step S403: According to the decoding instruction information, the decoding neural network is called to perform a decoding simplification operation to decode the code stream to reconstruct the image.
[0140] When the decoding neural network includes at least one convolutional layer, the decoding simplification operation includes a grouped convolution operation, and the decoding indication information is used to indicate that the grouped convolution operation is performed in the convolutional layer of the decoding neural network, calling the decoding neural network to decode the code stream according to the decoding indication information to reconstruct the image may include: calling the decoding neural network to decode the code stream according to the decoding indication information to obtain image transformation values; then calling at least one convolutional layer of the decoding neural network to perform a grouped convolution operation on the image transformation values to obtain grouped convolution results corresponding to the image transformation values; and decoding the grouped convolution results to reconstruct the image. Specifically, the decoding indication information includes decoding grouped convolution indication information, which is used to indicate the number of groups for the grouped convolution operation. Based on the number of groups, the decoding device can group the image transformation values to obtain one or more groups, and then perform a convolution operation on each group to obtain a grouped convolution result corresponding to the image transformation values.
[0141] When the decoding simplification operation includes a grouped convolution operation and a channel reorganization operation, and the decoding indication information is used to indicate that the grouped convolution operation and the channel reorganization operation are performed in the convolution layer of the decoding neural network, calling the decoding neural network to decode the code stream according to the decoding indication information to reconstruct the image may include: calling the decoding neural network to decode the code stream according to the decoding indication information to obtain the transformation value of the image; then calling at least one convolution layer of the decoding neural network to perform a grouped convolution operation on the transformation value of the image to obtain a grouped convolution result corresponding to the transformation value of the image; then, performing a channel reorganization operation on the grouped convolution result to obtain a channel reorganization result, and decoding the channel reorganization result to reconstruct the image.
[0142] The grouped convolution result includes convolution features of each of the multiple channels, and the decoding indication information includes decoding channel reorganization indication information, the decoding channel reorganization indication information further being used to indicate the number of channels for the channel reorganization operation. Performing the channel reorganization operation on the grouped convolution result to obtain the channel reorganization result may include reordering the grouped convolution result according to the number of channels indicated by the decoding channel reorganization indication information to obtain the channel reorganization result.
[0143] In an embodiment of the present application, decoding instructions are provided at the decoding end. This information is used to indicate the decoding simplification operations to be performed in the decoding neural network. The decoding neural network is then called according to the decoding instructions to decode the bitstream to reconstruct the image. During the image decoding process, the instructions in the decoding instructions can simplify the decoding operations in the decoding neural network at the decoding end, thereby effectively reducing the decoding complexity at the decoding end, improving decoding efficiency, and being well supported by mobile devices.
[0144] Please refer to Figure 5, which is a flowchart of an image processing method provided by another exemplary embodiment of the present application. The image processing method can be executed by the encoding device 301. The image processing method can include the following steps S501-S503.
[0145] Step S501: Acquire an image to be processed.
[0146] Step S502: Obtain encoding instruction information, where the encoding instruction information indicates an encoding simplification operation to be performed in the encoding neural network. The encoding simplification operation is used to group convolution operations performed by at least one convolutional layer included in the encoding neural network. Grouping convolution operations performed by a convolutional layer is called a grouped convolution operation.
[0147] The coding neural network may include at least one of the following: an analysis transformation subnetwork and a super coding subnetwork. The coding simplification operation may include a grouped convolution operation, or the coding simplification operation may include a grouped convolution operation and a channel reorganization operation.
[0148] (1) The coding simplification operation includes the group convolution operation.
[0149] The encoding indication information includes encoding grouped convolution indication information. The encoding neural network includes at least one convolution layer. The encoding grouped convolution indication information is used to indicate that a grouped convolution operation is performed in the convolution layer of the encoding neural network. The encoding grouped convolution indication information is set in the network structure of the convolution layer of the encoding neural network. The grouped convolution operation includes: grouping the feature map of the image according to the channel in the at least one convolution layer to obtain a plurality of groups; performing a convolution operation inside each group in the plurality of groups to obtain a grouped convolution feature of each group, wherein the grouped convolution feature of each group contains the convolution feature of the channel in the group. The output channel of each group in the plurality of groups only interacts with the input channel in the group and is independent of the channels in other groups in the plurality of groups.
[0150] Furthermore, the encoding group convolution indication information is further used to indicate the number of groups of group convolution operations performed in the convolution layer of the encoding neural network. This number of groups can be dynamically adjusted as network parameters of the convolution layer of the encoding neural network change. The network parameters here can include at least one of the following: the number of channels, feature data of the feature map input to the convolution layer, and the bit rate point during encoding and decoding.
[0151] ① Network parameters include the number of channels. When the number of channels in the convolutional layer of the encoding neural network changes, the number of groups in the grouped convolution operation needs to be dynamically adjusted. For example, if the number of channels in the convolutional layer of the encoding neural network changes from 128 to 64, the number of groups in the grouped convolution operation can be adjusted from 8 to 4. ② Network parameters include the feature data of the feature map input to the convolutional layer of the encoding neural network. For example, if the feature data of the feature map input to the convolutional layer of the encoding neural network includes the chrominance and luminance components of the image, the number of groups when performing grouped convolution on the chrominance component will be different from the number of groups when performing grouped convolution on the luminance component. ③ Network parameters include the bitrate used during encoding and decoding. When a new bitrate is used for encoding and decoding different images, the number of groups in the grouped convolution operation needs to be dynamically adjusted. When images come from different sources, different bitrates will be used for encoding and decoding different images. When different images are encoded at different bitrates, the number of groups in the grouped convolution operation needs to be dynamically adjusted.
[0152] The number of convolutional layers of the encoding neural network is M, where M is a positive integer. The encoding grouped convolution indication information is also used to indicate the m convolutional layers among the M convolutional layers that need to perform grouped convolution operations, where m is a positive integer and is less than or equal to M. Among them, the selection rules for the m convolutional layers include at least one of the following: ① Select the m convolutional layers that need to perform grouped convolution operations in descending order of distance from the output layer; ② Select the m convolutional layers that need to perform grouped convolution operations in descending order of complexity of the network structure. The selection rules for the m convolutional layers are similar to the selection rules for the n convolutional layers mentioned above. The difference is that the selection rules for the m convolutional layers are considered from the perspective of encoding and will not be repeated here.
[0153] Furthermore, the grouped convolution indication information is further used to indicate the number of groups for grouped convolution operations performed by each of the m convolutional layers. The number of groups for grouped convolution operations performed by different convolutional layers in the m convolutional layers may all be the same, or the number of groups for grouped convolution operations performed by different convolutional layers in the m convolutional layers may be partially different and partially the same; or the number of groups for grouped convolution operations performed by different convolutional layers in the m convolutional layers may all be different.
[0154] (2) The coding simplification operation includes group convolution operation and channel reorganization operation.
[0155] The coding indication information may include coding group convolution indication information and coding channel reorganization indication information. The coding group convolution indication information can refer to the relevant description of (1) above and will not be repeated here. The coding channel reorganization indication information is used to indicate the execution of a channel reorganization operation in the convolution layer of the coding neural network, wherein the coding channel reorganization indication information is set in the network structure of the convolution layer of the coding neural network. The channel reorganization operation includes: reordering the group convolution features of each group obtained by the group convolution operation to obtain the reordered group convolution features.
[0156] Furthermore, the encoding channel reorganization indication information is also used to indicate the number of channels of the channel reorganization operation performed in the convolution layer of the encoding neural network. The number of channels can be dynamically adjusted as the network parameters of the convolution layer of the encoding neural network change.
[0157] The convolutional layer includes I channels. The coded channel reorganization indication information is further used to indicate O channels among the I channels for which a channel reorganization operation is to be performed, where O is a positive integer and is less than or equal to 1. In other words, the coded channel reorganization indication information is used to indicate the number of channels for the channel reorganization operation and which channels among the I channels require the channel reorganization operation. In one implementation, when each channel corresponds to a channel index value, the coded channel reorganization indication information may indicate the index value of the channel for which the channel reorganization operation is to be performed.
[0158] The O channels selection rule includes at least one of the following: selecting the O channels to perform the channel operation from the I channels based on a rule for reducing memory resource consumption; or selecting the O channels to perform the channel reassembly operation from the I channels based on data communication requirements between the groups, where the O channels correspond to the multiple groups in the grouped convolution operation. The O channels selection rule is similar to the aforementioned p channels selection rule, except that the O channels selection rule is considered from a coding perspective and is not further described here.
[0159] When the number of convolutional layers of the encoding neural network is M, and the M convolutional layers include m convolutional layers that need to perform grouped convolution operations, M and m are both positive integers and m is less than or equal to M, the encoding channel reorganization indication information is also used to indicate q convolutional layers that need to perform reorganization operations among the m convolutional layers, where q is a positive integer and q is less than or equal to m.
[0160] Furthermore, the coded channel reorganization indication information is further used to indicate the number of channels for each of the q convolutional layers to perform the channel reorganization operation. The number of channels for different convolutional layers in the q convolutional layers to perform the channel reorganization operation is the same, or the number of channels for different convolutional layers in the q convolutional layers to perform the channel reorganization operation may be partially the same and partially different; or the number of channels for different convolutional layers in the q convolutional layers to perform the channel reorganization operation may be completely different.
[0161] Wherein, the coding neural network includes at least one of the following: an analysis transformation subnetwork and a super coding subnetwork. When the coding simplification operation includes a grouped convolution operation, if the coding indication information indicates that the convolution layers of each subnetwork need to perform the grouped convolution operation, then the number of groups for the grouped convolution operation of the convolution layers of each subnetwork indicated by the coding indication information may be all the same; or, the number of groups for the grouped convolution operation of the convolution layers of each subnetwork indicated by the coding indication information may be partially the same and partially different; or, the number of groups for the grouped convolution operation of the convolution layers of each subnetwork indicated by the coding indication information may be all different (e.g., the coding indication information is used to indicate that the number of groups for the grouped convolution operation of the convolution layers of the super coding subnetwork is 4, and the number of groups for the grouped convolution operation of the convolution layers of the analysis transformation subnetwork is 8).
[0162] When the coding simplification operation includes a grouped convolution operation and a channel reorganization operation, if the coding indication information indicates that the convolution layer of each sub-network performs a channel reorganization operation, then the number of channels of the convolution layer of the sub-network indicated by the coding indication information for the channel reorganization operation is all the same; or, the number of channels of the convolution layer of the sub-network indicated by the coding indication information for the channel reorganization operation is partially the same and partially different; or, the number of channels of the convolution layer of the sub-network indicated by the coding indication information for the channel reorganization operation is all different (for example, the coding indication information is used to indicate that the number of channels of the convolution layer of the super-coding sub-network for the channel reorganization operation is 16, and the number of channels of the convolution layer of the analysis and transformation sub-network for the channel reorganization operation is 32).
[0163] S503: According to the encoding instruction information, call the encoding neural network to perform encoding simplification operations on the image to obtain a code stream.
[0164] When the encoding neural network includes a convolution layer, and the encoding simplification operation includes a grouped convolution operation, the encoding indication information is used to indicate that when the grouped convolution operation is performed in the convolution layer of the encoding neural network, the encoding neural network is called according to the encoding indication information to perform encoding processing on the image, and obtaining the code stream may include: calling the encoding neural network to perform a grouped convolution operation on the image to obtain a grouped convolution result, and then encoding the grouped convolution result (such as quantization, etc.) to obtain the code stream of the image.
[0165] When the coding neural network includes a convolution layer, and the coding simplification operation includes a grouped convolution operation and a channel reorganization operation, the coding indication information is used to indicate that the grouped convolution operation and the channel reorganization operation are performed in the convolution layer of the coding neural network, and the coding neural network is called according to the coding indication information to perform encoding processing on the image, and obtaining the code stream may include: calling the coding neural network to perform a grouped convolution operation on the image to obtain a grouped convolution result, performing a channel reorganization operation on the grouped convolution result to obtain a channel reorganization result of the image, and performing encoding processing (such as quantization, etc.) on the channel reorganization result to obtain the code stream of the image.
[0166] In one embodiment, the encoding neural network includes at least one convolutional layer; the encoding simplification operation includes a grouped convolution operation; the encoding indication information includes encoding grouped convolution indication information, and the encoding grouped convolution indication information is used to indicate that a grouped convolution operation is performed in at least one convolutional layer of the encoding neural network. The grouped convolution operation includes: grouping the feature map of the image according to channels in the at least one convolutional layer to obtain multiple groups; performing a convolution operation within each of the multiple groups to obtain grouped convolution features of each group, wherein the grouped convolution features of each group include the convolution features of the channels in the group. The output channels of each of the multiple groups only interact with the input channels within the group and are independent of the channels in other groups in the multiple groups.
[0167] In one embodiment, the encoding group convolution indication information is further used to indicate the number of groups of group convolution operations performed in the convolution layer of the encoding neural network.
[0168] In one embodiment, the number of groups of the grouped convolution operation indicated by the encoding grouped convolution indication information supports dynamic adjustment as the network parameters of the convolution layer of the encoding neural network change.
[0169] In one embodiment, the number of the at least one convolutional layer is M, where M is a positive integer;
[0170] The encoded grouped convolution indication information is also used to indicate m convolution layers among the M convolution layers that need to perform grouped convolution operations, where m is a positive integer and m is less than or equal to M.
[0171] In one embodiment, the selection rules of the m convolutional layers include at least one of the following:
[0172] Select m convolutional layers that need to perform grouped convolution operations from the M convolutional layers in descending order of the distance between the M convolutional layers and the output layer;
[0173] Select m convolutional layers that need to perform group convolution operations from M convolutional layers in descending order of network structure complexity.
[0174] In one embodiment, the encoded group convolution indication information is further used to indicate the number of groups for performing group convolution operations on each of the m convolution layers;
[0175] Different convolutional layers in the m convolutional layers have the same or different numbers of groups for grouped convolution operations.
[0176] In one embodiment, the encoding simplification operation further includes a channel reorganization operation; the encoding instruction information includes encoding channel reorganization instruction information, and the encoding channel reorganization instruction information is used to instruct to perform the channel reorganization operation in at least one convolutional layer of the encoding neural network. The channel reorganization operation includes: reordering the grouped convolution features of each group obtained by the grouped convolution operation to obtain reordered grouped convolution features.
[0177] In one embodiment, the encoding channel reorganization indication information is set in the network structure of the convolutional layer of the encoding neural network.
[0178] In one embodiment, the encoding channel reorganization indication information is further used to indicate the number of channels and / or channel identifiers of the channel reorganization operation performed in the convolutional layer of the encoding neural network.
[0179] In one embodiment, the number of channels of the channel reorganization operation indicated by the encoding channel reorganization indication information supports dynamic adjustment as the network parameters of the convolutional layer of the encoding neural network change.
[0180] In one embodiment, one convolutional layer in at least one convolutional layer includes I channels; the encoded channel reorganization indication information is further used to indicate O channels among the I channels on which channel reorganization operations need to be performed, where O is a positive integer and O is less than or equal to I.
[0181] In one embodiment, the selection rules for the 0 channels include at least one of the following:
[0182] According to the rule of reducing memory resource consumption, select O channels to perform channel operations from I channels; or,
[0183] The O channels correspond to multiple groups of the group convolution operation respectively. According to the data communication requirements between the groups, the O channels that need to perform the channel reorganization operation are selected from the I channels.
[0184] In one embodiment, the number of convolutional layers is M, and the M convolutional layers include m convolutional layers that need to perform grouped convolution operations, where M and m are both positive integers and m is less than or equal to M;
[0185] The coding channel reorganization indication information is further used to indicate q convolutional layers among the m convolutional layers that need to perform the reorganization operation, where q is a positive integer and q is less than or equal to m.
[0186] In one embodiment, the coded grouped convolution indication information is further used to indicate the number of channels for each convolution layer in the q convolution layers to perform a channel reorganization operation;
[0187] The number of channels for channel reorganization operations performed by different convolutional layers in the q convolutional layers is the same or different.
[0188] In one embodiment, the encoding neural network includes at least one of the following: an analysis transformation subnetwork, a super encoding subnetwork;
[0189] The coding simplification operation includes a grouped convolution operation. If the coding indication information indicates that the convolution layer of each sub-network needs to perform the grouped convolution operation, the number of groups for the convolution layer of each sub-network indicated by the coding indication information to perform the grouped convolution operation is the same or different;
[0190] The coding simplification operation also includes a channel reorganization operation. If the coding indication information indicates that the convolutional layer of each sub-network performs a channel reorganization operation, the number of channels of the convolutional layer of each sub-network indicated by the coding indication information for performing the channel reorganization operation is the same or different.
[0191] In an embodiment of the present application, an image to be processed is obtained, and encoding instruction information is provided on the encoding device side. This encoding instruction information is used to indicate the encoding simplification operation to be performed in the encoding neural network. The encoding neural network is then called according to the encoding instruction information to perform encoding processing on the image to reconstruct the image. During the image encoding process, the instructions of the encoding instruction information can simplify the encoding operations in the encoding neural network of the encoding end, thereby effectively reducing the encoding complexity of the encoding end and improving encoding efficiency.
[0192] Next, the image processing device provided in the embodiment of the present application is described.
[0193] Please refer to Figure 6, which is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device can be a computer program (including program code) in a computing device, for example, the image processing device can be an application software in the computing device; the image processing device can be used to perform some or all of the steps in the method embodiment shown in Figure 4. Referring to Figure 6, the image processing device includes the following units:
[0194] An acquisition unit 601 is configured to acquire a code stream formed after the image is coded;
[0195] The acquisition unit 601 is further configured to acquire decoding indication information, where the decoding indication information is used to indicate a decoding simplification operation to be performed in the decoding neural network, where the decoding simplification operation is used to group convolution operations performed by at least one convolutional layer included in the decoding neural network.
[0196] The processing unit 602 is further configured to call a decoding neural network according to the decoding instruction information to perform a decoding simplification operation to decode the code stream to reconstruct an image.
[0197] In one embodiment, the decoding simplification operation includes a grouped convolution operation; the decoding indication information includes decoding grouped convolution indication information, and the decoding grouped convolution indication information is used to indicate that a grouped convolution operation is performed in at least one convolution layer of the decoding neural network. The grouped convolution operation includes: grouping the feature map of the image according to channels in the at least one convolution layer to obtain multiple groups; performing a convolution operation within each of the multiple groups to obtain grouped convolution features of each group, wherein the grouped convolution features of each group include the convolution features of the channels in the group. The output channels of each group in the multiple groups only interact with the input channels within the group and are independent of the channels in other groups in the multiple groups.
[0198] In one embodiment, the decoding group convolution indication information is set in the network structure of the convolution layer of the decoding neural network.
[0199] In one embodiment, the decoding group convolution indication information is further used to indicate the number of groups for the group convolution operation.
[0200] In one embodiment, the number of groups of the grouped convolution operation indicated by the decoding grouped convolution indication information supports dynamic adjustment as the network parameters of the convolution layer of the decoding neural network change.
[0201] In one embodiment, the number of the at least one convolutional layer of the decoding neural network is N, where N is a positive integer;
[0202] The decoding group convolution indication information is also used to indicate n convolution layers among the N convolution layers that need to perform group convolution operations, where n is a positive integer and n is less than or equal to N.
[0203] In one embodiment, the selection rules of the n convolutional layers include at least one of the following:
[0204] Select n convolutional layers to perform grouped convolution operations from the N convolutional layers in descending order of the distance between the N convolutional layers and the output layer;
[0205] Select n convolutional layers that need to perform group convolution operations from N convolutional layers in descending order of network structure complexity.
[0206] In one embodiment, the decoding group convolution indication information is further used to indicate the number of groups for performing group convolution operations on each convolution layer in the n convolution layers;
[0207] Different convolutional layers in the n convolutional layers have the same or different numbers of groups for grouped convolution operations.
[0208] In one embodiment, the decoding simplification operation further includes a channel reorganization operation; the decoding instruction information includes decoding channel reorganization instruction information, and the decoding channel reorganization instruction information is used to instruct to perform the channel reorganization operation in at least one convolutional layer of the decoding neural network. The channel reorganization operation includes: reordering the grouped convolution features of each group obtained by the grouped convolution operation to obtain reordered grouped convolution features.
[0209] In one embodiment, the decoding channel reorganization indication information is set in the network structure of the convolutional layer of the decoding neural network.
[0210] In one embodiment, the decoded channel reassembly indication information is further used to indicate the number of channels and / or channel identifiers for the channel reassembly operation.
[0211] In one embodiment, the number of channels of the channel reorganization operation indicated by the decoding channel reorganization indication information supports dynamic adjustment as the network parameters of the convolutional layer of the decoding neural network change.
[0212] In one embodiment, one convolutional layer in at least one convolutional layer includes P channels; the decoding channel reorganization indication information is further used to indicate p channels among the P channels on which channel reorganization operations need to be performed, where p is a positive integer and p is less than or equal to P.
[0213] In one embodiment, the selection rules for the p channels include at least one of the following:
[0214] According to the rule of reducing memory resource consumption, select p channels from P channels on which channel operations need to be performed; or,
[0215] The P channels correspond to multiple groups of the group convolution operation respectively. According to the data communication requirements between the groups, p channels that need to perform channel reorganization operations are selected from the P channels.
[0216] In one embodiment, the number of at least one convolutional layer of the decoding neural network is N, and the N convolutional layers include n convolutional layers that need to perform grouped convolution operations, N and n are both positive integers and n is less than or equal to N;
[0217] The decoding channel reorganization indication information is further used to indicate t convolutional layers among the n convolutional layers that need to perform channel reorganization operations, where t is a positive integer and t is less than or equal to n.
[0218] In one embodiment, the decoding channel reorganization indication information is further used to indicate the number of channels for performing the channel reorganization operation on each of the t convolutional layers;
[0219] The number of channels for channel reorganization operations performed by different convolutional layers in the t convolutional layers is the same or different.
[0220] In one embodiment, the decoding neural network includes at least one of the following: a super decoding subnetwork, a super-scale decoding subnetwork, and a synthetic transformation subnetwork; each subnetwork includes a convolutional layer;
[0221] The decoding simplification operation includes a group convolution operation. If the decoding indication information indicates that the convolution layer of each sub-network needs to perform the group convolution operation, the number of groups for the convolution layer of each sub-network indicated by the decoding indication information to perform the group convolution operation is the same or different;
[0222] The decoding simplification operation also includes a channel reorganization operation. If the decoding indication information indicates that the convolutional layer of each sub-network performs a channel reorganization operation, the number of channels of the convolutional layer of each sub-network indicated by the decoding indication information for performing the channel reorganization operation is the same or different.
[0223] In one embodiment, the decoding neural network includes a convolution layer, and the decoding simplification operation includes a grouped convolution operation and a channel reorganization operation; the processing unit 601 is specifically configured to:
[0224] Calling the decoding neural network to decode the code stream according to the decoding instruction information to obtain the transformed value of the image;
[0225] Calling at least one convolution layer to perform a grouped convolution operation on the transformed values of the image to obtain a grouped convolution result corresponding to the transformed values of the image;
[0226] Perform channel reorganization operation on the grouped convolution result to obtain the channel reorganization result;
[0227] The channel reorganization processing result is decoded to reconstruct the image.
[0228] In an embodiment of the present application, a code stream formed after image encoding is obtained, and decoding instruction information is obtained, which is used to indicate the decoding simplification operation to be performed in the decoding neural network; then, the decoding neural network is called according to the decoding instruction information to decode the code stream to reconstruct the image. During the image decoding process, the instructions of the decoding instruction information can simplify the decoding operations in the decoding neural network at the decoding end, thereby effectively reducing the decoding complexity at the decoding end and improving decoding efficiency.
[0229] Please refer to Figure 7, which is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device can be a computer program (including program code) in a computing device, for example, the image processing device can be an application software in the computing device; the image processing device can be used to perform some or all of the steps in the method embodiment shown in Figure 4. Referring to Figure 7, the image processing device includes the following units:
[0230] An acquisition unit 701 is used to acquire an image to be processed;
[0231] The acquiring unit 701 is further configured to acquire encoding indication information, where the encoding indication information is used to indicate an encoding simplification operation to be performed in the encoding neural network, where the encoding simplification operation is used to group convolution operations performed by at least one convolutional layer included in the encoding neural network.
[0232] The processing unit 702 is used to call the encoding neural network according to the encoding instruction information to perform encoding simplification operations on the image to obtain a code stream.
[0233] In one embodiment, the encoding simplification operation includes a grouped convolution operation; the encoding indication information includes encoding grouped convolution indication information, and the encoding grouped convolution indication information is used to indicate that a grouped convolution operation is performed in at least one convolution layer of the encoding neural network. The grouped convolution operation includes: grouping the feature map of the image according to channels in the at least one convolution layer to obtain multiple groups; performing a convolution operation within each of the multiple groups to obtain grouped convolution features of each group, wherein the grouped convolution features of each group include the convolution features of the channels in the group. The output channels of each of the multiple groups only interact with the input channels within the group and are independent of the channels in other groups in the multiple groups.
[0234] In one embodiment, the encoding group convolution indication information is further used to indicate the number of groups of group convolution operations performed in the convolution layer of the encoding neural network.
[0235] In one embodiment, the number of groups of the grouped convolution operation indicated by the encoding grouped convolution indication information supports dynamic adjustment as the network parameters of the convolution layer of the encoding neural network change.
[0236] In one embodiment, the number of the at least one convolutional layer is M, where M is a positive integer;
[0237] The encoded grouped convolution indication information is also used to indicate m convolution layers among the M convolution layers that need to perform grouped convolution operations, where m is a positive integer and m is less than or equal to M.
[0238] In one embodiment, the selection rules of the m convolutional layers include at least one of the following:
[0239] Select m convolutional layers that need to perform grouped convolution operations from the M convolutional layers in descending order of the distance between the M convolutional layers and the output layer;
[0240] Select m convolutional layers that need to perform group convolution operations from M convolutional layers in descending order of network structure complexity.
[0241] In one embodiment, the encoded group convolution indication information is further used to indicate the number of groups for performing group convolution operations on each of the m convolution layers;
[0242] Different convolutional layers in the m convolutional layers have the same or different numbers of groups for grouped convolution operations.
[0243] In one embodiment, the encoding simplification operation further includes a channel reorganization operation; the encoding instruction information includes encoding channel reorganization instruction information, and the encoding channel reorganization instruction information is used to instruct to perform the channel reorganization operation in at least one convolutional layer of the encoding neural network. The channel reorganization operation includes: reordering the grouped convolution features of each group obtained by the grouped convolution operation to obtain reordered grouped convolution features.
[0244] In one embodiment, the encoding channel reorganization indication information is set in the network structure of the convolutional layer of the encoding neural network.
[0245] In one embodiment, the encoding channel reorganization indication information is further used to indicate the number of channels and / or channel identifiers of the channel reorganization operation performed in the convolutional layer of the encoding neural network.
[0246] In one embodiment, the number of channels of the channel reorganization operation indicated by the encoding channel reorganization indication information supports dynamic adjustment as the network parameters of the convolutional layer of the encoding neural network change.
[0247] In one embodiment, one convolutional layer in at least one convolutional layer includes I channels; the encoded channel reorganization indication information is further used to indicate O channels among the I channels on which channel reorganization operations need to be performed, where O is a positive integer and O is less than or equal to I.
[0248] In one embodiment, the selection rules for the 0 channels include at least one of the following:
[0249] According to the rule of reducing memory resource consumption, select O channels to perform channel operations from I channels; or,
[0250] The O channels correspond to multiple groups of the group convolution operation respectively. According to the data communication requirements between the groups, the O channels that need to perform the channel reorganization operation are selected from the I channels.
[0251] In one embodiment, the number of the at least one convolutional layer is M, and the M convolutional layers include m convolutional layers that need to perform grouped convolution operations, M and m are both positive integers and m is less than or equal to M;
[0252] The coding channel reorganization indication information is further used to indicate q convolutional layers among the m convolutional layers that need to perform the reorganization operation, where q is a positive integer and q is less than or equal to m.
[0253] In one embodiment, the coded grouped convolution indication information is further used to indicate the number of channels for each convolution layer in the q convolution layers to perform a channel reorganization operation;
[0254] The number of channels for channel reorganization operations performed by different convolutional layers in the q convolutional layers is the same or different.
[0255] In one embodiment, the encoding neural network includes at least one of the following: an analysis transformation subnetwork, a super encoding subnetwork;
[0256] The coding simplification operation includes a grouped convolution operation. If the coding indication information indicates that the convolution layer of each sub-network needs to perform the grouped convolution operation, the number of groups for the convolution layer of each sub-network indicated by the coding indication information to perform the grouped convolution operation is the same or different;
[0257] The coding simplification operation also includes a channel reorganization operation. If the coding indication information indicates that the convolutional layer of each sub-network performs a channel reorganization operation, the number of channels of the convolutional layer of each sub-network indicated by the coding indication information for performing the channel reorganization operation is the same or different.
[0258] In an embodiment of the present application, an image to be processed is obtained, and encoding instruction information is provided on the encoding device side. This encoding instruction information is used to indicate the encoding simplification operation to be performed in the encoding neural network. The encoding neural network is then called according to the encoding instruction information to perform encoding processing on the image to reconstruct the image. During the image encoding process, the instructions of the encoding instruction information can simplify the encoding operations in the encoding neural network of the encoding end, thereby effectively reducing the encoding complexity of the encoding end and improving encoding efficiency.
[0259] Next, the computer device provided in the embodiments of the present application is described.
[0260] Furthermore, an embodiment of the present application provides a schematic diagram of the structure of a computer device, as shown in Figure 8 . The computer device may include a processor 801, an input device 802, an output device 803, and a memory 804. The processor 801, input device 802, output device 803, and memory 804 are connected via a bus. The memory 804 is used to store a computer program, which includes program instructions. The processor 801 is used to execute the program instructions stored in the memory 804.
[0261] When the computer device is the above-mentioned decoding device, in the embodiment of the present application, the processor 801 performs the following operations by running the program instructions in the memory 804:
[0262] Obtaining a code stream formed after the image is encoded;
[0263] Obtaining decoding indication information, where the decoding indication information is used to indicate a decoding simplification operation to be performed in the decoding neural network, where the decoding simplification operation is used to group convolution operations performed by at least one convolutional layer included in the decoding neural network;
[0264] According to the decoding instruction information, the decoding neural network is called to perform decoding simplification operations to decode the code stream to reconstruct the image.
[0265] In one embodiment, the decoding simplification operation includes a grouped convolution operation; the decoding indication information includes decoding grouped convolution indication information, and the decoding grouped convolution indication information is used to indicate that a grouped convolution operation is performed in at least one convolution layer of the decoding neural network. The grouped convolution operation includes: grouping the feature map of the image according to channels in the at least one convolution layer to obtain multiple groups; performing a convolution operation within each of the multiple groups to obtain grouped convolution features of each group, wherein the grouped convolution features of each group include the convolution features of the channels in the group. The output channels of each group in the multiple groups only interact with the input channels within the group and are independent of the channels in other groups in the multiple groups.
[0266] In one embodiment, the decoding group convolution indication information is set in the network structure of the convolution layer of the decoding neural network.
[0267] In one embodiment, the decoding group convolution indication information is further used to indicate the number of groups for the group convolution operation.
[0268] In one embodiment, the number of groups of the grouped convolution operation indicated by the decoding grouped convolution indication information supports dynamic adjustment as the network parameters of the convolution layer of the decoding neural network change.
[0269] In one embodiment, the number of the at least one convolutional layer of the decoding neural network is N, where N is a positive integer;
[0270] The decoding group convolution indication information is also used to indicate n convolution layers among the N convolution layers that need to perform group convolution operations, where n is a positive integer and n is less than or equal to N.
[0271] In one embodiment, the selection rules of the n convolutional layers include at least one of the following:
[0272] Select n convolutional layers to perform grouped convolution operations from the N convolutional layers in descending order of the distance between the N convolutional layers and the output layer;
[0273] Select n convolutional layers that need to perform group convolution operations from N convolutional layers in descending order of network structure complexity.
[0274] In one embodiment, the decoding group convolution indication information is further used to indicate the number of groups for performing group convolution operations on each convolution layer in the n convolution layers;
[0275] Different convolutional layers in the n convolutional layers have the same or different numbers of groups for grouped convolution operations.
[0276] In one embodiment, the decoding simplification operation further includes a channel reorganization operation; the decoding instruction information includes decoding channel reorganization instruction information, and the decoding channel reorganization instruction information is used to instruct to perform the channel reorganization operation in at least one convolutional layer of the decoding neural network. The channel reorganization operation includes: reordering the grouped convolution features of each group obtained by the grouped convolution operation to obtain reordered grouped convolution features.
[0277] In one embodiment, the decoding channel reorganization indication information is set in the network structure of the convolutional layer of the decoding neural network.
[0278] In one embodiment, the decoded channel reassembly indication information is further used to indicate the number of channels and / or channel identifiers for the channel reassembly operation.
[0279] In one embodiment, the number of channels of the channel reorganization operation indicated by the decoding channel reorganization indication information supports dynamic adjustment as the network parameters of the convolutional layer of the decoding neural network change.
[0280] In one embodiment, the convolution layer in at least one convolution layer includes P channels; the decoding channel reorganization indication information is further used to indicate p channels among the P channels on which channel reorganization operations need to be performed, where p is a positive integer and p is less than or equal to P.
[0281] In one embodiment, the selection rules for the p channels include at least one of the following:
[0282] According to the rule of reducing memory resource consumption, select p channels from P channels on which channel operations need to be performed; or,
[0283] The P channels correspond to multiple groups of the group convolution operation respectively. According to the data communication requirements between the groups, p channels that need to perform channel reorganization operations are selected from the P channels.
[0284] In one embodiment, the number of at least one convolutional layer of the decoding neural network is N, and the N convolutional layers include n convolutional layers that need to perform grouped convolution operations, N and n are both positive integers and n is less than or equal to N;
[0285] The decoding channel reorganization indication information is further used to indicate t convolutional layers among the n convolutional layers that need to perform channel reorganization operations, where t is a positive integer and t is less than or equal to n.
[0286] In one embodiment, the decoding channel reorganization indication information is further used to indicate the number of channels for performing the channel reorganization operation on each of the t convolutional layers;
[0287] The number of channels for channel reorganization operations performed by different convolutional layers in the t convolutional layers is the same or different.
[0288] In one embodiment, the decoding neural network includes at least one of the following: a super decoding subnetwork, a super-scale decoding subnetwork, and a synthetic transformation subnetwork; each subnetwork includes a convolutional layer;
[0289] The decoding simplification operation includes a group convolution operation. If the decoding indication information indicates that the convolution layer of each sub-network needs to perform the group convolution operation, the number of groups for the convolution layer of each sub-network indicated by the decoding indication information to perform the group convolution operation is the same or different;
[0290] The decoding simplification operation also includes a channel reorganization operation. If the decoding indication information indicates that the convolutional layer of each sub-network performs a channel reorganization operation, the number of channels of the convolutional layer of each sub-network indicated by the decoding indication information for performing the channel reorganization operation is the same or different.
[0291] In one embodiment, the decoding neural network includes at least one convolutional layer, and the decoding simplification operation includes a grouped convolution operation and a channel reorganization operation. When the processor 801 calls the decoding neural network to decode the bitstream according to the decoding instruction information to reconstruct the image, the processor 801 may specifically perform the following steps:
[0292] Calling the decoding neural network to decode the code stream according to the decoding instruction information to obtain the transformed value of the image;
[0293] Calling at least one convolution layer to perform a grouped convolution operation on the transformed values of the image to obtain a grouped convolution result corresponding to the transformed values of the image;
[0294] Perform channel reorganization operation on the grouped convolution result to obtain the channel reorganization result;
[0295] The channel reorganization processing result is decoded to reconstruct the image.
[0296] In an embodiment of the present application, a code stream formed after image encoding is obtained, and decoding instruction information is obtained, which is used to indicate the decoding simplification operation to be performed in the decoding neural network; then, the decoding neural network is called according to the decoding instruction information to decode the code stream to reconstruct the image. During the image decoding process, the instructions of the decoding instruction information can simplify the decoding operations in the decoding neural network at the decoding end, thereby effectively reducing the decoding complexity at the decoding end and improving decoding efficiency.
[0297] Optionally, when the computer device is the above-mentioned encoding device, in the embodiment of the present application, the processor 801 performs the following operations by running the program instructions in the memory 804:
[0298] Get the image to be processed;
[0299] Obtaining encoding indication information, where the encoding indication information is used to indicate an encoding simplification operation to be performed in the encoding neural network, where the encoding simplification operation is used to group convolution operations performed by at least one convolutional layer included in the encoding neural network;
[0300] According to the encoding instruction information, the encoding neural network is called to perform encoding simplification operations to encode the image and obtain a code stream.
[0301] In an embodiment of the present application, an image to be processed is obtained, and encoding instruction information is provided on the encoding device side. This encoding instruction information is used to indicate the encoding simplification operation to be performed in the encoding neural network. The encoding neural network is then called according to the encoding instruction information to perform encoding processing on the image to reconstruct the image. During the image encoding process, the instructions of the encoding instruction information can simplify the encoding operations in the encoding neural network of the encoding end, thereby effectively reducing the encoding complexity of the encoding end and improving encoding efficiency.
[0302] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and a computer program is stored in the computer-readable storage medium, and the computer program includes program instructions. When the processor executes the above program instructions, it can execute the method in the embodiment corresponding to Figures 4 and 5 above. Therefore, it will not be repeated here. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the program instructions can be deployed on a computer device, or executed on multiple computer devices located at one location, or, executed on multiple computer devices distributed at multiple locations and interconnected by a communication network.
[0303] According to one aspect of the present application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, so that the computer device can perform the methods described in the embodiments corresponding to FIG. 4 and FIG. 5 , and therefore, will not be described in detail herein.
[0304] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0305] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.
Claims
1. An image processing method, characterized in that: include: Obtaining a code stream formed after the image is encoded; Acquire decoding indication information, where the decoding indication information is used to indicate a decoding simplification operation to be performed in a decoding neural network, where the decoding simplification operation is used to group convolution operations performed by at least one convolution layer included in the decoding neural network; According to the decoding indication information, the decoding neural network is called to execute the decoding simplification operation to decode the code stream to reconstruct the image.
2. The method according to claim 1, characterized in that The decoding simplification operation includes a grouped convolution operation; the decoding indication information includes decoding grouped convolution indication information, and the decoding grouped convolution indication information is used to indicate that a grouped convolution operation is performed in at least one convolution layer of the decoding neural network; The grouped convolution operation includes: In the at least one convolutional layer, the feature map of the image is grouped according to channels to obtain a plurality of groups; A convolution operation is performed inside each of the multiple groups to obtain a grouped convolution feature of each group, wherein the grouped convolution feature of each group contains the convolution feature of the channel in the group.
3. The method according to claim 2, characterized in that The output channel of each group in the plurality of groups only interacts with the input channel in the group and is independent of the channels in other groups in the plurality of groups.
4. The method according to claim 2 or 3, characterized in that The decoding group convolution indication information is also used to indicate the number of groups of the group convolution operation.
5. The method according to claim 4, characterized in that The number of groups of the group convolution operation indicated by the decoding group convolution indication information supports dynamic adjustment as the network parameters of the convolution layer of the decoding neural network change.
6. The method according to any one of claims 2 to 5, characterized in that The number of at least one convolutional layer of the decoding neural network is N, where N is a positive integer; The decoding group convolution indication information is also used to indicate n convolution layers among N convolution layers that need to perform group convolution operations, where n is a positive integer and n is less than or equal to N.
7. The method according to claim 6, characterized in that The selection rules of the n convolutional layers include at least one of the following: Selecting n convolutional layers on which grouped convolution operations need to be performed from the N convolutional layers in descending order of distances between the N convolutional layers and the output layer; Select n convolutional layers that need to perform grouped convolution operations from the N convolutional layers in descending order of complexity of the network structure.
8. The method according to claim 6 or 7, characterized in that The decoding group convolution indication information is also used to indicate the number of groups for performing group convolution operations on each convolution layer in the n convolution layers; Different convolutional layers in the n convolutional layers have the same or different numbers of groups for performing grouped convolution operations.
9. The method according to any one of claims 2 to 8, characterized in that The decoding simplification operation further includes a channel reorganization operation; the decoding indication information includes decoding channel reorganization indication information, and the decoding channel reorganization indication information is used to indicate that a channel reorganization operation is performed in at least one convolutional layer of the decoding neural network, The channel reorganization operation includes: The grouped convolution features of each group obtained by the grouped convolution operation are reordered to obtain reordered grouped convolution features.
10. The method according to claim 9, characterized in that The decoding group convolution indication information and / or the decoding channel reorganization indication information are arranged in the network structure of the convolution layer of the decoding neural network.
11. The method according to claim 9 or 10, characterized in that The decoded channel reorganization indication information is also used to indicate the number of channels and / or channel identifiers of the channel reorganization operation.
12. The method according to claim 11, characterized in that The number of channels of the channel reorganization operation indicated by the decoding channel reorganization indication information supports dynamic adjustment as the network parameters of the convolutional layer of the decoding neural network change.
13. The method according to any one of claims 9 to 12, characterized in that: One convolution layer among the at least one convolution layer includes P channels; the decoding channel reorganization indication information is also used to indicate p channels among the P channels on which channel reorganization operations need to be performed, where p is a positive integer and p is less than or equal to P.
14. The method according to claim 13, characterized in that The selection rules of the p channels include at least one of the following: According to the rule of reducing memory resource consumption, p channels on which channel operations need to be performed are selected from the P channels; or, The P channels correspond to the multiple groups of the group convolution operation respectively, and according to the data communication requirements between the groups, p channels that need to perform channel reorganization operations are selected from the P channels.
15. The method according to any one of claims 9 to 14, characterized in that: The number of at least one convolutional layer of the decoding neural network is N, and the N convolutional layers include n convolutional layers that need to perform grouped convolution operations, N and n are both positive integers and n is less than or equal to N; The decoding channel reorganization indication information is also used to indicate t convolutional layers among the n convolutional layers that need to perform channel reorganization operations, where t is a positive integer and t is less than or equal to n.
16. The method according to claim 15, characterized in that The decoding channel reorganization indication information is also used to indicate the number of channels for each convolutional layer in the t convolutional layers to perform a channel reorganization operation; Different convolutional layers in the t convolutional layers have the same or different numbers of channels for performing channel reorganization operations.
17. The method according to any one of claims 1 to 16, characterized in that: The decoding neural network includes at least one of the following: a super decoding subnetwork, a super-scale decoding subnetwork, and a synthetic transformation subnetwork; each subnetwork includes a convolutional layer; The decoding simplification operation includes a group convolution operation. If the decoding indication information indicates that the convolution layer of each sub-network needs to perform a group convolution operation, the number of groups for performing the group convolution operation on the convolution layer of each sub-network indicated by the decoding indication information is the same or different; The decoding simplification operation also includes a channel reorganization operation. If the decoding indication information indicates that the convolutional layers of each sub-network perform a channel reorganization operation, the number of channels of the convolutional layers of each sub-network indicated by the decoding indication information for performing the channel reorganization operation is the same or different.
18. The method according to any one of claims 9 to 17, characterized in that The step of calling the decoding neural network according to the decoding instruction information and performing the decoding simplification operation to decode the code stream to reconstruct the image includes: Calling the decoding neural network to decode the bitstream according to the decoding instruction information to obtain a transformation value of the image; Calling the at least one convolution layer to perform a grouped convolution operation on the transformed values of the image to obtain a grouped convolution result corresponding to the transformed values of the image, wherein the transformed values of the image are included in a feature map of the image; Performing a channel reorganization operation on the grouped convolution result to obtain a channel reorganization result; The channel reorganization processing result is decoded to reconstruct the image.
19. An image processing method, characterized in that: include: Obtain the image to be processed; Acquire coding indication information, where the coding indication information is used to indicate a coding simplification operation to be performed in the coding neural network, where the coding simplification operation is used to group convolution operations performed by at least one convolution layer included in the coding neural network; According to the encoding instruction information, the encoding neural network is called to perform the encoding simplification operation to encode the image to obtain a code stream.
20. An image processing device, characterized in that: include: An acquisition unit, used for acquiring a code stream formed after the image is encoded; The acquisition unit is further used to acquire decoding indication information, wherein the decoding indication information is used to indicate a decoding simplification operation to be performed in the decoding neural network, and the decoding simplification operation is used to group convolution operations performed by at least one convolution layer included in the decoding neural network; A processing unit is used to call the decoding neural network according to the decoding indication information to perform the decoding simplification operation to decode the code stream to reconstruct the image.
21. An image processing device, characterized in that: include: An acquisition unit, used for acquiring an image to be processed; The acquisition unit is further used to acquire coding indication information, where the coding indication information is used to indicate a coding simplification operation to be performed in the coding neural network, where the coding simplification operation is used to group convolution operations performed by at least one convolution layer included in the coding neural network; A processing unit is used to call the encoding neural network according to the encoding indication information to perform the encoding simplification operation to encode the image and obtain a code stream.
22. A computer device, characterized in that: include: a processor suitable for executing a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the image processing method according to any one of claims 1 to 19 is executed.
23. A computer-readable storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by a processor, the image processing method according to any one of claims 1 to 19 is executed.
Citation Information
Patent Citations
Image processing method, image processing device and computer readable storage medium
CN108875904A
Efficient semantic segmentation method based on grouping asymmetric convolution
CN111401373A
Adaptive packet convolutional neural network structure design method based on attention mechanism
CN112801289A
Image face reconstruction method based on encoder-decoder structure
CN113887296A
Dynamic aggregation converter network and retinal vessel segmentation method
CN115272369A
