Image processing method and related equipment

By providing decoding instructions in image processing and adopting packet convolution and channel recombination operations, the problem of high decoding complexity in the prior art is solved, and efficient image decoding on mobile devices is realized.

CN119027520APending Publication Date: 2024-11-26TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310600970.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing end-to-end image compression neural network model has high decoding complexity and low decoding efficiency, and cannot effectively support mobile devices.

Method used

By providing decoding instructions, operations in decoding neural networks are simplified, packet convolution and channel recombination operations are adopted to reduce decoding complexity and improve decoding efficiency.

Benefits of technology

Effectively reduce the complexity of the decoding end, improve the decoding efficiency, and enable image processing to be supported more smoothly on mobile devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027520A_ABST
    Figure CN119027520A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides an image processing method and related equipment. The method comprises the following steps: acquiring a code stream formed by encoding an image; obtaining decoding indication information, wherein the decoding indication information is used for indicating a decoding simplification operation needing to be executed in the decoding neural network; and calling a decoding neural network to decode the code stream according to the decoding indication information so as to reconstruct an image. According to the embodiment of the invention, the decoding complexity of the decoding end can be reduced, and the decoding efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, an image processing apparatus, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In the neural network model of end-to-end image compression, the encoder maps the original image to a latent variable through the analysis transformation sub-network, and writes the latent variable into the bitstream through the entropy encoder; the decoder decodes the latent variable from the bitstream through the entropy decoder, and then inputs the latent variable into the synthesis transformation sub-network to obtain the reconstructed image. However, the current image encoding and decoding scheme has problems such as high decoding complexity, low decoding efficiency, and poor support by mobile devices. Summary of the invention

[0003] The embodiments of the present application provide an image processing method and related devices, which can reduce the decoding complexity of the decoding end and improve the decoding efficiency.

[0004] On the one hand, an embodiment of the present application provides an image processing method, the method comprising:

[0005] Obtain a code stream formed after the image is encoded;

[0006] Obtaining decoding indication information, where the decoding indication information is used to indicate a decoding simplification operation to be performed in a decoding neural network;

[0007] The decoding neural network is called according to the decoding instruction information to decode the bit stream to reconstruct the image.

[0008] On the one hand, an embodiment of the present application provides an image processing method, the method comprising:

[0009] Get the image to be processed;

[0010] Obtaining encoding indication information, where the encoding indication information is used to indicate an encoding simplification operation to be performed in the encoding neural network;

[0011] The encoding neural network is called according to the encoding instruction information to encode the image and obtain a code stream.

[0012] In one embodiment, the encoding neural network includes a convolution layer; the encoding simplification operation includes a grouped convolution operation; the encoding indication information includes encoding grouped convolution indication information, and the encoding grouped convolution indication information is used to indicate that a grouped convolution operation is performed in the convolution layer of the encoding neural network.

[0013] In one embodiment, the encoding grouped convolution indication information is also used to indicate the number of groups of grouped convolution operations performed in the convolution layer of the encoding neural network.

[0014] In one embodiment, the number of groups of the grouped convolution operation indicated by the encoding grouped convolution indication information supports dynamic adjustment as the network parameters of the convolution layer of the encoding neural network change.

[0015] In one embodiment, the number of convolutional layers is M, where M is a positive integer;

[0016] The encoded grouped convolution indication information is also used to indicate m convolution layers among the M convolution layers that need to perform grouped convolution operations, where m is a positive integer and m is less than or equal to M.

[0017] In one embodiment, the selection rules of the m convolutional layers include at least one of the following:

[0018] Select m convolutional layers that need to perform grouped convolution operations from the M convolutional layers in descending order of the distance between the convolutional layer and the output layer;

[0019] Select m convolutional layers that need to perform grouped convolution operations from M convolutional layers in descending order of the complexity of the network structure.

[0020] In one embodiment, the encoded grouped convolution indication information is further used to indicate the number of groups for performing grouped convolution operations in each of the m convolution layers;

[0021] Different convolutional layers in the m convolutional layers have the same or different numbers of groups for grouped convolution operations.

[0022] In one embodiment, the coding simplification operation also includes a channel reorganization operation; the coding indication information includes coding channel reorganization indication information, and the coding channel reorganization indication information is used to indicate that the channel reorganization operation is performed in the convolution layer of the coding neural network.

[0023] In one embodiment, the encoding channel reorganization indication information is set in the network structure of the convolutional layer of the encoding neural network.

[0024] In one embodiment, the encoding channel reorganization indication information is also used to indicate the number of channels of the channel reorganization operation performed in the convolutional layer of the encoding neural network.

[0025] In one embodiment, the number of channels of the channel reorganization operation indicated by the encoding channel reorganization indication information supports dynamic adjustment as the network parameters of the convolutional layer of the encoding neural network change.

[0026] In one embodiment, the convolution layer includes I channels; the encoded channel reorganization indication information is also used to indicate O channels among the I channels that need to perform channel reorganization operations, where O is a positive integer and O is less than or equal to I.

[0027] In one embodiment, the selection rules of the O channels include at least one of the following:

[0028] According to the rule of reducing memory resource consumption, select O channels to perform channel operations from I channels; or,

[0029] The O channels correspond to multiple groups of the group convolution operation respectively. According to the data communication requirements between the groups, O channels that need to perform channel reorganization operations are selected from the I channels.

[0030] In one embodiment, the number of convolutional layers is M, and the M convolutional layers include m convolutional layers that need to perform grouped convolution operations, and M and m are both positive integers and m is less than or equal to M;

[0031] The coded channel reorganization indication information is further used to indicate q convolutional layers among the m convolutional layers that need to perform the reorganization operation, where q is a positive integer and q is less than or equal to m.

[0032] In one embodiment, the coded grouped convolution indication information is further used to indicate the number of channels for each convolution layer in the q convolution layers to perform a channel reorganization operation;

[0033] Different convolutional layers in the q convolutional layers have the same or different numbers of channels for channel reorganization operations.

[0034] In one embodiment, the encoding neural network includes at least one of the following: an analysis transformation subnetwork, a super encoding subnetwork;

[0035] The coding simplification operation includes a group convolution operation. If the coding indication information indicates that the convolution layer of each sub-network needs to perform the group convolution operation, the number of groups for the group convolution operation performed by the convolution layer of each sub-network indicated by the coding indication information is the same or different;

[0036] The coding simplification operation also includes a channel reorganization operation. If the coding indication information indicates that the convolutional layers of each sub-network perform a channel reorganization operation, the number of channels of the convolutional layers of each sub-network indicated by the coding indication information for performing the channel reorganization operation is the same or different.

[0037] On the one hand, an embodiment of the present application provides an image processing device, the device comprising:

[0038] An acquisition unit, used for acquiring a code stream formed after the image is encoded;

[0039] The acquisition unit is further used to acquire decoding indication information, where the decoding indication information is used to indicate a decoding simplification operation to be performed in the decoding neural network;

[0040] The processing unit is also used to call the decoding neural network to decode the code stream according to the decoding instruction information to reconstruct the image.

[0041] In one embodiment, the decoding neural network includes a convolution layer; the decoding simplification operation includes a grouped convolution operation; the decoding indication information includes decoding grouped convolution indication information, and the decoding grouped convolution indication information is used to indicate that a grouped convolution operation is performed in the convolution layer of the decoding neural network.

[0042] In one embodiment, the decoding group convolution indication information is set in the network structure of the convolution layer of the decoding neural network.

[0043] In one embodiment, the decoded group convolution indication information is also used to indicate the number of groups for the group convolution operation.

[0044] In one embodiment, the number of groups of the grouped convolution operation indicated by the decoding grouped convolution indication information supports dynamic adjustment as the network parameters of the convolution layer of the decoding neural network change.

[0045] In one embodiment, the number of convolutional layers of the decoding neural network is N, where N is a positive integer;

[0046] The decoding group convolution indication information is also used to indicate n convolution layers among the N convolution layers that need to perform group convolution operations, where n is a positive integer and n is less than or equal to N.

[0047] In one embodiment, the selection rules of the n convolutional layers include at least one of the following:

[0048] Select n convolutional layers that need to perform grouped convolution operations from N convolutional layers in descending order of the distance between the convolutional layer and the output layer;

[0049] Select n convolutional layers that need to perform grouped convolution operations from N convolutional layers in descending order of the complexity of the network structure.

[0050] In one embodiment, the decoded group convolution indication information is further used to indicate the number of groups for performing group convolution operations on each convolution layer in the n convolution layers;

[0051] Different convolutional layers in the n convolutional layers may have the same or different numbers of groups for grouped convolution operations.

[0052] In one embodiment, the decoding simplification operation also includes a channel reorganization operation; the decoding indication information includes decoding channel reorganization indication information, and the decoding channel reorganization indication information is used to indicate that the channel reorganization operation is performed in the convolution layer of the decoding neural network.

[0053] In one embodiment, the decoding channel reorganization indication information is set in the network structure of the convolutional layer of the decoding neural network.

[0054] In one embodiment, the decoded channel reorganization indication information is further used to indicate the number of channels for the channel reorganization operation.

[0055] In one embodiment, the number of channels of the channel reorganization operation indicated by the decoding channel reorganization indication information supports dynamic adjustment as the network parameters of the convolutional layer of the decoding neural network change.

[0056] In one embodiment, the convolution layer includes P channels; the decoded channel reorganization indication information is further used to indicate p channels among the P channels that need to perform channel reorganization operations, where p is a positive integer and p is less than or equal to P.

[0057] In one embodiment, the selection rules of the p channels include at least one of the following:

[0058] According to the rule of reducing memory resource consumption, p channels to be used for channel operation are selected from P channels; or,

[0059] The P channels correspond to a plurality of groups of the group convolution operation respectively. According to the data communication requirements between the groups, p channels that need to perform the channel reorganization operation are selected from the P channels.

[0060] In one embodiment, the number of convolutional layers of the decoding neural network is N, and the N convolutional layers include n convolutional layers that need to perform grouped convolution operations, N and n are both positive integers and n is less than or equal to N;

[0061] The decoding channel reorganization indication information is also used to indicate t convolutional layers among the n convolutional layers that need to perform channel reorganization operations, where t is a positive integer and t is less than or equal to n.

[0062] In one embodiment, the decoding channel reorganization indication information is further used to indicate the number of channels for performing the channel reorganization operation on each of the t convolutional layers;

[0063] The number of channels for channel reorganization operations performed by different convolutional layers in the t convolutional layers is the same or different.

[0064] In one embodiment, the decoding neural network includes at least one of the following: a super decoding subnetwork, a super-scale decoding subnetwork, and a synthetic transformation subnetwork; each subnetwork includes a convolutional layer;

[0065] The decoding simplification operation includes a group convolution operation. If the decoding indication information indicates that the convolution layer of each sub-network needs to perform the group convolution operation, the number of groups for the group convolution operation performed by the convolution layer of each sub-network indicated by the decoding indication information is the same or different;

[0066] The decoding simplification operation also includes a channel reorganization operation. If the decoding indication information indicates that the convolutional layers of each sub-network perform a channel reorganization operation, the number of channels of the convolutional layers of each sub-network indicated by the decoding indication information for performing the channel reorganization operation is the same or different.

[0067] In one embodiment, the decoding neural network includes a convolution layer, and the decoding simplification operation includes a grouped convolution operation and a channel reorganization operation; the processing unit is specifically used to:

[0068] Calling a decoding neural network to decode the bitstream according to the decoding instruction information to obtain a transformation value of the image;

[0069] Call the convolution layer to perform grouped convolution operations on the transformed values ​​of the image to obtain grouped convolution results corresponding to the transformed values ​​of the image;

[0070] Perform channel reorganization operation on the grouped convolution result to obtain a channel reorganization result;

[0071] The channel reorganization processing result is decoded to reconstruct the image.

[0072] On the one hand, an embodiment of the present application provides an image processing device, the device comprising:

[0073] An acquisition unit, used for acquiring an image to be processed;

[0074] The acquisition unit is further used to acquire encoding indication information, where the encoding indication information is used to indicate the encoding simplification operation to be performed in the encoding neural network;

[0075] The processing unit is used to call the encoding neural network to encode the image according to the encoding instruction information to obtain a code stream.

[0076] In one embodiment, the encoding neural network includes a convolution layer; the encoding simplification operation includes a grouped convolution operation; the encoding indication information includes encoding grouped convolution indication information, and the encoding grouped convolution indication information is used to indicate that a grouped convolution operation is performed in the convolution layer of the encoding neural network.

[0077] In one embodiment, the encoding grouped convolution indication information is also used to indicate the number of groups of grouped convolution operations performed in the convolution layer of the encoding neural network.

[0078] In one embodiment, the number of groups of the grouped convolution operation indicated by the encoding grouped convolution indication information supports dynamic adjustment as the network parameters of the convolution layer of the encoding neural network change.

[0079] In one embodiment, the number of convolutional layers is M, where M is a positive integer;

[0080] The encoded grouped convolution indication information is also used to indicate m convolution layers among the M convolution layers that need to perform grouped convolution operations, where m is a positive integer and m is less than or equal to M.

[0081] In one embodiment, the selection rules of the m convolutional layers include at least one of the following:

[0082] Select m convolutional layers that need to perform grouped convolution operations from the M convolutional layers in descending order of the distance between the convolutional layer and the output layer;

[0083] Select m convolutional layers that need to perform grouped convolution operations from M convolutional layers in descending order of the complexity of the network structure.

[0084] In one embodiment, the encoded grouped convolution indication information is further used to indicate the number of groups for performing grouped convolution operations in each of the m convolution layers;

[0085] Different convolutional layers in the m convolutional layers have the same or different numbers of groups for grouped convolution operations.

[0086] In one embodiment, the coding simplification operation also includes a channel reorganization operation; the coding indication information includes coding channel reorganization indication information, and the coding channel reorganization indication information is used to indicate that the channel reorganization operation is performed in the convolution layer of the coding neural network.

[0087] In one embodiment, the encoding channel reorganization indication information is set in the network structure of the convolutional layer of the encoding neural network.

[0088] In one embodiment, the encoding channel reorganization indication information is also used to indicate the number of channels of the channel reorganization operation performed in the convolutional layer of the encoding neural network.

[0089] In one embodiment, the number of channels of the channel reorganization operation indicated by the encoding channel reorganization indication information supports dynamic adjustment as the network parameters of the convolutional layer of the encoding neural network change.

[0090] In one embodiment, the convolution layer includes I channels; the encoded channel reorganization indication information is also used to indicate O channels among the I channels that need to perform channel reorganization operations, where O is a positive integer and O is less than or equal to I.

[0091] In one embodiment, the selection rules of the O channels include at least one of the following:

[0092] According to the rule of reducing memory resource consumption, select O channels to perform channel operations from I channels; or,

[0093] The O channels correspond to multiple groups of the group convolution operation respectively. According to the data communication requirements between the groups, O channels that need to perform channel reorganization operations are selected from the I channels.

[0094] In one embodiment, the number of convolutional layers is M, and the M convolutional layers include m convolutional layers that need to perform grouped convolution operations, and M and m are both positive integers and m is less than or equal to M;

[0095] The coded channel reorganization indication information is further used to indicate q convolutional layers among the m convolutional layers that need to perform the reorganization operation, where q is a positive integer and q is less than or equal to m.

[0096] In one embodiment, the coded grouped convolution indication information is further used to indicate the number of channels for each convolution layer in the q convolution layers to perform a channel reorganization operation;

[0097] Different convolutional layers in the q convolutional layers have the same or different numbers of channels for channel reorganization operations.

[0098] In one embodiment, the encoding neural network includes at least one of the following: an analysis transformation subnetwork, a super encoding subnetwork;

[0099] The coding simplification operation includes a group convolution operation. If the coding indication information indicates that the convolution layer of each sub-network needs to perform the group convolution operation, the number of groups for the group convolution operation performed by the convolution layer of each sub-network indicated by the coding indication information is the same or different;

[0100] The coding simplification operation also includes a channel reorganization operation. If the coding indication information indicates that the convolutional layers of each sub-network perform a channel reorganization operation, the number of channels of the convolutional layers of each sub-network indicated by the coding indication information for performing the channel reorganization operation is the same or different.

[0101] In one aspect, an embodiment of the present application provides a computer device, the computer device comprising:

[0102] a processor suitable for executing a computer program;

[0103] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned image processing method is implemented.

[0104] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is loaded by a processor and executes the above-mentioned image processing method.

[0105] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program or computer instructions, and the computer program or computer instructions implement the above-mentioned image processing method when executed by a processor.

[0106] In the embodiment of the present application, a code stream formed after the image is encoded is obtained, and decoding indication information is obtained, the decoding indication information is used to indicate the decoding simplification operation to be performed in the decoding neural network; then the decoding neural network is called according to the decoding indication information to decode the code stream to reconstruct the image. In the process of decoding the image, the operation of decoding in the decoding neural network at the decoding end can be simplified through the indication of the decoding indication information, thereby effectively reducing the decoding complexity of the decoding end and improving the decoding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0108] Figure 1a A structural diagram of an end-to-end image coding model based on deep learning provided for an exemplary embodiment of the present application;

[0109] Figure 1b A structural diagram of an end-to-end image coding model based on deep learning provided by another exemplary embodiment of the present application;

[0110] Figure 2a A schematic diagram showing a network structure comparison of a synthesis transformation sub-network and a lightweight synthesis transformation sub-network when processing chrominance components and luminance components, respectively, provided as an exemplary embodiment of the present application;

[0111] Figure 2b A schematic diagram of a structural comparison between a residual block and a lightweight residual block provided for an exemplary embodiment of the present application;

[0112] Figure 2c A schematic diagram showing a network structure comparison of a super decoding sub-network and a lightweight super decoding sub-network when processing chrominance components and luminance components, respectively, provided as an exemplary embodiment of the present application;

[0113] Figure 2d A schematic diagram showing a network structure comparison of a super-scale decoding sub-network and a lightweight super-scale decoding sub-network when processing chrominance components and luminance components, respectively, provided as an exemplary embodiment of the present application;

[0114] Figure 2e A schematic diagram comparing a conventional convolution operation and a grouped convolution operation provided for an exemplary embodiment of the present application;

[0115] Figure 2fA schematic diagram of a channel reorganization provided for an exemplary embodiment of the present application;

[0116] Figure 3 An architecture diagram of an image processing system provided for an exemplary embodiment of the present application;

[0117] Figure 4 A flowchart of an image processing method provided for an exemplary embodiment of the present application;

[0118] Figure 5 A flowchart of an image processing method provided for another exemplary embodiment of the present application;

[0119] Figure 6 A schematic diagram of the structure of an image processing device provided for an exemplary embodiment of the present application;

[0120] Figure 7 A schematic diagram of the structure of an image processing device provided for another exemplary embodiment of the present application;

[0121] Figure 8 A schematic diagram of the structure of a computer device is provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0122] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0123] First, the relevant technical terms involved in this application are introduced.

[0124] 1. Image

[0125] Image refers to the image to be encoded, which can be any frame or multiple frames in the video. The pixel format of the image can be RGB or YUV. When the pixel format of the image is RGB, the image can include R (Red) component, G (Green) component and B (Blue) component; when the pixel format of the image to be processed is YUV, the image can include brightness (Y) component and chrominance (UV) component.

[0126] 2. End-to-end image coding model based on deep learning

[0127] The embodiments of the present application relate to an end-to-end image coding model based on deep learning. Schematically, the end-to-end image coding model based on deep learning can be applied to the JPEG (Joint Photographic Experts Group) AI (Artificial Intelligence) platform. Schematically, the end-to-end image coding model based on deep learning can be a VM model (Verification Model). Figure 1a As shown, it is a structural diagram of an end-to-end image coding model based on deep learning provided by an exemplary embodiment of the present application. The end-to-end image coding model based on deep learning adopts a transform-coding structure. The end-to-end image coding model based on deep learning mainly includes: Analysis Transform Net, Context Model Net, Hyper Encoder Net, Prediction Fusion Net, Hyper Decoder Net, Hyper Scale Decoder Net, Synthesis Transform Net. The above subnetworks are introduced in detail below:

[0128] ①Analysis transformation sub-network

[0129] The analysis transformation subnetwork is used to perform nonlinear transformation on the image to be encoded (x) to obtain the transformation result y (or hidden variable) of the image. Y ) and the chrominance component (x UV ), the analysis transformation subnetwork can perform nonlinear transformation on the luminance component and the chrominance component respectively, and obtain the transformation result corresponding to the luminance component (y Y ) and the corresponding transformation result of the chromaticity component (y UV ).

[0130] ②Context model sub-network

[0131] The context model subnetwork is used to transform the image Context processing is performed, and the output data of the context model sub-network is used as the input of the prediction fusion sub-network. It is obtained by superimposing the predicted value (u) of the image output by the prediction fusion subnetwork and the data output by the residual inverse gain unit (invG unitRes, inverse Gain unitresidual). The input data of the residual inverse gain unit is the image residual

[0132] The image residual The data output by the residual gain unit (G unitRes, Gain unit residual) can be quantized. Specifically, the data type output by the residual gain unit is a floating point type, and the quantization process includes a data type conversion process (i.e., rounding process); the data output by the residual gain unit is rounded to obtain an integer image residual. The input data of the residual gain unit is the residual (r) obtained by performing a difference operation between the transformed result (y) of the image and the predicted value (u) output by the prediction fusion sub-network.

[0133] When the image includes a luminance component (x Y ) and the chrominance component (x UV ), the residual error of the brightness component (r Y ) can be the transformed value of the brightness component (y Y ) and the predicted value of the brightness component output by the prediction fusion sub-network (u Y ) is obtained by performing a difference operation. The residual error of the chrominance component (r UV ) can be the transformed value of the chrominance component (y UV ) and the predicted value of the chrominance component output by the prediction fusion sub-network (u UV ) is obtained by performing a difference operation. Then r Y and r UV Quantize it and convert it into an integer ( and ) to generate a code stream.

[0134] ③ The prediction fusion sub-network is used to predict the image and obtain the predicted value of the image (u). The input data of the prediction fusion sub-network is the output data of the context model sub-network and the output data of the super decoding sub-network. Schematically, the prediction fusion sub-network is used to predict the chrominance component and the luminance component of the image and obtain the predicted value of the luminance component (u Y ), and the predicted value of the chrominance component (u UV ).

[0135] ③ The super coding sub-network is used to process the transformation result y of the image, and then quantize the output data of the super coding sub-network to obtain the quantization result of the image, and perform lossless encoding on the quantization result to obtain a bitstream. The quantization process includes rounding, and at this time, the output data of the super coding sub-network is quantized to obtain the quantization result of the image, which may include: rounding the output data of the super coding sub-network to obtain the quantization result of the image.

[0136] ④The input data of the super decoding sub-network is the hyperparameters obtained based on the bitstream The output data of the super decoding sub-network is used as the input data of the prediction fusion sub-network.

[0137] ⑤The super-scale decoding sub-network is used based on the hyper-parameters Determine the mean to be 0 and the variance to be Gaussian distribution The synthesis transformation subnetwork is used to transform the transformed values ​​decoded from the bitstream Perform synthetic transformation processing to reconstruct the image Schematically, the synthesis transformation subnetwork is used to transform the luminance component obtained from the bitstream decoding. Performs synthesis transformation processing and transforms the chrominance components obtained from the decoded bitstream Perform synthetic transformation processing to obtain the brightness component of the reconstructed image and chrominance components

[0138] In addition, the end-to-end image coding model based on deep learning can also be a lightweight VM model. Figure 1b As shown in FIG. 1 , a structural diagram of an end-to-end image coding model based on deep learning is provided as another exemplary embodiment of the present application. Figure 1b As shown in the figure, the end-to-end image coding model based on deep learning mainly includes a lightweight analysis transform sub-network (Light weighted Analysis Transform Net), a hyper encoding sub-network (Hyper EncoderNet), a lightweight hyper decoding sub-network (Light weighted Hyper Decoder Net), a lightweight hyper scale decoder sub-network (Light weighted Hyper Scale Decoder Net), and a lightweight synthesis transform sub-network (Lightweighted Synthesis Transform Net).

[0139] It should be understood that the lightweight analysis transformation subnetwork has the same function as the analysis transformation subnetwork, but the complexity of the lightweight analysis transformation subnetwork is lower than the complexity of the analysis transformation subnetwork; the lightweight super-decoding subnetwork has the same function as the super-decoding subnetwork, but the complexity of the lightweight super-decoding subnetwork is lower than the complexity of the super-decoding subnetwork; the lightweight super-scale decoding subnetwork has the same function as the super-scale decoding subnetwork, but the complexity of the lightweight super-scale decoding subnetwork is lower than the complexity of the super-scale decoding subnetwork; the lightweight synthesis transformation subnetwork has the same function as the synthesis transformation subnetwork, but the complexity of the lightweight synthesis transformation subnetwork is lower than the complexity of the synthesis transformation subnetwork. The following is an exemplary example of Figure 1a and Figure 1b The network structures of the corresponding sub-networks are compared:

[0140] (1) Comparison of the network structure of the synthetic transformation subnetwork and the network structure of the lightweight synthetic transformation subnetwork.

[0141] The synthetic transformation subnetwork can be used to perform synthetic transformation processing on the transformation values ​​of the brightness component and the transformation values ​​of the chrominance component of the image. The lightweight synthetic transformation subnetwork can be used to perform synthetic transformation processing on the transformation values ​​of the brightness component and the transformation values ​​of the chrominance component of the image. Figure 2a , which is a schematic diagram of the network structure comparison of a synthesis transformation subnetwork and a lightweight synthesis transformation subnetwork provided by an exemplary embodiment of the present application when processing chrominance components and luminance components respectively.

[0142] ① Comparison of the network structure of the synthesis transform subnetwork (Synthesis Transform Net_Y) when processing the brightness component and the network structure of the lightweight synthesis transform subnetwork (Light weighted Synthesis Transform NetY) when processing the brightness component.

[0143] like Figure 2aAs shown in 21, the synthetic transformation subnetwork when processing the brightness component mainly includes two residual blocks (ResBlock), four convolutional layers, four cropping layers (Crop), three residual nonlinear units (ResAU), and an attention module (residual non-local attention block, RNAB). Among them, the four convolutional layers are from top to bottom: DCnv128×3×3S2 (DConv means deconvolution, the number of channels is 128, the convolution kernel size is 3×3, and S2 means the convolution step is 2), DCnv128×3×3S2 (DConv means deconvolution, the number of channels is 128, the convolution kernel size is 3×3, and S2 means the convolution step is 2), DCnv128×3×3S2 (DConv means deconvolution, the number of channels is 128, the convolution kernel size is 3×3, and S2 means the convolution step is 2), and DCnv1×3×3S2 (DConv means deconvolution, the number of channels is 1, the convolution kernel size is 3×3, and S2 means the convolution step is 2).

[0144] like Figure 2a As shown in 22, the lightweight synthesis transformation subnetwork when processing the brightness component mainly includes a lightweight residual block (LightResBlock), four convolutional layers, three cropping layers (Crop), three residual non-linear units (ResAU) and 1 pixel reorganization (pixelshuffleS4). Among them, the four convolutional layers are from top to bottom: DCnv96×4×4S2 (DConv means deconvolution, the number of channels is 96, the convolution kernel size is 4×4, and S2 means the convolution step is 2), DCnv64×4×4S2 (DConv means deconvolution, the number of channels is 64, the convolution kernel size is 4×4, and S2 means the convolution step is 2), Conv64×3×3S1 (Conv means convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 means the convolution step is 1), and Conv16×1×1S1 (Conv means convolution, the number of channels is 16, the convolution kernel size is 1×1, and S1 means the convolution step is 1).

[0145] from Figure 2a Comparing 21 and 22 in , when processing the brightness component, the network structure of the lightweight synthetic transformation subnetwork only requires one lightweight residual block and does not require RNAB, while the synthetic transformation subnetwork requires two residual blocks, but requires the participation of RNAB. In addition, the network parameters of the lightweight synthetic transformation subnetwork and the synthetic transformation subnetwork in the convolutional layer have also changed. It can be seen that the network structure of the lightweight synthetic transformation subnetwork is less complex than that of the synthetic transformation subnetwork.

[0146] ② Comparison of the network structure of the synthesis transform subnetwork (Synthesis Transform Net_UV) when processing chrominance components and the network structure of the lightweight synthesis transform subnetwork (Light weighted Synthesis TransformNet_UV) when processing chrominance components.

[0147] like Figure 2a As shown in 23, the synthetic transformation subnetwork when processing the chrominance component mainly includes two residual blocks (ResBlock), four convolutional layers, four cropping layers (Crop), three residual nonlinear units (ResAU), and an attention module (RNAB). Among them, the four convolutional layers are from top to bottom: DCnv64×3×3S2 (Dconv means deconvolution, the number of channels is 64, the convolution kernel size is 3×3, and S2 means the convolution step is 2), DCnv64×3×3S2 (DConv means deconvolution, the number of channels is 64, the convolution kernel size is 3×3, and S2 means the convolution step is 2), DCnv64×3×3S2 (DConv means deconvolution, the number of channels is 64, the convolution kernel size is 3×3, and S2 means the convolution step is 2), and DCnv2×3×3S2 (DConv means deconvolution, the number of channels is 2, the convolution kernel size is 3×3, and S2 means the convolution step is 2).

[0148] like Figure 2a As shown in 24, the lightweight synthesis transformation subnetwork when processing the chrominance component mainly includes a lightweight residual block (LightResBlock), four convolutional layers, three cropping layers (Crop), three residual non-linear units (ResAU) and 1 pixel reorganization (pixelshuffleS4). Among them, the four convolutional layers are from top to bottom: DConv64×4×4S2 (Dconv means deconvolution, the number of channels is 64, the convolution kernel size is 4×4, and S2 means the convolution step is 2), DConv64×4×4S2 (Dconv means deconvolution, the number of channels is 64, the convolution kernel size is 4×4, and S2 means the convolution step is 2), Conv64×3×3S1 (Conv means convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 means the convolution step is 1), and Conv32×1×1S1 (Conv means convolution, the number of channels is 16, the convolution kernel size is 1×1, and S1 means the convolution step is 1).

[0149] from Figure 2aComparing 23 and 24 in , when processing the chrominance component, the network structure of the lightweight synthesis transformation subnetwork only requires one lightweight residual block and does not require RNAB, while the synthesis transformation subnetwork requires two residual blocks, but requires the participation of RNAB. In addition, the network parameters of the lightweight synthesis transformation subnetwork and the synthesis transformation subnetwork in the convolutional layer have also changed. It can be seen that the network structure of the lightweight synthesis transformation subnetwork is less complex than that of the synthesis transformation subnetwork.

[0150] It should be understood that the structure of the above residual block and the structure of the lightweight residual block are also different. Figure 2b , is a schematic diagram of a structural comparison between a residual block and a lightweight residual block provided by an exemplary embodiment of the present application. Figure 2b In , the residual block may include an input layer (such as an input image x), two convolutional layers (Conv3×3S1G1 and Conv3×3S1G1, respectively, where the number of channels is 3×3, S1 indicates a step size of 1, and G1 indicates a grouping number of convolution operations of 1), and an activation function (Relu). The lightweight residual block includes an input layer, a convolutional layer (Conv3×3S1G1), and an activation function (Relu). It can be seen that the structural complexity of the lightweight residual block is lower than that of the residual block.

[0151] (2) Comparison of the network structure of the hyperdecoding subnetwork (HyperDecoder) and the network structure of the lightweight hyperdecoding subnetwork (Lightweighted HyperDecoder).

[0152] See also Figure 2c , which is a schematic diagram of network structure comparison of a super decoding subnetwork and a lightweight super decoding subnetwork when processing chrominance components and luminance components respectively, provided by an exemplary embodiment of the present application.

[0153] ① Comparison of the network structure of the hyper decoding subnetwork (Hyper Decoder_Y) when processing the luminance component and the network structure of the lightweight hyper decoding subnetwork (Light weighted Hyper Decoder_Y) when processing the luminance component.

[0154] like Figure 2cAs shown in 25, the super decoding subnetwork when processing the brightness component mainly includes five convolutional layers, two cropping layers (Crop), and three activation functions (LeakyRelu). Among them, the five convolutional layers are from top to bottom: Conv128×3×3S1 (Conv means convolution, the number of channels is 128, the convolution kernel size is 3×3, and S1 means the convolution step is 1), DConv128×3×3S2 (DConv means deconvolution, the number of channels is 128, the convolution kernel size is 3×3, and S2 means the convolution step is 2), Conv128×3×3S1 (Conv means convolution, the number of channels is 128, the convolution kernel size is 3×3, and S1 means the convolution step is 1), DConv192×3×3S2 (DConv means deconvolution, the number of channels is 192, the convolution kernel size is 3×3, and S2 means the convolution step is 2), and Conv128×3×3S1 (Conv means convolution, the number of channels is 128, the convolution kernel size is 3×3, and S1 means the convolution step is 1).

[0155] like Figure 2c As shown in 26, the lightweight super decoding subnetwork when processing the brightness component mainly includes five convolutional layers, two cropping layers (Crop), and three activation functions (LeakyRelu). Among them, the five convolutional layers are from top to bottom: Conv128×1×1S1 (Conv means convolution, the number of channels is 128, the convolution kernel size is 1×1, and S1 means the convolution step is 1), DConv128×4×4S2 (DConv means deconvolution, the number of channels is 128, the convolution kernel size is 4×4, and S2 means the convolution step is 2), Conv128×3×3S1 (Conv means convolution, the number of channels is 128, the convolution kernel size is 3×3, and S1 means the convolution step is 1), DConv128×4×4S2 (DConv means deconvolution, the number of channels is 128, the convolution kernel size is 3×3, and S2 means the convolution step is 2), and Conv128×3×3S1 (Conv means convolution, the number of channels is 128, the convolution kernel size is 3×3, and S1 means the convolution step is 1).

[0156] ② Comparison of the network structure of the hyper decoding subnetwork (Hyper Decoder_UV) when processing chrominance components and the network structure of the lightweight hyper decoding subnetwork (Light weighted Hyper Decoder_UV) when processing chrominance components.

[0157] like Figure 2cAs shown in 27, the super decoding subnetwork when processing the chrominance component mainly includes five convolutional layers, two cropping layers (Crop), and three activation functions (LeakyRelu). Among them, the five convolutional layers are from top to bottom: Conv64×3×3S1 (Conv means convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 means the convolution step is 1), DConv64×3×3S2 (Dconv means deconvolution, the number of channels is 64, the convolution kernel size is 3×3, and S2 means the convolution step is 2), Conv64×3×3S1 (Conv means convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 means the convolution step is 1), DConv96×3×3S2 (DConv means deconvolution, the number of channels is 96, the convolution kernel size is 3×3, and S2 means the convolution step is 2), and Conv64×3×3S1 (Conv means convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 means the convolution step is 1).

[0158] like Figure 2c As shown in 28, the lightweight super decoding subnetwork when processing the chrominance component mainly includes five convolutional layers, two cropping layers (Crop), and three activation functions (LeakyRelu). Among them, the five convolutional layers are from top to bottom: Conv64×1×1S1 (Conv means convolution, the number of channels is 64, the convolution kernel size is 1×1, and S1 means the convolution step is 1), DConv64×4×4S2 (DConv means deconvolution, the number of channels is 64, the convolution kernel size is 4×4, and S2 means the convolution step is 2), Conv64×3×3S1 (Conv means convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 means the convolution step is 1), DConv64×4×4S2 (DConv means deconvolution, the number of channels is 64, the convolution kernel size is 4×4, and S2 means the convolution step is 2), and Conv64×3×3S1 (Conv means convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 means the convolution step is 1).

[0159] from Figure 2c By comparing 25 and 26, and 27 and 28 in , when processing the luminance component or the chrominance component, there are certain differences between the network structure of the super decoding subnetwork and the network structure of the lightweight super decoding subnetwork at the convolutional layer.

[0160] (3) Comparison of the network structure of the hyper-scale decoding sub-network (Hyper Scale Decoder) and the network structure of the light-weighted hyper-scale decoding sub-network (Light weighted Hyper Scale Decoder).

[0161] See also Figure 2d, which is a schematic diagram of network structure comparison of a super-scale decoding sub-network and a lightweight super-scale decoding sub-network when processing chrominance components and luminance components respectively, provided by an exemplary embodiment of the present application.

[0162] ① Comparison between the network structure of the hyper-scale decoding sub-network (Hyper Scale Decoder_Y) when processing the luminance component and the network structure of the light-weighted hyper-scale decoding sub-network (Light weighted Hyper Scale Decoder_Y) when processing the luminance component.

[0163] like Figure 2d As shown in Figure 29, the super-scale decoding subnetwork for processing the luminance component mainly includes six convolutional layers, two cropping layers (Crop), and five activation functions (LeakyRelu). Among them, the six convolutional layers are, from top to bottom: DCnv128×5×5S2 (DConv means deconvolution, the number of channels is 128, the convolution kernel size is 5×5, S2 means the convolution step is 2), DCnv192×5×5S2 (Dconv means deconvolution, the number of channels is 192, the convolution kernel size is 5×5, S2 means the convolution step is 2), Conv256×3×3S1 (Conv means convolution, the number of channels is 256, the convolution kernel size is 3×3, S2 means the convolution step is 2), and Conv256×3×3S1 (Conv means convolution, the number of channels is 256, the convolution kernel size is 3×3, S2 means the convolution step is 2). 1 means the convolution step is 1), Conv213×3×3S1 (Conv means convolution, the number of channels is 213, the convolution kernel size is 3×3, S1 means the convolution step is 1), Conv170×3×3S1 (Conv means convolution, the number of channels is 170, the convolution kernel size is 3×3, S1 means the convolution step is 1), Conv128×3×3S1 (Conv means convolution, the number of channels is 128, the convolution kernel size is 3×3, S1 means the convolution step is 1).

[0164] like Figure 2d 30 in the figure shows that the lightweight super decoding subnetwork for processing the brightness component mainly includes four convolutional layers, two cropping layers (Crop), and three activation functions (LeakyRelu). Among them, the four convolutional layers are from top to bottom: DCnv128×4×4S2 (DConv represents deconvolution, the number of channels is 128, the convolution kernel size is 4×4, and S2 represents the convolution step size is 2), Conv128×3×3S1 (Conv represents convolution, the number of channels is 128, the convolution kernel size is 3×3, and S1 represents the convolution step size is 1), DCnv128×4×4S2 (DConv represents deconvolution, the number of channels is 128, the convolution kernel size is 4×4, and S2 represents the convolution step size is 2), and Conv128×3×3S1 (Conv represents convolution, the number of channels is 128, the convolution kernel size is 3×3, and S1 represents the convolution step size is 1).

[0165] ② Comparison of the network structure of the hyper-scale decoding sub-network (Hyper Scale Decoder_UV) when processing chrominance components and the network structure of the lightweight hyper-scale decoding sub-network (Light weighted Hyper Scale Decoder_UV) when processing chrominance components.

[0166] like Figure 2d As shown in 31 in the figure, the super-scale decoding subnetwork for processing chrominance components mainly includes six convolutional layers, two cropping layers (Crop), and five activation functions (LeakyRelu). Among them, the six convolutional layers are, from top to bottom: DCnv64×5×5S2 (DConv means deconvolution, the number of channels is 64, the convolution kernel size is 5×5, S2 means the convolution step is 2), DCnv96×5×5S2 (DConv means deconvolution, the number of channels is 96, the convolution kernel size is 5×5, S2 means the convolution step is 2), Conv128×3×3S1 (Conv means convolution, the number of channels is 128, the convolution kernel size is 3×3, S2 means the convolution step is 2), Conv128×3×3S1 (Conv means convolution, the number of channels is 128, the convolution kernel size is 3×3, S2 means the convolution step is 2), Conv128×3×3S2 (Conv means convolution, the number of channels is 128, the convolution kernel size is 3×3, S2 means the convolution step is 2), Conv128×3×3S1 ... 1 means the convolution step is 1), Conv106×3×3S1 (Conv means convolution, the number of channels is 106, the convolution kernel size is 3×3, S1 means the convolution step is 1), Conv85×3×3S1 (Conv means convolution, the number of channels is 85, the convolution kernel size is 3×3, S1 means the convolution step is 1), Conv64×3×3S1 (Conv means convolution, the number of channels is 128, the convolution kernel size is 3×3, S1 means the convolution step is 1).

[0167] like Figure 2d As shown in 32 in the figure, the lightweight super decoding subnetwork for processing chrominance components mainly includes four convolutional layers, two cropping layers (Crop), and three activation functions (LeakyRelu). Among them, the four convolutional layers are, from top to bottom: DCnv64×4×4S2 (DConv means deconvolution, the number of channels is 64, the convolution kernel size is 4×4, and S2 means the convolution step is 2), Conv64×3×3S1 (Conv means convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 means the convolution step is 1), DCnv64×4×4S2 (DConv means deconvolution, the number of channels is 64, the convolution kernel size is 4×4, and S2 means the convolution step is 2), and Conv64×3×3S1 (Conv means convolution, the number of channels is 64, the convolution kernel size is 3×3, and S1 means the convolution step is 1).

[0168] from Figure 2dBy comparing 29 and 30, and 31 and 32 in , when processing the luminance component and the chrominance component, the complexity of the network structure of the super-scale decoding sub-network is higher than that of the lightweight super-scale decoding sub-network, and when processing the luminance component and the chrominance component, there are certain differences between the convolutional layers of the super-scale decoding sub-network and the convolutional layers of the lightweight super-scale decoding sub-network (such as different numbers of channels).

[0169] 3. Conventional convolution (or convolution) and grouped convolution

[0170] Conventional convolution refers to performing convolution processing on the feature maps of all input images in the convolution layer. It can be understood that all input feature maps are divided into a group for convolution processing. When performing conventional convolution operations, each output channel is connected to each input channel, and there is a dense connection between channels. It should be understood that the output channel is the output feature map after convolution, and the input channel is the input feature map. Schematically, Figure 2e A schematic diagram of a comparison between a conventional convolution operation and a grouped convolution operation provided for an exemplary embodiment of the present application. Figure 2e As shown in (1), all the output channels of the upper layer (i.e., output channel a-output channel h) are connected to the input channels of the lower layer (i.e., input channel 1-input channel 8). For example, output channel a is connected to input channel 1-input channel 8, and output channel b is connected to input channel 1-input channel 8, and so on. Therefore, when performing conventional convolution operations, it will inevitably bring greater computational complexity. It can be seen from the end-to-end encoding and decoding model based on deep learning that the entire model involves more convolutional layers. Therefore, using conventional convolution in the convolutional layer will cause the computational complexity of the entire image encoding and decoding process to be higher, thereby reducing the encoding and decoding efficiency.

[0171] Grouped convolution refers to grouping the input feature map according to the number of channels in the convolution layer to obtain multiple groups, then performing convolution operations within each group to obtain the convolution results of each group, and finally concatenating the convolution results of each group. Schematically, the input channels and output channels can be divided into multiple groups (groups), and the output channels of each group are only connected to the input channels in the corresponding group, and are independent of the channels in other groups. Figure 2eAs shown in (2), the convolutional layer includes 8 input channels (such as input channel 1-input channel 8) and 8 output channels (output channel a-output channel h). Then the 8 input channels and 8 output channels can be divided into 4 groups, each of which contains 2 input channels and 2 output channels. For example, group 1 includes input channel 1, input channel 2, output channel a and output channel b, and group 2 includes input channel 3, input channel 4, output channel c and output channel d. Group 3 includes input channel 5, input channel 6, output channel e and output channel f; group 4 includes input channel 7, input channel 8, output channel g and output channel h. It can be seen that in the above four groups, the 2 output channels in each group are only connected to the 2 input channels in the corresponding group.

[0172] It can be seen that group convolution can reduce the computational complexity to a certain extent. Therefore, in an embodiment of the present application, group convolution can be introduced in an end-to-end encoding and decoding model based on deep learning, thereby reducing the computational complexity of the entire encoding and decoding process and improving the encoding and decoding efficiency.

[0173] 4. Channel Reorganization

[0174] Channel reorganization can be used to achieve feature exchange between channels. After group convolution processing, channel reorganization can be used to achieve feature map exchange between groups, thereby effectively ensuring that the accuracy of feature extraction can be improved in subsequent processing. In the embodiment of the present application, channel reorganization may refer to: reordering the feature maps corresponding to the channels obtained after group convolution to achieve information flow between different groups. For example Figure 2f As shown in FIG. 1 , a schematic diagram of a channel reorganization provided by an exemplary embodiment of the present application is shown. Figure 2f As shown in (a) in the figure, the channel is divided into three groups, and each group is input into the first convolution layer for group convolution processing to obtain the group convolution features corresponding to each group, and then the group convolution features corresponding to each group are input into the second convolution layer for group convolution processing. Figure 2f In (a), since channel reorganization is not performed, the grouped convolutional features between different groups cannot communicate, which will reduce the feature extraction ability of the model.

[0175] To solve this problem, consider reordering the grouped convolution features output by the grouped convolution operation by using channel reorganization, such as Figure 2fAs shown in (b), multiple channels are divided into three groups, and each group is input into the first convolution layer for group convolution processing to obtain the group convolution features corresponding to each group, and then the group convolution features corresponding to each group are reordered to obtain the channel reorganization result. For example, the group convolution features corresponding to the first group include the convolution features corresponding to channel 1, the convolution features corresponding to channel 2, and the convolution features corresponding to channel 3; the group convolution features corresponding to the second group include the convolution features corresponding to channel 4, the convolution features corresponding to channel 5, and the convolution features corresponding to channel 6; the group convolution features corresponding to the second group include the convolution features corresponding to channel 7, the convolution features corresponding to channel 8, and the convolution features corresponding to channel 9; the group convolution features corresponding to the three groups are reordered to obtain the channel reordering result, which is: the convolution features corresponding to channel 1, the convolution features corresponding to channel 4, the convolution features corresponding to channel 7, the convolution features corresponding to channel 2, the convolution features corresponding to channel 5, the convolution features corresponding to channel 8, the convolution features corresponding to channel 3, the convolution features corresponding to channel 6, and the convolution features corresponding to channel 9. The channel rearrangement result is then input into the next convolutional layer for processing.

[0176] It can be seen that after channel reorganization, feature exchange between groups is realized, which can better ensure that the input of the next grouped convolution comes from different groups, and can improve the feature extraction ability of the model. Channel reorganization only reorders the feature maps corresponding to the channels, so using channel reorganization will not introduce additional computing operations.

[0177] From the above-mentioned end-to-end image coding model based on deep learning, when encoding and decoding an image, a large number of decoding operations are involved, and in the convolution layer, due to the number of channels and convolution kernels, there will be a large number of calculations, which have high computational complexity and low coding and decoding efficiency, and cannot be well supported by the mobile device and put into practical use. Therefore, the embodiment of the present application provides an image processing scheme, which can be applied to the end-to-end image coding model based on deep learning, and the image processing scheme can simplify the end-to-end image coding model based on deep learning. The image processing scheme is as follows: decoding indication information is provided at the decoding end, and the decoding indication information is used to indicate the decoding simplification operation to be performed in the decoding neural network; then the decoding neural network is called according to the decoding indication information to decode the code stream to reconstruct the image. In the process of decoding the image, the operation of decoding in the decoding neural network of the decoding end can be simplified through the indication of the decoding indication information, thereby effectively reducing the decoding complexity of the decoding end, improving the decoding efficiency, and at the same time being well supported by the mobile device (i.e., the decoding device).

[0178] Next, the image processing system provided in the embodiment of the present application is described.

[0179] See also Figure 3 , is a structural diagram of an image processing system provided by an exemplary embodiment of the present application. The image processing system may include an encoding device 301 and a decoding device 302. The encoding device 301 is located at the encoding end, and the decoding device 302 is located at the decoding end. The encoding device 301 can be a terminal or a server, and the decoding device 302 can be a terminal or a server, and a communication connection can be established between the encoding device 301 and the decoding device 302. Among them, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a car terminal, a smart TV, etc., but is not limited to this. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0180] (1) For the encoding device 301

[0181] The encoding device 301 can obtain an image to be processed, which can be obtained by shooting with a camera or generated by a computer. The camera can be a hardware component provided in the encoding device 301, for example, the camera can be a common camera, a stereo camera, a light field camera, etc. provided in a terminal. The camera can also refer to a hardware device connected to the encoding device 301, such as a camera connected to a server.

[0182] Then, the encoding device 301 can call the encoding neural network to encode the acquired image to obtain a code stream corresponding to the image. The encoding neural network may include: Figure 1a The analysis transformation subnetwork shown, etc., or the encoding neural network may include Figure 1b The lightweight analysis transformation subnetwork shown.

[0183] In one implementation, the encoded neural network may include: Figure 1a As an example of the analysis transformation subnetwork shown in FIG. 1 , the image includes a chrominance component and a luminance component; the encoding device can convert the chrominance component (x Y ) and the brightness component (x UV ) respectively through Figure 1a The analysis transformation subnetwork in the above example performs nonlinear transformation processing to obtain the transformation result of the brightness component (y Y) and the transformation result of the chrominance component (y UV ), and then perform a difference operation between the transformed result of the brightness component and the predicted value of the brightness component output by the prediction fusion subnetwork to obtain the residual error of the brightness component (r Y ) ; perform difference calculation on the transformed result of the chrominance component and the predicted value of the chrominance component output by the prediction fusion subnetwork to obtain the residual error (r UV ), then the residual of the luminance component and the residual of the chrominance component are quantized and converted into integer residual of the luminance component The residual of the integer chrominance component Form a code stream.

[0184] In another implementation, when encoding an image, encoding indication information may be pre-configured, and the encoding indication information is used to indicate the encoding simplification operation to be performed in the encoding neural network; then, the encoding device 301 calls the encoding neural network according to the encoding indication information to encode the image to obtain a code stream. In an exemplary embodiment, the encoding neural network may include: Figure 1a Taking the analysis transformation subnetwork shown as an example, the encoding indication information can be used to indicate the encoding simplification operation to be performed in the analysis transformation subnetwork. At this time, the encoding device 301 calls the analysis transformation subnetwork according to the encoding indication information to perform the encoding simplification operation on the image to obtain a code stream.

[0185] By performing coding simplification operations in the coding neural network, the complexity of the coding end can be reduced to a certain extent and the coding efficiency can be improved. It should be understood that the specific implementation method of encoding the image by calling the coding neural network according to the coding instruction information to obtain the code stream is described in the subsequent embodiments and will not be repeated here.

[0186] After obtaining the code stream, the encoding device 301 may transmit the code stream to the decoding device 302, so that the decoding device 302 decodes the code stream to reconstruct the image.

[0187] (2) Decoding device 302

[0188] Receive the code stream transmitted by the encoding device 301, and obtain decoding indication information, which is used to indicate the decoding simplification operation to be performed in the decoding neural network, and then call the decoding neural network to decode the code stream based on the decoding indication information to reconstruct the image. Specifically, the image includes a luminance component and a chrominance component. The decoding device 302 can decode the code stream to obtain the residual of the luminance component. and the residual of the chrominance component Then the residual of the brightness component and the predicted value of the brightness component output by the prediction fusion subnetwork are superimposed to obtain the transformed value of the brightness component And the residual of the chrominance component and the predicted value of the chrominance component output by the prediction fusion subnetwork are superimposed to obtain the transformed value of the chrominance component The decoding neural network is then called to perform a decoding simplification operation on the transformed values ​​of the luminance component and the transformed values ​​of the chrominance components to reconstruct the image.

[0189] As an implementation method, the decoding simplification operation may include a grouped convolution operation, and the decoding indication information is used to indicate that a grouped convolution operation needs to be performed in the decoding neural network; in this implementation method, calling the decoding neural network to perform a decoding simplification operation on the transformation value of the luminance component and the transformation value of the chrominance component to reconstruct the image can be: the decoding device 302 calls the decoding neural network according to the decoding indication information to perform grouped convolution operations on the transformation value of the luminance component and the transformation value of the chrominance component respectively to reconstruct the image.

[0190] As another implementation, the decoding simplification operation may include a grouped convolution operation and a channel reorganization operation, and the decoding indication information is used to indicate that the grouped convolution operation and the channel reorganization operation need to be performed in the decoding neural network; in this implementation, calling the decoding neural network to perform the decoding simplification operation on the transformation value of the luminance component and the transformation value of the chrominance component to reconstruct the image may be: the decoding device 302 calls the decoding neural network to perform a grouped convolution operation on the transformation value of the luminance component according to the decoding indication information to obtain a grouped convolution result of the luminance component; calling the decoding neural network to perform a grouped convolution operation on the transformation value of the chrominance component to obtain a grouped convolution result of the chrominance component, and performing a channel reorganization operation on the grouped convolution result of the luminance component and performing a channel reorganization operation on the grouped convolution result of the chrominance component to obtain a channel reorganization result of the luminance component and a channel reorganization result of the chrominance component; the channel reorganization result of the luminance component and the channel reorganization result of the chrominance component are used to reconstruct the image.

[0191] Among them, the decoding neural network may include Figure 1a The synthesis transformation subnetwork, super decoding subnetwork, super-scale decoding subnetwork, etc. are shown. Alternatively, the decoding neural network may include a lightweight synthesis transformation subnetwork, a lightweight super decoding subnetwork, a lightweight super-scale decoding subnetwork, etc. This application is not limited to this.

[0192] It should be understood that in the embodiment of the present application, encoding indication information can be configured on the encoding device 301 side, while decoding indication information is not configured on the decoding device 302 side, and the calculation complexity of the encoding end is simplified by the encoding indication information, thereby improving the encoding efficiency; decoding indication information can also be set on the decoding device 302 side, and encoding indication information is not configured on the encoding device 301 side, and the decoding complexity of the decoding end is simplified by the decoding indication information, thereby improving the decoding efficiency; of course, decoding indication information can be configured on the encoding device 301 side and decoding indication information can be configured on the decoding device 302 side, thereby reducing the complexity of the image encoding and decoding process and improving the encoding and decoding efficiency.

[0193] Next, an image processing method provided in an embodiment of the present application is described.

[0194] See also Figure 4 , Figure 4 A flowchart of an image processing method provided for an exemplary embodiment of the present application is shown in FIG. 1. The image processing method may be executed by a decoding device 302. The image processing method may include the following steps S401-S403.

[0195] S401, obtaining a code stream formed after the image is encoded.

[0196] S402. Obtain decoding indication information, where the decoding indication information is used to indicate a decoding simplification operation to be performed in a decoding neural network.

[0197] The decoding neural network may be at least one of a super decoding subnetwork, a super-scale decoding subnetwork, and a synthetic transformation subnetwork. The super decoding subnetwork may be Figure 1a The super decoding subnetwork shown or Figure 1b The lightweight super decoding sub-network shown; the super-scale decoding sub-network can be as follows Figure 1a The super-scale decoding subnetwork shown or Figure 1b The lightweight super-scale decoding sub-network shown in FIG. The synthesis transformation sub-network can be as follows Figure 1a The synthetic transformation subnetwork shown or Figure 1b The lightweight synthesis transformation subnetwork shown. The decoding simplification operation includes a grouped convolution operation; or, the decoding simplification operation includes a grouped convolution operation and a channel reorganization operation.

[0198] (1) The decoding simplification operation includes a group convolution operation.

[0199] The decoding indication information includes decoding grouped convolution indication information. The decoding neural network includes a convolution layer, and the number of the convolution layers may be N, where N is a positive integer. The decoding grouped convolution indication information is used to indicate that a grouped convolution operation is performed in the convolution layer of the decoding neural network. By indicating that a grouped convolution operation is performed in the convolution layer of the decoding neural network, the convolution operations involved in the convolution layer in the decoding neural network can be processed in groups, thereby avoiding the large computational complexity caused by performing convolution operations on all input features at the same time, reducing the computational complexity, and thereby reducing the decoding complexity and improving the decoding efficiency.

[0200] The decoding group convolution indication information can be set in the network structure of the convolution layer of the decoding neural network. Schematically, the decoding neural network is Figure 1a In the synthetic transformation subnetwork, the decoding indication information is used to indicate that a grouped convolution operation is performed in the convolution layer (DConv128×3×3S2). At this time, the decoding grouped convolution indication information can be set in the network structure of the convolution layer. Schematically, the convolution layer can be expressed as DCnv128×3×3S2G2, where G2 indicates that the number of groups performing the grouped convolution operation is 2.

[0201] Furthermore, the decoding group convolution indication information is also used to indicate the number of groups of the group convolution operation performed in the convolution layer of the decoding neural network. The number of groups can be set based on experience. For example, the number of groups can be set to 2, 4, 8, etc. For example, according to experience, when the number of groups is 4, the decoding complexity during decoding can be reduced while ensuring the decoding performance. Then the decoding group convolution indication information is used to indicate that the number of groups of the group convolution operation is 4.

[0202] The number of groups of the grouped convolution operation indicated by the decoding grouped convolution indication information can support dynamic changes with changes in network parameters of the convolution layer of the decoding neural network. The network parameters include at least one of the following: the number of channels, feature data of the input convolution layer, and the bit rate point when performing image encoding and decoding.

[0203] ① Network parameters include the number of channels. When the number of channels of the convolution layer that performs the grouped convolution operation in the decoding neural network changes, the number of groups of the grouped convolution operation needs to be dynamically adjusted. For example, when the number of channels of the convolution layer that performs the grouped convolution operation changes from 128 to 64, the number of groups of the grouped convolution operation can be adjusted from 8 to 4. ② Network parameters include the feature data of the input convolution layer. For example, when the feature data of the convolution layer of the decoding neural network includes the chrominance component and the luminance component of the image, the number of groups of the convolution layer of the decoding neural network when performing the grouped convolution operation on the chrominance component is different from the number of groups of the convolution layer when performing the grouped convolution operation on the luminance component. ③ Network parameters include the bit rate point when encoding and decoding. When a new bit rate point is used for encoding and decoding of different images, the number of groups of the grouped convolution operation needs to be dynamically adjusted. When the sources of the images are different, different bit rate points will correspond to different encoding and decoding processes. For example, an image derived from a computer-generated animated picture and another image derived from a screen content picture will correspond to different bit rate points when encoding and decoding.

[0204] In one embodiment, the decoding group convolution indication information is also used to indicate the n convolution layers among the N convolution layers that need to perform the group convolution operation. n is a positive integer and n is less than or equal to N. Schematically, the decoding neural network includes the following: Figure 1a The synthetic transformation subnetwork in Figure 1a The synthetic transformation subnetwork includes four (i.e., N=4) convolutional layers, and the decoded group convolution indication information can be used to indicate 2 (n=2) convolutional layers (i.e., DCnv128×3×3S2, DCnv128×3×3S2) among the N convolutional layers that need to perform group convolution operations, that is, group convolution operations need to be performed in the convolutional layer DCnv128×3×3S2 and the convolutional layer DCnv128×3×3S2.

[0205] Furthermore, the decoded grouped convolution indication information is also used to indicate the number of groups for grouped convolution operations performed by each of the n convolutional layers. The number of groups for grouped convolution operations performed by different convolutional layers in the n convolutional layers may be all the same, or the number of groups for grouped convolution operations performed by different convolutional layers in the n convolutional layers may be partially the same, or the number of groups for grouped convolution operations performed by different convolutional layers in the n convolutional layers may be all different. Schematically, the number of groups for grouped convolution operations performed by one convolutional layer may be 4, and the number of groups for grouped convolution operations performed by another convolutional layer may be 2.

[0206] The selection rules of the n convolutional layers may include at least one of the following: ① Selecting n convolutional layers to perform grouped convolution operations from the N convolutional layers in descending order of the distance between the convolutional layer and the output layer. For example, the decoding neural network is Figure 1bIn the lightweight synthetic transformation subnetwork shown, the distance between the convolution layer Conv64×3×3S1 and the output layer is greater than the distance between the convolution layer Conv16×1×1S1 and the output layer, so the convolution layer Conv64×3×3S1 can be selected as the convolution layer that needs to perform the grouped convolution operation. ② Select n convolution layers that need to perform the grouped convolution operation from N convolution layers in descending order of the complexity of the network structure. Schematically, the decoding neural network includes convolution layer 1, convolution layer 2 and convolution layer 3. The complexity of the network structure of convolution layer 1 is greater than the complexity of the network structure of convolution layer 2, and the complexity of the network structure of convolution layer 2 is greater than the complexity of the network structure of convolution layer 3. Then convolution layer 1 and convolution layer 2 can be selected as the convolution layers that need to perform the grouped convolution operation.

[0207] It should be understood that when selecting a convolution layer to perform a grouped convolution operation, a convolution layer with a higher complexity can be considered to perform the grouped convolution operation, thereby reducing the computational complexity of the convolution layer, and thus reducing the decoding complexity. In addition, when selecting a convolution layer to perform a grouped convolution operation, the complexity and the decoding performance of the decoding neural network can be considered to determine the convolution layer to perform the grouped convolution operation, thereby reducing the complexity while ensuring the decoding performance.

[0208] (2) Decoding simplification operations include channel reorganization and group convolution operations. Channel reorganization can realize information exchange between groups, and channel reorganization does not require the introduction of additional computing operations. While effectively ensuring the decoding neural network's ability to extract image features, it effectively guarantees decoding performance and reduces decoding complexity.

[0209] The decoding group convolution indication information includes decoding group convolution indication information and decoding channel reorganization indication information. The decoding group convolution indication information is used to indicate that a group convolution operation is performed in the convolution layer of the decoding neural network. The decoding group convolution indication information can refer to the above (1) and will not be repeated here. The channel indication information is used to indicate that a channel reorganization operation is performed in the convolution layer of the decoding neural network. The decoding channel reorganization indication information can be set in the network structure of the convolution layer of the decoding neural network.

[0210] It should be understood that the convolution layer that performs the channel reorganization operation is the convolution layer that performs the grouped convolution operation. Wherein, the number of convolution layers of the decoding neural network is N and the N convolution layers include n convolution layers that need to perform the grouped convolution operation, N and n are both positive integers and n is less than or equal to N; at this time, the decoding channel reorganization indication information is also used to indicate t convolution layers that need to perform the channel reorganization operation among the n convolution layers, t is a positive integer and t is less than or equal to n.

[0211] In one embodiment, the decoded channel reorganization indication information is also used to indicate the number of channels for the channel reorganization operation. When the number of convolutional layers that need to perform the channel reorganization operation is t, the channel reorganization indication information is also used to indicate the number of channels for each of the t convolutional layers to perform the channel reorganization operation. Among them, the number of channels for each of the t convolutional layers to perform the channel reorganization operation can be all the same, or, the number of channels for each of the t convolutional layers to perform the channel reorganization operation can be partially the same (such as the number of channels for convolutional layer 1 to perform the channel reorganization operation is 4, the number of channels for convolutional layer 2 to perform the channel reorganization operation is 4, and the number of channels for convolutional layer 3 to perform the channel reorganization operation is 6); or, the number of channels for each of the t convolutional layers to perform the channel reorganization operation can be all different.

[0212] The number of channels of the channel reorganization operation indicated by the decoding channel reorganization indication information can support dynamic adjustment as the network parameters of the convolution layer of the decoding neural network change. The network parameters include at least one of the following: the number of channels, the input feature data, and the bit rate point during encoding and decoding.

[0213] ① The network parameters include the number of channels. When the number of channels of the convolution layer that performs the channel reorganization operation changes, the number of channels for the channel reorganization operation needs to be dynamically adjusted. For example, when the number of channels of the convolution layer that performs the grouped convolution operation changes from 128 to 64, the number of channels for the channel reorganization operation can be adjusted from 128 to 64. ② The network parameters include the feature data input to the convolution layer. For example, when the feature data input to the convolution layer of the decoding neural network includes the chrominance component and the luminance component of the image, the number of channels of the convolution layer of the decoding neural network when performing the channel reorganization operation on the chrominance component is different from the number of channels of the convolution layer when performing the channel reorganization operation on the luminance component. ③ The network parameters include the bit rate point when encoding and decoding. When encoding and decoding different images correspond to different bit rate points, the number of channels for the channel reorganization operation needs to be dynamically adjusted.

[0214] Among them, the convolution layer may include P channels, and the decoded channel reorganization indication information is also used to indicate the p channels among the P channels that need to perform the channel reorganization operation. In other words, the decoded channel reorganization indication information is used to indicate the number of channels for the channel reorganization operation and indicate which channels among the P channels need to perform the channel reorganization operation. In one implementation, each channel corresponds to a channel index value. At this time, the index value of the channel for performing the channel reorganization operation can be indicated in the decoded channel reorganization indication information. The number of channels and the channel index are both parameters of the channel pixel. Schematically, the convolution layer includes 64 channels, and the index values ​​corresponding to the 64 channels are 0 to 63 in sequence. The decoded channel reorganization indication information is used to indicate that the number of channels for the channel reorganization operation is 32, and the 32 (p=32) channels that need to perform the channel reorganization operation are 0-3, 10-13, 20-23, and 30-33.

[0215] The selection rules of p channels may include at least one of the following: ① According to the rule of reducing memory resource consumption, p channels that need to perform channel reorganization operations are selected from P channels. Schematically, P = 64, and the 64 channels will be stored in different memory locations. When performing the channel reorganization operation, the memory resource consumption when obtaining 64 channels from the corresponding memory locations is different. Therefore, 32 channels that need to perform channel reorganization operations can be selected from the 64 channels according to the memory resource consumption from small to large. ② The P channels correspond to multiple groups of grouped convolution operations, for example, the number of groups is 4, P = 64. The 64 channels correspond to 4 groups of grouped convolution operations, and each group includes 16 channels. At this time, according to the data communication requirements between each group, p channels that need to perform channel reorganization operations can be selected from P channels. Exemplarily, the data communication requirement is to realize that the features between each group can come from other groups, so all 64 channels can be selected as channels for performing channel reorganization operations.

[0216] In one embodiment, the decoding neural network includes at least one of the following: a super decoding subnetwork, a super-scale decoding subnetwork, and a synthetic transformation subnetwork; each subnetwork includes a convolution layer; the decoding simplification operation includes a group convolution operation, and if the decoding indication information indicates that the convolution layer of each subnetwork needs to perform a group convolution operation, then the number of groups for the group convolution operation of the convolution layer of each subnetwork indicated by the decoding indication information may be all the same, the number of groups for the group convolution operation of the convolution layer of each subnetwork indicated by the decoding indication information may be partially the same, or the number of groups for the group convolution operation of the convolution layer of each subnetwork indicated by the decoding indication information may be all different.

[0217] Further, the decoding simplification operation includes a grouped convolution operation and a channel reorganization operation. If the decoding indication information indicates that the convolution layer of each subnetwork performs a channel reorganization operation, the number of channels of the convolution layer of each subnetwork indicated by the decoding indication information for the channel reorganization operation may be all the same; or, the number of channels of the convolution layer of each subnetwork indicated by the decoding indication information for the channel reorganization operation may be partially the same and partially different; or, the number of channels of the convolution layer of each subnetwork indicated by the decoding indication information for the channel reorganization operation may be all different (such as the decoding indication information is used to indicate that the number of channels of the convolution layer of the super decoding subnetwork for the channel reorganization operation is 4, and the number of channels of the synthesis transformation subnetwork for the channel reorganization operation is 32). This application does not impose any restrictions on this.

[0218] S403: Call a decoding neural network to decode the bitstream according to the decoding instruction information to reconstruct an image.

[0219] When the decoding neural network includes a convolution layer, and the decoding simplification operation includes a grouped convolution operation, and the decoding indication information is used to indicate that a grouped convolution operation is performed in the convolution layer of the decoding neural network, calling the decoding neural network to decode the code stream according to the decoding indication information to reconstruct the image may include: calling the decoding neural network to decode the code stream according to the decoding indication information to obtain the transformation value of the image; then calling the convolution layer of the decoding neural network to perform a grouped convolution operation on the transformation value of the image to obtain the grouped convolution result corresponding to the transformation value of the image; and decoding the grouped convolution result to reconstruct the image. Specifically, the decoding indication information includes decoding grouped convolution indication information, and the decoding grouped convolution indication information is used to indicate the number of groups of the grouped convolution operation. The decoding device can group the transformation value of the image based on the number of groups to obtain one or more groups, and then perform a convolution operation on each group to obtain the grouped convolution result corresponding to the transformation value of the image.

[0220] When the decoding simplification operation includes a grouped convolution operation and a channel reorganization operation, the decoding indication information is used to indicate that the grouped convolution operation and the channel reorganization operation are performed in the convolution layer of the decoding neural network. Calling the decoding neural network to decode the code stream according to the decoding indication information to reconstruct the image may include: calling the decoding neural network to decode the code stream according to the decoding indication information to obtain the transformation value of the image; then calling the convolution layer of the decoding neural network to perform a grouped convolution operation on the transformation value of the image to obtain a grouped convolution result corresponding to the transformation value of the image; then, performing a channel reorganization operation on the grouped convolution result to obtain a channel reorganization result, and decoding the channel reorganization result to reconstruct the image.

[0221] The grouped convolution result includes convolution features corresponding to each channel in the multiple channels, and the decoding indication information includes channel reorganization indication information, and the channel reorganization indication information is also used to indicate the number of channels for the channel reorganization operation. Performing a channel reorganization operation on the grouped convolution result to obtain the channel reorganization result may be: reordering the grouped convolution result according to the number of channels indicated by the channel reorganization indication information to obtain the channel reorganization result.

[0222] In an embodiment of the present application, decoding indication information is provided at the decoding end, and the decoding indication information is used to indicate the decoding simplification operation to be performed in the decoding neural network; then the decoding neural network is called according to the decoding indication information to decode the bitstream to reconstruct the image. In the process of decoding the image, the operation of decoding in the decoding neural network of the decoding end can be simplified through the indication of the decoding indication information, thereby effectively reducing the decoding complexity of the decoding end, improving the decoding efficiency, and being better supported by the mobile device.

[0223] See also Figure 5 , Figure 5 A flowchart of an image processing method provided for another exemplary embodiment of the present application is shown in FIG. 1. The image processing method may be executed by the encoding device 301. The image processing method may include the following steps S501-S503.

[0224] S501: Obtain an image to be processed.

[0225] S502. Obtain coding indication information, where the coding indication information is used to indicate a coding simplification operation to be performed in the coding neural network.

[0226] The coding neural network may include at least one of the following: an analysis transformation subnetwork, a super coding subnetwork. The coding simplification operation may include a grouped convolution operation, or the coding simplification operation may include a grouped convolution operation and a channel reorganization operation.

[0227] (1) The coding simplification operation includes a group convolution operation.

[0228] The coding indication information includes coding group convolution indication information, the coding neural network includes a convolution layer, and the coding group convolution indication information is used to indicate that a group convolution operation is performed in the convolution layer of the coding neural network. The coding group convolution indication information is set in the network structure of the convolution layer of the coding neural network.

[0229] Furthermore, the coding group convolution indication information is also used to indicate the number of groups of the group convolution operation performed in the convolution layer of the coding neural network. The number of groups supports dynamic adjustment as the network parameters of the convolution layer of the coding neural network change. The network parameters here may include at least one of the following: the number of channels, the feature data of the input convolution layer, and the bit rate point when encoding and decoding.

[0230] ① The network parameters include the number of channels. When the number of channels of the convolution layer of the coding neural network changes, the number of groups of the grouped convolution operation needs to be dynamically adjusted. For example, when the number of channels of the convolution layer of the coding neural network changes from 128 to 64, the number of groups of the grouped convolution operation can be adjusted from 8 to 4. ② The network parameters include the feature data of the convolution layer of the input coding neural network. For example, when the feature data of the convolution layer of the input coding neural network includes the chrominance component and the luminance component of the image, the number of groups when the convolution layer performs grouped convolution on the chrominance component is different from the number of groups when the convolution layer performs grouped convolution on the luminance component. ③ The network parameters include the bit rate point when encoding and decoding. When a new bit rate point is used for encoding and decoding of different images, the number of groups of the grouped convolution operation needs to be dynamically adjusted. When the sources of the images are different, different bit rate points will correspond to different encoding and decoding processes. At this time, when different bit rate points are used for encoding different images, the number of groups of the grouped convolution operation needs to be dynamically adjusted.

[0231] Among them, the number of convolutional layers of the encoding neural network is M, M is a positive integer; the encoding grouped convolution indication information is also used to indicate the m convolutional layers in the M convolutional layers that need to perform grouped convolution operations, m is a positive integer and m is less than or equal to M. Among them, the selection rules of the m convolutional layers include at least one of the following: ① Select the m convolutional layers that need to perform grouped convolution operations in order of distance from the output layer from large to small; ② Select the m convolutional layers that need to perform grouped convolution operations in order of complexity of the network structure from high to low. It should be understood that the selection rules of the m convolutional layers are similar to the selection rules of the above-mentioned n convolutional layers, which will not be repeated here.

[0232] Furthermore, the grouped convolution indication information is also used to indicate the number of groups for grouped convolution operations performed by each of the m convolutional layers; the number of groups for grouped convolution operations performed by different convolutional layers in the m convolutional layers may all be the same, or the number of groups for grouped convolution operations performed by different convolutional layers in the m convolutional layers may be partially different and partially the same; or the number of groups for grouped convolution operations performed by different convolutional layers in the m convolutional layers may all be different.

[0233] (2) The coding simplification operation includes group convolution operation and channel reorganization operation.

[0234] The coding indication information may include coding group convolution indication information and coding channel reorganization indication information, wherein the coding group convolution indication information can refer to the relevant description of (1) above and will not be repeated here. The coding channel reorganization indication information is used to indicate the execution of a channel reorganization operation in the convolution layer of the coding neural network, wherein the coding channel reorganization indication information is set in the network structure of the convolution layer of the coding neural network.

[0235] Furthermore, the coding channel reorganization indication information is also used to indicate the number of channels of the channel reorganization operation performed in the convolution layer of the coding neural network. The number of channels supports dynamic adjustment as the network parameters of the convolution layer of the coding neural network change.

[0236] The convolution layer includes I channels; the coded channel reorganization indication information is also used to indicate O channels among the I channels that need to perform channel reorganization operations, where O is a positive integer and O is less than or equal to I. In other words, the coded channel reorganization indication information is used to indicate the number of channels for the channel reorganization operation and to indicate which channels among the I channels need to perform the channel reorganization operation. In one implementation, each channel corresponds to a channel index value, and in this case, the index value of the channel for performing the channel reorganization operation can be indicated in the coded channel reorganization indication information.

[0237] The selection rule of the O channels includes at least one of the following: selecting the O channels to be subjected to the channel operation from the I channels according to the rule of reducing the memory resource consumption; or selecting the O channels to be subjected to the channel reorganization operation from the I channels according to the data communication requirements between the groups, wherein the selection rule of the O channels is similar to the selection rule of the p channels, and will not be described in detail here.

[0238] The number of convolutional layers of the encoding neural network is M, and the M convolutional layers include m convolutional layers that need to perform grouped convolution operations, M and m are both positive integers and m is less than or equal to M; at this time, the encoding channel reorganization indication information is also used to indicate q convolutional layers that need to perform reorganization operations among the m convolutional layers, q is a positive integer and q is less than or equal to m.

[0239] Furthermore, the encoded channel reorganization indication information is also used to indicate the number of channels for each of the q convolutional layers to perform channel reorganization operations; wherein, the number of channels for different convolutional layers in the q convolutional layers to perform channel reorganization operations is the same, or, the number of channels for different convolutional layers in the q convolutional layers to perform channel reorganization operations may be partially the same and partially different; or, the number of channels for different convolutional layers in the q convolutional layers to perform channel reorganization operations may be all different.

[0240] Among them, the coding neural network includes at least one of the following: an analysis transformation subnetwork, a super coding subnetwork; when the coding simplification operation includes a group convolution operation, if the coding indication information indicates that the convolution layers of each subnetwork need to perform a group convolution operation, then the number of groups for the group convolution operation of the convolution layers of each subnetwork indicated by the coding indication information may be all the same; or, the number of groups for the group convolution operation of the convolution layers of each subnetwork indicated by the coding indication information may be partially the same and partially different; or, the number of groups for the group convolution operation of the convolution layers of each subnetwork indicated by the coding indication information may be all different (such as the coding indication information is used to indicate that the number of groups for the group convolution operation of the convolution layer of the super coding subnetwork is 4, and the number of groups for the group convolution operation of the convolution layer of the analysis transformation subnetwork is 8).

[0241] When the coding simplification operation includes a grouped convolution operation and a channel reorganization operation, if the coding indication information indicates that the convolutional layers of each sub-network perform a channel reorganization operation, the number of channels for the convolutional layers of the sub-networks indicated by the coding indication information to perform the channel reorganization operation are all the same; or, the number of channels for the convolutional layers of the sub-networks indicated by the coding indication information to perform the channel reorganization operation are partially the same and partially different; or, the number of channels for the convolutional layers of the sub-networks indicated by the coding indication information to perform the channel reorganization operation are all different (e.g., the coding indication information is used to indicate that the number of channels for the convolutional layers of the super-coding sub-network to perform the channel reorganization operation is 16, and the number of channels for the convolutional layers of the analysis transformation sub-network to perform the channel reorganization operation is 32).

[0242] S503: Call the encoding neural network according to the encoding instruction information to encode the image to obtain a code stream.

[0243] When the coding neural network includes a convolution layer, and the coding simplification operation includes a grouped convolution operation, the coding indication information is used to indicate that when the grouped convolution operation is performed in the convolution layer of the coding neural network, the coding neural network is called according to the coding indication information to perform coding processing on the image, and obtaining the code stream may include: calling the coding neural network to perform a grouped convolution operation on the image to obtain a grouped convolution result, and then encoding the grouped convolution result (such as quantization, etc.) to obtain the code stream of the image.

[0244] When the coding neural network includes a convolution layer, and the coding simplification operation includes a grouped convolution operation and a channel reorganization operation, the coding indication information is used to indicate that the grouped convolution operation and the channel reorganization operation are performed in the convolution layer of the coding neural network, and the coding neural network is called according to the coding indication information to perform coding processing on the image to obtain the code stream, which may include: calling the coding neural network to perform a grouped convolution operation on the image to obtain a grouped convolution result, performing a channel reorganization operation on the grouped convolution result to obtain a channel reorganization result of the image, and performing coding processing (such as quantization, etc.) on the channel reorganization result to obtain the code stream of the image.

[0245] In an embodiment of the present application, an image to be processed is obtained, and coding indication information is provided on the coding device side, the coding indication information is used to indicate the coding simplification operation to be performed in the coding neural network; then the coding neural network is called according to the coding indication information to perform coding processing on the image to reconstruct the image. In the process of encoding the image, the instructions of the coding indication information can simplify the encoding operations in the coding neural network of the encoding end, thereby effectively reducing the coding complexity of the encoding end and improving the coding efficiency.

[0246] Next, the image processing device provided in the embodiment of the present application is described.

[0247] See also Figure 6 , Figure 6 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device may be a computer program (including program code) in a computing device. For example, the image processing device may be an application software in a computing device. The image processing device may be used to execute Figure 4 Some or all of the steps in the method embodiment shown. Figure 6 , the image processing device comprises the following units:

[0248] An acquisition unit 601 is used to acquire a code stream formed after the image is encoded;

[0249] The acquisition unit 601 is further used to acquire decoding indication information, where the decoding indication information is used to indicate a decoding simplification operation to be performed in the decoding neural network;

[0250] The processing unit 602 is further configured to call a decoding neural network to perform decoding processing on the bit stream according to the decoding instruction information to reconstruct an image.

[0251] In one embodiment, the decoding neural network includes a convolution layer; the decoding simplification operation includes a grouped convolution operation; the decoding indication information includes decoding grouped convolution indication information, and the decoding grouped convolution indication information is used to indicate that a grouped convolution operation is performed in the convolution layer of the decoding neural network.

[0252] In one embodiment, the decoding group convolution indication information is set in the network structure of the convolution layer of the decoding neural network.

[0253] In one embodiment, the decoded group convolution indication information is also used to indicate the number of groups for the group convolution operation.

[0254] In one embodiment, the number of groups of the grouped convolution operation indicated by the decoding grouped convolution indication information supports dynamic adjustment as the network parameters of the convolution layer of the decoding neural network change.

[0255] In one embodiment, the number of convolutional layers of the decoding neural network is N, where N is a positive integer;

[0256] The decoding group convolution indication information is also used to indicate n convolution layers among the N convolution layers that need to perform group convolution operations, where n is a positive integer and n is less than or equal to N.

[0257] In one embodiment, the selection rules of the n convolutional layers include at least one of the following:

[0258] Select n convolutional layers that need to perform grouped convolution operations from N convolutional layers in descending order of the distance between the convolutional layer and the output layer;

[0259] Select n convolutional layers that need to perform grouped convolution operations from N convolutional layers in descending order of the complexity of the network structure.

[0260] In one embodiment, the decoded group convolution indication information is further used to indicate the number of groups for performing group convolution operations on each convolution layer in the n convolution layers;

[0261] Different convolutional layers in the n convolutional layers may have the same or different numbers of groups for grouped convolution operations.

[0262] In one embodiment, the decoding simplification operation also includes a channel reorganization operation; the decoding indication information includes decoding channel reorganization indication information, and the decoding channel reorganization indication information is used to indicate that the channel reorganization operation is performed in the convolution layer of the decoding neural network.

[0263] In one embodiment, the decoding channel reorganization indication information is set in the network structure of the convolutional layer of the decoding neural network.

[0264] In one embodiment, the decoded channel reorganization indication information is further used to indicate the number of channels for the channel reorganization operation.

[0265] In one embodiment, the number of channels of the channel reorganization operation indicated by the decoding channel reorganization indication information supports dynamic adjustment as the network parameters of the convolutional layer of the decoding neural network change.

[0266] In one embodiment, the convolution layer includes P channels; the decoded channel reorganization indication information is further used to indicate p channels among the P channels that need to perform channel reorganization operations, where p is a positive integer and p is less than or equal to P.

[0267] In one embodiment, the selection rules of the p channels include at least one of the following:

[0268] According to the rule of reducing memory resource consumption, p channels to be used for channel operation are selected from P channels; or,

[0269] The P channels correspond to a plurality of groups of the group convolution operation respectively. According to the data communication requirements between the groups, p channels that need to perform the channel reorganization operation are selected from the P channels.

[0270] In one embodiment, the number of convolutional layers of the decoding neural network is N, and the N convolutional layers include n convolutional layers that need to perform grouped convolution operations, N and n are both positive integers and n is less than or equal to N;

[0271] The decoding channel reorganization indication information is also used to indicate t convolutional layers among the n convolutional layers that need to perform channel reorganization operations, where t is a positive integer and t is less than or equal to n.

[0272] In one embodiment, the decoding channel reorganization indication information is further used to indicate the number of channels for performing the channel reorganization operation on each of the t convolutional layers;

[0273] The number of channels for channel reorganization operations performed by different convolutional layers in the t convolutional layers is the same or different.

[0274] In one embodiment, the decoding neural network includes at least one of the following: a super decoding subnetwork, a super-scale decoding subnetwork, and a synthetic transformation subnetwork; each subnetwork includes a convolutional layer;

[0275] The decoding simplification operation includes a group convolution operation. If the decoding indication information indicates that the convolution layer of each sub-network needs to perform the group convolution operation, the number of groups for the group convolution operation performed by the convolution layer of each sub-network indicated by the decoding indication information is the same or different;

[0276] The decoding simplification operation also includes a channel reorganization operation. If the decoding indication information indicates that the convolutional layers of each sub-network perform a channel reorganization operation, the number of channels of the convolutional layers of each sub-network indicated by the decoding indication information for performing the channel reorganization operation is the same or different.

[0277] In one embodiment, the decoding neural network includes a convolution layer, and the decoding simplification operation includes a group convolution operation and a channel reorganization operation; the processing unit 601 is specifically used to:

[0278] Calling a decoding neural network to decode the bitstream according to the decoding instruction information to obtain a transformation value of the image;

[0279] Call the convolution layer to perform grouped convolution operations on the transformed values ​​of the image to obtain grouped convolution results corresponding to the transformed values ​​of the image;

[0280] Perform channel reorganization operation on the grouped convolution result to obtain a channel reorganization result;

[0281] The channel reorganization processing result is decoded to reconstruct the image.

[0282] In the embodiment of the present application, a code stream formed after the image is encoded is obtained, and decoding indication information is obtained, the decoding indication information is used to indicate the decoding simplification operation to be performed in the decoding neural network; then the decoding neural network is called according to the decoding indication information to decode the code stream to reconstruct the image. In the process of decoding the image, the operation of decoding in the decoding neural network at the decoding end can be simplified through the indication of the decoding indication information, thereby effectively reducing the decoding complexity of the decoding end and improving the decoding efficiency.

[0283] See also Figure 7 , Figure 7 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device may be a computer program (including program code) in a computing device. For example, the image processing device may be an application software in a computing device. The image processing device may be used to execute Figure 4 Some or all of the steps in the method embodiment shown. Figure 7 , the image processing device comprises the following units:

[0284] An acquisition unit 701 is used to acquire an image to be processed;

[0285] The acquisition unit 701 is further used to acquire coding indication information, where the coding indication information is used to indicate a coding simplification operation to be performed in the coding neural network;

[0286] The processing unit 702 is used to call the encoding neural network to encode the image according to the encoding instruction information to obtain a code stream.

[0287] In an embodiment of the present application, an image to be processed is obtained, and coding indication information is provided on the coding device side, the coding indication information is used to indicate the coding simplification operation to be performed in the coding neural network; then the coding neural network is called according to the coding indication information to perform coding processing on the image to reconstruct the image. In the process of encoding the image, the instructions of the coding indication information can simplify the encoding operations in the coding neural network of the encoding end, thereby effectively reducing the coding complexity of the encoding end and improving the coding efficiency.

[0288] Next, the computer device provided in the embodiment of the present application is described.

[0289] Furthermore, the present application embodiment also provides a schematic diagram of the structure of a computer device. The schematic diagram of the structure of the computer device can be found in Figure 8The computer device may include: a processor 801, an input device 802, an output device 803 and a memory 804. The processor 801, the input device 802, the output device 803 and the memory 804 are connected via a bus. The memory 804 is used to store a computer program, the computer program includes program instructions, and the processor 801 is used to execute the program instructions stored in the memory 804.

[0290] When the computer device is the above decoding device, in the embodiment of the present application, the processor 801 performs the following operations by running the program instructions in the memory 804:

[0291] Obtaining a code stream formed after the image is encoded;

[0292] Obtaining decoding indication information, where the decoding indication information is used to indicate a decoding simplification operation to be performed in a decoding neural network;

[0293] The decoding neural network is called according to the decoding instruction information to decode the bit stream to reconstruct the image.

[0294] In one embodiment, the decoding neural network includes a convolution layer; the decoding simplification operation includes a grouped convolution operation; the decoding indication information includes decoding grouped convolution indication information, and the decoding grouped convolution indication information is used to indicate that a grouped convolution operation is performed in the convolution layer of the decoding neural network.

[0295] In one embodiment, the decoding group convolution indication information is set in the network structure of the convolution layer of the decoding neural network.

[0296] In one embodiment, the decoded group convolution indication information is also used to indicate the number of groups for the group convolution operation.

[0297] In one embodiment, the number of groups of the grouped convolution operation indicated by the decoding grouped convolution indication information supports dynamic adjustment as the network parameters of the convolution layer of the decoding neural network change.

[0298] In one embodiment, the number of convolutional layers of the decoding neural network is N, where N is a positive integer;

[0299] The decoding group convolution indication information is also used to indicate n convolution layers among the N convolution layers that need to perform group convolution operations, where n is a positive integer and n is less than or equal to N.

[0300] In one embodiment, the selection rules of the n convolutional layers include at least one of the following:

[0301] Select n convolutional layers that need to perform grouped convolution operations from N convolutional layers in descending order of the distance between the convolutional layer and the output layer;

[0302] Select n convolutional layers that need to perform grouped convolution operations from N convolutional layers in descending order of the complexity of the network structure.

[0303] In one embodiment, the decoded group convolution indication information is further used to indicate the number of groups for performing group convolution operations on each convolution layer in the n convolution layers;

[0304] Different convolutional layers in the n convolutional layers may have the same or different numbers of groups for grouped convolution operations.

[0305] In one embodiment, the decoding simplification operation also includes a channel reorganization operation; the decoding indication information includes decoding channel reorganization indication information, and the decoding channel reorganization indication information is used to indicate that the channel reorganization operation is performed in the convolution layer of the decoding neural network.

[0306] In one embodiment, the decoding channel reorganization indication information is set in the network structure of the convolutional layer of the decoding neural network.

[0307] In one embodiment, the decoded channel reorganization indication information is further used to indicate the number of channels for the channel reorganization operation.

[0308] In one embodiment, the number of channels of the channel reorganization operation indicated by the decoding channel reorganization indication information supports dynamic adjustment as the network parameters of the convolutional layer of the decoding neural network change.

[0309] In one embodiment, the convolution layer includes P channels; the decoded channel reorganization indication information is further used to indicate p channels among the P channels that need to perform channel reorganization operations, where p is a positive integer and p is less than or equal to P.

[0310] In one embodiment, the selection rules of the p channels include at least one of the following:

[0311] According to the rule of reducing memory resource consumption, p channels to be used for channel operation are selected from P channels; or,

[0312] The P channels correspond to a plurality of groups of the group convolution operation respectively. According to the data communication requirements between the groups, p channels that need to perform the channel reorganization operation are selected from the P channels.

[0313] In one embodiment, the number of convolutional layers of the decoding neural network is N, and the N convolutional layers include n convolutional layers that need to perform grouped convolution operations, N and n are both positive integers and n is less than or equal to N;

[0314] The decoding channel reorganization indication information is also used to indicate t convolutional layers among the n convolutional layers that need to perform channel reorganization operations, where t is a positive integer and t is less than or equal to n.

[0315] In one embodiment, the decoding channel reorganization indication information is further used to indicate the number of channels for performing the channel reorganization operation on each of the t convolutional layers;

[0316] The number of channels for channel reorganization operations performed by different convolutional layers in the t convolutional layers is the same or different.

[0317] In one embodiment, the decoding neural network includes at least one of the following: a super decoding subnetwork, a super-scale decoding subnetwork, and a synthetic transformation subnetwork; each subnetwork includes a convolutional layer;

[0318] The decoding simplification operation includes a group convolution operation. If the decoding indication information indicates that the convolution layer of each sub-network needs to perform the group convolution operation, the number of groups for the group convolution operation performed by the convolution layer of each sub-network indicated by the decoding indication information is the same or different;

[0319] The decoding simplification operation also includes a channel reorganization operation. If the decoding indication information indicates that the convolutional layers of each sub-network perform a channel reorganization operation, the number of channels of the convolutional layers of each sub-network indicated by the decoding indication information for performing the channel reorganization operation is the same or different.

[0320] In one embodiment, the decoding neural network includes a convolution layer, and the decoding simplification operation includes a group convolution operation and a channel reorganization operation; when the processor 801 calls the decoding neural network to decode the bitstream according to the decoding instruction information to reconstruct the image, the following steps may be specifically performed:

[0321] Calling a decoding neural network to decode the bitstream according to the decoding instruction information to obtain a transformation value of the image;

[0322] Call the convolution layer to perform grouped convolution operations on the transformed values ​​of the image to obtain grouped convolution results corresponding to the transformed values ​​of the image;

[0323] Perform channel reorganization operation on the grouped convolution result to obtain a channel reorganization result;

[0324] The channel reorganization processing result is decoded to reconstruct the image.

[0325] In the embodiment of the present application, a code stream formed after the image is encoded is obtained, and decoding indication information is obtained, the decoding indication information is used to indicate the decoding simplification operation to be performed in the decoding neural network; then the decoding neural network is called according to the decoding indication information to decode the code stream to reconstruct the image. In the process of decoding the image, the operation of decoding in the decoding neural network at the decoding end can be simplified through the indication of the decoding indication information, thereby effectively reducing the decoding complexity of the decoding end and improving the decoding efficiency.

[0326] Optionally, when the computer device is the above-mentioned encoding device, in the embodiment of the present application, the processor 801 performs the following operations by running the program instructions in the memory 804:

[0327] Obtain the image to be processed;

[0328] Obtaining encoding indication information, where the encoding indication information is used to indicate an encoding simplification operation to be performed in the encoding neural network;

[0329] The encoding neural network is called according to the encoding instruction information to encode the image and obtain a code stream.

[0330] In an embodiment of the present application, an image to be processed is obtained, and coding indication information is provided on the coding device side, the coding indication information is used to indicate the coding simplification operation to be performed in the coding neural network; then the coding neural network is called according to the coding indication information to perform coding processing on the image to reconstruct the image. In the process of encoding the image, the instructions of the coding indication information can simplify the encoding operations in the coding neural network of the encoding end, thereby effectively reducing the coding complexity of the encoding end and improving the coding efficiency.

[0331] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and the computer program includes program instructions. When the processor executes the above program instructions, it can execute the above Figure 4 and Figure 5 The method in the corresponding embodiment, therefore, will not be repeated here. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the program instructions can be deployed on a computer device, or executed on multiple computer devices located in one location, or, executed on multiple computer devices distributed in multiple locations and interconnected by a communication network.

[0332] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device can perform the above Figure 4 and Figure 5 The method in the corresponding embodiment will therefore not be described in detail here.

[0333] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0334] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An image processing method, characterized in that: include: Obtaining a code stream formed after the image is encoded; Obtaining decoding indication information, where the decoding indication information is used to indicate a decoding simplification operation to be performed in a decoding neural network; The decoding neural network is called according to the decoding instruction information to perform decoding processing on the code stream to reconstruct the image.

2. The method according to claim 1, characterized in that The decoding neural network includes a convolution layer; the decoding simplification operation includes a grouped convolution operation; the decoding indication information includes decoding grouped convolution indication information, and the decoding grouped convolution indication information is used to indicate that a grouped convolution operation is performed in the convolution layer of the decoding neural network.

3. The method according to claim 2, characterized in that The decoding group convolution indication information is set in the network structure of the convolution layer of the decoding neural network.

4. The method according to claim 2, characterized in that The decoding group convolution indication information is also used to indicate the number of groups of the group convolution operation.

5. The method according to claim 4, characterized in that The number of groups of the group convolution operation indicated by the decoding group convolution indication information supports dynamic adjustment as the network parameters of the convolution layer of the decoding neural network change.

6. The method according to claim 2, characterized in that The number of convolutional layers of the decoding neural network is N, where N is a positive integer; The decoding group convolution indication information is also used to indicate n convolution layers among N convolution layers that need to perform group convolution operations, where n is a positive integer and n is less than or equal to N.

7. The method according to claim 6, characterized in that The selection rules of the n convolutional layers include at least one of the following: Selecting n convolutional layers on which grouped convolution operations need to be performed from the N convolutional layers in descending order of the distance between the convolutional layer and the output layer; Select n convolutional layers that need to perform grouped convolution operations from the N convolutional layers in descending order of complexity of the network structure.

8. The method according to claim 6, characterized in that The decoding group convolution indication information is also used to indicate the number of groups for performing group convolution operations on each convolution layer in the n convolution layers; Different convolutional layers in the n convolutional layers have the same or different numbers of groups for performing grouped convolution operations.

9. The method according to claim 2, characterized in that The decoding simplification operation also includes a channel reorganization operation; the decoding indication information includes decoding channel reorganization indication information, and the decoding channel reorganization indication information is used to indicate that a channel reorganization operation is performed in a convolutional layer of the decoding neural network.

10. The method according to claim 9, characterized in that The decoding channel reorganization indication information is set in the network structure of the convolutional layer of the decoding neural network.

11. The method according to claim 9, characterized in that The decoded channel reorganization indication information is also used to indicate the number of channels for the channel reorganization operation.

12. The method according to claim 11, characterized in that The number of channels of the channel reorganization operation indicated by the decoding channel reorganization indication information supports dynamic adjustment as the network parameters of the convolutional layer of the decoding neural network change.

13. The method according to claim 9, characterized in that The convolution layer includes P channels; the decoding channel reorganization indication information is also used to indicate p channels among the P channels that need to perform channel reorganization operations, where p is a positive integer and p is less than or equal to P.

14. The method according to claim 13, characterized in that The selection rules of the p channels include at least one of the following: According to the rule of reducing memory resource consumption, p channels on which channel operations need to be performed are selected from the P channels; or, The P channels correspond to the multiple groups of the group convolution operation respectively, and according to the data communication requirements between the groups, p channels that need to perform channel reorganization operations are selected from the P channels.

15. The method according to claim 9, characterized in that The number of convolutional layers of the decoding neural network is N, and the N convolutional layers include n convolutional layers that need to perform grouped convolution operations, N and n are both positive integers and n is less than or equal to N; The decoding channel reorganization indication information is also used to indicate t convolutional layers among the n convolutional layers that need to perform channel reorganization operations, where t is a positive integer and t is less than or equal to n.

16. The method according to claim 15, characterized in that The decoding channel reorganization indication information is also used to indicate the number of channels for each convolutional layer in the t convolutional layers to perform a channel reorganization operation; Different convolutional layers in the t convolutional layers have the same or different numbers of channels for performing channel reorganization operations.

17. The method according to claim 1, wherein: The decoding neural network includes at least one of the following: a super decoding subnetwork, a super-scale decoding subnetwork, and a synthetic transformation subnetwork; each subnetwork includes a convolutional layer; The decoding simplification operation includes a group convolution operation. If the decoding indication information indicates that the convolution layer of each sub-network needs to perform a group convolution operation, the number of groups for performing the group convolution operation on the convolution layer of each sub-network indicated by the decoding indication information is the same or different; The decoding simplification operation also includes a channel reorganization operation. If the decoding indication information indicates that the convolutional layers of each sub-network perform a channel reorganization operation, the number of channels of the convolutional layers of each sub-network indicated by the decoding indication information for performing the channel reorganization operation is the same or different.

18. The method of claim 1, wherein: The decoding neural network includes a convolution layer, the decoding simplification operation includes a group convolution operation and a channel reorganization operation; calling the decoding neural network according to the decoding instruction information to decode the code stream to reconstruct the image includes: Calling the decoding neural network to decode the bitstream according to the decoding instruction information to obtain a transformation value of the image; Calling the convolution layer to perform a grouped convolution operation on the transformed values ​​of the image to obtain a grouped convolution result corresponding to the transformed values ​​of the image; Performing a channel reorganization operation on the grouped convolution result to obtain a channel reorganization result; The channel reorganization processing result is decoded to reconstruct the image.

19. An image processing method, characterized in that: include: Obtain the image to be processed; Acquire coding indication information, where the coding indication information is used to indicate a coding simplification operation to be performed in the coding neural network; The encoding neural network is called according to the encoding instruction information to encode the image to obtain a code stream.

20. An image processing device, characterized in that: include: An acquisition unit, used for acquiring a code stream formed after the image is encoded; The acquisition unit is further used to acquire decoding indication information, where the decoding indication information is used to indicate a decoding simplification operation to be performed in the decoding neural network; A processing unit is used to call the decoding neural network to decode the code stream according to the decoding indication information to reconstruct the image.

21. An image processing device, characterized in that: include: An acquisition unit, used for acquiring an image to be processed; The acquisition unit is further used to acquire encoding indication information, where the encoding indication information is used to indicate an encoding simplification operation to be performed in the encoding neural network; A processing unit is used to call the encoding neural network to encode the image according to the encoding instruction information to obtain a code stream.

22. A computer device, characterized in that: include: a processor suitable for executing a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the image processing method according to any one of claims 1 to 19 is executed.

23. A computer-readable storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by a processor, the image processing method according to any one of claims 1 to 19 is executed.