Nonlinear processing method and apparatus for image compression
Through the nonlinear transformation method of multiple convolution processing and point-by-point multiplication operation, the problem of limited receptive field and large calculation volume in the image/video compression algorithm is solved, the rate distortion performance and computing power balance are improved, and local attention is enhanced force.
Patent Information
- Application Number
- PCT/CN2024/096013
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-29
- Filing Date
- 2024-05-29
- Publication Date
- 2024-12-05
AI Technical Summary
The existing image/video compression algorithms have problems with limited receptive fields and large calculations in the design of nonlinear transformation units, which affect the performance of rate distortion.
The nonlinear transformation method of multiple convolution processing and point-by-point multiplication operation is adopted to expand the receptive field or reduce the calculation amount, enhance local attention, and balance the calculation force by configuring the convolution kernel size and the number of intermediate layers of channels.
Improve the rate distortion performance of the image/video compression algorithm, maintain model performance, effectively balance computing power, and enhance local attention.
Smart Images

Figure CN2024096013_05122024_PF_FP_ABST
Abstract
Description
Nonlinear processing method and device for image compression Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a nonlinear processing method and device for image compression. Background Art
[0002] As convolutional neural networks (CNNs) surpass traditional algorithms in computer vision tasks like image recognition and object detection, a growing number of researchers are exploring deep learning-based image / video compression methods. Some researchers have designed end-to-end deep learning image / video compression algorithms. For example, these algorithms simultaneously optimize the encoding network, entropy estimation network, entropy encoding network, entropy decoding network, and decoding network as a whole. The encoding and decoding networks, also known as the transform and inverse transform modules, typically consist of convolutional layers and nonlinear transformation units.
[0003] Nonlinear transform units (NTUs) are one of the fundamental components of image / video compression networks. Their nonlinear characteristics directly impact the rate-distortion performance of compression algorithms. Therefore, designing more efficient NTUs is key to further improving the rate-distortion performance of image / video compression algorithms.
[0004] Summary of the Invention
[0005] The embodiments of the present application provide a nonlinear processing method and device for image compression, which can expand the receptive field, reduce the amount of calculation, and further enhance local attention, maintain model performance and effectively balance computing power.
[0006] In a first aspect, the present application provides a nonlinear processing method for image compression, comprising: obtaining a first image feature to be processed; performing a nonlinear transformation on the first image feature to obtain a processed image feature, wherein the nonlinear transformation sequentially comprises a first convolution process, a second convolution process, and a point-by-point multiplication operation.
[0007] The embodiments of the present application change the nonlinear transformation processing in image compression and perform multiple convolution processes to expand the receptive field, or reduce the amount of calculation while keeping the receptive field unchanged, further enhance local attention, maintain model performance and effectively balance computing power.
[0008] At the encoding end, the first image feature is obtained by converting the image to be processed from the image domain to the feature domain after the encoding end obtains it. The above-mentioned conversion may include: (1) convolution processing, using convolution layers to extract features, with local receptive fields and weight sharing mechanisms (i.e., each filter slides to process input features); (2) using multilayer perceptrons (MLP) or fully connected layers to extract features, with global receptive field characteristics and no weight sharing; (3) transformer processing, which includes matrix multiplication, MLP and normalization processing, with global receptive field characteristics and strong ability to capture long-distance dependencies.
[0009] At the decoding end, the first image feature can be obtained by performing entropy decoding on the bitstream and then performing convolution processing, deconvolution processing, transposed convolution processing, interpolation + convolution processing, Transformer processing, or other methods corresponding to the encoding end. The aforementioned entropy decoding can be implemented using the entropy decoding network in the above embodiment and will not be described in detail here.
[0010] The first image features can be represented as a two-dimensional matrix (L×C, where L represents length and C represents channel) or a three-dimensional matrix (C×H×W, where C represents the number of channels, H represents height, and W represents width). The specific form is related to the above-mentioned conversion method. For example, the first image features obtained by convolution processing or MLP extraction generally correspond to a three-dimensional matrix, while the first image features obtained by transformer processing generally correspond to a two-dimensional matrix.
[0011] It should be noted that the embodiment of the present application does not specifically limit the representation method of the first image feature.
[0012] The image to be processed may be a picture, an image frame in a video, or an image block obtained by segmenting the aforementioned picture or image frame, and there is no specific limitation on this.
[0013] In one possible implementation, the nonlinear transformation process may include: performing a first convolution process on the first image feature to obtain a second image feature; performing a second convolution process on the second image feature to obtain a third image feature; and performing a point-by-point multiplication operation on the first image feature and the third image feature to obtain a processed image feature.
[0014] In the embodiment of the present application, the nonlinear transformation processing includes a first convolution processing, a second convolution processing and a point-by-point multiplication operation.
[0015] First, perform the first convolution on the first image feature and output the second image feature. The second image feature can be considered as the local response of the first image feature (corrected value), that is, the second image feature is the response signal obtained by the first convolution of the first image feature. Since the receptive field of the first convolution is limited, the response value of each position in the second image feature output by the first convolution is only related to the input feature value of the adjacent position, so it is called a local response. The first convolution can be expressed as the following formula: conv1(x)=β1+∑γ1×x
[0016] Among them, y1 represents the weight of the convolutional layer, and β1 represents the bias parameter of the convolutional layer.
[0017] The second image feature is then subjected to a second convolution process to output a third image feature. The third image feature can be considered as a local response (corrected value) of the second image feature, that is, the third image feature is the response signal obtained by the second convolution process of the second image feature. Since the receptive field of the second convolution process is limited, the response value of each position in the third image feature output by the second convolution process is only related to the input feature value of the adjacent position of the position, so it is called a local response. The second convolution process can be expressed as the following formula: conv2(x)=β2+∑γ2×x
[0018] Among them, γ2 represents the weight of the convolutional layer, and β2 represents the bias parameter of the convolutional layer.
[0019] The first convolution process and the second convolution process satisfy the following conditions: 1<=mid_chs<=chs
[0020] Where mid_chs represents the number of output channels of the first convolution process or the number of input channels of the second convolution process; chs represents the number of input channels of the first convolution process or the number of output channels of the second convolution process. This shows that the third image feature output by the second convolution process has the same number of channels as the first image feature input by the first convolution process.
[0021] Optionally, the above chs=mid_chs; or, chs=2×mid_chs.
[0022] In an embodiment of the present application, the configuration constraints for the number of channels may include: chs and mid_chs can be multiples of 2 to the power of N. When chs = 2 × mid_chs, in terms of theoretical computing power, two convolutions are equivalent to a single convolution with both the number of input channels and the number of output channels being chs. At this time, the model performance and computing power constraints are optimally balanced. For example, in common codec network models, chs can take values such as 256, 192, 128, 96, 64, 32, and 16.
[0023] Optionally, the convolution kernel size ks1_h*w of the first convolution processing and the convolution kernel size ks2_h*w of the second convolution processing can be the same or different.
[0024] In the embodiments of the present application, the convolution kernel size can be configured according to the actual model performance requirements and hardware computing power constraints. For example, the convolution kernel size can be set to symmetrical 1×1, 3×3, 5×5, or the convolution kernel size can be set to asymmetrical 1×3, 3×1, 5×1, 1×5. This embodiment of the present application does not specifically limit this.
[0025] In the embodiment of the present application, the receptive field can be expanded by two convolution processes, or the amount of calculation can be reduced while the receptive field remains unchanged, thereby further enhancing local attention, and by configuring the kernel size of the two convolutions and the number of channels in the intermediate layer, the overall computing power can be effectively balanced. For example, the convolution kernels of the first convolution process and the second convolution process are both 3×3, and two convolution processes can obtain a receptive field of 5×5. The input of the first convolution process is 8 channels, and the output of the first convolution process after processing can be 4 channels. The input of the second convolution process is 4 channels, and the output of the second convolution process after processing can be 8 channels. The number of input / output channels remains unchanged, but only processing 4 channels in the intermediate calculation process can reduce the amount of calculation. If only one convolution process is used, the convolution kernel of the convolution process needs to be 5×5 so as not to change the receptive field, and its input / output is both 8 channels, and the amount of calculation is also relatively high.
[0026] The corresponding multiple feature values in the first image feature and the third image feature are multiplied point by point, that is, the input of the nonlinear transformation and the output of the two convolution processes are multiplied point by point, which can be expressed as the following formula: c(i, j) = a(i, j) × b(i, j)
[0027] Among them, (i, j) represents the index of the eigenvalue in the image feature in which it is located, a(i, j) represents the eigenvalue of the first image feature, b(i, j) represents the eigenvalue of the second image feature, and c(i, j) represents the eigenvalue of the processed image feature.
[0028] Optionally, the size of the matrix corresponding to the third image feature is the same as the size of the matrix corresponding to the first image feature.
[0029] The two convolutions can produce a third image feature of the same size as the first. This depends on the internal structure of the convolution process, particularly the length, width, and number of channels of the filters used in the two convolutions. If the third image feature is the same size as the first, the elements in the matrix corresponding to the third image feature will correspond one-to-one with the elements in the matrix corresponding to the first. In this case, point-by-point multiplication can multiply the values of elements at the same position to obtain the final value of the element at that position.
[0030] Optionally, the size of the matrix corresponding to the third image feature is different from the size of the matrix corresponding to the first image feature.
[0031] The double convolution process can also produce a third image feature with a different size from the first image feature. This also depends on the internal structure of the convolution process, especially the length, width, and number of channels of the filter used in the double convolution process. If the third image feature is of different sizes from the first image feature, the elements in the matrix corresponding to the third image feature will not correspond one-to-one with the elements in the matrix corresponding to the first image feature. In this case, the point-by-point multiplication operation can only multiply the values of the elements at the same position to obtain the final value of the element at that position, while the elements at non-corresponding positions retain their original values.
[0032] The nonlinear transformation process of this embodiment can be expressed as the following formula: y = x*Conv2(Conv1(x))
[0033] Among them, x represents the first image feature, Conv1(x) represents the second image feature, Conv2(Conv1(x)) represents the third image feature, and y represents the processed image feature.
[0034] The above formula uses point-by-point multiplication operations. On the one hand, there is no need to require that the convolution parameters γ and β must be positive numbers to accommodate the restriction that the square root must be a positive number. On the other hand, there is no need to accommodate the restriction that the denominator cannot be 0 in division, which also constitutes a restriction on the values of the convolution parameters γ and β.
[0035] Through the above-mentioned first convolution processing, second convolution processing and point-by-point multiplication operation, a nonlinear transformation processing with an attention mechanism is realized. The attention mechanism means that all eigenvalues in the first image feature are important or redundant. The output of the convolution processing can be the weight of each eigenvalue in the image feature, which can correct the original eigenvalue, highlight the important eigenvalue, and suppress the redundant eigenvalue; the point-by-point multiplication operation uses the above-mentioned correction information to correct the value of each eigenvalue on the first image feature, avoiding the limitation of the convolution parameters.
[0036] In one possible implementation, the above-mentioned nonlinear transformation processing process may include: performing a first convolution processing on the first image feature to obtain a second image feature; performing a second convolution processing on the second image feature to obtain a third image feature; performing a point-by-point multiplication operation on the first image feature and the third image feature to obtain a fourth image feature; and performing a point-by-point addition operation on the first image feature and the fourth image feature to obtain a processed image feature.
[0037] In the embodiment of the present application, the nonlinear transformation processing includes a first convolution processing, a second convolution processing, a point-by-point multiplication operation and a point-by-point addition operation.
[0038] After obtaining the fourth image feature, the corresponding multiple feature values in the first image feature and the fourth image feature are added point by point. That is, the input of the nonlinear transformation and the output of the point-by-point multiplication operation are added point by point. This can be expressed as the following formula: sum(i, j) = a(i, j) + c(i, j)
[0039] Among them, (i, j) represents the index of the eigenvalue in the image feature in which it is located, a(i, j) represents the eigenvalue of the first image feature, c(i, j) represents the eigenvalue of the fourth image feature, and sum(i, j) represents the eigenvalue of the processed image feature.
[0040] The point-by-point addition operation is a residual structure of first addition, which can make the codec network using the above processing process more easily converged during training.
[0041] Optionally, the size of the matrix corresponding to the fourth image feature is the same as the size of the matrix corresponding to the first image feature.
[0042] For the same reason as the third image feature, the fourth image feature can have the same size as the first image feature, and the elements in the matrix corresponding to the fourth image feature have a one-to-one correspondence with the elements in the matrix corresponding to the first image feature. In this case, a point-by-point addition operation can add the values of elements at the same position to obtain the final value of the element at that position.
[0043] Optionally, the size of the matrix corresponding to the fourth image feature is different from the size of the matrix corresponding to the first image feature.
[0044] For the same reason as for the third image feature, the fourth image feature can have different sizes than the first image feature, and the elements in the matrix corresponding to the fourth image feature may not correspond one-to-one with the elements in the matrix corresponding to the first image feature. In this case, the point-by-point addition operation can only add the values of elements at the same position to obtain the final value of the element at that position, while elements at non-corresponding positions retain their original values.
[0045] The nonlinear transformation process of this embodiment can be expressed as the following formula: y = x + x * Conv2 (Conv1 (x))
[0046] Among them, x represents the first image feature, Conv1(x) represents the second image feature, Conv2(Conv1(x)) represents the third image feature, x*Conv2(Conv1(x)) represents the fourth image feature, and y represents the processed image feature.
[0047] In one possible implementation, the above-mentioned nonlinear transformation processing may include: performing a first convolution processing on the first image feature to obtain a second image feature; performing a first activation processing on the second image feature to obtain a fifth image feature; performing a second convolution processing on the fifth image feature to obtain a sixth image feature; and performing a point-by-point multiplication operation on the first image feature and the sixth image feature to obtain a processed image feature.
[0048] In an embodiment of the present application, the nonlinear transformation processing includes a first convolution processing, a first activation processing, a second convolution processing and a point-by-point multiplication operation.
[0049] The first activation processing is a nonlinear operation performed on each eigenvalue in the second image feature after the first convolution processing, which may include but is not limited to taking the absolute value operation, the rectified linear unit (Rectified Linear Unit, ReLU) processing series, Sigmoid, Tanh, identity operation (Identity) or parametric rectified linear unit (PReLU) processing, etc.
[0050] It should be noted that the present application may also adopt other nonlinear operations, and the embodiments of the present application do not specifically limit this.
[0051] After the above-mentioned first activation processing, the second image feature is converted into the fifth image feature. The fifth image feature can also be expressed in the form of a matrix like the first image feature. Since the first activation processing is for each eigenvalue in the second image feature, each eigenvalue of the second image feature has an eigenvalue corresponding to it in the fifth image feature value. Therefore, the matrix corresponding to the fifth image feature and the matrix corresponding to the second image feature have the same size, and the eigenvalues (values of matrix elements) at the same position correspond. For example, the second image feature is expressed as a 3×3 matrix like the above-mentioned matrix A, then the fifth image feature can also be expressed as a 3×3 matrix, but since the second image feature and the fifth image feature have undergone the first activation processing, the eigenvalues in the second image feature and the fifth image feature are not exactly the same. Accordingly, the element values in the matrices corresponding to the two are also not exactly the same.
[0052] The nonlinear transformation process of this embodiment can be expressed as the following formula: y = x*Conv2(φ1(Conv1(x)))
[0053] Among them, x represents the first image feature, Conv1(x) represents the second image feature, φ1(Conv1(x)) represents the fifth image feature, Conv2(φ1(Conv1(x))) represents the sixth image feature, and y represents the processed image feature.
[0054] In one possible implementation, the above-mentioned nonlinear transformation processing process may include: performing a first convolution processing on the first image feature to obtain a second image feature; performing a first activation processing on the second image feature to obtain a fifth image feature; performing a second convolution processing on the fifth image feature to obtain a sixth image feature; performing a point-by-point multiplication operation on the first image feature and the sixth image feature to obtain a seventh image feature; and performing a point-by-point addition operation on the first image feature and the seventh image feature to obtain a processed image feature.
[0055] In an embodiment of the present application, the nonlinear transformation processing includes a first convolution processing, a first activation processing, a second convolution processing, a point-by-point multiplication operation, and a point-by-point addition operation.
[0056] The nonlinear transformation process of this embodiment can be expressed as the following formula: y = x + x * Conv2 (φ1 (Conv1 (x)))
[0057] Among them, x represents the first image feature, Conv1(x) represents the second image feature, φ1(Conv1(x)) represents the fifth image feature, Conv2(φ1(Conv1(x))) represents the sixth image feature, x*Conv2(φ1(Conv1(x))) represents the seventh image feature, and y represents the processed image feature.
[0058] In one possible implementation, the above-mentioned nonlinear transformation processing process may include: performing a first convolution processing on the first image feature to obtain a second image feature; performing a first activation processing on the second image feature to obtain a fifth image feature; performing a second convolution processing on the fifth image feature to obtain a sixth image feature; performing a second activation processing on the sixth image feature to obtain an eighth image feature; and performing a point-by-point multiplication operation on the first image feature and the eighth image feature to obtain a processed image feature.
[0059] In an embodiment of the present application, the nonlinear transformation processing includes a first convolution processing, a first activation processing, a second convolution processing, a second activation processing and a point-by-point multiplication operation.
[0060] The nonlinear transformation process of this embodiment can be expressed as the following formula: y = x*φ2(Conv2(φ1(Conv1(x))))
[0061] Among them, x represents the first image feature, Conv1(x) represents the second image feature, φ1(Conv1(x)) represents the fifth image feature, Conv2(φ1(Conv1(x))) represents the sixth image feature, φ2(Conv2(φ1(Conv1(x)))) represents the eighth image feature, and y represents the processed image feature.
[0062] In one possible implementation, the above-mentioned nonlinear transformation processing process may include: performing a first convolution processing on the first image feature to obtain a second image feature; performing a first activation processing on the second image feature to obtain a fifth image feature; performing a second convolution processing on the fifth image feature to obtain a sixth image feature; performing a second activation processing on the sixth image feature to obtain an eighth image feature; performing a point-by-point multiplication operation on the first image feature and the eighth image feature to obtain a ninth image feature; and performing a point-by-point addition operation on the first image feature and the ninth image feature to obtain a processed image feature.
[0063] In an embodiment of the present application, the nonlinear transformation processing includes a first convolution processing, a first activation processing, a second convolution processing, a second activation processing, a point-by-point multiplication operation, and a point-by-point addition operation.
[0064] The nonlinear transformation process of this embodiment can be expressed as the following formula: y = x + x * φ2 (Conv2 (φ1 (Conv1 (x))))
[0065] Among them, x represents the first image feature, Conv1(x) represents the second image feature, φ1(Conv1(x)) represents the fifth image feature, Conv2(φ1(Conv1(x))) represents the sixth image feature, φ2(Conv2(φ1(Conv1(x)))) represents the eighth image feature, x*φ2(Conv2(φ1(Connv1(x)))) represents the ninth image feature, and y represents the processed image feature.
[0066] The embodiment of the present application uses the operation of fusing two layers of convolution processing and activation processing to achieve channel-adaptive nonlinear transformation processing.
[0067] It should be noted that the above embodiments are only several examples of nonlinear transformation processing, but do not constitute a limitation on the process of nonlinear transformation processing. In some possible implementations, nonlinear transformation processing may also include more activation processing and convolution processing. The embodiments of the present application do not specifically limit the process of nonlinear transformation processing.
[0068] At the encoding end, encoding can be performed based on the processed image features to obtain a code stream.
[0069] After obtaining the processed image features, the encoding side can continue to perform convolution processing on them, or can further perform nonlinear transformation processing on the output of the convolution processing after the convolution processing, and then perform entropy coding on the results of the above processing to obtain a bitstream. Entropy coding can be implemented using the entropy coding network in the above embodiment, which will not be described in detail here. Other coding methods can also be used to encode the results of the above processing to obtain a bitstream, which is not limited by this application.
[0070] At the decoding end, a reconstructed image can be obtained based on the processed image features.
[0071] After obtaining the processed image features, the decoding side may continue to perform convolution processing on them, or may perform nonlinear transformation processing on the output of the convolution processing again after the convolution processing, thereby converting it from the feature domain to the image domain to obtain a reconstructed image.
[0072] In second aspect, the present application provides a nonlinear processing device for image compression, comprising: an acquisition module for acquiring a first image feature to be processed; a transformation module for performing nonlinear transformation processing on the first image feature to obtain a processed image feature, wherein the nonlinear transformation processing sequentially includes a first convolution processing, a second convolution processing and a point-by-point multiplication operation.
[0073] In a possible implementation, the nonlinear transformation process further includes a point-by-point addition operation after the point-by-point multiplication operation.
[0074] In a possible implementation, the nonlinear transformation process further includes a first activation process after the first convolution process.
[0075] In a possible implementation, the nonlinear transformation process further includes a second activation process after the second convolution process.
[0076] In a possible implementation, the first activation processing and the second activation processing respectively include an identity operation Identity, a rectified linear unit ReLU processing, a leaky rectified linear unit LeakyReLU processing, or a parameter rectified linear unit PReLU processing.
[0077] In one possible implementation, the transformation module is specifically used to perform the first convolution processing on the first image feature to obtain the second image feature; perform the second convolution processing on the second image feature to obtain the third image feature; and perform the point-by-point multiplication operation on the first image feature and the third image feature to obtain the processed image feature.
[0078] In one possible implementation, the transformation module is specifically used to perform the first convolution processing on the first image feature to obtain a second image feature; perform the second convolution processing on the second image feature to obtain a third image feature; perform the point-by-point multiplication operation on the first image feature and the third image feature to obtain a fourth image feature; and perform the point-by-point addition operation on the first image feature and the fourth image feature to obtain the processed image feature.
[0079] In a possible implementation, the first convolution process and the second convolution process satisfy the following conditions: 1<=mid_chs<=chs
[0080] Among them, mid_chs represents the number of output channels of the first convolution processing or the number of input channels of the second convolution processing; chs represents the number of input channels of the first convolution processing or the number of output channels of the second convolution processing.
[0081] In a possible implementation, chs=mid_chs; or, chs=2×mid_chs.
[0082] In a possible implementation, the convolution kernel size of the first convolution processing and the convolution kernel size of the second convolution processing are the same as or different from each other.
[0083] In a possible implementation, when the nonlinear processing device for image compression is used in an encoding end, it further includes: an encoding module for encoding according to the processed image features to obtain a code stream
[0084] In a possible implementation, when the nonlinear processing device for image compression is used at a decoding end, it further includes: a reconstruction module, configured to obtain a reconstructed image based on the processed image features.
[0085] In a third aspect, an embodiment of the present application provides an encoder comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the processor and storing a program executed by the processor, wherein when the program is executed by the processor, the decoder performs the method described in any one of the above-mentioned first aspects.
[0086] In a fourth aspect, an embodiment of the present application provides a decoder comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the processor and storing a program executed by the processor, wherein when the program is executed by the processor, the decoder executes any one of the methods described in the first aspect above.
[0087] In a fifth aspect, an embodiment of the present application provides a computer program product comprising a program code, which, when executed on a computer or a processor, is used to execute any of the methods described in the first aspect above.
[0088] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium comprising instructions, which, when executed on a computer, enables the computer to execute any one of the methods described in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS
[0089] Figure 1 is an example diagram of a typical deep learning image compression framework;
[0090] Figure 2 is an example diagram of a typical deep learning video compression framework;
[0091] Figure 3 is an example diagram of an image / video storage / transmission application scenario;
[0092] FIG4 is an example diagram of an image / video storage application scenario;
[0093] Figure 5 is an example diagram of Huawei Cloud application scenarios;
[0094] FIG6 is a flow chart of a process 600 of a nonlinear processing method for image compression according to an embodiment of the present application;
[0095] Figure 7a is a schematic diagram of the nonlinear transformation processing process with an attention mechanism;
[0096] Figure 7b is a schematic diagram of the residual nonlinear transformation processing process with an attention mechanism;
[0097] Figure 7c is a schematic diagram of the nonlinear transformation processing process with the attention mechanism;
[0098] Figure 7d is a schematic diagram of the residual nonlinear transformation processing process with an attention mechanism;
[0099] Figure 7e is a schematic diagram of the nonlinear transformation processing process with attention mechanism;
[0100] Figure 7f is a schematic diagram of the residual nonlinear transformation processing process with the attention mechanism;
[0101] FIG8 is a schematic diagram of residual nonlinear transformation processing with an attention mechanism;
[0102] FIG9 is a schematic diagram of nonlinear transformation processing with an attention mechanism;
[0103] FIG10 is a schematic diagram of nonlinear transformation processing with an attention mechanism;
[0104] FIG11 is a schematic diagram of an exemplary structure of an encoding device 1100 according to an embodiment of the present application;
[0105] FIG12 is a schematic diagram of an exemplary structure of a decoding device 1200 according to an embodiment of the present application. DETAILED DESCRIPTION
[0106] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.
[0107] The terms "first," "second," and the like in the description, embodiments, claims, and drawings of this application are used solely for descriptive purposes and are not to be construed as indicating or implying relative importance or order. Furthermore, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions, such as, for example, inclusion of a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0108] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0109] The following is a description of the terms involved in the embodiments of this application:
[0110] 1. Bit rate: In image compression, the average coding length required to encode a unit pixel.
[0111] 2. Rate-distortion performance: An indicator used to measure the performance of a compression algorithm, which comprehensively considers both bit rate and decoded image distortion.
[0112] 3. Attention mechanism: A method that uses limited attention resources to filter out high-value information from a large amount of information. It can make the neural network pay more attention to the relevant parts of the input and less attention to the irrelevant parts.
[0113] 4. Residual structure: A common connection structure in neural networks, calculated as H(x) = x + f(x). This structure prevents the vanishing and exploding gradient problems that can occur when the network depth increases.
[0114] 5. Nonlinear transformation unit: A network unit that contains nonlinear operations (for example, ReLU, Sigmoid, Tanh, etc.), and the overall calculation method of this unit does not conform to linear characteristics.
[0115] Since the embodiments of the present application involve the application of neural networks, for ease of understanding, the following first explains the relevant nouns or terms used in the embodiments of the present application:
[0116] 1. Neural Networks
[0117] A neural network (NN) is a machine learning model. A neural network can be composed of neural units. A neural unit can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be:
[0118] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a nonlinear function such as ReLU. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.
[0119] 2. Multi-layer perception (MLP)
[0120] MLP is a simple deep neural network (DNN) (different layers are fully connected), also known as a multi-layer neural network, which can be understood as a neural network with many hidden layers. The "many" here does not have a special metric. According to the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since there are many DNN layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscript corresponds to the output of the third layer index 2 and the input of the second layer index 4. In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).
[0121] 3. Convolutional Neural Networks
[0122] A convolutional neural network (CNN) is a deep neural network with a convolutional structure and a deep learning architecture. A deep learning architecture involves multiple levels of learning at different levels of abstraction using machine learning algorithms. As a deep learning architecture, a CNN is a feed-forward artificial neural network in which individual neurons respond to input images. A CNN consists of a feature extractor consisting of convolutional and pooling layers. The feature extractor can be thought of as a filter, and the convolution process can be thought of as convolving an input image or feature map with a trainable filter.
[0123] A convolutional layer is a layer of neurons in a convolutional neural network that performs convolution on the input signal. A convolutional layer can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter that extracts specific information from the input image matrix. A convolution operator is essentially a weight matrix, which is usually predefined. During the convolution operation, the weight matrix is typically applied horizontally to the input image, pixel by pixel (or two pixels by two pixels, depending on the stride), to extract specific features from the image. The size of the weight matrix should be proportional to the image size. It is important to note that the depth dimension of the weight matrix is the same as the depth dimension of the input image; during the convolution operation, the weight matrix extends across the entire depth of the input image. Therefore, convolution with a single weight matrix produces a convolved output with a single depth dimension. However, in most cases, multiple weight matrices of the same size (rows × columns) are applied instead. This is known as multiple homogeneous matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. The dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from the image. For example, one weight matrix is used to extract image edge information, another weight matrix is used to extract specific colors in the image, and yet another weight matrix is used to blur unwanted noise in the image. The multiple weight matrices have the same size (rows × columns), and the feature maps extracted by these multiple weight matrices of the same size are also the same size. The extracted feature maps of the same size are then merged to form the output of the convolution operation. In practical applications, the weight values in these weight matrices require extensive training. The weight matrices formed by the trained weight values can be used to extract information from the input image, allowing the convolutional neural network to make accurate predictions. When a convolutional neural network has multiple convolutional layers, the initial convolutional layers often extract more general features, which can also be called low-level features. As the depth of the convolutional neural network increases, the features extracted by subsequent convolutional layers become increasingly complex, such as high-level semantic features. Features with higher semantics are more applicable to the problem being solved.
[0124] Because it's often necessary to reduce the number of trainable parameters, pooling layers are often periodically introduced after convolutional layers. This can be done in a single convolutional layer followed by a pooling layer, or in a multi-layered system followed by one or more pooling layers. In image processing, the sole purpose of a pooling layer is to reduce the spatial size of an image. Pooling layers can include average pooling and / or max pooling operators, which are used to downsample the input image to produce a smaller image. The average pooling operator calculates the average value of pixel values within a specific range, producing the average pooling result. The max pooling operator takes the pixel with the largest value within a specific range as the max pooling result. Furthermore, just as the size of the weight matrix used in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The output image size after processing by a pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel in the output image represents the average or maximum value of the corresponding subregion of the image input to the pooling layer.
[0125] After being processed by the convolution layer / pooling layer, the convolutional neural network is still not sufficient to output the required output information. Because as mentioned above, the convolution layer / pooling layer only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network needs to use the neural network layer to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer may include multiple hidden layers, and the parameters contained in the multiple hidden layers can be pre-trained based on relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.
[0126] Optionally, after the multiple hidden layers in the neural network layer, an output layer of the entire convolutional neural network is also included. The output layer has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network is completed, the backpropagation will begin to update the weight values and biases of the aforementioned layers to reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network through the output layer and the ideal result.
[0127] 4. Recurrent Neural Networks
[0128] Recurrent neural networks (RNNs) are designed to process sequential data. In traditional neural network models, layers are fully connected, from the input layer to the hidden layer to the output layer, while nodes within each layer are disconnected. While these conventional neural networks have solved many difficult problems, they are still inadequate for many others. For example, to predict the next word in a sentence, you generally need to use the previous words, as the previous and next words in a sentence are not independent. RNNs are called recurrent neural networks because the current output of a sequence is dependent on the previous output. Specifically, the network memorizes previous information and applies it to the calculation of the current output. This means that nodes within the hidden layer are no longer disconnected but connected, and the input to a hidden layer includes not only the output of the input layer but also the output of the previous hidden layer. In theory, RNNs can process sequence data of any length. Training an RNN is similar to training a traditional CNN or DNN. This approach also uses the backpropagation algorithm, but with one key difference: if the RNN is expanded, its parameters, such as W, are shared; this is not the case with traditional neural networks, as in the example above. Furthermore, when using gradient descent, the output of each step depends not only on the state of the network at the current step but also on the state of the network at several previous steps. This learning algorithm is called backpropagation through time (BPTT).
[0129] Since we already have convolutional neural networks, why do we still need recurrent neural networks? The reason is simple. Convolutional neural networks assume that elements are independent of each other, and that inputs and outputs are also independent, such as cats and dogs. However, in the real world, many elements are interconnected, such as the changes in stock prices over time. Or, for example, someone says, "I love traveling, and my favorite place is Yunnan. I must visit it someday." Humans should know to fill in the blank with "Yunnan." This is because humans make inferences based on context, but how can machines do the same? That's where RNNs come in. RNNs aim to give machines the ability to remember, like humans do. Therefore, the output of an RNN depends on both the current input and historical memory.
[0130] 5. Loss Function
[0131] During the training of a deep neural network, because we want the output of the deep neural network to be as close as possible to the desired predicted value, we can compare the current network's predicted value with the desired target value and then update the weight vector of each layer of the neural network based on the difference between the two. (Of course, before the first update, there is usually an initialization process, which pre-configures the parameters for each layer in the deep neural network.) For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value. This adjustment is continued until the deep neural network can predict the desired target value or a value very close to the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function (or objective function), which is an important equation used to measure the difference between the predicted value and the target value. For example, the loss function output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss.
[0132] 6. Backpropagation algorithm
[0133] Convolutional neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during training, reducing the reconstruction error loss of the super-resolution model. Specifically, the forward propagation of the input signal to the output generates an error loss. This error loss information is then backpropagated to update the parameters of the initial super-resolution model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by the error loss, aiming to obtain the optimal super-resolution model parameters, such as the weight matrix.
[0134] 7. Generative Adversarial Networks
[0135] Generative adversarial networks (GANs) are a type of deep learning model. They consist of at least two modules: a generative model and a discriminative model. These two modules learn through interaction to produce better outputs. Both the generative and discriminative models can be neural networks, specifically deep neural networks or convolutional neural networks. The basic principle of a GAN is as follows: For example, consider a GAN that generates images. Suppose there are two networks, G (Generator) and D (Discriminator). G is the image generator network, which receives random noise z and generates an image from it, denoted as G(z). D is the discriminator network, which determines whether an image is "real." Its input parameter is x, representing an image. Its output, D(x), represents the probability that x is real. A value of 1 indicates a 130% probability of authenticity, while a value of 0 indicates an impossibility of authenticity. During the training of this generative adversarial network, the goal of the generative network G is to generate realistic images as much as possible to deceive the discriminative network D, while the goal of the discriminative network D is to distinguish the images generated by G from real images as much as possible. This creates a dynamic "game" between G and D, which is the "game" in "generative adversarial network." Ultimately, under ideal conditions, G can generate images G(z) that are sufficiently realistic, while D has difficulty determining whether the images generated by G are real, i.e., D(G(z)) = 0.5. This results in an excellent generative model G that can be used to generate images.
[0136] As convolutional neural networks (CNNs) surpass traditional algorithms in computer vision tasks like image recognition and object detection, a growing number of researchers are exploring deep learning-based image / video compression methods. Some researchers have designed end-to-end deep learning image / video compression algorithms. For example, these algorithms simultaneously optimize the encoding network, entropy estimation network, entropy encoding network, entropy decoding network, and decoding network as a whole. The encoding and decoding networks, also known as the transform and inverse transform modules, typically consist of convolutional layers and nonlinear transformation units.
[0137] FIG1 is an example diagram of a typical deep learning image compression framework. As shown in FIG1 , the image compression framework is an implementation of current artificial intelligence (AI) image compression technology.
[0138] On the encoding side, the original image is transformed from the image domain to the feature domain through the encoding network. The transformed image features are then processed by the quantization module and the entropy coding network and encoded into a bitstream for transmission or storage. On the decoding side, the bitstream is decoded into image features through the entropy decoding network. The image features are then transformed from the feature domain to the image domain through the decoding network to produce a reconstructed image. The entropy estimation network estimates the image features and derives an estimated probability value for each feature element for processing by the entropy coding and decoding networks.
[0139] In this embodiment, both the encoding network (Encoder) and the decoding network (Decoder) have nonlinear transformation units.
[0140] FIG2 is an example diagram of a typical deep learning video compression framework. As shown in FIG2 , the video compression framework includes a prediction module (predict model) and a residual compression module (residual compress).
[0141] The prediction module uses the reconstructed image of the previous frame to predict the current frame and generate a predicted image. The residual compression module compresses the residual between the original image and the predicted image of the current frame and decompresses it to obtain a reconstructed residual. The reconstructed residual and the predicted image are summed to obtain the reconstructed image of the current frame. Both the encoding and decoding subnetworks in the prediction and residual compression modules have nonlinear transformation units.
[0142] In this embodiment, both the prediction module (predict model) and the residual compression module (residual compress) have nonlinear transformation units.
[0143] Figure 3 is an example diagram of an image / video storage / transmission application scenario. As shown in Figure 3, the application scenario can be a business involving image / video acquisition, storage, or transmission in a terminal, Huawei Cloud, or video surveillance, such as terminal photo / video recording, photo album, Huawei Cloud photo album, video surveillance, etc.
[0144] Encoding: The camera captures images / videos. The AI image / video encoding network extracts features from the images / video to obtain low-redundancy image features, and then compresses them to produce a bitstream / image file.
[0145] Decoding end: When reconstructed images / videos need to be output, the AI image / video decoding network decompresses the bitstream / image file to obtain image features, and then performs inverse feature extraction on the image features to obtain the reconstructed image / video.
[0146] The storage / transmission module stores the compressed code stream / image files for different services (for example, terminal photography, video surveillance, cloud servers, etc.) or transmits them (for example, cloud services, live broadcast technology, etc.).
[0147] Figure 4 is an example diagram of an image / video storage application scenario. As shown in Figure 4, the application scenario can be a terminal album, Cyberverse (a new "digital reality" technology based on the integration of virtual and real released by Huawei), Huawei Cloud Album, video surveillance, live broadcast, etc.
[0148] Encoding: The camera captures digital images / videos. The AI encoding unit transforms the images / videos into image features with lower redundancy. This unit typically includes a nonlinear transformation unit and exhibits nonlinear characteristics. The entropy estimation unit calculates the encoding probability of each data element in the image features. The entropy coding network losslessly encodes the image features based on the corresponding probabilities to generate a bitstream / image file, further reducing the amount of data transmitted during the image compression process. The storage module saves the bitstream / image file to the corresponding storage location on the terminal.
[0149] Decoding: The loading module loads the bitstream / image file from the corresponding storage location on the terminal and inputs it into the entropy decoding network. The entropy estimation unit is responsible for calculating the decoding probability of each data point in the image feature. Based on the corresponding probabilities of each data point, the entropy decoding network performs lossless decoding on the bitstream / image file to reconstruct the image features. The AI decoding unit performs an inverse transformation on the image features output by entropy decoding, parsing them into a reconstructed image / video. Similar to the AI encoding unit, it typically includes a nonlinear transformation unit and exhibits nonlinear characteristics.
[0150] Optionally, the AI encoding unit, the AI decoding unit and the entropy estimation unit may adopt a network processor (NPU), and the entropy encoding network, the entropy decoding network, the saving module and the loading module may adopt a central processing unit (CPU).
[0151] Figure 5 is an example diagram of Huawei Cloud application scenarios. As shown in Figure 5, the application scenarios can be terminal albums, Cyberverse, Huawei Cloud albums, video surveillance, live broadcasts, etc.
[0152] Encoding end: The original image is obtained locally, image (JPEG) encoded on it to obtain a compressed image, and then the compressed image is sent to the cloud. The cloud performs JPEG decoding on the compressed image to obtain an image, and then the AI encoding unit transforms the image into image features with lower redundancy. It usually includes a nonlinear transformation unit and has nonlinear characteristics. The entropy estimation unit is responsible for calculating the encoding probability of each data in the image feature. The entropy coding network performs lossless encoding on the image features according to the probability corresponding to each data to obtain a code stream / image file, further reducing the amount of data transmitted during the image compression process. The saving module saves the code stream / image file to the corresponding storage location of the terminal.
[0153] Decoding: When a local device needs to obtain an image from the cloud, the cloud loading module loads the bitstream / image file from the corresponding storage location on the terminal and inputs it into the entropy decoding network. The entropy estimation unit is responsible for calculating the decoding probability of each data point in the image features. The entropy decoding network losslessly decodes the bitstream / image file based on the corresponding probabilities of each data point to obtain reconstructed image features. The AI decoding unit inversely transforms the image features output by entropy decoding and parses them into a reconstructed image. Similar to the AI encoding unit, it typically includes a nonlinear transformation unit and has nonlinear characteristics. The cloud then performs JPEG encoding on the image to obtain a compressed image, which is then sent to the local device. The local device performs JPEG decoding on the compressed image to obtain and display the image.
[0154] Optionally, the entropy estimation unit can adopt NPU, the JPEG encoding module, JPEG decoding module, entropy encoding network, entropy decoding network, saving module and loading module can adopt CPU, and the AI encoding unit and AI decoding unit can adopt graphics processing unit (GPU).
[0155] Based on the above application scenarios, an embodiment of the present application provides a nonlinear processing method for image compression to achieve efficient nonlinear transformation processing and improve the rate-distortion performance in image / video compression algorithms.
[0156] FIG6 is a flow chart of process 600 of a nonlinear processing method for image compression according to an embodiment of the present application. Process 600 may be performed by the encoding end or the decoding end in the above-described embodiments. Process 600 is described as a series of steps or operations. It should be understood that process 600 may be performed in various orders and / or concurrently, and is not limited to the execution order shown in FIG6. Process 600 includes the following steps:
[0157] Step 601: Obtain a first image feature to be processed.
[0158] At the encoding end, the first image feature is obtained by converting the image to be processed from the image domain to the feature domain after the encoding end obtains it. The above-mentioned conversion may include: (1) convolution processing, using convolution layers to extract features, with local receptive fields and weight sharing mechanisms (i.e., each filter slides to process input features); (2) using multilayer perceptrons (MLP) or fully connected layers to extract features, with global receptive field characteristics and no weight sharing; (3) transformer processing, which includes matrix multiplication, MLP and normalization processing, with global receptive field characteristics and strong ability to capture long-distance dependencies.
[0159] At the decoding end, the first image feature can be obtained by performing entropy decoding on the bitstream and then performing convolution processing, deconvolution processing, transposed convolution processing, interpolation + convolution processing, Transformer processing, or other methods corresponding to the encoding end. The aforementioned entropy decoding can be implemented using the entropy decoding network in the above embodiment and will not be described in detail here.
[0160] The first image features can be represented as a two-dimensional matrix (L×C, where L represents length and C represents channel) or a three-dimensional matrix (C×H×W, where C represents the number of channels, H represents height, and W represents width). The specific form is related to the above-mentioned conversion method. For example, the first image features obtained by convolution processing or MLP extraction generally correspond to a three-dimensional matrix, while the first image features obtained by transformer processing generally correspond to a two-dimensional matrix.
[0161] For example, the first image feature is represented as a two-dimensional matrix:
[0162] The two-dimensional matrix A is a 3×3 matrix containing 9 elements, each element a(i, j) corresponds to an eigenvalue of the first image feature, where i represents the length corresponding to the element a(i, j), and j represents the channel where the element a(i, j) is located.
[0163] For another example, the first image feature is represented as a three-dimensional matrix:
[0164] The three-dimensional matrix B is a 3×3×2 matrix containing 18 elements, each element a(i, j, l) corresponds to an eigenvalue of the first image feature, where i represents the row where the element a(i, j, l) is located, j represents the column where the element a(i, j, l) is located, and l represents the channel where the element a(i, j, l) is located.
[0165] It should be noted that the embodiment of the present application does not specifically limit the representation method of the first image feature.
[0166] The image to be processed may be a picture, an image frame in a video, or an image block obtained by segmenting the aforementioned picture or image frame, and there is no specific limitation on this.
[0167] Step 602: Perform nonlinear transformation processing on the first image feature to obtain processed image features, where the nonlinear transformation processing sequentially includes a first convolution processing, a second convolution processing, and a point-by-point multiplication operation.
[0168] In one possible implementation, the nonlinear transformation process may include: performing a first convolution process on the first image feature to obtain a second image feature; performing a second convolution process on the second image feature to obtain a third image feature; and performing a point-by-point multiplication operation on the first image feature and the third image feature to obtain a processed image feature.
[0169] Figure 7a is a schematic diagram of the nonlinear transformation processing process with an attention mechanism. As shown in Figure 7a, in an embodiment of the present application, the nonlinear transformation processing includes a first convolution processing, a second convolution processing and a point-by-point multiplication operation.
[0170] First, perform the first convolution on the first image feature and output the second image feature. The second image feature can be considered as the local response of the first image feature (corrected value), that is, the second image feature is the response signal obtained by the first convolution of the first image feature. Since the receptive field of the first convolution is limited, the response value of each position in the second image feature output by the first convolution is only related to the input feature value of the adjacent position, so it is called a local response. The first convolution can be expressed as the following formula: conv1(x)=β1+∑γ1×x
[0171] Among them, γ1 represents the weight of the convolutional layer, and β1 represents the bias parameter of the convolutional layer.
[0172] The second image feature is then subjected to a second convolution process to output a third image feature. The third image feature can be considered as a local response (corrected value) of the second image feature, that is, the third image feature is the response signal obtained by the second convolution process of the second image feature. Since the receptive field of the second convolution process is limited, the response value of each position in the third image feature output by the second convolution process is only related to the input feature value of the adjacent position of the position, so it is called a local response. The second convolution process can be expressed as the following formula: conv2(x)=β2+∑γ2×x
[0173] Among them, γ2 represents the weight of the convolutional layer, and β2 represents the bias parameter of the convolutional layer.
[0174] The first convolution process and the second convolution process satisfy the following conditions: 1<=mid_chs<=chs
[0175] Where mid_chs represents the number of output channels of the first convolution process or the number of input channels of the second convolution process; chs represents the number of input channels of the first convolution process or the number of output channels of the second convolution process. This shows that the third image feature output by the second convolution process has the same number of channels as the first image feature input by the first convolution process.
[0176] Optionally, the above chs=mid_chs; or, chs=2×mid_chs.
[0177] In an embodiment of the present application, the configuration constraints for the number of channels may include: chs and mid_chs can be multiples of 2 to the power of N. When chs = 2 × mid_chs, in terms of theoretical computing power, two convolutions are equivalent to a single convolution with both the number of input channels and the number of output channels being chs. At this time, the model performance and computing power constraints are optimally balanced. For example, in common codec network models, chs can take values such as 256, 192, 128, 96, 64, 32, and 16.
[0178] Optionally, the convolution kernel size ks1_h*w of the first convolution processing and the convolution kernel size ks2_h*w of the second convolution processing can be the same or different.
[0179] In the embodiments of the present application, the convolution kernel size can be configured according to the actual model performance requirements and hardware computing power constraints. For example, the convolution kernel size can be set to symmetrical 1×1, 3×3, 5×5, or the convolution kernel size can be set to asymmetrical 1×3, 3×1, 5×1, 1×5. This embodiment of the present application does not specifically limit this.
[0180] In the embodiment of the present application, the receptive field can be expanded by two convolution processes, or the amount of calculation can be reduced while the receptive field remains unchanged, thereby further enhancing local attention, and by configuring the kernel size of the two convolutions and the number of channels in the intermediate layer, the overall computing power can be effectively balanced. For example, the convolution kernels of the first convolution process and the second convolution process are both 3×3, and two convolution processes can obtain a receptive field of 5×5. The input of the first convolution process is 8 channels, and the output of the first convolution process after processing can be 4 channels. The input of the second convolution process is 4 channels, and the output of the second convolution process after processing can be 8 channels. The number of input / output channels remains unchanged, but only processing 4 channels in the intermediate calculation process can reduce the amount of calculation. If only one convolution process is used, the convolution kernel of the convolution process needs to be 5×5 so as not to change the receptive field, and its input / output is both 8 channels, and the amount of calculation is also relatively high.
[0181] The corresponding multiple feature values in the first image feature and the third image feature are multiplied point by point, that is, the input of the nonlinear transformation and the output of the two convolution processes are multiplied point by point, which can be expressed as the following formula: c(i, j) = a(i, j) × b(i, j)
[0182] Among them, (i, j) represents the index of the eigenvalue in the image feature in which it is located, a(i, j) represents the eigenvalue of the first image feature, b(i, j) represents the eigenvalue of the second image feature, and c(i, j) represents the eigenvalue of the processed image feature.
[0183] Optionally, the size of the matrix corresponding to the third image feature is the same as the size of the matrix corresponding to the first image feature.
[0184] The two convolutions can produce a third image feature of the same size as the first. This depends on the internal structure of the convolution process, particularly the length, width, and number of channels of the filters used in the two convolutions. If the third image feature is the same size as the first, the elements in the matrix corresponding to the third image feature will correspond one-to-one with the elements in the matrix corresponding to the first. In this case, point-by-point multiplication can multiply the values of elements at the same position to obtain the final value of the element at that position.
[0185] Optionally, the size of the matrix corresponding to the third image feature is different from the size of the matrix corresponding to the first image feature.
[0186] The double convolution process can also produce a third image feature with a different size from the first image feature. This also depends on the internal structure of the convolution process, especially the length, width, and number of channels of the filter used in the double convolution process. If the third image feature is of different sizes from the first image feature, the elements in the matrix corresponding to the third image feature will not correspond one-to-one with the elements in the matrix corresponding to the first image feature. In this case, the point-by-point multiplication operation can only multiply the values of the elements at the same position to obtain the final value of the element at that position, while the elements at non-corresponding positions retain their original values.
[0187] The nonlinear transformation process of this embodiment can be expressed as the following formula: y = x*Conv2(Conv1(x))
[0188] Among them, x represents the first image feature, Conv1(x) represents the second image feature, Conv2(Conv1(x)) represents the third image feature, and y represents the processed image feature.
[0189] The above formula uses point-by-point multiplication operations. On the one hand, there is no need to require that the convolution parameters γ and β must be positive numbers to accommodate the restriction that the square root must be a positive number. On the other hand, there is no need to accommodate the restriction that the denominator cannot be 0 in division, which also constitutes a restriction on the values of the convolution parameters γ and β.
[0190] Through the above-mentioned first convolution processing, second convolution processing and point-by-point multiplication operation, a nonlinear transformation processing with an attention mechanism is realized. The attention mechanism means that all eigenvalues in the first image feature are important or redundant. The output of the convolution processing can be the weight of each eigenvalue in the image feature, which can correct the original eigenvalue, highlight the important eigenvalue, and suppress the redundant eigenvalue; the point-by-point multiplication operation uses the above-mentioned correction information to correct the value of each eigenvalue on the first image feature, avoiding the limitation of the convolution parameters.
[0191] In one possible implementation, the above-mentioned nonlinear transformation processing process may include: performing a first convolution processing on the first image feature to obtain a second image feature; performing a second convolution processing on the second image feature to obtain a third image feature; performing a point-by-point multiplication operation on the first image feature and the third image feature to obtain a fourth image feature; and performing a point-by-point addition operation on the first image feature and the fourth image feature to obtain a processed image feature.
[0192] Figure 7b is a schematic diagram of the residual nonlinear transformation processing process with an attention mechanism. As shown in Figure 7b, in an embodiment of the present application, the nonlinear transformation processing includes a first convolution processing, a second convolution processing, a point-by-point multiplication operation, and a point-by-point addition operation.
[0193] In the embodiment of the present application, the first convolution processing, the second convolution processing and the point-by-point multiplication operation can all be described with reference to the embodiment shown in Figure 7a, and will not be repeated here.
[0194] After obtaining the fourth image feature, the corresponding multiple feature values in the first image feature and the fourth image feature are added point by point. That is, the input of the nonlinear transformation and the output of the point-by-point multiplication operation are added point by point. This can be expressed as the following formula: sum(i, j) = a(i, j) + c(i, j)
[0195] Among them, (i, j) represents the index of the eigenvalue in the image feature in which it is located, a(i, j) represents the eigenvalue of the first image feature, c(i, j) represents the eigenvalue of the fourth image feature, and sum(i, j) represents the eigenvalue of the processed image feature.
[0196] The point-by-point addition operation is a residual structure of first addition, which can make the codec network using the above processing process more easily converged during training.
[0197] Optionally, the size of the matrix corresponding to the fourth image feature is the same as the size of the matrix corresponding to the first image feature.
[0198] For the same reason as the third image feature, the fourth image feature can have the same size as the first image feature, and the elements in the matrix corresponding to the fourth image feature have a one-to-one correspondence with the elements in the matrix corresponding to the first image feature. In this case, a point-by-point addition operation can add the values of elements at the same position to obtain the final value of the element at that position.
[0199] Optionally, the size of the matrix corresponding to the fourth image feature is different from the size of the matrix corresponding to the first image feature.
[0200] For the same reason as for the third image feature, the fourth image feature can have different sizes than the first image feature, and the elements in the matrix corresponding to the fourth image feature may not correspond one-to-one with the elements in the matrix corresponding to the first image feature. In this case, the point-by-point addition operation can only add the values of elements at the same position to obtain the final value of the element at that position, while elements at non-corresponding positions retain their original values.
[0201] The nonlinear transformation process of this embodiment can be expressed as the following formula: y = x + x * Conv2 (Conv1 (x))
[0202] Among them, x represents the first image feature, Conv1(x) represents the second image feature, Conv2(Conv1(x)) represents the third image feature, x*Conv2(Conv1(x)) represents the fourth image feature, and y represents the processed image feature.
[0203] In one possible implementation, the above-mentioned nonlinear transformation processing may include: performing a first convolution processing on the first image feature to obtain a second image feature; performing a first activation processing on the second image feature to obtain a fifth image feature; performing a second convolution processing on the fifth image feature to obtain a sixth image feature; and performing a point-by-point multiplication operation on the first image feature and the sixth image feature to obtain a processed image feature.
[0204] Figure 7c is a schematic diagram of the nonlinear transformation processing process with an attention mechanism. As shown in Figure 7c, in an embodiment of the present application, the nonlinear transformation processing includes a first convolution processing, a first activation processing, a second convolution processing and a point-by-point multiplication operation.
[0205] The first activation processing is a nonlinear operation performed on each eigenvalue in the second image feature after the first convolution processing, which may include but is not limited to taking the absolute value operation, the rectified linear unit (Rectified Linear Unit, ReLU) processing series, Sigmoid, Tanh, identity operation (Identity) or parametric rectified linear unit (PReLU) processing, etc.
[0206] Taking the absolute value means taking the absolute value of the input eigenvalue. It can be expressed as the following formula:
[0207] Identity refers to taking the original value of the input feature value. It can be expressed as the following formula: y = x
[0208] Piecewise linear mapping includes rectified linear unit (ReLU) processing or leaky rectified linear unit (LeakyReLU) processing. ReLU is a piecewise linear mapping method. For the input eigenvalue, the output is 0 if the eigenvalue is less than 0, and the output is constant if the eigenvalue is greater than or equal to 0. It can be expressed as follows:
[0209] LeakyReLU is a piecewise linear mapping method that scales input feature values less than 0 with a pre-set weight based on ReLU, which is usually 0.01. It can be expressed as follows:
[0210] Here, a represents a preset value, which is usually set to 0.01.
[0211] The definition of PReLU is similar to LeakyReLU:
[0212] The difference from LeakyReLU is that a is a trainable parameter that can be obtained through training.
[0213] Sigmoid can be expressed as the following operation:
[0214] Tanh can be expressed as the following operation:
[0215] It should be noted that the piecewise linear mapping can also adopt other deformation methods of ReLU, or other new first activation processing methods, which are not specifically limited in the embodiments of the present application.
[0216] After the above-mentioned first activation processing, the second image feature is converted into the fifth image feature. The fifth image feature can also be expressed in the form of a matrix like the first image feature. Since the first activation processing is for each eigenvalue in the second image feature, each eigenvalue of the second image feature has an eigenvalue corresponding to it in the fifth image feature value. Therefore, the matrix corresponding to the fifth image feature and the matrix corresponding to the second image feature have the same size, and the eigenvalues (values of matrix elements) at the same position correspond. For example, the second image feature is expressed as a 3×3 matrix like the above-mentioned matrix A, then the fifth image feature can also be expressed as a 3×3 matrix, but since the second image feature and the fifth image feature have undergone the first activation processing, the eigenvalues in the second image feature and the fifth image feature are not exactly the same. Accordingly, the element values in the matrices corresponding to the two are also not exactly the same.
[0217] In the embodiment of the present application, in addition to the above-mentioned first activation processing, the first convolution processing, the second convolution processing and the point-by-point multiplication operation can all refer to the description in the embodiment shown in Figure 7a, and will not be repeated here.
[0218] The nonlinear transformation process of this embodiment can be expressed as the following formula: y = x*Conv2(φ1(Conv1(x)))
[0219] Among them, x represents the first image feature, Conv1(x) represents the second image feature, φ1(Conv1(x)) represents the fifth image feature, Conv2(φ1(Conv1(x))) represents the sixth image feature, and y represents the processed image feature.
[0220] In one possible implementation, the above-mentioned nonlinear transformation processing process may include: performing a first convolution processing on the first image feature to obtain a second image feature; performing a first activation processing on the second image feature to obtain a fifth image feature; performing a second convolution processing on the fifth image feature to obtain a sixth image feature; performing a point-by-point multiplication operation on the first image feature and the sixth image feature to obtain a seventh image feature; and performing a point-by-point addition operation on the first image feature and the seventh image feature to obtain a processed image feature.
[0221] Figure 7d is a schematic diagram of the residual nonlinear transformation processing process with an attention mechanism. As shown in Figure 7d, in an embodiment of the present application, the nonlinear transformation processing includes a first convolution processing, a first activation processing, a second convolution processing, a point-by-point multiplication operation, and a point-by-point addition operation.
[0222] In the embodiment of the present application, the first convolution processing, the second convolution processing and the point-by-point multiplication operation can all refer to the description in the embodiment shown in Figure 7a, the point-by-point addition operation can refer to the description in the embodiment shown in Figure 7b, and the first activation processing can refer to the description in the embodiment shown in Figure 7c.
[0223] The nonlinear transformation process of this embodiment can be expressed as the following formula: y = x + x * Conv2 (φ1 (Conv1 (x)))
[0224] Where x represents the first image feature, Conv1(x) represents the second image feature, φ1(Conv1(x)) represents the fifth image feature, Conv2(φ1(Conv1(x))) represents the sixth image feature, and x*Conv2(φ1(Conv1(x))) represents the seventh image feature. y Represents the processed image features.
[0225] In one possible implementation, the above-mentioned nonlinear transformation processing process may include: performing a first convolution processing on the first image feature to obtain a second image feature; performing a first activation processing on the second image feature to obtain a fifth image feature; performing a second convolution processing on the fifth image feature to obtain a sixth image feature; performing a second activation processing on the sixth image feature to obtain an eighth image feature; and performing a point-by-point multiplication operation on the first image feature and the eighth image feature to obtain a processed image feature.
[0226] Figure 7e is a schematic diagram of the nonlinear transformation processing process with an attention mechanism. As shown in Figure 7e, in an embodiment of the present application, the nonlinear transformation processing includes a first convolution processing, a first activation processing, a second convolution processing, a second activation processing and a point-by-point multiplication operation.
[0227] In the embodiment of the present application, the first convolution processing, the second convolution processing and the point-by-point multiplication operation can all refer to the description in the embodiment shown in Figure 7a, and the first activation processing and the second activation processing can refer to the description in the embodiment shown in Figure 7c.
[0228] The nonlinear transformation process of this embodiment can be expressed as the following formula: y = x*φ2(Conv2(φ1(Conv1(x))))
[0229] Among them, x represents the first image feature, Conv1(x) represents the second image feature, φ1(Conv1(x)) represents the fifth image feature, Conv2(φ1(Conv1(x))) represents the sixth image feature, φ2(Conv2(φ1(Conv1(x)))) represents the eighth image feature, and y represents the processed image feature.
[0230] In one possible implementation, the above-mentioned nonlinear transformation processing process may include: performing a first convolution processing on the first image feature to obtain a second image feature; performing a first activation processing on the second image feature to obtain a fifth image feature; performing a second convolution processing on the fifth image feature to obtain a sixth image feature; performing a second activation processing on the sixth image feature to obtain an eighth image feature; performing a point-by-point multiplication operation on the first image feature and the eighth image feature to obtain a ninth image feature; and performing a point-by-point addition operation on the first image feature and the ninth image feature to obtain a processed image feature.
[0231] Figure 7f is a schematic diagram of the residual nonlinear transformation processing process with an attention mechanism. As shown in Figure 7f, in an embodiment of the present application, the nonlinear transformation processing includes a first convolution processing, a first activation processing, a second convolution processing, a second activation processing, a point-by-point multiplication operation, and a point-by-point addition operation.
[0232] In the embodiment of the present application, the first convolution processing, the second convolution processing and the point-by-point multiplication operation can all refer to the description in the embodiment shown in Figure 7a, the point-by-point addition operation can refer to the description in the embodiment shown in Figure 7b, and the first activation processing and the second activation processing can refer to the description in the embodiment shown in Figure 7c.
[0233] The nonlinear transformation process of this embodiment can be expressed as the following formula: y = x + x * φ2 (Conv2 (φ1 (Conv1 (x))))
[0234] Among them, x represents the first image feature, Conv1(x) represents the second image feature, φ1(Conv1(x)) represents the fifth image feature, Conv2(φ1(Conv1(x))) represents the sixth image feature, φ2(Conv2(φ1(Conv1(x)))) represents the eighth image feature, x*φ2(Conv2(φ1(Conv1(x)))) represents the ninth image feature, and y represents the processed image feature.
[0235] The embodiment of the present application uses the operation of fusing two layers of convolution processing and activation processing to achieve channel-adaptive nonlinear transformation processing.
[0236] It should be noted that the embodiments shown in Figures 7a to 7f are only several examples of nonlinear transformation processing, but do not constitute a limitation on the process of nonlinear transformation processing. In some possible implementations, nonlinear transformation processing may also include more activation processing and convolution processing. For example, after the second activation processing in the embodiments shown in Figures 7e and 7f, a third convolution processing, or a third convolution processing and a third activation processing may also be included, and so on. The embodiments of the present application do not specifically limit the process of nonlinear transformation processing.
[0237] At the encoding end, encoding can be performed based on the processed image features to obtain a code stream.
[0238] After obtaining the processed image features, the encoding side can continue to perform convolution processing on them, or can further perform nonlinear transformation processing on the output of the convolution processing after the convolution processing, and then perform entropy coding on the results of the above processing to obtain a bitstream. Entropy coding can be implemented using the entropy coding network in the above embodiment, which will not be described in detail here. Other coding methods can also be used to encode the results of the above processing to obtain a bitstream, which is not limited by this application.
[0239] At the decoding end, a reconstructed image can be obtained based on the processed image features.
[0240] After obtaining the processed image features, the decoding side may continue to perform convolution processing on them, or may perform nonlinear transformation processing on the output of the convolution processing again after the convolution processing, thereby converting it from the feature domain to the image domain to obtain a reconstructed image.
[0241] The embodiments of the present application change the nonlinear transformation processing in image compression and perform multiple convolution processes to expand the receptive field, or reduce the amount of calculation while keeping the receptive field unchanged, further enhance local attention, maintain model performance and effectively balance computing power.
[0242] The nonlinear processing method for image compression is described below through several specific embodiments.
[0243] Example 1
[0244] The number of input channels and output channels of the first convolution process satisfies chs = mid_chs. Both the first activation process and the second activation process can be set to PReLU with learnable parameters. This is shown in Figure 8 (Figure 8 is a schematic diagram of the residual nonlinear transformation process with an attention mechanism).
[0245] Example 2
[0246] The number of input channels and output channels in the first convolution process satisfies chs = mid_chs. The first activation process can be set to PReLU with learnable parameters, and the second activation process can be set to fixed Identity. This is shown in Figure 9 (Figure 9 is a schematic diagram of nonlinear transformation processing with an attention mechanism).
[0247] In this embodiment, the second activation process is set to a fixed Identity. Since Identity takes the original value, the processing process shown in Figure 9 can be transformed, that is, as shown in Figure 10 (Figure 10 is a schematic diagram of nonlinear transformation processing with attention mechanism), the second activation process is removed, so that the nonlinear transformation process is similar to Figure 7c.
[0248] Example 3
[0249] The number of input channels and output channels in the first convolution process satisfies chs = 2 × mid_chs. Both the first activation process and the second activation process can be set to PReLU with learnable parameters, as shown in Figure 8.
[0250] Example 4
[0251] The number of input channels and output channels of the first convolution process satisfies chs = 2 × mid_chs. The first activation process can be set to PReLU with learnable parameters, and the second activation process can be set to fixed Identity, as shown in Figure 9 or Figure 10.
[0252] FIG11 is a schematic diagram of an exemplary structure of an encoding device 1100 according to an embodiment of the present application. As shown in FIG11 , the device 1100 according to the present embodiment can be applied to an encoding end. The device 1100 may include: an acquisition module 1101, a transformation module 1102, and an encoding module 1103.
[0253] The acquisition module 1101 is used to obtain the first image feature to be processed; the transformation module 1102 is used to perform nonlinear transformation processing on the first image feature to obtain a processed image feature, and the nonlinear transformation processing includes a first convolution processing, a second convolution processing and a point-by-point multiplication operation in sequence.
[0254] In a possible implementation, the nonlinear transformation process further includes a point-by-point addition operation after the point-by-point multiplication operation.
[0255] In a possible implementation, the nonlinear transformation process further includes a first activation process after the first convolution process.
[0256] In a possible implementation, the nonlinear transformation process further includes a second activation process after the second convolution process.
[0257] In a possible implementation, the first activation processing and the second activation processing respectively include an identity operation Identity, a rectified linear unit ReLU processing, a leaky rectified linear unit LeakyReLU processing, or a parameter rectified linear unit PReLU processing.
[0258] In one possible implementation, the transformation module 1102 is specifically used to perform the first convolution processing on the first image feature to obtain the second image feature; perform the second convolution processing on the second image feature to obtain the third image feature; and perform the point-by-point multiplication operation on the first image feature and the third image feature to obtain the processed image feature.
[0259] In one possible implementation, the transformation module 1102 is specifically used to perform the first convolution processing on the first image feature to obtain a second image feature; perform the second convolution processing on the second image feature to obtain a third image feature; perform the point-by-point multiplication operation on the first image feature and the third image feature to obtain a fourth image feature; and perform the point-by-point addition operation on the first image feature and the fourth image feature to obtain the processed image feature.
[0260] In a possible implementation, the first convolution process and the second convolution process satisfy the following conditions: 1<=mid_chs<=chs
[0261] Among them, mid_chs represents the number of output channels of the first convolution processing or the number of input channels of the second convolution processing; chs represents the number of input channels of the first convolution processing or the number of output channels of the second convolution processing.
[0262] In a possible implementation, chs=mid_chs; or, chs=2×mid_chs.
[0263] In a possible implementation, the convolution kernel size of the first convolution processing and the convolution kernel size of the second convolution processing are the same as or different from each other.
[0264] In a possible implementation, the encoding module 1103 is configured to encode the processed image features to obtain a code stream.
[0265] The device of this embodiment can be used to execute the technical solution of the method embodiment shown in Figure 6. Its implementation principle and technical effects are similar and will not be repeated here.
[0266] FIG12 is a schematic diagram of an exemplary structure of a decoding device 1200 according to an embodiment of the present application. As shown in FIG12 , the device 1200 according to the present embodiment can be applied to a decoding end. The device 1200 may include: an acquisition module 1201, a transformation module 1202, and a reconstruction module 1203.
[0267] The acquisition module 1201 is used to obtain the first image feature to be processed; the transformation module 1202 is used to perform nonlinear transformation processing on the first image feature to obtain a processed image feature, and the nonlinear transformation processing includes a first convolution processing, a second convolution processing and a point-by-point multiplication operation in sequence.
[0268] In a possible implementation, the nonlinear transformation process further includes a point-by-point addition operation after the point-by-point multiplication operation.
[0269] In a possible implementation, the nonlinear transformation process further includes a first activation process after the first convolution process.
[0270] In a possible implementation, the nonlinear transformation process further includes a second activation process after the second convolution process.
[0271] In a possible implementation, the first activation processing and the second activation processing respectively include an identity operation Identity, a rectified linear unit ReLU processing, a leaky rectified linear unit LeakyReLU processing, or a parameter rectified linear unit PReLU processing.
[0272] In one possible implementation, the transformation module 1202 is specifically used to perform the first convolution processing on the first image feature to obtain the second image feature; perform the second convolution processing on the second image feature to obtain the third image feature; and perform the point-by-point multiplication operation on the first image feature and the third image feature to obtain the processed image feature.
[0273] In one possible implementation, the transformation module 1202 is specifically used to perform the first convolution processing on the first image feature to obtain a second image feature; perform the second convolution processing on the second image feature to obtain a third image feature; perform the point-by-point multiplication operation on the first image feature and the third image feature to obtain a fourth image feature; and perform the point-by-point addition operation on the first image feature and the fourth image feature to obtain the processed image feature.
[0274] In a possible implementation, the first convolution process and the second convolution process satisfy the following conditions: 1<=mid_chs<=chs
[0275] Among them, mid_chs represents the number of output channels of the first convolution processing or the number of input channels of the second convolution processing; chs represents the number of input channels of the first convolution processing or the number of output channels of the second convolution processing.
[0276] In a possible implementation, chs=mid_chs; or, chs=2×mid_chs.
[0277] In a possible implementation, the convolution kernel size of the first convolution processing and the convolution kernel size of the second convolution processing are the same as or different from each other.
[0278] In a possible implementation, the reconstruction module 1203 is configured to obtain a reconstructed image according to the processed image features.
[0279] The device of this embodiment can be used to execute the technical solution of the method embodiment shown in Figure 6. Its implementation principle and technical effects are similar and will not be repeated here.
[0280] During implementation, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware coding processor, or can be executed by a combination of hardware and software modules in the coding processor. The software module can be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0281] The memory mentioned in the above embodiments may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0282] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.
[0283] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0284] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0285] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.
[0286] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0287] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to make a computer device (personal computer, server, or network equipment, etc.) perform all or part of the steps of the method described in each embodiment of the embodiment of the present application. Or the computer software product can be transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instruction can be transmitted from a website, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center that includes one or more available media integrations. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD), a read-only memory (ROM), a random access memory (RAM)), etc.
[0288] The above description is merely a specific implementation of the embodiments of the present application, but the scope of protection of the embodiments of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the embodiments of the present application should be included in the scope of protection of the embodiments of the present application. Therefore, the scope of protection of the embodiments of the present application should be based on the scope of protection of the claims.
Claims
1. A nonlinear processing method for image compression, It is characterized in that include: Acquire a first image feature to be processed; A nonlinear transformation process is performed on the first image feature to obtain a processed image feature, wherein the nonlinear transformation process sequentially includes a first convolution process, a second convolution process and a point-by-point multiplication operation.
2. The method according to claim 1, It is characterized in that The non-linear transformation process further includes a point-by-point addition operation after the point-by-point multiplication operation.
3. The method according to claim 1 or 2, It is characterized in that The nonlinear transformation process also includes a first activation process after the first convolution process.
4. The method according to claim 3, It is characterized in that The nonlinear transformation process further includes a second activation process after the second convolution process.
5. The method according to claim 4, It is characterized in that The first activation processing and the second activation processing respectively include an identity operation Identity, a rectified linear unit ReLU processing, a leaky rectified linear unit LeakyReLU processing or a parameter rectified linear unit PReLU processing.
6. The method according to claim 1, It is characterized in that The performing nonlinear transformation processing on the first image feature to obtain a processed image feature includes: Performing the first convolution process on the first image feature to obtain a second image feature; performing the second convolution process on the second image feature to obtain a third image feature; The first image feature and the third image feature are subjected to the point-by-point multiplication operation to obtain the processed image feature.
7. The method according to claim 2, It is characterized in that The performing nonlinear transformation processing on the first image feature to obtain a processed image feature includes: Performing the first convolution process on the first image feature to obtain a second image feature; performing the second convolution process on the second image feature to obtain a third image feature; Performing the point-by-point multiplication operation on the first image feature and the third image feature to obtain a fourth image feature; The first image feature and the fourth image feature are subjected to the point-by-point addition operation to obtain the processed image feature.
8. The method according to any one of claims 1 to 7, It is characterized in that The first convolution process and the second convolution process satisfy the following conditions: 1<=mid_chs<=chs Among them, mid_chs represents the number of output channels of the first convolution processing or the number of input channels of the second convolution processing; chs represents the number of input channels of the first convolution processing or the number of output channels of the second convolution processing.
9. The method according to claim 8, It is characterized in that chs=mid_chs; or, chs=2×mid_chs.
10. The method according to any one of claims 1 to 9, It is characterized in that The convolution kernel size of the first convolution processing and the convolution kernel size of the second convolution processing are the same or different.
11. The method according to any one of claims 1 to 10, It is characterized in that After performing nonlinear transformation processing on the first image feature to obtain processed image features, the method further includes: Encoding is performed according to the processed image features to obtain a code stream.
12. The method according to any one of claims 1 to 10, It is characterized in that After performing nonlinear transformation processing on the first image feature to obtain processed image features, the method further includes: A reconstructed image is obtained according to the processed image features.
13. A nonlinear processing device for image compression, It is characterized in that include: An acquisition module, used for acquiring a first image feature to be processed; The transformation module is used to perform nonlinear transformation processing on the first image feature to obtain a processed image feature, wherein the nonlinear transformation processing sequentially includes a first convolution processing, a second convolution processing and a point-by-point multiplication operation.
14. The device according to claim 13, It is characterized in that The non-linear transformation process further includes a point-by-point addition operation after the point-by-point multiplication operation.
15. The device according to claim 13 or 14, It is characterized in that The nonlinear transformation process also includes a first activation process after the first convolution process.
16. The device according to claim 15, It is characterized in that The nonlinear transformation process further includes a second activation process after the second convolution process.
17. The device according to claim 16, It is characterized in that The first activation processing and the second activation processing respectively include an identity operation Identity, a rectified linear unit ReLU processing, a leaky rectified linear unit LeakyReLU processing or a parameter rectified linear unit PReLU processing.
18. The device according to claim 13, It is characterized in that The transformation module is specifically used to perform the first convolution processing on the first image feature to obtain the second image feature; perform the second convolution processing on the second image feature to obtain the third image feature; and perform the point-by-point multiplication operation on the first image feature and the third image feature to obtain the processed image feature.
19. The device according to claim 14, It is characterized in that The transformation module is specifically used to perform the first convolution processing on the first image feature to obtain a second image feature; perform the second convolution processing on the second image feature to obtain a third image feature; perform the point-by-point multiplication operation on the first image feature and the third image feature to obtain a fourth image feature; and perform the point-by-point addition operation on the first image feature and the fourth image feature to obtain the processed image feature.
20. The device according to any one of claims 13 to 19, It is characterized in that The first convolution process and the second convolution process satisfy the following conditions: 1<=mid_chs<=chs Among them, mid_chs represents the number of output channels of the first convolution processing or the number of input channels of the second convolution processing; chs represents the number of input channels of the first convolution processing or the number of output channels of the second convolution processing.
21. The device according to claim 20, It is characterized in that chs=mid_chs; or, chs=2×mid_chs.
22. The device according to any one of claims 13 to 21, It is characterized in that The convolution kernel size of the first convolution processing and the convolution kernel size of the second convolution processing are the same or different.
23. An encoder, It is characterized in that include: one or more processors; A non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the decoder to perform the method of any one of claims 1-11.
24. A decoder, It is characterized in that include: one or more processors; A non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the decoder to perform the method of any one of claims 1-10 and 12.
25. A computer program product, It is characterized in that The method comprises a program code, and when the program code is executed on a computer or a processor, it is used to execute the method according to any one of claims 1 to 12.
26. A computer-readable storage medium comprising instructions, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 12.
Citation Information
Patent Citations
Image processing method and device, electronic equipment and storage medium
CN111008924A
Image processing method and device
CN113949867A
Image compression method and apparatus thereof
US20220329807A1
Image processing method and device using feature map compression
WO2019143026A1
Cited By
Data encoding method and related device for lossless compression using trained volume preserving flow model
US12626416B2
