Image feature processing method, encoding device and decoding device

By introducing content gain attention module and mobile window attention module in the verification model of JPEG AI, the problem of residual non-local attention block not taking into account parameters β and image content is solved, and the accuracy and long image feature extraction are improved. The capture capability of distance correlation improves image encoding and decoding performance.

CN119094769AActive Publication Date: 2024-12-06HISENSE VISUAL TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202310863406.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-06-06
Filing Date
2023-07-13
Publication Date
2024-12-06
Estimated Expiration
2043-07-13

AI Technical Summary

Technical Problem

In the verification model of JPEG AI, when analyzing transform networks and synthetic transform networks extract deep features through residual non-local attention blocks, the influence of different parameters β on image content is not considered, and the ability of mask branches to capture pixel long-distance correlation is limited. .

Method used

The content gain attention module is introduced, and the content gain weight is generated based on the parameter β through the full connection layer, and combined with the mobile window attention module, attention weighting of image content is performed, improving the accuracy of feature extraction and the capture ability of long-distance correlation.

Benefits of technology

The performance of the JPEG AI verification model in the image encoding and decoding process is improved, the adaptability to different parameters β and image content is ensured, and the accuracy and reconstruction quality of image feature extraction are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119094769A_ABST
    Figure CN119094769A_ABST
Patent Text Reader

Abstract

The invention relates to an image feature processing method, encoding equipment and decoding equipment, and particularly relates to the technical field of image encoding and decoding. Comprising the steps of obtaining a first to-be-transformed image feature; performing at least one first processing on the first to-be-transformed image feature to obtain a first input image feature; processing the first input image feature based on a content gain attention block to obtain a first output image feature, the content gain attention block being used for performing attention weighting on the content of the image; and performing the first processing, down-sampling processing and convolution processing on the first output image feature for at least one time to obtain an analysis transformation image feature.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on June 6, 2023, with application number 2023106645409. The entire contents of this application are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image encoding and decoding technology, and in particular to an image feature processing method, an encoding device, and a decoding device. Background Art

[0003] Image compression is one of the basic tasks in computer vision and image processing. The development of lossy image compression technology has brought benefits to various fields. Image compression technology can greatly reduce the amount of data for image storage and image transmission, but the image quality obtained after reconstructing the image may be slightly reduced. In recent years, deep neural networks have achieved great success in various tasks, and image compression based on deep neural networks has begun to surpass traditional image compression technology. JPEG AI is a learning-based image coding standard created by the Joint Photographic Experts Group (JPEG). The verification model of JPEG AI is based on human vision and achieves a lower bit rate at the same subjective quality compared with traditional image coding.

[0004] In the verification model of JPEG AI, both the analysis transformation network and the synthesis transformation network use residual non-local attention blocks to extract deep features. This residual non-local attention block can reduce the pixel redundancy of the feature map by capturing the correlation between pixels. The residual non-local attention block has two branches: the trunk branch and the mask branch. The trunk branch contains 3 residual blocks to extract deep features; the mask branch contains upsampling convolution, downsampling convolution and a series of residual blocks to scale deep features.

[0005] However, in the current JPEG AI verification model, both the analysis transformation network and the synthesis transformation network extract deep features through residual non-local attention blocks and do not consider the impact of different parameters β on different image contents. Summary of the invention

[0006] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides an image feature processing method, an encoding device and a decoding device.

[0007] In order to achieve the above objectives, some embodiments of the present application provide the following technical solutions:

[0008] In a first aspect, an image feature processing method is provided, which is applied to an encoding device, comprising:

[0009] Acquire a first image feature to be transformed;

[0010] Performing at least one first processing on the first image feature to be transformed to obtain a first input image feature, wherein the first processing includes downsampling processing and residual activation processing;

[0011] Processing the first input image feature based on a content gain attention block to obtain a first output image feature, wherein the content gain attention block is used to perform attention weighting on the content of the image;

[0012] The first output image feature is subjected to at least one first processing, a downsampling processing and a convolution processing to obtain an analysis transformation image feature.

[0013] In a second aspect, an image feature processing method is provided, which is applied to a decoding device, comprising:

[0014] Acquire features of a second image to be transformed;

[0015] Performing at least one residual block processing, upsampling processing, and at least one second processing on the second image feature to be transformed to obtain a second input image feature, wherein the second processing includes residual activation processing and upsampling processing;

[0016] Processing the second input image feature based on the content gain attention block to obtain a second output image feature, wherein the content gain attention block is used to perform attention weighting on the content of the image;

[0017] The second output image feature is subjected to at least one second processing to obtain a synthetic transformed image feature.

[0018] In a third aspect, a coding device is provided, including:

[0019] A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the image feature processing method described in the first aspect or any one of its optional embodiments.

[0020] In a fourth aspect, a decoding device is provided, including:

[0021] A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the image feature processing method described in the second aspect or any one of its optional embodiments.

[0022] In a fifth aspect, a computer-readable storage medium is provided, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the image feature processing method described in the first aspect or any one of its optional embodiments is implemented, or the image feature processing method described in the second aspect or any one of its optional embodiments is implemented.

[0023] In a sixth aspect, a computer program product is provided, wherein a computer program is stored in the computer program product, and when the computer program is executed by a processor, the image feature processing method described in the first aspect or any one of its optional embodiments is implemented, or the image feature processing method described in the second aspect or any one of its optional embodiments is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0025] In order to more clearly illustrate the technical solutions in some embodiments of the present application, the drawings required for describing the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 A schematic diagram of the architecture of a JPEG AI verification model in the related technology provided in an embodiment of the present application;

[0027] Figure 2 A schematic diagram of an analysis and transformation network 101 in the related technology provided in an embodiment of the present application;

[0028] Figure 3 A schematic diagram of a residual activation unit in the related technology provided in an embodiment of the present application;

[0029] Figure 4 A schematic diagram of a residual non-local attention block 203 in the related art provided in an embodiment of the present application;

[0030] Figure 5 A schematic diagram of a residual block in the related technology provided in an embodiment of the present application;

[0031] Figure 6 A schematic diagram of a masked convolution kernel used in a context model of an autoregressive convolution in the related technology provided in an embodiment of the present application;

[0032] Fig. 7A A schematic diagram of a mask convolution kernel used by a context model network 105 in a JPEG AI verification model in the related technology provided in an embodiment of the present application;

[0033] Figure 7B A schematic diagram of a wavefront parallel processing mechanism of a latent space in a related technology provided in an embodiment of the present application;

[0034] Figure 8 A schematic diagram of a synthesis transformation network 117 in a related technology provided in an embodiment of the present application;

[0035] Fig. 9 A schematic diagram of another JPEG AI verification model in the related art provided in an embodiment of the present application;

[0036] Fig.10 A schematic diagram of an ICCI sub-network provided in an embodiment of the present application;

[0037] Fig.11 A schematic diagram of a content gain attention module provided in an embodiment of the present application;

[0038] Fig.12 A method based on the embodiment of the present application is provided Fig.11 A flowchart of an image feature processing method performed by a content gain attention module shown;

[0039] Fig.13 A schematic diagram of a moving window attention module provided in an embodiment of the present application;

[0040] Fig.14A A schematic diagram of an analysis transformation network 101 for embedding a content gain attention module into a verification model of JPEG AI provided in an embodiment of the present application;

[0041] Fig. 14B A schematic diagram of another analysis transformation network 101 for embedding a content gain attention module into a verification model of JPEG AI provided in an embodiment of the present application;

[0042] Fig. 14C A flowchart of an image feature processing method performed by an analysis transformation network based on an embedded content gain attention module provided in an embodiment of the present application;

[0043] Fig.15A A schematic diagram of a synthetic transformation network 117 for embedding a content gain attention module into a verification model of JPEG AI provided in an embodiment of the present application;

[0044] Fig. 15B A schematic diagram of another synthetic transformation network 117 for embedding a content gain attention module into a verification model of JPEG AI provided in an embodiment of the present application;

[0045] Fig. 15CA flowchart of an image feature processing method performed by a synthetic transformation network based on an embedded content gain attention module provided in an embodiment of the present application;

[0046] Fig.16A A schematic diagram of replacing a residual non-local attention block in an analysis transformation network 101 with a content gain attention module provided in an embodiment of the present application;

[0047] Fig. 16B A flowchart of an image feature processing method performed by an analysis transformation network based on replacing a residual non-local attention block with a content gain attention module provided in an embodiment of the present application;

[0048] Fig.17A A schematic diagram of replacing a residual non-local attention block in a synthesis transformation network 117 with a content gain attention module provided in an embodiment of the present application;

[0049] Fig. 17B A flowchart of an image feature processing method performed by a synthetic transformation network based on replacing a residual non-local attention block with a content gain attention module provided in an embodiment of the present application;

[0050] Fig.18A A schematic diagram of a content gain attention module provided in an embodiment of the present application that combines replacement and embedding in an analysis transformation network 101;

[0051] Fig.18B A flowchart of an image feature processing method performed by an analysis transformation network based on a combination of content gain attention module embedding and replacement provided in an embodiment of the present application;

[0052] Fig.19A A schematic diagram of a content gain attention module provided in an embodiment of the present application that combines replacement and embedding in a synthetic transformation network 117;

[0053] Fig.19B A flowchart of an image feature processing method performed by a synthetic transformation network based on a combination of content gain attention module embedding and replacement provided in an embodiment of the present application;

[0054] Fig. 20 A schematic diagram of the structure of a decoding device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0056] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0057] The terms "first" and "second" in the specification and claims of this application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first context feature and a second context feature are used to distinguish different context features rather than to describe a specific order of context features.

[0058] Image compression is one of the basic tasks in computer vision and image processing. The development of lossy image compression technology has brought benefits to various fields. Image compression technology can greatly reduce the amount of data for image storage and image transmission, but the image quality obtained after reconstructing the image may be slightly reduced. In the field of lossy image compression technology, traditional codecs including the Joint Photographic Experts Group (JPEG), Better Portable Graphics (BPG) and Versatile Video Coding (VVC) use similar transforms, quantization and entropy coding to compress images.

[0059] In recent years, deep neural networks have achieved great success in various tasks, and image compression based on deep neural networks has begun to surpass traditional image compression technology. JPEG AI is a learning-based image coding standard created by the Joint Photographic Experts Group (JPEG). JPEG AI targets human vision and achieves a lower bit rate than traditional image coding at the same subjective quality.

[0060] Figure 1A schematic diagram of the architecture of a JPEG AI verification model (Verification Model 1.0) in the related technology provided in an embodiment of the present application. The JPEG AI verification model includes: an analysis transform network (Analysis Transform Net) 101, a gain unit 102, a first quantization unit 103, a first inverse gain unit 104, a context model network 105, a prediction fusion network 106, a super priori encoding network 107, a super priori decoding network 108, a second quantization unit 109, a first lossless encoder 110, a first lossless decoder 111, a decomposition entropy model 112, a super priori scale decoding network 113, a second lossless encoder 114, a second lossless decoder 115, a second inverse gain unit 116, and a synthesis transform network 117. The JPEG AI verification model can compress the input original image x into a code stream, and reconstruct the compressed code stream into a reconstructed image In the verification model of JPEGAI, the network modules Analysis Transform Net 101, gain unit 102, first quantization unit 103, first inverse gain unit 104, super priori decoding network 108, second quantization unit 109, first lossless encoder 110, and second lossless encoder 114 only exist in the encoder; the network modules second lossless decoder 115, second inverse gain unit 116, and synthesis transform network 117 only exist in the decoder. The network modules context model network 105, prediction fusion network 106, super priori coding network 107, first lossless decoder 111, decomposition entropy model 112, and super priori scale decoding network 113 exist in both the encoder and the decoder.

[0061] The above-mentioned analysis transformation network 101 is used to transform the original image x input into the verification model of the JPEG AI into a latent tensor y. The analysis transformation network 101 only works in the encoder. The latent tensor is a high-dimensional representation of the data. The latent tensor cannot be directly observed from the input data, but they can be determined by the information learned from the input data. The above-mentioned latent tensor y is determined based on the information learned from the original image x.

[0062] Figure 2 A schematic diagram of an analysis transformation network 101 in the related technology provided in the embodiment of the present application is shown as follows: Figure 2As shown, the analysis transformation network 101 includes, from the input end to the output end, a 3×3 downsampling convolution 201, a residual activation unit 202, a residual non-local attention block (Residual non-local attention networks for image restoration) 203, a 3×3 downsampling convolution 204, a residual activation unit 205, a 3×3 downsampling convolution 206, a residual activation unit 207, a 3×3 downsampling convolution 208 and a 1×1 convolution 209.

[0063] Among them, the analysis transformation network 101 uses a residual activation unit with an attention mechanism, for example, Figure 2 The residual activation unit 202, the residual activation unit 205 and the residual activation unit 207 in are all residual activation units with attention mechanism. Figure 2 The 3×3 downsampling convolution 201, the 3×3 downsampling convolution 204, the 3×3 downsampling convolution 206 and the 3×3 downsampling convolution 208 are all 3×3 downsampling convolutions with the same structure.

[0064] Figure 3 Schematic diagram of a residual activation unit 202 in the related technology provided in the embodiment of the present application. Figure 3 As shown, the residual activation unit 202 may include: a LeakyReLU activation function 301, a convolution layer 302, a tanh activation function 303 and a residual connection unit 304. The residual activation unit is used to add nonlinear transformation.

[0065] In the JPEG AI verification model, the attention mechanism can be used Figure 2 The residual non-local attention block 203 shown is implemented. The residual non-local attention block 203 can extract deep features. In the process of extracting deep features, by capturing the correlation between pixels, the pixel redundancy of the feature map can be reduced. Figure 4 Schematic diagram of a residual non-local attention block 203 in the related art provided in the embodiment of the present application. Figure 4 As shown, the residual non-local attention block 203 includes two branches: a trunk branch 41 and a mask branch 42. The trunk branch 41 contains three residual blocks for extracting deep features. The mask branch 42 contains a 3×3 downsampling convolution 401, a 3×3 upsampling convolution 402, a 1×1 convolution 403, a sigmoid activation function 404, and a series of residual blocks. Figure 4The three residual blocks of the trunk branch 41 are respectively represented as residual block 1, residual block 2, and residual block 3; the series of residual blocks included in the mask branch 42 are respectively represented as residual block 4, residual block 5, residual block 6, residual block 7, residual block 8, and residual block 9. It should be noted that the structure and function of each residual block in the above-mentioned residual block 1 to residual block 9 are the same.

[0066] Figure 5 for Figure 4 A schematic diagram of a residual block in the embodiment shown, each residual block includes a LeakyReLU activation function 501 and two 3×3 convolutions, wherein the two 3×3 convolutions are Figure 5 They are represented as 3×3 convolution 502 and 3×3 convolution 503 respectively.

[0067] like Figure 1 As shown, in the verification model (Verification Model 1.0) of JPEG AI, the interpolation gain unit is used to realize the variable bit rate. The parameter of the interpolation gain unit is controlled by the parameter β to scale the channel dimension of the potential residual variable. The interpolation gain unit includes the above-mentioned gain unit 102 and the first inverse gain unit 104.

[0068] The gain unit 102 is used to perform gain processing on the residual tensor r, and multiply each element of the residual tensor r by the forward gain vector m t The gain factor specified in is used to obtain the residual tensor r′ after gain processing, where the residual tensor r is the difference between the potential tensor y and the predicted value μ, where the predicted value μ is the predicted value based on the elements in the potential tensor y. The gain unit 102 is used to implement the variable rate capability of the verification model of JPEG AI. The gain unit 102 uses a parameter β to control the compression ratio. The parameter β specifies the weight between the rate and the distortion in the loss function during training. During model training, for multiple β∈{β t}Get the paired forward gain vector and reverse gain vector, which can be expressed as Among them, m t is the forward gain vector, is the reverse gain vector. During encoding and decoding, the forward gain vector and the reverse gain vector are known to the encoder and decoder (they are part of the model), and the gain vector m t and The dimension of is equal to the number of channels of the residual tensor r. The gain unit 102 works only in the encoder.

[0069] The first quantization unit 103 is used to quantize the residual tensor after gain processing to obtain a quantized residual tensor The quantization process refers to the process of approximating the continuous value (or a large number of possible discrete values) of the signal to a finite number of discrete values ​​(or a small number of discrete values). The first quantization unit 103 only works in the encoder.

[0070] The first inverse gain unit 104 is used to quantize the residual tensor After inverse gain processing, it is the quantized residual tensor Each element of is multiplied by the inverse gain vector The gain factor specified in , to obtain the residual tensor after inverse gain processing The first inverse gain unit 104 can realize the variable rate capability of the verification model of JPEG AI. The first inverse gain unit 104 uses a parameter β to control the compression ratio, and the parameter β specifies the weight between rate and distortion in the loss function during training. The first inverse gain unit 104 only works in the encoder.

[0071] The context model network 105 is used to extract context features. The context model network 105 is present in both the encoder and the decoder. In the encoder, the context model network 105 is used to extract the context features. Perform context feature extraction to obtain the latent tensor The context features of . Among them, the latent tensor in the encoder The residual tensor after processing according to the inverse gain In the decoder, the context model network 105 is used to calculate the potential tensor Perform context feature extraction to obtain the latent tensor Context features. In the decoder, the bitstream ( Figure 1 The bitstream shown in 2) is the residual tensor after lossless decoding and inverse gain processing Adding it to the predicted value μ, we can get the potential tensor

[0072] In some learning-based image coding methods, an autoregressive convolutional context model is used to extract contextual features. Figure 6 A schematic diagram of a mask convolution kernel used in a context model of an autoregressive convolution in the related technology provided in the embodiment of the present application. Figure 6 As shown, the context model of the autoregressive convolution uses the encoded elements before the current encoding position 601 in the latent space to predict the context features corresponding to the current encoding position, and uses the context features to subsequently predict the probability distribution of the current encoding position. Among them, the encoded elements are Figure 6The elements at the encoding position shown in the dotted box in . The context model of the autoregressive convolution can significantly improve the rate-distortion performance of the super-prior model. However, due to its autoregressive structure, the context model of the autoregressive convolution can only be decoded in a fixed raster scanning order, and the latent space decoded later always depends on the latent space decoded earlier, resulting in a very slow decoding speed for this model.

[0073] To solve the above problems, in some embodiments, the context model network 105 in the verification model of JPEG AI can be further used in the context model network 105 to achieve parallel processing. The Wavefront Parallel Processing (WPP) mechanism of the latent space is used in the context model network 105, in which the decoded elements are used for latent tensors. The prediction process is an autoregressive process and is for the potential tensor Elements in different rows of can be processed independently in a wavefront-parallel manner, where the delay in processing consecutive rows depends on the shape of the masked convolution kernel.

[0074] Fig. 7A A schematic diagram of a masked convolution kernel used by a context model network 105 in a JPEG AI verification model in the related technology provided in an embodiment of the present application. Fig. 7A In the column coordinates, the current coding position is the origin of the coordinates. The current coding position is in row 0 and column 0. n=0 indicates the row where the current coding position is located, n=-1 indicates the row before the row where the current coding position is located, and n=-2 indicates the row before the row where the current coding position is located. Fig. 7A The "-2", "-1", "0", "1" and "2" in the table indicate different columns, where "0" indicates the column at the current encoding position, "-1" indicates 1 column before the column at the current encoding position, "-2" indicates 2 columns before the column at the current encoding position, "1" indicates 1 column after the column at the current encoding position, and "2" indicates 2 columns after the column at the current encoding position.

[0075] Figure 7B A schematic diagram of a wavefront parallel processing mechanism of a latent space in the related technology provided by the embodiment of the present application. In order to improve the utilization rate of computing resources, the wavefront parallel processing mechanism is introduced. Figure 7B In the figure, “columu” means column and “row” means row. The figure shows a potential space with 5 rows and 10 columns as an example. Figure 7B The elements of all channels of the tensor of the current pixel are predicted independently (simultaneously) and can be predicted based on the pixels that have been processed in the preset process. Fig. 7A The masked convolution kernel shown is designed in such a way that Figure 7BEach row of samples (i.e., the prediction of each row of pixels) can be processed in parallel, with the next row being processed delayed by one element compared to the previous row, i.e., delayed by the time of one wave movement (T).

[0076] The super-a priori coding network 107 is used to perform super-a priori coding on the potential tensor y to obtain the potential tensor z, and can extract additional auxiliary information during the super-a priori coding process so that an accurate probability model of entropy coding can be obtained based on the coding result. The super-a priori coding network 107 only works in the encoder, and the super-a priori coding network 107 may include two downsampling convolutions, three convolutions and a ReLU activation function. The above-mentioned ReLU activation function is a neural network activation function. When the input value is greater than 0, it remains unchanged, and when the input value is less than 0, it will output 0.

[0077] The second quantization unit 109 is used to quantize the potential tensor z to obtain a quantized coding potential tensor The second quantization unit 109 only works in the encoder.

[0078] The first lossless encoder 110 is used to quantize the potential tensor Entropy encoding is performed to obtain a bit stream 1. Entropy encoding is a lossless operation used to encode a bit stream. The entropy encoding and entropy decoding process takes the cumulative distribution function (CDF) output by the decomposition entropy model 112 as input. The cumulative distribution function (CDF) is essentially the integration of the probability density function (PDF) and the symbol to be encoded at the encoding end, or the integration of the probability density function (PDF) and the bit stream to be decoded at the decoding end. The first lossless encoder 110 only works at the encoder.

[0079] The first lossless decoder 111 is used to entropy decode the bit stream 1 obtained after entropy coding to obtain a potential tensor after entropy decoding. Among them, entropy decoding is a lossless operation for decoding a bit stream. The cumulative distribution function (CDF) can be received as input during the entropy decoding process. Entropy decoding is essentially the integral of the probability density function (PDF) and the bit stream to be decoded (at the decoder end). The first lossless decoder 111 only works at the decoder.

[0080] The decomposition entropy model 112 is used to input a cumulative distribution function (CDF) to the first lossless encoder 110 or the first lossless decoder 111, and the cumulative distribution function (CDF) is used for entropy encoding or entropy decoding. The decomposition entropy model 112 works in the decoder and also in the encoder. The decomposition entropy model 112 is integerized to solve the problem of device interoperability so that the decomposition entropy model 112 can be used on any device. Among them, the cumulative distribution function (CDF) is also called a probability table, which is pre-calculated and quantized, rather than trained based on trainable parameters (matrix, bias and factor). The table is stored in integer form, so there is no need to store the above Parameters of the distribution.

[0081] The super-a priori decoding network 108 is used to perform super-a priori decoding. The super-a priori decoding network 108 is used to decode the potential tensor after entropy decoding. Decode to get the potential tensor The super-prior decoding network 108 includes a series of two inverse convolutions with upsampling stride 2, three convolutions without tensor size change, and a ReLU activation function. The super-prior decoding network 108 can receive a tensor of size C×h×w as input and generate an output tensor with dual channels 4C×h×w. The super-prior decoding network 108 is symmetrical to the super-prior encoding network 107, and each upsampling convolution layer is followed by a clipping operation. The super-prior decoding network 108 works in the decoder and also in the encoder.

[0082] The prediction fusion network 106 can receive two inputs, the first input is the potential tensor obtained by the super prior decoding network 108 after super prior decoding The second input is the context feature output by the context model network 105. The prediction fusion network 106 generates a prediction value μ (size C×h×w) of size C×h×w based on the two inputs of size 4C×h×w. In the encoder, the prediction value μ can be added to the residual tensor after the inverse gain processing. To obtain the latent tensor As the input of the context model network 105. In the decoder, the bitstream ( Figure 1 The bitstream shown in 2) is the residual tensor after lossless decoding and inverse gain processing Adding it to the predicted value μ, we can get the potential tensor As input to the context model network 105, the prediction fusion network 106 works in the decoder and also in the encoder.

[0083] Super prior scale decoding network 113, used to decode the potential tensor after entropy The inverse operation of the super-prior coding network is performed, and the variance of the Gaussian distribution N(0,σ) with a probability distribution of 0 mean is output. The super-prior scale decoding network 113 includes two 5×5 inverse convolution sequences with an upsampling stride of 2, two 3×3 convolutions and a LeakyReLU activation function. The super-prior scale decoding network 113 is symmetrical to the super-prior coding network 107, and each upsampling convolution layer is followed by a cropping operation. The prediction fusion network 106 works in the decoder and also in the encoder.

[0084] The second lossless encoder 114 is used to quantize the residual tensor based on N(0,σ) Entropy encoding is performed to obtain a code stream 2. It should be noted that the second lossless encoder 114 has a similar structure and function to the first lossless encoder 110. The second lossless encoder 114 only works in the encoder.

[0085] The second lossless decoder 115 is used to perform entropy decoding on the bitstream 2 based on N(0,σ) to obtain a residual tensor after entropy decoding. It should be noted that the structure and function of the second lossless decoder 115 are similar to those of the first lossless decoder 111. The second lossless decoder 115 only works in the decoder.

[0086] The second inverse gain unit 116 is used to perform inverse gain processing on the residual tensor after entropy decoding, and multiply each element of the residual tensor after entropy decoding by the inverse gain vector The gain factor specified in , to obtain the residual tensor after inverse gain processing The second inverse gain unit 116 operates only in the decoder.

[0087] Synthetic Transformation Network 117, used to transform the latent tensor Representation is converted into the reconstructed image Among them, the potential tensor is the residual tensor after inverse gain processing The result is added to the prediction value μ. The synthesis transformation network 117 only works in the decoder.

[0088] Figure 8 FIG. 1 is a schematic diagram of a synthesis transformation network 117 in the related art provided in an embodiment of the present application. Figure 8 As shown, the synthesis transformation network 117 includes, from the input end to the output end, a residual block 801, a residual block 802, a 3×3 upsampling convolution 803, a residual activation unit 804, a 3×3 upsampling convolution 805, a residual activation unit 806, a 3×3 upsampling convolution 807, a residual non-local attention block 808, a residual activation unit 809, and a 3×3 upsampling convolution 810. Among them, two residual blocks are placed at the beginning of the synthesis transformation, which enhance the reconstructed potential tensor Reduced quantization error.

[0089] It should be noted that each residual activation unit in the synthetic coding network 117 is similar to Figure 3 The structure and function of the residual activation unit shown in FIG. 8 are the same as those of the residual non-local attention block 808 in the synthetic coding network 117. Figure 4 The structure and function of the residual non-local attention block 203 shown in FIG. 1 are the same. Each residual block in the synthetic coding network 117 is the same as Figure 4 The structure and function of the residual blocks shown are the same.

[0090] Fig. 9 A schematic diagram of another JPEG AI verification model in the related technology provided in an embodiment of the present application.

[0091] like Fig. 9 As shown in the above Figure 1 The content adaptive filter 118 is added to the verification model of JPEG AI shown in the figure. The content adaptive filter 118 is used to enhance the Y, U, and V components and synthesize the reconstructed image output by the transformation network 117. Enhance the Y, U, and V components to obtain the enhanced reconstructed image The adaptive filter 118 includes a content adaptive Inter Channel Correlation Information filter sub network (ICCI) sub network for reconstructing and enhancing the three components Y, U, and V of the image. The purpose of enhancing the Y, U, and V components is achieved through this ICCI sub network.

[0092] Fig.10 A schematic diagram of an ICCI sub-network provided in an embodiment of the present application. Fig.10 As shown in FIG. 1 , the ICCI subnetwork includes three different ICCI processing modules. The input of the ICCI subnetwork is the Y, U, and V components of the image. Fig.10 The Y component of the image is expressed as The U component of the image is expressed as The V component of the image is expressed as The U component of the image is bicubicly interpolated to obtain a U component of the same shape as Y. The V component of the image is bicubicly interpolated to obtain a V component of the same shape as Y Then the Y component U Component and V component All of them are forward wavelet transformed, and the three components obtained by the transformation are expressed as Fig.10 Components 1001, 1002 and 1003 in the image are spliced ​​in the channel dimension, and the spliced ​​component 1004 and their respective components are respectively sent to three ICCI processing modules as inputs. The three ICCI processing modules process the Y, U and V components respectively to obtain three processed components, so as to obtain processed components 1008, 1009 and 1010. The processed components 1008, 1009 and 1010 are all subjected to inverse wavelet transform to obtain the reconstructed Y, U and V components. The reconstructed Y component is The U component after reconstruction is The reconstructed V component is The number of residual blocks in the three ICCI processing modules is Nb, the number of residual blocks in the ICCI processing module 1005 for the Y component is Nb=8, the number of residual blocks in the ICCI processing module 1006 for the U component is Nb=13, and the number of residual blocks in the ICCI processing module 1007 for the V component is Nb=13. The bicubic interpolation is a method for enlarging the image size and increasing the pixel density, and the bicubic difference can be used to double the number of image pixels, wherein these expanded pixels are predicted based on the existing pixels.

[0093] Above Fig.10 In the ICCI subnetwork shown, in the process of image feature enhancement, in order to consider the characteristics of different Y, U, and V components, three different ICCI processing modules are set to enhance different components to achieve the purpose of adaptive processing.

[0094] Regarding the above Figure 1 or Fig. 9 In the verification model of JPEG AI shown in the figure, the above-mentioned analysis transformation network 101 and the synthesis transformation network 117 both use residual non-local attention blocks to extract deep features. The residual non-local attention block can reduce the pixel redundancy of the feature map by capturing the correlation between pixels. The residual non-local attention block has two branches: a trunk branch 41 and a mask branch 42. The trunk branch 41 contains 3 residual blocks for extracting deep features; the mask branch contains upsampling convolution, downsampling convolution and a series of residual blocks for scaling deep features. However, in the current verification model of JPEG AI, the above-mentioned analysis transformation network 101 and the synthesis transformation network 117 both use residual non-local attention blocks to extract deep features, and there is at least one of the following problems:

[0095] (1) The residual non-local attention block ignores the impact of different parameters β on the feature map, and treats the feature maps equally for different parameters β;

[0096] (2) The mask branch cannot be connected across windows and can only analyze the correlation of pixels within the same window. Therefore, the ability of the mask branch to capture the long-distance correlation of pixels is limited.

[0097] (3) The influence of different parameters β on different image contents is not considered. In some embodiments, in the verification model of JPEG AI, an interpolation gain unit is used to achieve variable bit rate. The parameters of the interpolation gain unit are controlled by the parameter β to scale the channel dimension of the potential residual tensor. It can be seen that although the influence of different parameters β is considered in the scaling of the interpolation gain unit, it only involves the influence of different parameters β on the feature map when the channel dimension is scaled.

[0098] In order to solve the above problems (1), (2) and (3), the embodiment of the present application sets up a content gain attention module, which can adaptively weight the content of the image according to the parameter β and can effectively capture the long-distance correlation of pixels.

[0099] Fig.11 A schematic diagram of a content gain attention module provided in an embodiment of the present application. Fig.11 The content gain attention module includes: a trunk branch and a mask branch. The trunk branch includes: a residual block 1101, a residual block 1102, and a residual block 1103, and the mask branch includes: a moving window attention module 1104, a moving window attention module 1105, a fully connected layer 1107, and a Sigmoid activation function 1106. The content gain attention module is used to extract deep image features. The Sigmoid activation function is an activation function with an S-shaped curve, which can map the input value to a range between [0,1].

[0100] Fig.12 A method based on the embodiment of the present application is provided Fig.11 The flowchart of the image feature processing method performed by the content gain attention module shown in FIG. Fig.12 As shown, the process of the image feature processing method includes but is not limited to the following steps:

[0101] 1201. Input the parameter β into the fully connected layer, and obtain the content gain weight output by the fully connected layer.

[0102] Here, the parameter β specifies the weight between rate and distortion in the loss function during training.

[0103] The above process of obtaining content gain weight can be expressed as:

[0104] W=f c (β) (1)

[0105] The value range of parameter β is B, B = [0.0018, 0.0483]. Parameter β∈B, W represents the content gain weight generated after the fully connected layer, f c (β) represents the process of generating content gain weights through full connection. The parameter β is mapped into a vector with the same dimension as the content image feature X through the fully connected layer. Different bit rates can be compressed according to different parameters β.

[0106] like Fig.11 As shown in , after the parameter β is input into the fully connected layer 1107, the fully connected layer 1107 can output the content gain weight W.

[0107] 1202. Perform feature extraction on input image features through at least one residual block to obtain extracted image features.

[0108] It should be noted that in the embodiment of the present application, feature extraction of the input image features can be performed through one or more residual blocks.

[0109] Fig.11 In the figure, the feature extraction of the input image feature y is performed through three residual blocks to obtain the extracted image feature Y. The three residual blocks are Fig.12 The residual blocks 1101, 1102, and 1103 in FIG. 1 have the same structure. The input image features are extracted by the residual blocks 1101, 1102, and 1103 to generate extracted image features Y.

[0110] The process of extracting image feature Y can be expressed as: Y = f RB3 (f RB2 (f RB1 (y))) (2)

[0111] Among them, f RB1 、f RB2 、f RB3 These are the calculation processes of residual block 1101, residual block 1102, and residual block 1103 respectively.

[0112] The residual block in the embodiment of the present application may include a neural network convolution layer and a residual connection. Figure 5 The structure of the residual block shown.

[0113] 1203. Extract content image features corresponding to the input image features.

[0114] In an embodiment of the present application, content image features may be extracted from input image features through at least one moving window attention module to extract content image features corresponding to the input image features.

[0115] Fig.11 The content gain attention module shown in is described by taking two moving window attention modules as an example. The input image feature y is extracted from the content image feature by the moving window attention module 1104 and the moving window attention module 1105 to obtain the content image feature X.

[0116] The above process of obtaining the content image feature X can be expressed as: X = f s2 (f s1 (y)) (3)

[0117] Among them, f s1 and f s2 They are Fig.11 The calculation process of the moving window attention module 1104 and the moving window attention module 1105 shown in FIG.

[0118] Fig.13 Schematic diagram of a moving window attention module provided in an embodiment of the present application. Fig.13 As shown, the moving window attention module includes: a multi-layer perceptron (MLP) fully connected layer 1301, a SoftMax activation function 1302, a 1×1 convolution 1303, a 1×1 convolution 1304, a 1×1 convolution 1305, and a 1×1 convolution 1306. The SoftMax activation function is an activation function with a normalized property, which can map the input value to a range between [0, 1], and the sum of all output values ​​is 1. The input image feature y is convolved by 1×1 convolution 1303 and 1×1 convolution 1304 respectively, and the image feature Q obtained by the convolution processing is element-wise multiplied with the image feature K, and processed by the SoftMax activation function 1302. The processing result of the SoftMax activation function 1302 is element-wise multiplied with the image feature V obtained by the input image feature y after the 1×1 convolution 1305. The result of the dot multiplication operation is processed by 1×1 convolution 1306 and then dot-wise multiplied with the input image feature y, and input to the MLP fully connected layer to obtain the output f s (y).f s (y) is the result of the input image feature y processed by the moving window attention module, which can be expressed as:

[0119] f s(y)=y+MLP(y+Attention(Q,K,V)) (4)

[0120]

[0121] Among them, f s (y) is a general representation of the calculation process of the moving window attention module. Q, K, V are the representations of the input image feature y after 1×1 convolution, SoftMax() is the SoftMax activation function, d is the number of channels of Q, K, V, B is the learnable position code, and Attention(Q,K,V) means calculating the correlation of each position element of Q, K and V to generate the attention matrix.

[0122] 1204. Perform channel dimension operations based on content image features and content gain weights to obtain a joint spatial channel attention mask.

[0123] like Fig.11 As shown in , the input image feature y is converted into a joint spatial channel attention mask M, which may include but is not limited to first passing the input image feature y through the moving window attention module 1104 and the moving window attention module 1105 to generate the content image feature X, and then performing a point multiplication operation of the content image feature X and the content gain weight W in the channel dimension, and finally passing through the Sigmoid activation function to generate the joint spatial channel attention mask M. The process of generating the joint spatial channel attention mask M can be expressed as:

[0124] M=sigmoid(f s2 (f s1 (y))⊙W) (6)

[0125] Among them, f s1 and f s2 is the calculation process of the moving window attention module 1104 and the moving window attention module 1105. ⊙ is the dot multiplication operation of the channel dimension. Sigmoid() is the Sigmoid activation function.

[0126] 1205. Perform element-wise multiplication operation on the extracted image features and the joint spatial channel attention mask to obtain the target image features.

[0127] 1206. Generate output image features according to the target image features and the input image features.

[0128] The process of generating the output image feature can be specifically as follows: extract the image feature y and the joint spatial channel attention mask M, perform element-wise multiplication, add the result of the operation to the input image feature y, and generate the output image feature y o .

[0129] The above process can be expressed as:

[0130]

[0131] in, It is the element-wise multiplication operation.

[0132] Combining the above formulas (1) to (7), it can be seen that the overall calculation process of the content gain attention module can be expressed as:

[0133]

[0134] In the embodiment of the present application, in the content gain attention module, since the content gain weight is obtained by inputting the parameter β into the fully connected layer when performing deep feature extraction, compared with the original residual non-local attention block, the influence of different parameters β on the image feature map is considered, thereby solving the above-mentioned problem (1).

[0135] In some embodiments, since the embodiments of the present application also perform calculations based on the content gain weights and the extracted image content features to obtain a joint spatial channel attention mask, the influence of different parameters β on different image contents is considered in the content gain attention module. When the content gain attention module is applied to the verification model of JPEG AI, the above-mentioned problem (3) can be solved.

[0136] In some embodiments of the present application, the moving window attention module can divide the features into multiple windows in a non-overlapping manner, and calculate the attention map of each window respectively to obtain the attention of the local window, and by moving the window to different positions in different moving window attention blocks, the elements originally in different windows can be divided into the same window, so that cross-window connection can be achieved. The moving window attention module calculates the attention of the local window and allows cross-window connection at the same time, thereby improving the ability of the mask branch to capture the long-distance correlation of pixels, that is, solving the above-mentioned problem (2).

[0137] Exemplarily, the above-mentioned moving window attention module can divide the features into 8×8 windows in a non-overlapping manner, and calculate the attention map of each window respectively. In the case where at least one moving window attention module is 2 moving window attention modules, in the second moving window attention module, compared with the first moving window attention module, the divided window can be moved 4 pixels to the right and downward respectively, so that the elements in different windows can be divided into the same window to achieve cross-window connection.

[0138] The above-mentioned content gain attention module can act on the analysis transformation network 101 and the synthesis transformation network 117 in the verification model of JPEG AI.

[0139] In some embodiments of the present application, the content gain attention module is embedded into the above-mentioned analysis transformation network 101 and the synthesis transformation network 117 in the verification model of JPEG AI.

[0140] For example, Fig.14A A schematic diagram of an analysis transformation network 101 for embedding a content gain attention module into a verification model of JPEGAI provided in an embodiment of the present application. Fig.14A exist Figure 2 Based on this, a content gain attention module 1401 is embedded between the residual activation unit 205 and the 3×3 downsampling convolution 206.

[0141] Fig.14A In the analysis transformation network shown, the original image (i.e., the first image feature to be transformed in the present application) input into the verification model of the JPEG AI is processed by 3×3 downsampling convolution 201, residual activation unit 202, residual non-local attention block 203, and 3×3 downsampling convolution 204 and residual activation unit 205 to obtain the first input image feature, and the first input image feature is used as the input image feature of the content gain attention block 1401. In the content gain attention block 1401, Fig.12 After the input image features are processed by the process shown, the first output image features output by the content gain attention module 1401 can be obtained, and the first output image features are further processed in sequence through 3×3 downsampling convolution 206, residual activation unit 207, 3×3 downsampling convolution 208 and 1×1 convolution 209 to obtain the potential tensor corresponding to the original image (i.e., the analysis transformation image features in this application).

[0142] For example, Fig. 14B A schematic diagram of another analysis transformation network 101 for embedding a content gain attention module into a verification model of JPEG AI provided in an embodiment of the present application. Fig. 14B exist Figure 2 Based on this, a content gain attention module 1401 is embedded between the residual activation unit 207 and the 3×3 downsampling convolution 208.

[0143] Fig.14A and Fig. 14B The difference between the analysis transformation network 101 shown is that the content gain attention module 1401 is embedded in a different position. Based on this difference, Fig.14AThe analysis transformation network shown in FIG. 1 performs feature processing on the feature before the 3×3 downsampling convolution 206 and the residual activation unit 207 through the content gain attention module 1401. Fig. 14B The analysis transformation network diagram will perform feature processing through the content gain attention module 1401 before feature processing after the 3×3 downsampling convolution 206 and the residual activation unit 207. Since downsampling will reduce the receptive field of the analysis transformation network 101 and reduce the computing resource consumption of the analysis transformation network 101, it can be selected when the quality of the analysis transformation image features is lower and the computing resources are less. Fig. 14B The analysis transformation network shown in Figure 1 can be used when the quality of the analysis transformation image features is higher and the computing resources are sufficient. Fig.14A The analytical transformation network shown.

[0144] Above Fig.14A and Fig. 14B The content gain attention module in Fig.13 as shown in .

[0145] Fig. 14B In the analysis transformation network shown, the original image input into the verification model of the JPEG AI is processed by 3×3 downsampling convolution 201, residual activation unit 202, residual non-local attention block 203, 3×3 downsampling convolution 204, residual activation unit 205, 3×3 downsampling convolution 206, and residual activation unit 207 to obtain the first input image feature, and the first input image feature is used as the input image feature of the content gain attention block 1401. In the content gain attention block 1401, as shown in FIG. Fig.12 After the first input image feature is processed by the process shown, the first output image feature output by the content gain attention module 1401 can be obtained, and the first output image feature is further processed by 3×3 downsampling convolution 208 and 1×1 convolution 209 in sequence to obtain the potential tensor corresponding to the original image (i.e., the analysis transformation image feature in this application).

[0146] Fig. 14C A flowchart of an image feature processing method performed by an analysis transformation network based on an embedded content gain attention module provided in an embodiment of the present application. The image feature processing method is applied to an encoding device, and the method may include but is not limited to the following steps:

[0147] 141. Obtain a first image feature to be transformed.

[0148] The first image feature to be transformed may be a feature of an input analysis transformation network. Figure 1 or Fig. 9As shown, the first image feature to be transformed may be the original image x input to the analysis transformation network 101 .

[0149] 142. Perform a first processing, a processing based on a residual non-local attention block, and at least one of the first processing on a first image feature to be transformed to obtain a first input image feature.

[0150] Among them, the first processing includes downsampling processing and residual activation processing, for example, downsampling processing is performed by 3×3 downsampling convolution, and then residual activation processing is performed on the output of the 3×3 downsampling convolution through a residual activation unit.

[0151] The at least one first treatment is one first treatment, or the at least one first treatment is two first treatments.

[0152] For example, Fig.14A As shown, the above-mentioned at least one first processing is a first processing. After the first image feature to be transformed is input into the analysis transformation network, it will first be processed by 3×3 downsampling convolution 201 and residual activation unit 202, and then be processed by residual non-local attention block 203, 3×3 downsampling convolution 204 and residual activation unit 205 again, so that the first input image feature can be obtained, and then continue to be input into content gain attention block 1401 for subsequent processing.

[0153] For example, Fig. 14B As shown, the at least one first processing mentioned above is two first processings. After the first image feature to be transformed is input into the analysis transformation network, it will first be processed once by 3×3 downsampling convolution 201 and residual activation unit 202, and then by residual non-local attention block 203, and then by 3×3 downsampling convolution 204 and residual activation unit 205 for a first processing, and then by 3×3 downsampling convolution 206 and residual activation unit 207 for a first processing (i.e., two first processings), so that the first input image feature can be obtained, and then continue to be input into the content gain attention block 1401 for subsequent processing.

[0154] 143. Process the first input image feature based on the content gain attention block to obtain a first output image feature.

[0155] The content gain attention block is used to weight the content of the image. The process of the content gain attention block processing the first input image feature to obtain the first output image feature is as described above. Fig.12 shown.

[0156] 144. Perform downsampling and convolution processing on the first output image feature to obtain analysis transformation image features.

[0157] The above analysis transformation image features can be output features of the analysis transformation network. For example, Figure 1 or Fig. 9 As shown, the analysis transformation image feature may be the potential tensor y output by the analysis transformation network 101 .

[0158] For example, Fig. 14B As shown, after the content gain attention block 1401 processes the first input image feature, it outputs the first output image feature, and is down-sampled by a 3×3 down-sampling convolution 208 and convolved by a 1×1 convolution 209 to obtain the analyzed transformed image feature.

[0159] In the case where the at least one first processing is a first processing, performing downsampling processing and convolution processing on the first output image feature to obtain the analysis transformation image feature can be: first performing the first processing on the first output image feature, and then performing downsampling processing and convolution processing to obtain the analysis transformation image feature.

[0160] For example, Fig.14A As shown, the at least one first processing mentioned above is a first processing. After the first input image feature is input to the content gain attention block 1401 for processing, the first output image feature will be output. Then, the first output image feature will first undergo a first processing through a 3×3 downsampling convolution 206 and a residual activation unit 207, and then undergo a downsampling processing through a 3×3 downsampling convolution 208, and undergo a convolution processing through a 1×1 convolution 209 to obtain an analysis transformation image feature.

[0161] In the above embodiment, by embedding a content gain attention module in the analysis transformation network, the analysis transformation network improves the ability to capture long-distance correlation of pixels in the process of converting the input image into a latent tensor, and considers the influence of different parameters β on the image feature map and the influence of different parameters on different image contents, so as to obtain a more accurate latent tensor and improve the performance of the JPEG AI verification model.

[0162] Fig.15A A schematic diagram of a synthetic transformation network 117 that embeds a content gain attention module into a verification model of JPEG AI provided in an embodiment of the present application. Fig.15A exist Figure 8 Based on this, a content gain attention block 1501 is embedded between the 3×3 upsampling convolution 805 and the residual activation unit 806.

[0163] Fig.15AIn the synthetic transformation network shown in FIG. 1 , the potential tensor (i.e., the second image feature to be transformed) input into the synthetic transformation network in the verification model of the JPEG AI is processed by the residual block 801, the residual block 802, the 3×3 upsampling convolution 803, the residual activation unit 804, and the 3×3 upsampling convolution 805 to obtain the third image feature (i.e., the second input image feature). The obtained third image feature is used as the input image feature of the content gain attention block 1501. The content gain attention block 1501 is processed as follows: Fig.12 After the input image features are processed by the process shown, the output image features (i.e., the second output image features) output by the content gain attention module 1501 can be obtained, and the output image features are further processed in sequence by the residual activation unit 806, 3×3 upsampling convolution 807, residual non-local attention block 808, residual activation unit 809 and 3×3 upsampling convolution 810 to obtain a reconstructed image (i.e., a synthetic transformed image feature).

[0164] Fig. 15B A schematic diagram of another synthetic transformation network 117 provided in an embodiment of the present application that embeds a content gain attention module into the verification model of JPEG AI. Fig. 15B exist Figure 8 Based on this, a content gain attention block 1501 is embedded between the 3×3 upsampling convolution 803 and the residual activation unit 804.

[0165] Above Fig.15A and Fig. 15B The content gain attention module in Fig.13 as shown in .

[0166] Fig. 15B In the synthetic transformation network shown in FIG. 1 , the potential tensor input into the synthetic transformation network in the verification model of the JPEG AI (i.e., the second image feature to be transformed in the present application) is processed by the residual block 801, the residual block 802, and the 3×3 upsampling convolution 803 to obtain the second input image feature, and the obtained second input image feature is used as the input image feature of the content gain attention block 1501. In the content gain attention block 1501, as shown in FIG. Fig.12 After the second input image feature is processed by the process shown, the second output image feature output by the content gain attention module 1501 can be obtained, and the second output image feature is further processed in sequence by the residual activation unit 804, 3×3 upsampling convolution 805, residual activation unit 806, 3×3 upsampling convolution 807, residual non-local attention block 808, residual activation unit 809 and 3×3 upsampling convolution 810 to obtain a reconstructed image (i.e., the synthetic transformation image feature in this application).

[0167] Fig. 15C A flowchart of an image feature processing method performed by a synthetic transformation network based on an embedded content gain attention module provided in an embodiment of the present application. The image feature processing method is applied to a decoding device, and the method may include but is not limited to the following steps:

[0168] 151. Obtain features of a second image to be transformed.

[0169] The second image feature to be transformed is a feature input to the synthetic transformation network. Figure 1 or Fig. 9 As shown, the second image feature to be transformed can be a potential tensor input to the synthetic transformation network 117

[0170] 152. Perform at least one residual block processing and upsampling processing on the second to-be-transformed image feature to obtain a second input image feature.

[0171] Exemplarily, in some embodiments of the present application, the at least one residual block for processing the second image feature to be transformed may be two residual blocks; and the upsampling process may be a process performed by a 3×3 upsampling convolution.

[0172] For example, Fig. 15B As shown, after the second image feature to be transformed is input into the synthetic transformation network, it is first processed by residual block 801 and residual block 802, and then processed by 3×3 upsampling convolution 803 to obtain the second input image feature, which is then input into the content gain attention block 1501 to continue the subsequent processing process.

[0173] In some embodiments of the present application, in the process of performing at least one residual block processing and upsampling processing on the second image feature to be transformed to obtain the second input image feature, the second image feature to be transformed can be firstly subjected to at least one residual block processing and upsampling processing, and then the second processing can be performed to obtain the second input image feature.

[0174] For example, Fig.15A As shown, after the second image feature to be transformed is input into the synthetic transformation network, it is first processed by residual block 801 and residual block 802, and then processed by 3×3 upsampling convolution 803, and then processed by residual activation unit 804 and upsampling convolution 805 for the second processing to obtain the second input image feature, and the second input image feature is input into the content gain attention block 1501 to continue the subsequent processing process.

[0175] 153. Process the second input image feature based on the content gain attention block to obtain a second output image feature.

[0176] The content gain attention block is used to perform attention weighting on the content of the image. The process of the content gain attention block processing the second input image feature to obtain the second output image feature is as described above. Fig.12 shown.

[0177] 154. Perform at least one second processing on the second output image feature, perform processing based on the residual non-local attention block, and perform a second processing to obtain a synthetic transformed image feature.

[0178] The above-mentioned second process includes residual activation processing and upsampling processing.

[0179] The synthetic transformation image features may be output features of the synthetic transformation network. For example, Figure 1 or Fig. 9 As shown, the synthetic transformation image feature can be a reconstructed image output by the synthetic transformation network 117

[0180] The at least one second treatment is one second treatment, or the at least one second treatment is two second treatments.

[0181] For example, Fig.15A As shown, the at least one second processing mentioned above is a second processing. After the content gain attention block 1501 outputs the second output image feature, it will be processed once by the residual activation unit 806 and the 3×3 upsampling convolution 807, and will be processed based on the residual non-local attention block 808, and will be processed once by the residual activation unit 809 and the upsampling convolution 810.

[0182] For example, Fig. 15B As shown, the at least one second processing mentioned above is two second processings. After the content gain attention block 1501 outputs the second output image feature, it will be subjected to a second processing through the residual activation unit 804 and the 3×3 upsampling convolution 805, and will be subjected to a second processing through the residual activation unit 806 and the 3×3 upsampling convolution 807 (i.e., two second processings), and then processed based on the residual non-local attention block 808, and will be subjected to a second processing through the residual activation unit 809 and the upsampling convolution 810.

[0183] In the above embodiment, by embedding a content gain attention module in the synthetic transformation network, the synthetic transformation network improves the ability to capture long-distance correlation of pixels in the process of reconstructing the potential tensor into an output image, and considers the influence of different parameters β on the image feature map and the influence of different parameters on different image contents, so as to obtain a more accurate output image and improve the performance of the JPEG AI verification model.

[0184] In the above embodiments of the present application, the verification model of JPEG AI may include: Fig.14A and Fig. 14B Any analytical transformation network in , and including Fig.15A and Fig. 15B By embedding the content gain attention module into the analysis transformation network and the synthesis transformation network in the verification model of JPEG AI, the performance of the verification model of JPEG AI in the image encoding and image decoding processes can be improved.

[0185] In some embodiments of the present application, the residual non-local attention block (i.e., Figure 2 The residual non-local attention block 203 shown in FIG.

[0186] Fig.16A A schematic diagram of a content gain attention module provided in an embodiment of the present application replacing the residual non-local attention block in the above-mentioned analysis transformation network 101. Fig.16A exist Figure 2 Based on the content gain attention block 1601, Figure 2 The original residual non-local attention block 203.

[0187] Fig.16A The content gain attention block 1601 in Fig.13 as shown in .

[0188] Fig.16A In the analysis transformation network shown in FIG. 1 , the original image (i.e., the first image feature to be transformed in this embodiment) input into the verification model of the JPEG AI is processed by a 3×3 downsampling convolution 201 and a residual activation unit 202 to obtain a first input image feature. The obtained first input image feature is used as the input image feature of the content gain attention block 1601. The content gain attention block 1601 is processed as follows: Fig.12After the input image feature is processed by the process shown, the first output image feature output by the content gain attention module 1601 can be obtained, and the first output image feature is further processed twice by 3×3 downsampling convolution 204, residual activation unit 205, 3×3 downsampling convolution 206, and residual activation unit 207, and then processed in sequence by 3×3 downsampling convolution 208 and 1×1 convolution 209 to obtain the potential tensor corresponding to the original image (that is, the analysis transformation image feature in this application).

[0189] Fig. 16B A flowchart of an image feature processing method performed by an analysis transformation network based on a content gain attention module replacing a residual non-local attention block provided in an embodiment of the present application. The image feature processing method is applied to an encoding device, and the method may include but is not limited to the following steps:

[0190] 161. Obtain a first image feature to be transformed.

[0191] The first image feature to be transformed may be a feature of an input analysis transformation network. Figure 1 or Fig. 9 As shown, the first image feature to be transformed may be the original image x input to the analysis transformation network 101 .

[0192] 162. Perform at least one first processing on the first image feature to be transformed to obtain a first input image feature.

[0193] Among them, the first processing includes downsampling processing and residual activation processing.

[0194] In some embodiments of the present application, performing at least one first processing on the first image feature to be transformed may include: performing one first processing on the first image feature to be transformed, or performing two CIA Yang and residual activation processes on the first image feature to be transformed.

[0195] Exemplarily, as shown in FIG. 16 , a first process is performed on the first image feature to be transformed.

[0196] 163. Process the first input image feature based on the content gain attention block to obtain a first output image feature.

[0197] The content gain attention block is used to weight the content of the image. The process of the content gain attention block processing the first input image feature to obtain the first output image feature is as described above. Fig.12 shown.

[0198] 164. Perform at least one first processing, downsampling processing and convolution processing on the first output image feature to obtain an analysis transformation image feature.

[0199] The above analysis transformation image features may be output features of the analysis transformation network. For example, Figure 1 or Fig. 9 As shown, the analysis transformation image feature may be the potential tensor y output by the analysis transformation network 101 .

[0200] In some embodiments of the present application, performing at least one first processing on the first output image feature includes: performing one first processing on the first output image feature, or performing two first processing on the first output image feature.

[0201] In some embodiments of the present application, if the first image feature to be transformed is first processed once in the above step 162, then the first output image feature may be first processed twice in step 164; if the first image feature to be transformed is first processed twice in the above step 162, then the first output image feature may be first processed once in step 164. In the image feature processing method executed by the entire analysis transformation network, the first processing may be performed three times, so that more accurate image features can be obtained.

[0202] For example, Fig.16A What is shown in FIG. 1 is that the first processing is performed once on the first image feature to be transformed, and the first processing is performed twice on the first output image feature.

[0203] In the above embodiment, the analysis transformation network replaces the residual non-local attention block with the content gain attention module, so that the analysis transformation network improves the ability to capture long-distance correlation of pixels in the process of converting the input image into a latent tensor, and considers the influence of different parameters β on the image feature map and the influence of different parameters on different image contents, so as to obtain a more accurate latent tensor and improve the performance of the JPEG AI verification model.

[0204] In some embodiments of the present application, the residual non-local attention block (i.e., Figure 8 The residual non-local attention block 808 shown in FIG.

[0205] Fig.17A A schematic diagram of a content gain attention module replacing the residual non-local attention block in the synthesis transformation network 117 provided in an embodiment of the present application. Fig.17A exist Figure 8 Based on the content gain attention block 1701, Figure 8The original residual non-local attention block 808.

[0206] Fig.17A The content gain attention block 1701 in Fig.13 as shown in .

[0207] Fig.17A In the synthetic transformation network shown in FIG. 1 , the potential tensor input into the synthetic transformation network in the verification model of the JPEG AI (i.e., the second image feature to be transformed in the present application) is processed by the residual block 801, the residual block 802, the 3×3 upsampling convolution 803, the residual activation unit 804, the 3×3 upsampling convolution 805, the residual activation unit 806, and the 3×3 upsampling convolution 807 to obtain the second input image feature, and the second input image feature is input into the content gain attention block 1701, and the content gain attention block 1701 is processed as follows: Fig.12 After the second input image feature is processed by the process shown, a second output image feature can be obtained, and the second output image feature is further processed by the residual activation unit 809 and the 3×3 upsampling convolution 810 in sequence to obtain a reconstructed image (i.e., the synthetic transformation image feature in this application).

[0208] Fig. 17B A flowchart of an image feature processing method performed by a synthetic transformation network based on a content gain attention module replacing a residual non-local attention block provided in an embodiment of the present application. The image feature processing method is applied to a decoding device, and the method may include but is not limited to the following steps:

[0209] 171. Obtain features of a second image to be transformed.

[0210] The second image feature to be transformed is a feature input to the synthetic transformation network. Figure 1 or Fig. 9 As shown, the second image feature to be transformed can be a potential tensor input to the synthetic transformation network 117

[0211] 172. Perform at least one residual block processing, upsampling processing, and at least one second processing on the second image feature to be transformed to obtain a second input image feature.

[0212] Exemplarily, in some embodiments of the present application, the at least one residual block for processing the second image feature to be transformed may be two residual blocks; and the upsampling process may be a process performed by a 3×3 upsampling convolution.

[0213] In some embodiments of the present application, the at least one second processing mentioned above may include: one second processing, or two second processings.

[0214] For example, Fig.17A What is shown in FIG. 1 is a case where the first processing is performed twice on the second image feature to be transformed.

[0215] 173. Process the second input image feature based on the content gain attention block to obtain a second output image feature.

[0216] The content gain attention block is used to perform attention weighting on the content of the image. The process of the content gain attention block processing the second input image feature to obtain the second output image feature is as described above. Fig.12 shown.

[0217] 174. Perform at least one second processing on the second output image feature to obtain a synthetic transformed image feature.

[0218] The synthetic transformation image features may be output features of the synthetic transformation network. For example, Figure 1 or Fig. 9 As shown, the synthetic transformation image feature can be a reconstructed image output by the synthetic transformation network 117

[0219] In some embodiments of the present application, performing at least one second process on the second output image feature includes: performing one second process on the second output image feature, or performing two first processes on the first output image feature.

[0220] In some embodiments of the present application, if the first processing is performed once on the second image feature to be transformed in the above step 172, then the first processing may be performed twice on the second output image feature in step 174; if the first processing is performed twice on the second image feature to be transformed in the above step 172, then the first processing may be performed once on the second output image feature in step 174. In the image feature processing method performed by the entire synthetic transformation network, the first processing may be performed three times, so that more accurate image features can be obtained.

[0221] For example, Fig.17A What is shown in FIG. 1 is that the first processing is performed twice on the second image feature to be transformed, and the first processing is performed once on the first output image feature.

[0222] In the above embodiment, the synthetic transformation network replaces the residual non-local attention block with the content gain attention module, so that the synthetic transformation network improves the ability to capture long-distance correlation of pixels in the process of reconstructing the potential tensor into an output image, and considers the influence of different parameters β on the image feature map and the influence of different parameters on different image contents, so as to obtain a more accurate output image and improve the performance of the JPEG AI verification model.

[0223] In the above-mentioned embodiments of the present application, the verification model of JPEG AI may include an analysis transformation network as shown in Figure 16 and a synthesis transformation network as shown in Figure 17. By replacing the residual non-local attention block with a content gain attention module and embedding it into the verification model of JPEG AI, the performance of the verification model of JPEG AI during image encoding and image decoding can be improved.

[0224] In some embodiments of the present application, not only a content gain attention module is used to replace the residual non-local attention block (i.e. Figure 2 The residual non-local attention block 203 shown in FIG. 1 is further configured to embed a content gain attention module in the above-mentioned analysis transformation network 101.

[0225] Fig.18A A schematic diagram of a content gain attention module provided in an embodiment of the present application that combines replacement and embedding in an analysis transformation network 101. Fig.18A Zhongzai Figure 2 On the basis of Figure 2 The residual non-local attention block 203 shown in FIG is replaced by the content gain attention module 1801, and the content gain attention module 1802 is embedded between the residual activation unit 205 and the 3×3 downsampling convolution 206. The content gain attention module 1801 and the content gain attention module 1802 are both as shown in FIG. Fig.13 as shown in .

[0226] Fig.18A In the analysis transformation network shown in FIG. 1 , the original image (i.e., the first image feature to be transformed in the present application) input into the verification model of the JPEG AI is processed by a 3×3 downsampling convolution 201 and a residual activation unit 202 to obtain a first initial input image feature, and the first initial input image feature is used as the input image feature of the content gain attention block 1801. Fig.12 After the first initial input image feature is processed by the process shown in the figure, the first initial output image feature can be obtained. The first initial output image feature is processed by 3×3 downsampling convolution 204 and residual activation unit 205 to obtain the first re-input image feature, and the first re-input image feature is input into the image feature of the content gain attention block 1802. The content gain attention block 1802 is as follows Fig.12After the first re-input image feature is processed by the process shown, the first re-output image feature can be obtained, and then the first re-output image feature is further processed through 3×3 downsampling convolution 206, residual activation unit 207, 3×3 downsampling convolution 208 and 1×1 convolution 209 in sequence to obtain the potential tensor corresponding to the original image (that is, the analysis transformation image feature in this application).

[0227] Fig.18B A flowchart of an image feature processing method performed by an analysis transformation network based on a combination of content gain attention module embedding and replacement provided in an embodiment of the present application. The image feature processing method is applied to an encoding device, and the method may include but is not limited to the following steps:

[0228] 181. Obtain a first image feature to be transformed.

[0229] The first image feature to be transformed may be a feature of an input analysis transformation network. Figure 1 or Fig. 9 As shown, the first image feature to be transformed may be the original image x input to the analysis transformation network 101 .

[0230] 182. Perform a first process on the first image feature to be transformed to obtain a first initial input image feature.

[0231] For example, Fig.18A As shown, the 3×3 downsampling convolution 201 and the residual activation unit 202 perform a first process on the input first image feature to be transformed to obtain a first initial input image feature.

[0232] 183. Process the first initial input image feature based on the first content gain attention block to obtain a first initial output image feature.

[0233] The first content gain attention block is used to perform attention weighting on the content of the image. The first content gain attention block processes the first initial input image feature to obtain the first initial output image feature as described above. Fig.12 shown.

[0234] 184. Perform a first processing on the first initial output image feature to obtain a first re-input image feature.

[0235] For example, Fig.18A As shown, the first content gain attention block is represented as content gain attention block 1801. The first initial input image feature is input into content gain attention block 1801. Content gain attention block 1801 is Fig.12After the input image feature is processed by the process shown, the first initial output image feature output by the content gain attention module 1801 can be obtained, and the first initial output image feature is processed by the 3×3 downsampling convolution 204 and the residual activation unit 205 to obtain the first re-input image feature.

[0236] 185. Process the first re-input image feature based on the second content gain attention block to obtain a first re-output image feature.

[0237] The second content gain attention block is used to perform attention weighting on the content of the image. The second content gain attention block processes the first re-input image feature to obtain the first re-output image feature as described above. Fig.12 shown.

[0238] 186. Perform a first processing, a downsampling processing, and a convolution processing on the first re-output image feature to obtain an analysis transformation image feature.

[0239] The above analysis transformation image features may be output features of the analysis transformation network. For example, Figure 1 or Fig. 9 As shown, the analysis transformation image feature may be the potential tensor y output by the analysis transformation network 101 .

[0240] For example, Fig.18A As shown, the second content gain attention block is represented as content gain attention block 1802, and the first re-input image feature is used as the input image feature of the content gain attention block 1802. Fig.12 After the input image feature is processed by the process shown, a first re-output image feature can be obtained, and the first re-output image feature is further processed in sequence through 3×3 downsampling convolution 206, residual activation unit 207, 3×3 downsampling convolution 208 and 1×1 convolution 209 to obtain an analysis transformation image feature.

[0241] In the above embodiment, the analysis transformation network not only replaces the residual non-local attention block with the content gain attention module, but also embeds the residual non-local attention block in the analysis transformation network. In this way, the analysis transformation network improves the ability to capture long-distance correlation of pixels in the process of converting the input image into a potential tensor, and considers the influence of different parameters β on the image feature map and the influence of different parameters on different image contents, so as to obtain a more accurate potential tensor and improve the performance of the JPEG AI verification model.

[0242] In some embodiments of the present application, not only a content gain attention module is used to replace the residual non-local attention block in the above-mentioned synthesis transformation network 117, but also a content gain attention module (i.e. Figure 8 The residual non-local attention block 808 shown in FIG. Fig.19A A schematic diagram of a content gain attention module provided in an embodiment of the present application that combines replacement and embedding in a synthetic transformation network 117. Fig.19A Zhongzai Figure 8 On the basis of Figure 8 The residual non-local attention block 808 shown in FIG is replaced by the content gain attention module 1901, and the content gain attention module 1902 is embedded between the 3×3 upsampling convolution 805 and the residual activation unit 806. The content gain attention module 1902 and the content gain attention module 1901 are both as shown in FIG. Fig.13 as shown in .

[0243] Fig.19A In the synthetic transformation network shown in FIG. 1 , the potential tensor input into the synthetic transformation network in the verification model of the JPEG AI (i.e., the second image feature to be transformed in the present application) is processed by the residual block 801, the residual block 802, the 3×3 upsampling convolution 803, the residual activation unit 804, and the 3×3 upsampling convolution 805 to obtain the second initial input image feature, and the second initial input image feature is input into the content gain attention block 1902. Fig.12 After the second initial input image feature is processed in the process shown, the second initial output image feature can be obtained. After the second initial output image feature is processed by the residual activation unit 806 and the 3×3 upsampling convolution 807, the second re-input image feature is obtained, and the second re-output image feature is obtained by inputting the content gain attention block 1901 for processing, and the second re-output image feature is processed by the residual non-local attention block 808, the residual activation unit 809 and the 3×3 upsampling convolution 810 in sequence to obtain a reconstructed image (i.e., a synthetic transformation image feature).

[0244] Fig.19B A flowchart of an image feature processing method performed by a synthetic transformation network based on a combination of content gain attention module embedding and replacement provided in an embodiment of the present application. The image feature processing method is applied to a decoding device, and the method may include but is not limited to the following steps:

[0245] 191. Obtain features of a second image to be transformed.

[0246] The second image feature to be transformed is a feature input to the synthetic transformation network. Figure 1 or Fig. 9 As shown, the second image feature to be transformed can be a potential tensor input to the synthetic transformation network 117

[0247] 192. Perform at least one residual block processing, upsampling processing, and second processing on the second image feature to be transformed to obtain a second initial input image feature.

[0248] Exemplarily, in some embodiments of the present application, the at least one residual block for processing the second image feature to be transformed may be two residual blocks; and the upsampling process may be a process performed by a 3×3 upsampling convolution.

[0249] For example, Fig.19A As shown, the second image feature to be transformed is input into the synthetic transformation network, and the second image feature to be transformed will undergo two residual processes through residual blocks 801 and 802, and then will undergo upsampling processing through 3×3 upsampling convolution 803, and then will undergo a second processing through residual activation unit 804 and 3×3 upsampling convolution 805 to obtain the second initial input image feature.

[0250] 193. Process the second initial input image features based on the third content gain attention block to obtain second initial output image features.

[0251] The third content gain attention block is used to perform attention weighting on the content of the image. The process of processing the second initial input image feature by the third content gain attention block to obtain the second initial output image feature is as described above. Fig.12 shown.

[0252] 194. Perform a second process on the second initial output image feature to obtain a second re-input image feature.

[0253] like Fig.19A As shown, the third content gain attention block is Fig.19A The content gain attention block 1902 is used to represent that the content gain attention block 1902 is Fig.12 After the second initial output image feature is processed by the process shown, the second initial output image feature can be obtained. After the output image feature output by the content gain attention module 1902 is processed by the residual activation unit 806 and the 3×3 upsampling convolution 807, the second input image feature is obtained.

[0254] 195. Process the second re-input image feature based on the fourth content gain attention block to obtain the second re-output image feature.

[0255] The fourth content gain attention block is used to perform attention weighting on the content of the image. The fourth content gain attention block processes the second re-input image feature to obtain the second re-output image feature as described above. Fig.12 shown.

[0256] like Fig.19A As shown, the fourth content gain attention block is Fig.19A The content gain attention block 1901 is used to represent that the content gain attention block 1901 is Fig.12 After the second re-input image feature is processed in the process shown, the second re-output image feature can be obtained.

[0257] 196. Perform a second process on the second re-output image feature to obtain a synthetic transformed image feature.

[0258] like Fig.19A As shown, the second re-output image features are subjected to a second processing by a residual activation unit 809 and an upsampling convolution 810 to synthesize the transformed image features.

[0259] The synthetic transformation image features may be output features of the synthetic transformation network. For example, Figure 1 or Fig. 9 As shown, the synthetic transformation image feature can be a reconstructed image output by the synthetic transformation network 117

[0260] In the above embodiment, the synthetic transformation network not only replaces the residual non-local attention block with the content gain attention module, but also embeds the residual non-local attention block in the synthetic transformation network. In this way, the synthetic transformation network improves the ability to capture long-distance correlation of pixels in the process of reconstructing the potential tensor into an output image, and considers the influence of different parameters β on the image feature map and the influence of different parameters on different image contents, so as to obtain a more accurate output image and improve the performance of the JPEG AI verification model.

[0261] In the above-mentioned embodiments of the present application, the verification model of JPEG AI may include an analysis transformation network as shown in Figure 18 and a synthesis transformation network as shown in Figure 19. By replacing the residual non-local attention block with a content gain attention module and embedding it into the verification model of JPEG AI, the performance of the verification model of JPEG AI during image encoding and image decoding can be improved.

[0262] Among them, the first image feature, the second image feature, the third image feature, the fourth image feature, the fifth image feature, the sixth image feature, the seventh image feature, the eighth image feature, the ninth image feature, and the tenth image feature are used to distinguish image features obtained after various processing is performed on the image.

[0263] It should be noted that in the above Fig.13 The content gain attention module shown in Figure 2 is applied to the analysis transformation network, as shown in Figure 2. Fig.12 The method flow shown can be applied to an encoder; in the above Fig.13 When the content gain attention module shown in Fig. Fig.12 The method flow shown can be applied to a decoder.

[0264] An embodiment of the present application provides a decoding device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the image feature processing method in the above method embodiment is implemented.

[0265] For example, Fig. 20 A schematic diagram of the structure of a decoding device provided in an embodiment of the present application, the decoding device includes: a processor 2001, a memory 2002;

[0266] For the case where the decoding device is the encoding device:

[0267] In some embodiments, the processor 2001 is configured to: obtain a first image feature to be transformed; perform a first processing on the first image feature to be transformed at least once to obtain a first input image feature, wherein the first processing includes a downsampling process and a residual activation process; process the first input image feature based on a content gain attention block to obtain a first output image feature, wherein the content gain attention block is used to perform attention weighting on the content of the image; perform the first processing on the first output image feature at least once, as well as a downsampling process and a convolution process, to obtain an analysis transformation image feature.

[0268] In some embodiments, the processor 2001 is specifically configured to: process the first input image feature based on the content gain attention block to obtain the first output image feature, including: inputting the parameter β into the fully connected layer, and obtaining the content gain weight output by the fully connected layer; performing feature extraction on the first input image feature through at least one residual block to obtain a first extracted image feature; extracting the first content image feature corresponding to the first input image feature; performing channel dimension operations based on the first content image feature and the first content gain weight to obtain a first joint spatial channel attention mask; performing element-wise dot multiplication operations on the first extracted image feature and the first joint spatial channel attention mask to obtain a first target image feature; and generating a first output image feature based on the first target image feature and the first input image feature.

[0269] In some embodiments, the processor 2001 is specifically configured to: extract a first content image feature corresponding to a first input image feature, including: performing content feature extraction on the input image feature through at least one moving window attention module to obtain a first content image feature corresponding to the first input image feature.

[0270] In some embodiments, at least one moving window attention module is 2 moving window attention modules, and at least one residual block is 3 residual blocks.

[0271] In some embodiments, the at least one first treatment is two first treatments, or the at least one first treatment is one first treatment.

[0272] For the case where the decoding device is a decoding device:

[0273] In some embodiments, the processor 2001 is configured to: obtain a second image feature to be transformed; perform at least one residual block processing, upsampling processing, and at least one second processing on the second image feature to be transformed to obtain a second input image feature, the second processing including residual activation processing and upsampling processing; process the second input image feature based on a content gain attention block to obtain a second output image feature, the content gain attention block is used to perform attention weighting on the content of the image; perform the second processing on the second output image feature at least once to obtain a synthetic transformed image feature.

[0274] In some embodiments, the processor 2001 is specifically configured to: process the second input image feature based on the content gain attention block to obtain the second output image feature, including: inputting the parameter β into the fully connected layer, and obtaining the content gain weight output by the fully connected layer; performing feature extraction on the second input image feature through at least one residual block to obtain a second extracted image feature; extracting the second content image feature corresponding to the second input image feature; performing channel dimension operations based on the second content image feature and the content gain weight to obtain a second joint spatial channel attention mask; performing element-wise dot multiplication operations on the second extracted image feature and the second joint spatial channel attention mask to obtain a second target image feature; and generating a second output image feature based on the second target image feature and the second input image feature.

[0275] In some embodiments, the processor 2001 is specifically configured to: extract a second content image feature corresponding to the second input image feature, including: performing content feature extraction on the second input image feature through at least one moving window attention module to obtain a second content image feature corresponding to the second input image feature.

[0276] In some embodiments, at least one moving window attention module is 2 moving window attention modules, and at least one residual block is 3 residual blocks.

[0277] In some embodiments, the at least one second treatment is a one second treatment.

[0278] Some embodiments of the present application provide a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, the image feature processing method in the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here. In the present application, computer-readable media include permanent and non-permanent, removable and non-removable storage media. The storage medium can implement information storage by any method or technology, and the information can be a computer-readable instruction, a data structure, a program module or other data. Among them, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0279] Some embodiments of the present application provide a computer program product, which stores a computer program. When the computer program is executed by a processor, the image feature processing method in the above method embodiment is implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0280] In the present application, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0281] In the present application, the memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM.

[0282] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0283] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image feature processing method for an encoding device, characterized in that: include: Acquire a first image feature to be transformed; Performing at least one first processing on the first image feature to be transformed to obtain a first input image feature, wherein the first processing includes downsampling processing and residual activation processing; Processing the first input image feature based on a content gain attention block to obtain a first output image feature, wherein the content gain attention block is used to perform attention weighting on content of the image; The first output image feature is subjected to the first processing at least once, downsampling processing and convolution processing to obtain an analysis transformation image feature.

2. The method according to claim 1, characterized in that The content gain-based attention block processes the first input image feature to obtain a first output image feature, including: Inputting parameter β into a fully connected layer, and obtaining a content gain weight output by the fully connected layer; Extracting the first input image feature by using at least one residual block to obtain a first extracted image feature; Extracting a first content image feature corresponding to the first input image feature; performing a channel dimension operation according to the first content image feature and the first content gain weight to obtain a first joint spatial channel attention mask; Performing an element-wise multiplication operation on the first extracted image feature and the first joint spatial channel attention mask to obtain a first target image feature; The first output image feature is generated according to the first target image feature and the first input image feature.

3. The method according to claim 2, characterized in that The extracting the first content image feature corresponding to the first input image feature includes: Content feature extraction is performed on the input image feature through at least one moving window attention module to obtain a first content image feature corresponding to the first input image feature.

4. The method according to claim 3, characterized in that The at least one moving window attention module is 2 moving window attention modules, and the at least one residual block is 3 residual blocks.

5. The method according to claim 1, characterized in that The at least one first treatment is two first treatments, or the at least one first treatment is one first treatment.

6. A method for processing image features of a decoding device, characterized in that: include: Acquire features of a second image to be transformed; Performing at least one residual block processing, upsampling processing, and at least one second processing on the second image feature to be transformed to obtain a second input image feature, wherein the second processing includes residual activation processing and upsampling processing; Processing the second input image feature based on a content gain attention block to obtain a second output image feature, wherein the content gain attention block is used to perform attention weighting on content of the image; The second output image feature is subjected to the second processing at least once to obtain a synthetic transformed image feature.

7. The method according to claim 6, characterized in that The content gain-based attention block processes the second input image feature to obtain a second output image feature, including: Inputting parameter β into a fully connected layer, and obtaining a content gain weight output by the fully connected layer; Extracting features of the second input image by using at least one residual block to obtain second extracted image features; extracting a second content image feature corresponding to the second input image feature; performing a channel dimension operation according to the second content image feature and the content gain weight to obtain a second joint spatial channel attention mask; performing an element-wise multiplication operation on the second extracted image feature and the second joint spatial channel attention mask to obtain a second target image feature; A second output image feature is generated according to the second target image feature and the second input image feature.

8. The method according to claim 7, characterized in that The extracting the second content image feature corresponding to the second input image feature includes: Content feature extraction is performed on the second input image feature through at least one moving window attention module to obtain a second content image feature corresponding to the second input image feature.

9. The method according to claim 8, characterized in that The at least one moving window attention module is 2 moving window attention modules, and the at least one residual block is 3 residual blocks.

10. The method according to claim 6, characterized in that The at least one second process is one second process.

11. A coding device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the image feature processing method according to any one of claims 1 to 5.

12. A decoding device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the image feature processing method according to any one of claims 6 to 10.

Citation Information

Patent Citations

  • Ultrasonic image denoising method based on multi-attention fusion

    CN114663310A

  • Image compression method and device based on deep learning

    CN115484459A

  • Image processing method and device, computer equipment and storage medium

    CN115908086A

  • Image processing method and processing device

    US20190108411A1