Image feature processing method, encoding device and decoding device

Abstract

The application relates to an image feature processing method, an encoding device and a decoding device, in particular to the field of image coding and decoding technology. The method comprises the following steps: acquiring a first to-be-transformed image feature; performing first processing, residual non-local attention block-based processing and at least one first processing on the first to-be-transformed image feature, so as to acquire a first input image feature; performing processing on the first input image feature based on a content gain attention block, so as to obtain a first output image feature, the content gain attention block is used for attention weighting of the content of the image; performing down-sampling processing and convolution processing on the first output image feature, so as to obtain an analysis transformed image feature.

Description

[0001] This application claims priority to Chinese Patent Application No. 2023106645409, filed with the Chinese Patent Office on June 6, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image encoding and decoding technology, and in particular to an image feature processing method, encoding device and decoding device. Background Technology

[0003] Image compression is one of the fundamental tasks in computer vision and image processing. The development of lossy image compression techniques has brought benefits to various fields, significantly reducing the amount of data required for image storage and transmission; however, the image quality after reconstruction may slightly decrease. In recent years, deep neural networks have achieved great success in various tasks, and image compression based on deep neural networks has begun to surpass traditional image compression techniques. JPEG AI is a learning-based image coding standard created by the Joint Picture Experts Group (JPEG). JPEG AI's validation model, designed for human vision, achieves a lower bitrate compared to traditional image coding while maintaining the same subjective quality.

[0004] In the validation model of JPEG AI, both the analytical transform network and the synthetic transform network use residual nonlocal attention blocks to extract deep features. These residual nonlocal attention blocks reduce pixel redundancy in the feature map by capturing the correlation between pixels. The residual nonlocal attention block has two branches: a backbone branch and a mask branch. The backbone branch contains three residual blocks used to extract deep features; the mask branch contains upsampling convolutions, downsampling convolutions, and a series of residual blocks used to scale deep features.

[0005] However, in the current JPEG AI validation model, both the analytical transform network and the synthetic transform network extract deep features through residual nonlocal attention blocks, without considering the impact of different parameters β on different image content. Summary of the Invention

[0006] To solve the above-mentioned technical problems, or at least partially solve them, this application provides an image feature processing method, an encoding device, and a decoding device.

[0007] To achieve the above objectives, some embodiments of this application provide the following technical solutions: In a first aspect, an image feature processing method for an encoding device is provided, comprising: Obtain the features of the first image to be transformed; The first image features to be transformed are subjected to a first processing, a processing based on residual nonlocal attention blocks, and at least one first processing to obtain the first input image features. The first processing includes downsampling processing and residual activation processing. The first input image features are processed based on the content gain attention block to obtain the first output image features. The content gain attention block is used to perform attention weighting on the content of the image. The features of the first output image are downsampled and convolved to obtain the analyzed and transformed image features.

[0008] Secondly, an image feature processing method for a decoding device is provided, including: Obtain the features of the second image to be transformed; The second image features to be transformed are processed by at least one residual block and upsampling process to obtain the second input image features; The second input image features are processed based on the content gain attention block to obtain the second output image features. The content gain attention block is used to perform attention weighting on the content of the image. The second output image features are subjected to at least one second processing, which is based on residual nonlocal attention blocks, and a second processing is performed to obtain synthetic transformed image features. The second processing includes residual activation processing and upsampling processing.

[0009] Thirdly, an encoding device is provided, comprising: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the image feature processing method of the first aspect or any optional embodiment thereof, or the image feature processing method of the second aspect or any optional embodiment thereof.

[0010] Fourthly, a decoding device is provided, comprising: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the image feature processing method described in the second aspect or any of its optional embodiments.

[0011] Fifthly, a computer-readable storage medium is provided, comprising: storing a computer program on the computer-readable storage medium, wherein the computer program, when executed by a processor, implements the image feature processing method described in the first aspect or any optional embodiment thereof, or the image feature processing method described in the second aspect or any optional embodiment thereof.

[0012] In a sixth aspect, a computer program product is provided, wherein a computer program is stored in the computer program product, and when the computer program is executed by a processor, it implements the image feature processing method described in the first aspect or any optional embodiment thereof, or the image feature processing method described in the second aspect or any optional embodiment thereof. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0014] To more clearly illustrate the technical solutions in some embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0015] Figure 1 A schematic diagram of the architecture of a JPEG AI verification model provided in the embodiments of this application; Figure 2 A schematic diagram of an analytical transformation network 101 provided in the embodiments of this application; Figure 3 A schematic diagram of a residual activation unit in the related technology provided in the embodiments of this application; Figure 4 A schematic diagram of a residual nonlocal attention block 203 provided in the related technology of this application embodiment; Figure 5 A schematic diagram of a residual block in the related art provided in the embodiments of this application; Figure 6 A schematic diagram of a masked convolution kernel used in a context model of autoregressive convolution provided in the embodiments of this application; Figure 7A A schematic diagram of the masked convolutional kernel used in the context model network 105 of a JPEG AI verification model provided in the embodiments of this application; Figure 7B A schematic diagram of a wavefront parallel processing mechanism for latent space in a related art provided for an embodiment of this application; Figure 8 A schematic diagram of a synthetic transformation network 117 in a related art provided in an embodiment of this application; Figure 9 A schematic diagram of another JPEG AI verification model provided in the related technology for embodiments of this application; Figure 10A schematic diagram of an ICCI subnetwork provided for an embodiment of this application; Figure 11 A schematic diagram of a content gain attention module provided in an embodiment of this application; Figure 12 A method based on the embodiments of this application is provided. Figure 11 The flowchart shown is a schematic diagram of the image feature processing method performed by the content gain attention module. Figure 13 A schematic diagram of a moving window attention module provided in an embodiment of this application; Figure 14A A schematic diagram of an analysis transform network 101 that embeds a content gain attention module into a verification model of JPEG AI, as provided in an embodiment of this application; Figure 14B A schematic diagram of another analysis transform network 101 that embeds a content gain attention module into a verification model of JPEG AI, provided for an embodiment of this application; Figure 14C A flowchart illustrating an image feature processing method based on an analysis transform network with an embedded content gain attention module, provided in an embodiment of this application; Figure 15A A schematic diagram of a synthetic transform network 117 that embeds a content gain attention module into a verification model of JPEG AI, as provided in an embodiment of this application; Figure 15B A schematic diagram of another synthetic transform network 117 that embeds a content gain attention module into a verification model of JPEG AI, provided for an embodiment of this application; Figure 15C A flowchart illustrating an image feature processing method performed by a synthetic transform network based on an embedded content gain attention module, provided in an embodiment of this application; Figure 16A A schematic diagram of a residual nonlocal attention block in a content gain attention module replacement analysis transform network 101 provided in an embodiment of this application; Figure 16B A flowchart illustrating an image feature processing method performed by an analysis transform network based on replacing residual nonlocal attention blocks with content gain attention modules, as provided in this application embodiment; Figure 17A A schematic diagram illustrating the replacement of a residual nonlocal attention block in a synthesis transform network 117 with a content gain attention module, as provided in an embodiment of this application. Figure 17B A flowchart illustrating an image feature processing method performed by a synthetic transform network that replaces residual nonlocal attention blocks with content gain attention modules, as provided in an embodiment of this application. Figure 18A A schematic diagram illustrating the combination of replacement and embedding methods of a content gain attention module in an analysis transformation network 101, as provided in an embodiment of this application; Figure 18B A flowchart illustrating an image feature processing method performed by an analysis transform network based on a combination of content gain attention module embedding and replacement methods, provided in an embodiment of this application; Figure 19A A schematic diagram illustrating the combination of replacement and embedding methods of a content gain attention module in a synthesis transform network 117, as provided in an embodiment of this application; Figure 19B A flowchart illustrating an image feature processing method performed by a synthetic transform network based on a combination of content gain attention module embedding and replacement methods, provided in an embodiment of this application; Figure 20 This is a schematic diagram of the structure of a decoding device provided in an embodiment of this application. Detailed Implementation

[0016] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0017] Many specific details are set forth in the following description in order to provide a full understanding of this application, but this application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of this application, and not all embodiments.

[0018] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first contextual feature" and "second contextual feature," etc., are used to distinguish different contextual features, not to describe a specific order of contextual features.

[0019] Image compression is one of the fundamental tasks in computer vision and image processing. The development of lossy image compression techniques has brought benefits to various fields. Image compression can significantly reduce the amount of data required for image storage and transmission, but the quality of the reconstructed image may slightly decrease. In the field of lossy image compression, traditional codecs, including the Joint Photographic Experts Group (JPEG), Better Portable Graphics (BPG), and Versatile Video Coding (VVC), use similar transform, quantization, and entropy coding techniques for image compression.

[0020] In recent years, deep neural networks have achieved great success in various tasks, and image compression based on deep neural networks has begun to surpass traditional image compression techniques. JPEG AI is a learning-based image coding standard created by the Joint Picture Experts Group (JPEG). Compared to traditional image coding, JPEG AI, designed for human vision, achieves a lower bitrate while maintaining the same subjective quality.

[0021] Figure 1 This is a schematic diagram of the architecture of a JPEG AI verification model (Verification Model 1.0) provided in the embodiments of this application. The JPEG AI verification model includes: an AnalysisTransform Net 101, a gain unit 102, a first quantization unit 103, a first inverse gain unit 104, a context model network 105, a prediction fusion network 106, a super-prior coding network 107, a super-prior decoding network 108, a second quantization unit 109, a first lossless encoder 110, a first lossless decoder 111, a decomposition entropy model 112, a super-prior scale decoding network 113, a second lossless encoder 114, a second lossless decoder 115, a second inverse gain unit 116, and a synthesis transform network 117. This JPEG AI verification model can compress the input original image x into a bitstream and reconstruct the reconstructed image from the compressed bitstream. In the validation model of this JPEG AI, the network modules Analysis Transform Net 101, Gain Unit 102, First Quantization Unit 103, First Inverse Gain Unit 104, Super Prior Decoding Network 108, Second Quantization Unit 109, First Lossless Encoder 110, and Second Lossless Encoder 114 exist only in the encoder; the network modules Second Lossless Decoder 115, Second Inverse Gain Unit 116, and Synthesis Transform Network 117 exist only in the decoder. The network modules Context Model Network 105, Prediction Fusion Network 106, Super Prior Encoding Network 107, First Lossless Decoder 111, Decomposition Entropy Model 112, and Super Prior Scale Decoding Network 113 exist in both the encoder and decoder.

[0022] The aforementioned analysis transform network 101 is used to transform the original image x input to the validation model of the JPEG AI into a latent tensor. The analysis transform network 101 operates only in the encoder. Latent tensors are high-dimensional representations of data; they cannot be directly observed from the input data, but they can be determined through information learned from the input data. The aforementioned latent tensors... That is, it is determined based on the information learned from the original image x.

[0023] Figure 2 A schematic diagram of an analytical transformation network 101 provided in the embodiments of this application is shown below. Figure 2 As shown, the analysis transformation network 101 includes, from input to output, the following components in sequence: a 3×3 downsampled convolution 201, a residual activation unit 202, a residual non-local attention block 203, a 3×3 downsampled convolution 204, a residual activation unit 205, a 3×3 downsampled convolution 206, a residual activation unit 207, a 3×3 downsampled convolution 208, and a 1×1 convolution 209.

[0024] In this analysis, the transformation network 101 uses residual activation units with attention mechanisms, for example, Figure 2 The residual activation units 202, 205, and 207 mentioned above are all residual activation units with attention mechanisms. Figure 2 The 3×3 downsampling convolutions 201, 204, 206, and 208 are all 3×3 downsampling convolutions with the same structure.

[0025] Figure 3This is a schematic diagram of a residual activation unit 202 provided in the related technology of this application. For example... Figure 3 As shown, the residual activation unit 202 may include: a LeakyReLU activation function 301, a convolutional layer 302, a tanh activation function 303, and a residual connection unit 304. This residual activation unit is used to add nonlinear transformations.

[0026] In the JPEG AI validation model, attention mechanisms can be used. Figure 2 The residual nonlocal attention block 203 shown is implemented. The residual nonlocal attention block 203 can extract deep features. In the process of extracting deep features, by capturing the correlation between pixels, pixel redundancy in the feature map can be reduced. Figure 4 This is a schematic diagram of a residual nonlocal attention block 203 provided in the related technology of this application embodiment. For example... Figure 4 As shown, the residual nonlocal attention block 203 includes two branches: a main branch 41 and a mask branch 42. The main branch 41 contains three residual blocks used to extract deep features. The mask branch 42 contains a 3×3 downsampling convolution 401, a 3×3 upsampling convolution 402, a 1×1 convolution 403, a sigmoid activation function 404, and a series of residual blocks. Figure 4 The three residual blocks in the main branch 41 are represented as: residual block 1, residual block 2, and residual block 3; the series of residual blocks included in the mask branch 42 are represented as: residual block 4, residual block 5, residual block 6, residual block 7, residual block 8, and residual block 9. It should be noted that the structure and function of each residual block in the above residual blocks 1 to 9 are the same.

[0027] Figure 5 for Figure 4 The illustrated embodiment shows a schematic diagram of a residual block, each residual block including a LeakyReLU activation function 501 and two 3×3 convolutions, wherein the two 3×3 convolutions are in Figure 5 These are represented as 3×3 convolution 502 and 3×3 convolution 503, respectively.

[0028] like Figure 1 As shown, in the JPEG AI Verification Model 1.0, the extrapolation gain unit is used to implement the variable bit rate, through parameters... The parameters of the extrapolation gain unit are controlled to scale the potential residual variable by the channel dimension. The extrapolation gain unit includes the aforementioned gain unit 102 and the first inverse gain unit 104.

[0029] Gain unit 102 is used for the residual tensor Perform gain processing to convert it into a residual tensor. Multiply each element by The specified gain factor is used to obtain the residual tensor after gain processing. Among them, residual tensor For potential tensors Compared with the predicted value The difference, where the predicted value For latent tensors The predicted values ​​of the elements. Gain unit 102 is used to implement the variable rate capability of the JPEG AI verification model. Gain unit 102 uses parameters to control the compression ratio. This parameter Specifies the weights between rate and distortion in the loss function during training. Obtaining the paired positive and negative gain vectors can be represented as follows: ,in, It is the positive gain vector. This is the inverse gain vector. During encoding and decoding, the forward and inverse gain vectors are known to the encoder and decoder (they are part of the model), and the gain vector... and The dimension is equal to the residual tensor. The number of channels. This gain unit 102 only operates in the encoder.

[0030] The first quantization unit 103 is used to quantize the residual tensor after gain processing to obtain the quantized residual tensor. Quantization refers to the process of approximating a signal's continuous values ​​(or a large number of possible discrete values) to a finite number of discrete values ​​(or a small number of discrete values). This first quantization unit 103 operates only in the encoder.

[0031] The first inverse gain unit 104 is used to process the quantized residual tensor. Inverse gain processing was performed to obtain the quantized residual tensor. Each element is multiplied by the inverse gain vector The specified gain factor is used to obtain the residual tensor after inverse gain processing. The first inverse gain unit 104 enables the variable rate capability of the JPEG AI verification model. The first inverse gain unit 104 uses parameters to control the compression ratio. This parameter Specifies the weights between rate and distortion in the loss function during training. This first inverse gain unit 104 operates only in the encoder.

[0032] The aforementioned context model network 105 is used to extract contextual features, and such a network module exists in both the encoder and decoder. In the encoder, the context model network 105 is used to process the latent tensor. Contextual feature extraction is performed to obtain the latent tensor. Contextual features. Among them, the latent tensor in the encoder. Based on the residual tensor after inverse gain processing and predicted value The sum is obtained by addition. In the decoder, the context model network 105 is used to process the latent tensors. Contextual feature extraction is performed to obtain the latent tensor. Contextual features. Specifically, in the decoder, the bitstream ( Figure 1 The residual tensor of the bitstream 2) shown in the figure after lossless decoding and inverse gain processing Compared with the predicted value By adding them together, we can obtain the potential tensor. .

[0033] In some learning-based image coding methods, autoregressive convolutional context models are used to extract contextual features. Figure 6 This is a schematic diagram of a masked convolution kernel used in a context model of autoregressive convolution provided in the embodiments of this application. Figure 6 As shown, this autoregressive convolutional context model uses the encoded elements 601 before the current encoding position in the latent space to predict the context features corresponding to the current encoding position, and uses these context features to predict the probability distribution of the current encoding position in subsequent calculations. Here, the encoded elements are... Figure 6 The elements at the encoded positions are shown in the dashed box. Autoregressive convolutional context models can significantly improve the rate-distortion performance of super-prior models. However, due to their autoregressive structure, these context models can only decode according to a fixed raster scan order. The latent space of later decoded models always depends on the latent space of earlier decoded models, resulting in very slow decoding speeds.

[0034] To address the aforementioned issues, in some embodiments, the context model network 105 in the JPEG AI verification model further employs a wavefront parallel processing (WPP) mechanism in the latent space to enable parallel processing. This mechanism uses already decoded elements to generate latent tensors. The element prediction is an autoregressive process, and it is performed on the latent tensor. Elements in different rows can be processed independently in a wavefront parallel manner, where the delay in processing consecutive rows depends on the shape of the masked convolution kernel.

[0035] Figure 7A This is a schematic diagram of the masked convolutional kernel used in the context model network 105 of a JPEG AI verification model provided in the embodiments of this application. Figure 7A The coordinates are established with the current encoding position as the origin. The current encoding position is in row 0 and column 0. n=0 indicates the row where the current encoding position is located, n=-1 indicates the row before the current encoding position, and n=-2 indicates the row before the current encoding position. Figure 7A The symbols “-2”, “-1”, “0”, “1” and “2” indicate different columns. “0” indicates the column where the current encoding position is located, “-1” indicates the column before the current encoding position, “-2” indicates the column before the current encoding position, “1” indicates the column after the current encoding position, and “2” indicates the column after the current encoding position.

[0036] Figure 7B This diagram illustrates a wavefront parallel processing mechanism for a potential space, as provided in the embodiments of this application. The wavefront parallel processing mechanism is introduced to improve the utilization of computing resources. Figure 7B In this context, "column" represents a column and "row" represents a row. The diagram illustrates a potential space of 5 rows and 10 columns. Figure 7B The tensor of the current pixel shows that all channels are predicted independently (simultaneously), and prediction can be based on previously processed pixels during the preset process. For example... Figure 7A The design of the masked convolution kernel shown makes Figure 7B Each row of samples (i.e., the prediction of each row of pixels) can be processed in parallel, only with a delay of one element compared to the previous processing time when processing the next row, which is a delay of one wave shift (T).

[0037] Super-prior coding network 107, used for latent tensors Perform prior encoding to obtain the latent tensor. Furthermore, additional auxiliary information can be extracted during the super-prior encoding process, enabling the subsequent acquisition of an accurate probabilistic model based on the entropy encoding result. The super-prior encoding network 107 operates only in the encoder and can include two downsampling convolutions, three convolutions, and a ReLU activation function. The ReLU activation function is a neural network activation function. It remains unchanged when the input value is greater than 0, and outputs 0 when the input value is less than 0.

[0038] The second quantization unit 109 is used for quantization of potential tensors. Quantization is performed to obtain the quantized encoded latent tensor. The second quantization unit 109 only operates in the encoder.

[0039] The first lossless encoder 110 is used to process the quantized latent tensor. Entropy encoding is performed to obtain bitstream 1. Entropy encoding is a lossless operation used to encode the bitstream. The entropy encoding and decoding processes take the cumulative distribution function (CDF) output by the decomposed entropy model 112 as input. The cumulative distribution function (CDF) is essentially the integration of the probability density function (PDF) with the symbol to be encoded at the encoding end, or the integration of the probability density function (PDF) with the bitstream to be decoded at the decoding end. This first lossless encoder 110 operates only as an encoder.

[0040] The first lossless decoder 111 is used to perform entropy decoding on the bitstream 1 obtained after entropy encoding to obtain the latent tensor after entropy decoding. Entropy decoding is a lossless operation used to decode bitstreams. It can receive a cumulative distribution function (CDF) as input and is essentially the integral of the probability density function (PDF) and the bitstream to be decoded (at the decoder). This first lossless decoder 111 operates only at the decoder.

[0041] The decomposition entropy model 112 is used to input the cumulative distribution function (CDF) into the first lossless encoder 110 or the first lossless decoder 111. The CDF is used for entropy encoding or decoding. The decomposition entropy model 112 operates in both the decoder and the encoder. The decomposition entropy model 112 is integerized to address device interoperability issues, ensuring its usability on any device. The cumulative distribution function (CDF), also known as a probability table, is pre-computed and quantized, rather than trained based on trainable parameters (matrices, biases, and factors). This table is stored in integer form, thus eliminating the need to store the aforementioned parameters. Parameters of the distribution.

[0042] The super-prior decoding network 108 is used for super-prior decoding, specifically for processing the latent tensor after entropy decoding. Decoding is performed to obtain the latent tensor. The hyperprior decoding network 108 comprises a series of two deconvolutions with an upsampling stride of 2, three convolutions without tensor size changes, and a ReLU activation function. The hyperprior decoding network 108 can accept a tensor of size C×h×w as input and generate an output tensor of 4C×h×w with two channels. This hyperprior decoding network 108 is symmetric to the hyperprior coding network 107, with a pruning operation following each upsampling convolutional layer. The hyperprior decoding network 108 operates in both the decoder and encoder.

[0043] The predictive fusion network 106 can receive two inputs. The first input is the latent tensor obtained after super-prior decoding by the super-prior decoding network 108. The second input is the contextual features output by the context model network 105. The prediction fusion network 106 generates a predicted value of size C×h×w based on the two inputs of size 4C×h×w. (Size is C×h×w). In the encoder, this predicted value... It can be added to the residual tensor after inverse gain processing. To obtain the potential tensor This is used as input to the context model network 105. In the decoder, the bitstream ( Figure 1 The residual tensor of the bitstream 2) shown in the figure after lossless decoding and inverse gain processing Compared with the predicted value Adding them together yields the potential tensor. As input to the context model network 105, the prediction fusion network 106 operates in both the decoder and the encoder.

[0044] Super-prior scale decoding network 113 is used for the latent tensor after entropy decoding. Perform the inverse operation of the hyperprior coding network and output a Gaussian distribution with zero mean. The variance of the super-prior scale decoding network 113 is calculated. The super-prior scale decoding network 113 consists of two 5×5 deconvolutional sequences with an upsampling stride of 2, two 3×3 convolutions, and a LeakyReLU activation function. The super-prior scale decoding network 113 is symmetric to the super-prior encoder network 107, with a pruning operation following each upsampling convolutional layer. The prediction fusion network 106 operates in both the decoder and encoder.

[0045] The second lossless encoder 114 is used for... For the quantized residual tensor Entropy encoding is performed to obtain bitstream 2. It should be noted that the second lossless encoder 114 has a similar structure and function to the first lossless encoder 110. The second lossless encoder 114 only operates within the encoder.

[0046] The second lossless decoder 115 is used for... Entropy decoding is performed on bitstream 2 to obtain the residual tensor after entropy decoding. It should be noted that the second lossless decoder 115 has a similar structure and function to the first lossless decoder 111. The second lossless decoder 115 only operates within the decoder.

[0047] The second inverse gain unit 116 is used to perform inverse gain processing on the residual tensor after entropy decoding, multiplying each element of the residual tensor after entropy decoding by an inverse gain vector. The specified gain factor is used to obtain the residual tensor after inverse gain processing. The second inverse gain unit 116 only operates in the decoder.

[0048] Synthetic transformation network 117 is used to transform latent tensors This indicates conversion to a reconstructed image. Among them, the latent tensor The residual tensor after inverse gain processing Compared with the predicted value The result obtained by addition. The synthetic transform network 117 only operates in the decoder.

[0049] Figure 8 This is a schematic diagram of a synthetic transformation network 117 provided in the embodiments of this application. For example... Figure 8 As shown, the synthesis transform network 117 includes, from input to output, the following components in sequence: residual block 801, residual block 802, 3×3 upsampling convolution 803, residual activation unit 804, 3×3 upsampling convolution 805, residual activation unit 806, 3×3 upsampling convolution 807, residual nonlocal attention block 808, residual activation unit 809, and 3×3 upsampling convolution 810. Two residual blocks are placed at the beginning of the synthesis transform, enhancing the reconstructed latent tensor. This reduces quantization error.

[0050] It should be noted that each residual activation unit in the synthetic coding network 117 is related to, for example, Figure 3 The residual activation unit shown has the same structure and function. The residual nonlocal attention block 808 in the synthetic coding network 117 is the same as described above. Figure 4 The residual nonlocal attention block 203 shown has the same structure and function. Each residual block in the synthetic coding network 117 is similar to... Figure 4 The residual blocks shown have the same structure and function.

[0051] Figure 9 This is a schematic diagram of another JPEG AI verification model provided in the related technology of this application embodiment.

[0052] like Figure 9 As shown above, in the above Figure 1 The JPEG AI validation model shown is supplemented with a content-adaptive filter 118. This content-adaptive filter 118 is used to enhance the Y, U, and V components, and to reconstruct the image output by the synthetic transform network 117. Enhancement is performed on the Y, U, and V components to obtain the enhanced reconstructed image. The adaptive filter 118 includes a Content Adaptive Inter Channel Correlation Information filter subnetwork (ICCI) for reconstructing and enhancing the Y, U, and V components of the image. This ICCI subnetwork is used to enhance the Y, U, and V components.

[0053] Figure 10 This is a schematic diagram of an ICCI subnetwork provided in an embodiment of this application. (See diagram below.) Figure 10 As shown, this ICCI sub-network includes three different ICCI processing modules. The input to this ICCI sub-network is the Y, U, and V components of the image. Figure 10 The Y component of the image is represented as The U component of the image is represented as The V component of the image is represented as The U component of the image is obtained by bicubic interpolation to obtain a U component with the same shape as the Y component. The V component of the image is obtained by bicubic interpolation to obtain a V component with the same shape as the Y component. Then the Y component U component and V component All are subjected to forward wavelet transform, and the three components obtained by the transform are represented as follows: Figure 10 Components 1001, 1002, and 1003 are concatenated along the channel dimension. The concatenated component 1004, along with the individual components, is then fed into three ICCI processing modules as input. These modules process the Y, U, and V components respectively, resulting in processed components 1008, 1009, and 1010. These components are then subjected to inverse wavelet transform to obtain the reconstructed Y, U, and V components. The reconstructed Y component is... The reconstructed U component is The reconstructed V component is In the three ICCI processing modules mentioned above, the number of residual blocks is Nb. Specifically, the number of residual blocks in ICCI processing module 1005 (for the Y component) is Nb=8, in ICCI processing module 1006 (for the U component) is Nb=13, and in ICCI processing module 1007 (for the V component) is Nb=13. The aforementioned bicubic interpolation is a method for enlarging image size and increasing pixel density. It can double the number of image pixels using bicubic interpolation, where these expanded pixels are predicted based on existing pixels.

[0054] The above Figure 10 In the ICCI subnetwork shown, during the image feature enhancement process, in order to take into account the characteristics of different Y, U, and V components, three different ICCI processing modules are set up to enhance different components, thereby achieving the purpose of adaptive processing.

[0055] In response to the above Figure 1 or Figure 9 The JPEG AI validation model shown above uses residual nonlocal attention blocks (RNIGs) in both the analysis transform network 101 and the synthesis transform network 117 to extract deep features. These RIGIGs can reduce pixel redundancy in the feature map by capturing the correlation between pixels. The RIGIGs have two branches: a main branch 41 and a mask branch 42. The main branch 41 contains three residual blocks used to extract deep features; the mask branch contains upsampling convolutions, downsampling convolutions, and a series of residual blocks used to scale deep features. However, the current JPEG AI validation model, where both the analysis transform network 101 and the synthesis transform network 117 extract deep features using RIGIGs, suffers from at least one of the following problems: (1) The residual nonlocal attention block ignores the influence of different parameters β on the feature map and treats the feature map equally for different parameters β; (2) Mask branches cannot connect across windows and can only analyze the correlation of pixels within a unified window. Therefore, the ability of mask branches to capture long-distance correlation of pixels is limited.

[0056] (3) The influence of different parameters β on different image content is not considered. In some embodiments, the JPEG AI verification model uses an extrapolation gain unit to implement variable bitrate. The parameters of the extrapolation gain unit are controlled by the parameter β to scale the potential residual tensor in the channel dimension. It can be seen that although the scaling of the extrapolation gain unit considers the influence of different parameters β, it only involves the influence of different parameters β on the feature map when scaling in the channel dimension.

[0057] To address the aforementioned problems (1), (2), and (3), this application embodiment provides a content gain attention module, which can adaptively weight the content of the image according to parameter β, and effectively capture the long-distance correlation of pixels.

[0058] Figure 11 This is a schematic diagram of a content gain attention module provided in an embodiment of this application. Figure 11 The content gain attention module includes a main branch and a mask branch. The main branch includes residual blocks 1101, 1102, and 1103, while the mask branch includes moving window attention modules 1104 and 1105, a fully connected layer 1107, and a Sigmoid activation function 1106. This content gain attention module is used to extract deep image features. The Sigmoid activation function is an activation function with an S-shaped curve that maps input values ​​to the range [0,1].

[0059] Figure 12 A method based on the embodiments of this application is provided. Figure 11 The diagram shows a flowchart illustrating the image feature processing method performed by the content gain attention module. Figure 12 As shown, the process of this image feature processing method includes, but is not limited to, the following steps: 1201. Set parameters Input the fully connected layer and obtain the content gain weights of the fully connected layer's output.

[0060] Among them, parameters Specifies the weights between rate and distortion in the loss function during training.

[0061] The above process of obtaining content gain weights can be expressed as: (1) Among them, parameters The range of values ​​is , .parameter W represents the content gain weight generated after passing through the fully connected layer. This represents the process of generating content gain weights using a fully connected layer. The parameters are then processed through the fully connected layer. Mapped to a vector with the same dimension as the content image feature X. This can be done based on different parameters. Compress at different bitrates.

[0062] like Figure 11 As shown, when the parameters are... After inputting into the fully connected layer 1107, the fully connected layer 1107 can output content gain weights. .

[0063] 1202. Extract features from the input image using at least one residual block to obtain the extracted image features.

[0064] It should be noted that, in the embodiments of this application, feature extraction of the input image features can be performed through one or more residual blocks.

[0065] Figure 11 The middle part uses three residual blocks to process the features of the input image. Feature extraction is performed to obtain the extracted image features. For example, the three residual blocks are as follows: Figure 12 The residual blocks 1101, 1102, and 1103 are identical in structure. Input image features are extracted through residual blocks 1101, 1102, and 1103 to generate extracted image features. .

[0066] Extracting image features The process can be represented as: (2) in, , , The calculation processes for residual blocks 1101, 1102, and 1103 are respectively.

[0067] The residual block in this application embodiment may include a neural network convolutional layer and residual connections. For example, it may be as follows: Figure 5 The structure of the residual block is shown.

[0068] 1203. Extract the content image features corresponding to the input image features.

[0069] In this embodiment, at least one moving window attention module can be used to extract content image features from the input image features, so as to extract the content image features corresponding to the input image features.

[0070] Figure 11 The content gain attention module shown is illustrated using an example that includes two moving window attention modules. The input image features... The content image features X are obtained by extracting the content image features through the moving window attention module 1104 and the moving window attention module 1105.

[0071] The process of obtaining the content image feature X described above can be represented as: X = (3) in, and They are respectively Figure 11 The calculation process of the moving window attention module 1104 and the moving window attention module 1105 shown.

[0072] Figure 13 This is a schematic diagram of a mobile window attention module provided in an embodiment of this application. Figure 13 As shown, the moving window attention module includes: a fully connected layer 1301 of a Multi-Layer Perceptron (MLP), a SoftMax activation function 1302, a 1×1 convolution 1303, a 1×1 convolution 1304, a 1×1 convolution 1305, and a 1×1 convolution 1306. The SoftMax activation function is a normalization function that maps input values ​​to the range [0,1], and the sum of all output values ​​is 1. The input image features... The image features are convolved using 1×1 convolutions 1303 and 1×1 convolutions 1304, respectively. The resulting image features Q and K are then multiplied element-wise and processed using the SoftMax activation function 1302. The result of the SoftMax activation function 1302 is then compared with the input image features. The image features V obtained after 1×1 convolution (1305) are then subjected to element-wise multiplication. The result of this multiplication is then subjected to 1×1 convolution (1306) before being combined with the input image features. After dot product operation, the result is fed into the fully connected layer of the MLP to obtain the output. . Input image features The result after processing by the moving window attention module can be represented as: (4) (5) in, This is a general representation of the computation process of the moving window attention module. Q, K, and V represent the input image features. The representation after 1×1 convolution, where SoftMax() is the SoftMax activation function, d is the number of channels in Q, K, and V, and B is the learnable positional encoding. This indicates that the correlation of each element at position Q, K, and V is calculated to generate the attention matrix.

[0073] 1204. Based on the content image features and content gain weights, perform channel dimension calculations to obtain the joint spatial channel attention mask.

[0074] like Figure 11 As shown, input image features The transformation into a joint spatial channel attention mask M can include, but is not limited to, input image features. First, the content image features X are generated through the moving window attention modules 1104 and 1105. Then, the content image features X and the content gain weights W are multiplied by a dot product along the channel dimension. Finally, the content image features X and the content gain weights W are activated by a sigmoid activation function to generate a joint spatial channel attention mask M. The process of generating the joint spatial channel attention mask M can be represented as follows: (6) in, and The calculation process for the moving window attention module 1104 and the moving window attention module 1105. This is the dot product operation for the channel dimension. Sigmoid() is the Sigmoid activation function.

[0075] 1205. Perform element-wise multiplication on the extracted image features and the joint spatial channel attention mask to obtain the target image features.

[0076] 1206. Generate output image features based on the target image features and the input image features.

[0077] Specifically, the process of generating output image features can be as follows: extracting image features y and performing element-wise multiplication on the joint spatial channel attention mask M, and then adding the result to the input image features y to generate the output image features. .

[0078] The above process can be represented as: (7) in, This is the element-wise multiplication operation.

[0079] Combining formulas (1) to (7) above, we can see that the overall calculation process of the content gain attention module can be expressed as follows: (8) In this embodiment of the application, in the content gain attention module, since the parameters are adjusted during deep feature extraction... The content gain weights are obtained by inputting the fully connected layer. Therefore, compared with the original residual nonlocal attention block, the influence of different parameters β on the image feature map is considered, which solves the above problem (1).

[0080] In some embodiments, since the joint spatial channel attention mask is obtained by performing calculations based on the content gain weight and the extracted image content features in this application embodiment, the influence of different parameters β on different image content is considered in the content gain attention module. When the content gain attention module is applied to the verification model of JPEG AI, the above problem (3) can be solved.

[0081] In some embodiments of this application, the moving window attention module can divide the features into multiple windows in a non-overlapping manner and calculate the attention map of each window to obtain the attention of the local window. By moving the window to different positions in different moving window attention blocks, elements that were originally in different windows can be divided into the same window, thus enabling cross-window connections. This moving window attention module improves the ability of mask branches to capture long-distance correlations of pixels by calculating the attention of local windows and allowing cross-window connections, thereby solving the above-mentioned problem (2).

[0082] For example, the aforementioned moving window attention module can divide the features into 8×8 windows in a non-overlapping manner and calculate the attention map for each window separately. In the case where at least one moving window attention module consists of two moving window attention modules, the second moving window attention module can move the divided windows 4 pixels to the right and 4 pixels down compared to the first, allowing elements that were originally in different windows to be grouped into the same window, thus achieving cross-window connectivity.

[0083] The aforementioned content gain attention module can be applied to the analysis transform network 101 and the synthesis transform network 117 in the verification model of JPEG AI.

[0084] In some embodiments of this application, the content gain attention module is embedded into the aforementioned analysis transform network 101 and synthesis transform network 117 in the verification model of JPEG AI.

[0085] For example, Figure 14A This is a schematic diagram of an analysis transform network 101 that embeds a content gain attention module into the verification model of JPEGAI, as provided in an embodiment of this application. Figure 14A exist Figure 2 Based on this, a content gain attention module 1401 is embedded between the residual activation unit 205 and the 3×3 downsampling convolution 206.

[0086] Figure 14AIn the analysis and transformation network shown, the original image input to the verification model of the JPEG AI (i.e., the first image feature to be transformed in this application) is processed by a 3×3 downsampling convolution 201, a residual activation unit 202, a residual nonlocal attention block 203, and a 3×3 downsampling convolution 204 and a residual activation unit 205 to obtain the first input image feature. The first input image feature is used as the input image feature of the content gain attention block 1401. In the content gain attention block 1401, the image features are processed as follows: Figure 12 After the input image features are processed by the process shown, the first output image features output by the content gain attention module 1401 can be obtained. The first output image features are further processed by 3×3 downsampling convolution 206, residual activation unit 207, 3×3 downsampling convolution 208 and 1×1 convolution 209 in sequence to obtain the latent tensor corresponding to the original image (i.e. the analysis transformation image features in this application).

[0087] For example, Figure 14B This is a schematic diagram of another analysis transform network 101 that embeds a content gain attention module into the verification model of JPEG AI, as provided in an embodiment of this application. Figure 14B exist Figure 2 Based on this, a content gain attention module 1401 is embedded between the residual activation unit 207 and the 3×3 downsampling convolution 208.

[0088] Figure 14A and Figure 14B The difference in the analysis transformation network 101 shown is that the content gain attention module 1401 is embedded in a different location. Based on this difference, Figure 14A The analysis transformation network shown performs feature processing via the content gain attention module 1401 before the 3×3 downsampling convolution 206 and residual activation unit 207. Figure 14B The analysis and transformation network graph performs feature processing via the content gain attention module 1401 after the 3×3 downsampling convolution 206 and residual activation unit 207, but before feature processing. Since downsampling reduces the receptive field of the analysis and transformation network 101 and also reduces its computational resource consumption, it can be used when the quality requirements for the analysis and transformation image features are low and computational resources are limited. Figure 14B The analysis transform network shown can be selected when high quality is required for the analyzed transform image features and sufficient computational resources are available. Figure 14A The analysis transformation network is shown.

[0089] The above Figure 14A and Figure 14B Content gain attention module in such a module Figure 13 As shown in the image.

[0090] Figure 14B In the analysis and transformation network shown, the original image input to the verification model of the JPEG AI is processed through a 3×3 downsampling convolution 201, a residual activation unit 202, a residual nonlocal attention block 203, a 3×3 downsampling convolution 204, a residual activation unit 205, a 3×3 downsampling convolution 206, and a residual activation unit 207 to obtain the first input image features. These first input image features are then used as the input image features of the content gain attention block 1401. The content gain attention block 1401 then processes the image through the following steps: Figure 12 After the process shown processes the first input image features, the first output image features output by the content gain attention module 1401 can be obtained. The first output image features are then further processed by 3×3 downsampling convolution 208 and 1×1 convolution 209 in sequence to obtain the latent tensor corresponding to the original image (i.e., the analysis and transformation image features in this application).

[0091] Figure 14C This is a flowchart illustrating an image feature processing method based on an analysis transform network with an embedded content gain attention module, provided as an embodiment of this application. This image feature processing method is applied to an encoding device and may include, but is not limited to, the following steps: 141. Obtain the features of the first image to be transformed.

[0092] The first image feature to be transformed can be a feature input to the analysis and transformation network. For example, such as... Figure 1 or Figure 9 As shown, the first image feature to be transformed can be the original image x input to the analysis and transformation network 101.

[0093] 142. Perform a first processing on the first image features to be transformed, a processing based on residual nonlocal attention blocks, and at least one of the first processing steps to obtain the first input image features.

[0094] The first processing includes downsampling and residual activation processing. For example, downsampling is performed by a 3×3 downsampling convolution, and then residual activation is performed on the output of the 3×3 downsampling convolution by a residual activation unit.

[0095] The above-mentioned at least one first processing is one first processing, or the above-mentioned at least one first processing is two first processings.

[0096] For example, such as Figure 14AAs shown, the above-mentioned at least one first processing step constitutes one first processing step. After the first image feature to be transformed is input into the analysis and transformation network, it first undergoes a first processing step through a 3×3 downsampling convolution 201 and a residual activation unit 202, and then undergoes another first processing step through a residual nonlocal attention block 203, a 3×3 downsampling convolution 204, and a residual activation unit 205. In this way, the first input image feature can be obtained and then input into the content gain attention block 1401 for further processing.

[0097] For example, such as Figure 14B As shown, the above-mentioned at least one first processing is equivalent to two first processings. After the first image feature to be transformed is input into the analysis and transformation network, it first undergoes a first processing through a 3×3 downsampling convolution 201 and a residual activation unit 202, then passes through a residual nonlocal attention block 203, followed by a first processing through a 3×3 downsampling convolution 204 and a residual activation unit 205, and then undergoes a first processing through a 3×3 downsampling convolution 206 and a residual activation unit 207 (i.e., two first processings). This yields the first input image feature, which is then input into the content gain attention block 1401 for further processing.

[0098] 143. Process the first input image features based on content gain attention blocks to obtain the first output image features.

[0099] The content gain attention block is used to apply attention weights to the content of the image. The process by which this content gain attention block processes the features of the first input image to obtain the features of the first output image is as described above. Figure 12 As shown.

[0100] 144. Perform downsampling and convolution processing on the features of the first output image to obtain the analyzed and transformed image features.

[0101] The above-described analysis of transformed image features can be used to analyze the output features of the transform network. For example, such as... Figure 1 or Figure 9 As shown, the analyzed image features can be used as the latent tensor output by the analyzed transformation network 101. .

[0102] For example, such as Figure 14B As shown, after the content gain attention block 1401 processes the first input image features, it outputs the first output image features, which are then downsampled by a 3×3 downsampling convolution 208 and convolved by a 1×1 convolution 209 to obtain the analyzed and transformed image features.

[0103] In the case where at least one of the above-mentioned first processing is a first processing, performing downsampling processing and convolution processing on the first output image features to obtain the analyzed transformed image features may be as follows: firstly, perform first processing on the first output image features, and then perform downsampling processing and convolution processing to obtain the analyzed transformed image features.

[0104] For example, such as Figure 14A As shown, the above-mentioned at least one first processing is one first processing. After the first input image features are input to the content gain attention block 1401 for processing, the first output image features will be output. Then, the first output image features will first undergo a first processing through a 3×3 downsampling convolution 206 and a residual activation unit 207, then undergo downsampling processing through a 3×3 downsampling convolution 208, and then undergo convolution processing through a 1×1 convolution 209 to obtain the analysis and transformation image features.

[0105] In the above embodiments, the analysis transformation network incorporates a content gain attention module, which improves the network's ability to capture long-range correlations of pixels during the process of transforming the input image into a latent tensor. It considers the influence of different parameters β on the image feature map and the influence of different parameters on different image content, thereby obtaining a more accurate latent tensor and improving the performance of the JPEG AI verification model.

[0106] Figure 15A This is a schematic diagram of a synthetic transform network 117 that embeds a content gain attention module into a verification model of JPEG AI, as provided in an embodiment of this application. Figure 15A exist Figure 8 Based on this, a content gain attention block 1501 is embedded between the 3×3 upsampling convolution 805 and the residual activation unit 806.

[0107] Figure 15A In the synthetic transformation network shown, the latent tensor (i.e., the second image feature to be transformed) input to the synthetic transformation network in the validation model of the JPEG AI is processed by residual block 801, residual block 802, 3×3 upsampling convolution 803, residual activation unit 804, and 3×3 upsampling convolution 805 to obtain the third image feature (i.e., the second input image feature). The obtained third image feature is used as the input image feature of the content gain attention block 1501. In the content gain attention block 1501, the image features are processed as follows: Figure 12After the input image features are processed by the process shown, the output image features (i.e., the second output image features) output by the content gain attention module 1501 can be obtained. The output image features are then further processed by the residual activation unit 806, the 3×3 upsampling convolution 807, the residual nonlocal attention block 808, the residual activation unit 809, and the 3×3 upsampling convolution 810 in sequence to obtain the reconstructed image (i.e., the synthetic transformed image features).

[0108] Figure 15B This is a schematic diagram of the above-described synthetic transform network 117, which embeds a content gain attention module into a verification model of JPEG AI, as provided in an embodiment of this application. Figure 15B exist Figure 8 Based on this, a content gain attention block 1501 is embedded between the 3×3 upsampling convolution 803 and the residual activation unit 804.

[0109] The above Figure 15A and Figure 15B Content gain attention module in such a module Figure 13 As shown in the image.

[0110] Figure 15B In the synthetic transformation network shown, the latent tensor input to the synthetic transformation network in the JPEG AI verification model (i.e., the second image feature to be transformed in this application) is processed by residual block 801, residual block 802, and 3×3 upsampling convolution 803 to obtain the second input image feature. The obtained second input image feature is used as the input image feature of the content gain attention block 1501. In the content gain attention block 1501, the image features are processed as follows: Figure 12 After processing the second input image features as shown in the process, the second output image features output by the content gain attention module 1501 can be obtained. The second output image features are then further processed sequentially by the residual activation unit 804, the 3×3 upsampling convolution 805, the residual activation unit 806, the 3×3 upsampling convolution 807, the residual nonlocal attention block 808, the residual activation unit 809, and the 3×3 upsampling convolution 810 to obtain the reconstructed image (i.e., the synthetic transformation image features in this application).

[0111] Figure 15C This is a flowchart illustrating an image feature processing method performed by a synthetic transform network with an embedded content gain attention module, provided as an embodiment of this application. This image feature processing method is applied to a decoding device and may include, but is not limited to, the following steps: 151. Obtain the features of the second image to be transformed.

[0112] The second image feature to be transformed is the feature input to the synthetic transformation network. For example, such as... Figure 1or Figure 9 As shown, the second image feature to be transformed can be a latent tensor input to the synthetic transformation network 117. .

[0113] 152. Perform at least one residual block processing and upsampling processing on the second image features to be transformed to obtain the second input image features.

[0114] For example, in some embodiments of this application, the at least one residual block used to process the second image features to be transformed may be two residual blocks; the upsampling process may be a process performed by a 3×3 upsampling convolution.

[0115] For example, such as Figure 15B As shown, after the second image feature to be transformed is input into the synthesis transformation network, it is first processed by residual block 801 and residual block 802, and then processed by 3×3 upsampling convolution 803 to obtain the second input image feature. The second input image feature is then input into content gain attention block 1501 for further processing.

[0116] In some embodiments of this application, in the process of performing at least one residual block processing and upsampling processing on the second image features to be transformed to obtain the second input image features, the second image features to be transformed may first be subjected to at least one residual block processing and upsampling processing, and then the second processing may be performed to obtain the second input image features.

[0117] For example, such as Figure 15A As shown, after the second image feature to be transformed is input into the synthesis transformation network, it is first processed by residual block 801 and residual block 802, then processed by 3×3 upsampling convolution 803, and then processed by residual activation unit 804 and upsampling convolution 805 for a second processing, before the second input image feature is obtained. The second input image feature is then input into content gain attention block 1501 for further processing.

[0118] 153. Process the second input image features based on content gain attention blocks to obtain the second output image features.

[0119] The content gain attention block is used to apply attention weights to the content of the image. The process by which this content gain attention block processes the features of the second input image to obtain the features of the second output image is as described above. Figure 12 As shown.

[0120] 154. Perform at least one second processing on the second output image features, processing based on residual nonlocal attention blocks, and perform the second processing to obtain synthetic transformed image features.

[0121] The second processing described above includes residual activation processing and upsampling processing.

[0122] The aforementioned synthesized image features can be the output features of the synthesized transform network. For example, such as... Figure 1 or Figure 9 As shown, the synthetic transformation image features can be the reconstructed image output by the synthetic transformation network 117. .

[0123] The above-mentioned at least one second processing is one second processing, or the above-mentioned at least one second processing is two second processings.

[0124] For example, such as Figure 15A As shown, the above-mentioned at least one second processing is a second processing. After the content gain attention block 1501 outputs the second output image features, it will undergo a second processing through the residual activation unit 806 and the 3×3 upsampling convolution 807, and will be processed based on the residual nonlocal attention block 808, and will undergo a second processing through the residual activation unit 809 and the upsampling convolution 810.

[0125] For example, such as Figure 15B As shown, the above-mentioned at least one second processing is actually two second processings. After the content gain attention block 1501 outputs the second output image features, it will undergo a second processing through the residual activation unit 804 and the 3×3 upsampling convolution 805, and then undergo another second processing through the residual activation unit 806 and the 3×3 upsampling convolution 807 (i.e., two second processings). Then, it will be processed based on the residual nonlocal attention block 808, and then undergo a second processing through the residual activation unit 809 and the upsampling convolution 810.

[0126] In the above embodiments, the synthetic transformation network incorporates a content gain attention module, which improves the network's ability to capture long-range correlations of pixels during the process of reconstructing the latent tensor into the output image. It also considers the influence of different parameters β on the image feature map and the influence of different parameters on different image content, thereby obtaining a more accurate output image and improving the performance of the JPEG AI verification model.

[0127] In the above embodiments of this application, the verification model of JPEG AI may include, for example: Figure 14A and Figure 14B Any analytical transformation network in the, and including such Figure 15A and Figure 15BBy embedding the content gain attention module into the analytical and synthetic transform networks in the JPEG AI validation model, the performance of the JPEG AI validation model during image encoding and decoding can be improved.

[0128] In some embodiments of this application, the residual nonlocal attention block in the analysis transformation network 101 described above is replaced by a content gain attention module (i.e., Figure 2 The residual nonlocal attention block 203 shown in the figure.

[0129] Figure 16A This is a schematic diagram of a content gain attention module replacing the residual nonlocal attention block in the analysis transformation network 101 described above, as provided in an embodiment of this application. Figure 16A exist Figure 2 Based on this, the content gain attention block 1601 was used to replace... Figure 2 The original residual nonlocal attention block 203.

[0130] Figure 16A Content gain attention block 1601 in the middle Figure 13 As shown in the image.

[0131] Figure 16A In the analysis and transformation network shown, the original image input to the verification model of the JPEG AI (i.e., the first image feature to be transformed in this case) is processed by a 3×3 downsampling convolution 201 and a residual activation unit 202 to obtain the first input image feature. The obtained first input image feature is used as the input image feature of the content gain attention block 1601. In the content gain attention block 1601, the image features are processed as follows: Figure 12 After processing the input image features as shown in the process, the first output image features output by the content gain attention module 1601 can be obtained. The first output image features are then further processed twice by a 3×3 downsampling convolution 204, a residual activation unit 205, a 3×3 downsampling convolution 206, and a residual activation unit 207. After that, they are processed sequentially by a 3×3 downsampling convolution 208 and a 1×1 convolution 209 to obtain the latent tensor corresponding to the original image (that is, the analysis and transformation image features in this application).

[0132] Figure 16B This is a flowchart illustrating an image feature processing method performed by an analysis transform network based on replacing residual nonlocal attention blocks with content gain attention modules, as provided in an embodiment of this application. This image feature processing method is applied to an encoding device and may include, but is not limited to, the following steps: 161. Obtain the features of the first image to be transformed.

[0133] The first image feature to be transformed can be a feature input to the analysis and transformation network. For example, such as... Figure 1 or Figure 9 As shown, the first image feature to be transformed can be the original image x input to the analysis and transformation network 101.

[0134] 162. Perform at least one first processing on the first image features to be transformed to obtain the first input image features.

[0135] The first process includes downsampling and residual activation.

[0136] In some embodiments of this application, performing at least one first processing on the first image features to be transformed may include: performing one first processing on the first image features to be transformed, or performing two downsampling and residual activation processing on the first image features to be transformed.

[0137] For example, such as Figure 16A The image shown illustrates the case of performing a first processing step on the first image feature to be transformed.

[0138] 163. Process the first input image features based on content gain attention blocks to obtain the first output image features.

[0139] The content gain attention block is used to apply attention weights to the content of the image. The process by which this content gain attention block processes the features of the first input image to obtain the features of the first output image is as described above. Figure 12 As shown.

[0140] 164. Perform at least one first processing step, as well as downsampling and convolution processing, on the first output image features to obtain the analyzed transformed image features.

[0141] The aforementioned analysis of transformed image features can be the output features of the analysis transform network. For example, such as... Figure 1 or Figure 9 As shown, the analyzed image features can be used as the latent tensor output by the analyzed transformation network 101. .

[0142] In some embodiments of this application, performing at least one first processing on the first output image features includes performing one first processing on the first output image features, or performing two first processing on the first output image features.

[0143] In some embodiments of this application, if the first image feature to be transformed is processed once in step 162, then the first output image feature can be processed twice in step 164; conversely, if the first image feature to be transformed is processed twice in step 162, then the first output image feature can be processed once in step 164. In the entire image feature processing method executed by the analysis and transformation network, three first processes can be performed, thus obtaining more accurate image features.

[0144] For example, such as Figure 16A The diagram shows a first processing step performed on the first image features to be transformed, and a second first processing step performed on the first output image features.

[0145] In the above embodiments, the analysis transform network replaces the residual nonlocal attention block with a content gain attention module. This improves the ability of the analysis transform network to capture long-range correlations of pixels during the process of transforming the input image into a latent tensor. It considers the influence of different parameters β on the image feature map and the influence of different parameters on different image content, thereby obtaining a more accurate latent tensor and improving the performance of the JPEG AI verification model.

[0146] In some embodiments of this application, the residual nonlocal attention block in the above-described synthesis transformation network 117 is replaced by a content gain attention module (i.e., Figure 8 The residual nonlocal attention block 808 shown in the figure.

[0147] Figure 17A This is a schematic diagram illustrating how a content gain attention module replaces a residual nonlocal attention block in a synthetic transform network 117, as provided in an embodiment of this application. Figure 17A exist Figure 8 Based on this, the content gain attention block 1701 was used to replace... Figure 8 The original residual nonlocal attention block 808.

[0148] Figure 17A Content gain attention block 1701 in the middle Figure 13 As shown in the image.

[0149] Figure 17AIn the synthetic transformation network shown, the latent tensor input to the synthetic transformation network in the JPEG AI verification model (i.e., the second image feature to be transformed in this application) is processed by residual block 801, residual block 802, 3×3 upsampling convolution 803, residual activation unit 804, 3×3 upsampling convolution 805, residual activation unit 806, and 3×3 upsampling convolution 807 to obtain the second input image feature. The second input image feature is then input to content gain attention block 1701, where it is processed by... Figure 12 After processing the second input image features as shown, the second output image features can be obtained. The second output image features are then further processed by the residual activation unit 809 and the 3×3 upsampling convolution 810 in sequence to obtain the reconstructed image (i.e., the synthetic transformation image features in this application).

[0150] Figure 17B This is a flowchart illustrating an image feature processing method performed by a synthetic transform network that replaces residual nonlocal attention blocks with content gain attention modules, as provided in an embodiment of this application. This image feature processing method is applied to a decoding device and may include, but is not limited to, the following steps: 171. Obtain the features of the second image to be transformed.

[0151] The second image feature to be transformed is the feature input to the synthetic transformation network. For example, such as... Figure 1 or Figure 9 As shown, the second image feature to be transformed can be a latent tensor input to the synthetic transformation network 117. .

[0152] 172. Perform at least one residual block processing, upsampling processing, and at least one second processing on the second image features to be transformed to obtain the second input image features.

[0153] For example, in some embodiments of this application, the at least one residual block used to process the second image features to be transformed may be two residual blocks; the upsampling process may be a process performed by a 3×3 upsampling convolution.

[0154] In some embodiments of this application, the above-mentioned at least one second process may include: one second process, or two second processes.

[0155] For example, such as Figure 17A The image shown illustrates the case where the first processing is performed twice on the features of the second image to be transformed.

[0156] 173. Process the features of the second input image based on the content gain attention block to obtain the features of the second output image.

[0157] The content gain attention block is used to apply attention weights to the content of the image. The process by which this content gain attention block processes the features of the second input image to obtain the features of the second output image is as described above. Figure 12 As shown.

[0158] 174. Perform at least one second processing on the second output image features to obtain the synthesized transformed image features.

[0159] The aforementioned synthesized image features can be the output features of the synthesized transform network. For example, such as... Figure 1 or Figure 9 As shown, the synthetic transformation image features can be the reconstructed image output by the synthetic transformation network 117. .

[0160] In some embodiments of this application, performing at least one second processing on the second output image features includes performing one second processing on the second output image features, or performing two first processing on the first output image features.

[0161] In some embodiments of this application, if the second image feature to be transformed is processed once in step 172, then the second output image feature can be processed twice in step 174; conversely, if the second image feature to be transformed is processed twice in step 172, then the second output image feature can be processed once in step 174. In the image feature processing method executed by the entire synthetic transform network, three first processes can be performed, thus obtaining more accurate image features.

[0162] For example, such as Figure 17A The diagram shows two first processing steps applied to the second image features to be transformed, and one first processing step applied to the first output image features.

[0163] In the above embodiments, the synthetic transformation network replaces the residual nonlocal attention block with a content gain attention module, which improves the ability of the synthetic transformation network to capture long-distance correlation of pixels in the process of reconstructing the latent tensor into the output image. It takes into account the influence of different parameters β on the image feature map and the influence of different parameters on different image content, thereby obtaining a more accurate output image and improving the performance of the JPEG AI verification model.

[0164] In the above embodiments of this application, the verification model of JPEG AI may include, for example: Figure 16A Analysis of transformation networks, and such as Figure 17AThe synthetic transform network in JPEG AI, by replacing the residual nonlocal attention block with a content gain attention module and embedding it into the JPEG AI verification model, can improve the performance of the JPEG AI verification model during image encoding and decoding.

[0165] In some embodiments of this application, not only is a content gain attention module used to replace the residual nonlocal attention block in the analysis transformation network 101 described above (i.e., ... Figure 2 The residual nonlocal attention block 203 shown in the figure is further embedded in the analysis transformation network 101 described above, and a content gain attention module is also embedded in the analysis transformation network 101 described above.

[0166] Figure 18A This is a schematic diagram illustrating the combination of replacement and embedding methods of a content gain attention module in an analysis transformation network 101, as provided in an embodiment of this application. Figure 18A In China Figure 2 Based on this, Figure 2 The residual nonlocal attention block 203 shown is replaced with a content gain attention module 1801, and a content gain attention module 1802 is embedded between the residual activation unit 205 and the 3×3 downsampling convolution 206. Both the content gain attention module 1801 and the content gain attention module 1802 are as follows: Figure 13 As shown in the image.

[0167] Figure 18A In the analysis and transformation network shown, the original image input to the verification model of the JPEG AI (i.e., the first image feature to be transformed in this application) is processed by a 3×3 downsampling convolution 201 and a residual activation unit 202 to obtain the first initial input image feature. This first initial input image feature is used as the input image feature of the content gain attention block 1801. In the content gain attention block 1801, the image features are processed as follows: Figure 12 After processing the first initial input image features as shown in the flowchart, the first initial output image features can be obtained. The first initial output image features are then processed by a 3×3 downsampling convolution 204 and a residual activation unit 205 to obtain the first re-input image features. These first re-input image features are then input into the image features of the content gain attention block 1802. The content gain attention block 1802 processes the image features as follows: Figure 12 After the process shown in the diagram processes the first re-input image features, the first re-output image features can be obtained. Then, the first re-output image features are further processed sequentially through a 3×3 downsampling convolution 206, a residual activation unit 207, a 3×3 downsampling convolution 208, and a 1×1 convolution 209 to obtain the latent tensor corresponding to the original image (i.e., the analysis-transformed image features in this application).

[0168] Figure 18BThis is a flowchart illustrating an image feature processing method based on a combination of content gain attention module embedding and replacement methods performed by an analysis transform network, as provided in this application embodiment. This image feature processing method is applied to an encoding device and may include, but is not limited to, the following steps: 181. Obtain the features of the first image to be transformed.

[0169] The first image feature to be transformed can be a feature input to the analysis and transformation network. For example, such as... Figure 1 or Figure 9 As shown, the first image feature to be transformed can be the original image x input to the analysis and transformation network 101.

[0170] 182. Perform a first process on the first image features to be transformed to obtain the first initial input image features.

[0171] For example, such as Figure 18A As shown, the 3×3 downsampling convolution 201 and the residual activation unit 202 perform a first processing on the first input image features to be transformed, so as to obtain the first initial input image features.

[0172] 183. The first initial input image features are processed based on the first content gain attention block to obtain the first initial output image features.

[0173] The first content gain attention block is used to apply attention weights to the content of the image. The process by which the first content gain attention block processes the features of the first initial input image to obtain the features of the first initial output image is as described above. Figure 12 As shown.

[0174] 184. Perform a first process on the first initial output image features to obtain the first re-input image features.

[0175] For example, such as Figure 18A As shown, the first content gain attention block is represented as content gain attention block 1801. The features of the first initial input image are input into content gain attention block 1801, and content gain attention block 1801 processes them as follows: Figure 12 After the input image features are processed by the process shown, the first initial output image features output by the content gain attention module 1801 can be obtained. The first initial output image features are then processed by the 3×3 downsampling convolution 204 and the residual activation unit 205 to obtain the first re-input image features.

[0176] 185. The first re-input image features are processed based on the second content gain attention block to obtain the first re-output image features.

[0177] The second content-gain attention block is used to apply attention weights to the content of the image. The process by which this second content-gain attention block processes the features of the first re-input image to obtain the features of the first re-output image is as described above. Figure 12 As shown.

[0178] 186. Perform first processing on the first re-output image features, including downsampling and convolution, to obtain the analyzed transformed image features.

[0179] The aforementioned analysis of transformed image features can be the output features of the analysis transform network. For example, such as... Figure 1 or Figure 9 As shown, the analyzed image features can be used as the latent tensor output by the analyzed transformation network 101. .

[0180] For example, such as Figure 18A As shown, the second content gain attention block is represented as content gain attention block 1802. The first re-input image features are used as the input image features of content gain attention block 1802. Content gain attention block 1802 then processes the input image features as follows: Figure 12 The process shown above processes the input image features to obtain the first output image features. The first output image features are then further processed sequentially through a 3×3 downsampling convolution 206, a residual activation unit 207, a 3×3 downsampling convolution 208, and a 1×1 convolution 209 to obtain the analysis and transformation image features.

[0181] In the above embodiments, the analysis transform network not only replaces the residual nonlocal attention block with the content gain attention module, but also embeds the residual nonlocal attention block in the analysis transform network. This improves the ability of the analysis transform network to capture long-range correlation of pixels in the process of transforming the input image into a latent tensor. It takes into account the influence of different parameters β on the image feature map and the influence of different parameters on different image content, thereby obtaining a more accurate latent tensor and improving the performance of the JPEG AI verification model.

[0182] In some embodiments of this application, not only is a content gain attention module used to replace the residual nonlocal attention block in the synthesis transform network 117, but a content gain attention module is also embedded in the synthesis transform network 117 (i.e., Figure 8 The residual nonlocal attention block 808 shown in the figure. Figure 19A This is a schematic diagram illustrating the combination of replacement and embedding methods of a content gain attention module in a synthesis transform network 117, as provided in an embodiment of this application. Figure 19A In China Figure 8 Based on this, Figure 8The residual nonlocal attention block 808 shown is replaced with a content gain attention module 1901, and a content gain attention module 1902 is embedded between the 3×3 upsampling convolution 805 and the residual activation unit 806. Both the content gain attention module 1902 and the content gain attention module 1901 are as follows: Figure 13 As shown in the image.

[0183] Figure 19A In the synthetic transformation network shown, the latent tensor input to the synthetic transformation network in the JPEG AI verification model (i.e., the second image feature to be transformed in this application) is processed by residual block 801, residual block 802, 3×3 upsampling convolution 803, residual activation unit 804, and 3×3 upsampling convolution 805 to obtain the second initial input image feature. The second initial input image feature is then input into content gain attention block 1902, where it is processed by residual block 801, residual block 802, 3×3 upsampling convolution 803, residual activation unit 804, and 3×3 upsampling convolution 805. Figure 12 The process shown above processes the second initial input image features to obtain the second initial output image features. The second initial output image features are then processed by the residual activation unit 806 and the 3×3 upsampling convolution 807 to obtain the second re-input image features. These features are then processed by the content gain attention block 1901 to obtain the second re-output image features. Finally, the second re-output image features are processed sequentially by the residual nonlocal attention block 808, the residual activation unit 809, and the 3×3 upsampling convolution 810 to obtain the reconstructed image (i.e., the synthesized transformed image features).

[0184] Figure 19B This is a flowchart illustrating an image feature processing method performed by a synthetic transform network based on a combination of content gain attention module embedding and replacement, as provided in an embodiment of this application. This image feature processing method is applied to a decoding device and may include, but is not limited to, the following steps: 191. Obtain the features of the second image to be transformed.

[0185] The second image feature to be transformed is the feature input to the synthetic transformation network. For example, such as... Figure 1 or Figure 9 As shown, the second image feature to be transformed can be a latent tensor input to the synthetic transformation network 117. .

[0186] 192. Perform at least one residual block processing, upsampling processing, and a second processing on the second image features to be transformed to obtain the second initial input image features.

[0187] For example, in some embodiments of this application, the at least one residual block used to process the second image features to be transformed may be two residual blocks; the upsampling process may be a process performed by a 3×3 upsampling convolution.

[0188] For example, such as Figure 19A As shown, the second image feature to be transformed is input into the synthesis transformation network. The second image feature to be transformed will undergo two residual processing steps through residual block 801 and residual block 802. After that, it will undergo upsampling processing through 3×3 upsampling convolution 803. Then, it will undergo the second processing through residual activation unit 804 and 3×3 upsampling convolution 805 to obtain the second initial input image feature.

[0189] 193. The second initial input image features are processed based on the third content gain attention block to obtain the second initial output image features.

[0190] The third content-gain attention block is used to apply attention weights to the image content. The process by which this third content-gain attention block processes the second initial input image features to obtain the second initial output image features is as described above. Figure 12 As shown.

[0191] 194. Perform a second processing on the second initial output image features to obtain the second re-input image features.

[0192] like Figure 19A As shown, the third content gain attention block is in Figure 19A The content gain attention block 1902 is used in the representation. The content gain attention block 1902 is represented by, for example... Figure 12 The process shown above processes the second initial output image features to obtain the second initial output image features. The output image features output by the content gain attention module 1902 are then processed by the residual activation unit 806 and the 3×3 upsampling convolution 807 to obtain the second re-input image features.

[0193] 195. The second re-input image features are processed based on the fourth content gain attention block to obtain the second re-output image features.

[0194] The fourth content-gain attention block is used to apply attention weights to the image content. The process by which this fourth content-gain attention block processes the second re-input image features to obtain the second re-output image features is as described above. Figure 12 As shown.

[0195] like Figure 19A As shown, the fourth content gain attention block is in Figure 19A The content gain attention block 1901 is used to represent this, and the content gain attention block 1901 is represented by, for example, Figure 12 The process shown above processes the second re-input image features to obtain the second re-output image features.

[0196] 196. Perform a second processing on the second output image features to obtain the synthetic transformed image features.

[0197] like Figure 19A As shown, the second output image features are processed by the residual activation unit 809 and the upsampling convolution 810 to synthesize the transformed image features.

[0198] The aforementioned synthesized image features can be the output features of the synthesized transform network. For example, such as... Figure 1 or Figure 9 As shown, the synthetic transformation image features can be the reconstructed image output by the synthetic transformation network 117. .

[0199] In the above embodiments, the synthetic transformation network not only replaces the residual nonlocal attention block with the content gain attention module, but also embeds the residual nonlocal attention block in the synthetic transformation network. This improves the synthetic transformation network's ability to capture long-range correlations of pixels in the process of reconstructing the latent tensor into the output image. It takes into account the influence of different parameters β on the image feature map and the influence of different parameters on different image content, thereby obtaining a more accurate output image and improving the performance of the JPEG AI verification model.

[0200] In the above embodiments of this application, the verification model of JPEG AI may include, for example: Figure 18A Analysis of transformation networks, and such as Figure 19A The synthetic transform network in JPEG AI, by replacing the residual nonlocal attention block with a content gain attention module and embedding it into the JPEG AI verification model, can improve the performance of the JPEG AI verification model during image encoding and decoding.

[0201] Among them, the first image feature, second image feature, third image feature, fourth image feature, fifth image feature, sixth image feature, seventh image feature, eighth image feature, ninth image feature, and tenth image feature are used to distinguish the image features obtained after various processing of the image.

[0202] It should be noted that, in the above-mentioned... Figure 13 The content gain attention module shown is used in analysis transform networks, such as... Figure 12 The method flow shown can be applied to encoders; in the above... Figure 13 When the content gain attention module shown is applied in a synthesis transform network, such as Figure 12The method flow shown can be applied to the decoder.

[0203] This application provides a decoding device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the image feature processing method described in the above method embodiments.

[0204] For example, Figure 20 This is a schematic diagram of a decoding device provided in an embodiment of the present application. The decoding device includes: a processor 2001 and a memory 2002. Regarding the case where the decoding device is an encoding device: In some embodiments, the processor 2001 is configured to: acquire first image features to be transformed; perform a first processing, a residual nonlocal attention block-based processing, and at least one first processing on the first image features to be transformed to acquire first input image features, wherein the first processing includes downsampling processing and residual activation processing; process the first input image features based on a content gain attention block to obtain first output image features, wherein the content gain attention block is used to perform attention weighting on the content of the image; and perform downsampling processing and convolution processing on the first output image features to obtain analytically transformed image features.

[0205] In some embodiments, the at least one first process is one first process, or the at least one first process is two first processes.

[0206] In some embodiments, the at least one first process is a single first process; The processor 2001 is specifically configured to: perform downsampling and convolution processing on the first output image features to obtain the analyzed transformed image features, including: performing the first processing, downsampling, and convolution processing on the first output image features to obtain the analyzed transformed image features.

[0207] In some embodiments, the processor 2001 is specifically configured to: process the first input image features based on content gain attention blocks to obtain first output image features, including: obtaining parameters... The first input image features are used as input to the fully connected layer, and the content gain weights of the fully connected layer are used as input. Feature extraction is performed on the first input image features using at least one residual block to obtain first extracted image features. First content image features corresponding to the first input image features are extracted. Channel dimension operations are performed based on the first content image features and the first content gain weights to obtain a first joint spatial channel attention mask. Element-wise multiplication is performed on the first extracted image features and the first joint spatial channel attention mask to obtain a first target image feature. The first output image feature is generated based on the first target image feature and the first input image features.

[0208] In some embodiments, the processor 2001 is specifically configured to: extract the first content image features corresponding to the first input image features, including: extracting content features from the first input image features through at least one moving window attention module to obtain the content image features corresponding to the first input image features.

[0209] In some embodiments, the at least one moving window attention module is two moving window attention modules, and the at least one residual block is three residual blocks.

[0210] Regarding the case where the decoding device is a decoding device: In some embodiments, the processor 2001 is configured to: acquire a second image feature to be transformed; perform at least one residual block processing and upsampling processing on the second image feature to be transformed to acquire a second input image feature; process the second input image feature based on a content gain attention block to obtain a second output image feature, wherein the content gain attention block is used to perform attention weighting on the content of the image; perform at least one second processing on the second output image feature, processing based on a residual nonlocal attention block, and performing the second processing to obtain a synthesized transformed image feature.

[0211] In some embodiments, the at least one second process is one second process, or the at least one second process is two second processes.

[0212] In some embodiments, the at least one second process is a single second process; The processor 2001 is specifically configured to perform at least one residual block processing and upsampling processing on the second image features to be transformed to obtain the second input image features, including: performing at least one residual block processing, upsampling processing, and a second processing on the second image features to be transformed to obtain the second input image features.

[0213] In some embodiments, the processor 2001 is specifically configured to: process the second input image features based on content gain attention blocks to obtain second output image features, including: obtaining the content gain weights of the fully connected layer output when parameter β is the input of the fully connected layer; extracting features from the second input image features through at least one residual block to obtain second extracted image features; extracting second content image features corresponding to the second input image features; performing channel dimension operations based on the second content image features and the content gain weights to obtain a second joint spatial channel attention mask; performing element-wise multiplication operations on the second extracted image features and the second joint spatial channel attention mask to obtain second target image features; and generating second output image features based on the second target image features and the second input image features.

[0214] In some embodiments, the processor 2001 is specifically configured to: extract the second content image features corresponding to the second input image features, including: extracting content features from the second input image features through at least one moving window attention module to obtain the second content image features corresponding to the second input image features.

[0215] In some embodiments, the at least one mobile window attention module is two mobile window attention modules.

[0216] This application provides a computer-readable storage medium in some embodiments, characterized in that a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, it implements the image feature processing method described in the above-described method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here. In this application, the computer-readable medium includes permanent and non-permanent, removable and non-removable storage media. The storage medium can implement information storage by any method or technology, and the information can be computer-readable instructions, data structures, program modules, or other data. The computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0217] Some embodiments of this application provide a computer program product that stores a computer program. When the computer program is executed by a processor, it implements the image feature processing method in the above-described method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0218] In this application, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0219] In this application, the memory may include non-permanent memory in a computer-readable medium, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM.

[0220] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0221] The above are merely specific embodiments of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image feature processing method for an encoding device, characterized in that, include: Obtain the features of the first image to be transformed; The first image features to be transformed are first processed, then processed based on residual nonlocal attention blocks, and then processed at least once more to obtain the first input image features. The first processing includes downsampling processing and residual activation processing. Obtain the first content gain weight of the fully connected layer output when parameter β is used as input to the fully connected layer; wherein, parameter β is used to specify the weight between rate and distortion in the loss function during training; The first input image features are extracted by using at least one residual block to obtain the first extracted image features; Extract the first content image features corresponding to the features of the first input image; Based on the first content image features and the first content gain weight, channel dimension operations are performed to obtain the first joint spatial channel attention mask; The first extracted image features and the first joint spatial channel attention mask are multiplied element-wise to obtain the first target image features; Generate a first output image feature based on the first target image feature and the first input image feature; The features of the first output image are downsampled and convolved to obtain the analyzed and transformed image features.

2. The image feature processing method according to claim 1, characterized in that, The at least one first process is one first process, or the at least one first process is two first processes.

3. The image feature processing method according to claim 2, characterized in that, The at least one first process is one first process; The step of performing downsampling and convolution processing on the features of the first output image to obtain the analyzed transformed image features includes: The first output image features are subjected to the first processing, downsampling processing, and convolution processing to obtain the analyzed and transformed image features.

4. The method according to claim 1, characterized in that, The step of extracting the first content image features corresponding to the first input image features includes: Content features are extracted from the first input image features using at least one moving window attention module to obtain the content image features corresponding to the first input image features.

5. The method according to claim 4, characterized in that, The at least one moving window attention module is two moving window attention modules, and the at least one residual block is three residual blocks.

6. An image feature processing method for a decoding device, characterized in that, include: Obtain the features of the second image to be transformed; The second image features to be transformed are processed by at least one residual block and upsampling process to obtain the second input image features; Obtain the second content gain weight of the fully connected layer output when parameter β is used as input to the fully connected layer; wherein, parameter β is used to specify the weight between rate and distortion in the loss function during training; The second input image features are extracted by using at least one residual block to obtain the second extracted image features; Extract the second content image features corresponding to the features of the second input image; Based on the second content image features and the second content gain weight, channel dimension operations are performed to obtain the second joint spatial channel attention mask; The second extracted image features and the second joint spatial channel attention mask are multiplied element-wise to obtain the second target image features; Based on the second target image features and the second input image features, generate the second output image features; The second output image features are subjected to at least one second processing step, which involves processing based on residual nonlocal attention blocks and performing the second processing step to obtain synthetic transformed image features. The second processing step includes residual activation processing and upsampling processing.

7. The method according to claim 6, characterized in that, The at least one second process is one second process, or the at least one second process is two second processes.

8. The method according to claim 7, characterized in that, The at least one second process is one second process; The process of performing at least one residual block processing and upsampling processing on the second image features to be transformed to obtain the second input image features includes: The second image features to be transformed are subjected to at least one residual block processing, upsampling processing, and a second processing to obtain the second input image features.

9. The method according to claim 6, characterized in that, The step of extracting the second content image features corresponding to the second input image features includes: Content features are extracted from the second input image features using at least one moving window attention module to obtain the second content image features corresponding to the second input image features.

10. The method according to claim 9, characterized in that, The at least one mobile window attention module is two mobile window attention modules.

11. An encoding device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the image feature processing method as described in any one of claims 1 to 5.

12. A decoding device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the image feature processing method as described in any one of claims 6 to 10.

Images