Image processing method, electronic device, storage medium and program product

Abstract

The invention provides an image processing method, electronic equipment, a storage medium and a program product, and relates to the technical field of image processing. The method comprises the following steps: acquiring a code stream, wherein the code stream is obtained by performing entropy coding on a potential variable of a target image; performing entropy decoding on the code stream to obtain a potential variable; the denoising mode of the potential variable is determined according to the frequency characteristic of the potential variable, denoising decoding is conducted on the potential variable according to the denoising mode, reconstructed image information is obtained, the frequency characteristic of the potential variable is used for representing the pixel value change rate of the potential variable, and the reconstructed image information is used for reconstructing the target image. According to the technical scheme, denoising decoding in different denoising modes is carried out on potential variables with different frequency characteristics, and the overall perception quality of a user is improved while image details and texture definition are reserved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of network and mobile communication technologies, the demand for image processing applications is increasing, such as live streaming, video-on-demand, real-time communication (RTC), media player playback, and image conferencing. To reduce the transmission and storage costs of image data, compression is typically required. Traditional image compression techniques focus on minimizing file size while preserving as much image information as possible. However, this approach often ignores the differences in human perception of image quality, especially at the texture and detail levels, even at high compression ratios, leading to noticeable distortion and artifacts. Summary of the Invention

[0003] This application provides an image processing method, an electronic device, a computer-readable storage medium, and a computer program product to improve image quality at the texture and detail levels.

[0004] In a first aspect, embodiments of this application provide an image processing method, comprising: acquiring a bitstream, wherein the bitstream is obtained by entropy encoding of latent variables of a target image; performing entropy decoding on the bitstream to obtain the latent variables; determining a denoising method for the latent variables based on the frequency characteristics of the latent variables, and performing denoising decoding on the latent variables according to the denoising method to obtain reconstructed image information, wherein the frequency characteristics of the latent variables are used to represent the rate of change of pixel values ​​of the latent variables, and the reconstructed image information is used to reconstruct the target image.

[0005] Secondly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0006] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0007] Fourthly, embodiments of this application provide a computer program product, including a computer program, which, when executed by a processor, implements any of the methods described in the embodiments of this application.

[0008] In the technical solution of this application embodiment, the encoding end extracts latent variables from the target image (compression encoding) and encodes the latent variable entropy into a bitstream; the decoding end is used to decode the bitstream entropy into latent variables and perform targeted denoising decoding processing according to the frequency characteristics of the latent variables. Different denoising decoding methods can be used for latent variables with different frequency characteristics to fully capture the differences in human eye perception of image quality, especially the differences in perception of high-frequency regions, thereby improving the overall perceived quality of the user while preserving image details and texture clarity.

[0009] Optionally, the checkerboard-based attention mechanism is optimized into a self-attention mechanism based on receptive field perception. This is achieved by utilizing learnable Laplace-shaped relative position encoding to compute local and global contextual information. Thus, each channel block has its own receptive field in the self-attention computation, which dynamically adjusts according to changes in channel block entropy. This allows for dynamic adaptation to the characteristics of different data blocks, enabling faster and more accurate processing of complex data structures and achieving rapid decoding.

[0010] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0012] Figure 1 A system architecture diagram of an exemplary scenario for the image processing method provided in the embodiments of this application;

[0013] Figure 2 A flowchart of the image processing method provided in the embodiments of this application;

[0014] Figure 3 This is a schematic diagram illustrating the application of context information in the decoding process of the i-th channel block;

[0015] Figure 4A , Figure 4B , Figure 4C and Figure 4D A schematic diagram illustrating the generation process of Laplace relative position encoding is shown;

[0016] Figure 5 This is a schematic diagram of the functional modules of the image processing method according to an embodiment of this application;

[0017] Figure 6 This is a schematic diagram of the network structure of the image processing method according to an embodiment of this application.

[0018] Figure 7A This is a flowchart of the encoding method according to an embodiment of this application;

[0019] Figure 7B This is a flowchart of the decoding method according to an embodiment of this application;

[0020] Figure 8 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0021] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0022] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and they all fall within the protection scope of the embodiments of this application.

[0023] To more clearly illustrate the image processing method provided in the embodiments of this application, we first introduce the application scenarios that can be used to implement the method.

[0024] The technical solution of this application can be applied to scenarios that require image compression and high-quality requirements for the details of the reconstructed image, such as high-definition image transmission, high-definition streaming media playback or real-time communication, medical image analysis, etc. Figure 1 This is a system architecture diagram of an exemplary scenario for the image processing method provided in this application. The encoding end is used to extract latent variables from the target image and encode the latent variable entropy into a bitstream; the decoding end is used to decode the bitstream entropy into latent variables and perform targeted denoising decoding processing based on the frequency characteristics of the latent variables to reconstruct the target image.

[0025] Frequency characteristics represent the rate of change of pixel values. Based on different frequency characteristics, latent variables can be classified into low-frequency latent variables or high-frequency latent variables. Therefore, different denoising decoding methods can be used for latent variables with different frequency characteristics to fully capture the differences in human perception of image quality, especially the differences in perception of high-frequency regions. This achieves the goal of improving the overall perceived quality for users while preserving image details and texture clarity.

[0026] For example, the encoding and decoding ends can be applications, services, instances, software modules, virtual machines (VMs), containers, or cloud servers, or hardware devices (such as servers or terminal devices) or hardware chips with data processing capabilities. Terminal devices can be fixed terminals, such as personal computers, or mobile terminals, such as mobile phones or tablets. Furthermore, the encoding and decoding ends can be deployed on the same side. For example, they can be deployed as software modules in the same terminal device, or they can be deployed as cloud products in the same cloud server or cluster.

[0027] It should be noted that the application scenarios or examples provided in the embodiments of this application are for ease of understanding, and the embodiments of this application do not specifically limit the application of the technical solutions. In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0028] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0029] Figure 2 This is a flowchart illustrating an image processing method provided in an embodiment of this application. This method can be applied to an image processing apparatus, which can be deployed at a decoding end. Figure 2 As shown, the image processing method includes:

[0030] Step S201: Obtain the bitstream, which is obtained by entropy encoding of the latent variables of the target image.

[0031] The execution entity in this embodiment can be a decoding end, which can obtain the bitstream from the encoding end. The target image x is the original image input to the encoding end, and may include one or more channels. Channels may include color channels, depth channels, or other types of channels. The target image x is typically represented as a multidimensional array. The latent variable y is a tensor with shape (C, H, W), where C is the number of channels, and H and W are the height and width. This tensor contains multi-level, multi-scale features extracted from the target image x, including global features (such as background) and local features (such as texture and edges).

[0032] The encoding layer inputs the target image or a preprocessed target image into a Convolutional Neural Network (CNN) for feature extraction and representation. This process, known as compression or encoding, can be viewed as a mapping of the target image from a high-dimensional space to a low-dimensional space, aiming to retain the most important information while removing redundant information. For example, the CNN convolutional layers progressively learn feature maps representing the edges, textures, and higher-level abstractions of the target image x from its raw pixel values. Pooling layers reduce the size of these feature maps, decreasing computational cost and the risk of overfitting. Fully connected layers integrate these feature maps to form the feature representation of the target image x in the latent space, i.e., the latent variable y.

[0033] The encoding end performs quantization and entropy encoding on the latent variable y. For example, the quantization process maps the continuous feature value of the latent variable y to a finite discrete set, thereby further reducing the amount of data. The quantized latent variable can be represented as... Right now Entropy coding, for example, is arithmetic coding (AE). For instance, the encoder utilizes quantized latent variables. The probability distribution in the target image, for the quantized latent variables Each quantization symbol in the image is entropy encoded, and symbols with higher probability of occurrence are given shorter codewords, thereby outputting a bitstream to further compress the target image.

[0034] The probability information includes quantified latent variables. The probability distribution in the target image can be output by the entropy model. Entropy coding can also be other types of coding, which are not limited in this application.

[0035] Step S202: Perform entropy decoding on the bitstream to obtain the latent variables.

[0036] Entropy decoding can be viewed as the inverse process of entropy encoding. When entropy encoding is arithmetic encoding, entropy decoding corresponds to arithmetic decoding (AD). For example, the decoding end uses probabilistic information to recover the quantized latent variables from the bitstream. And further restored to the latent variable y.

[0037] Step S203: Determine the denoising method of the latent variables based on the frequency characteristics of the latent variables, and perform denoising decoding on the latent variables according to the denoising method to obtain the reconstructed image information, wherein the reconstructed image information is used to reconstruct the target image.

[0038] The frequency characteristics of latent variables are used to represent the rate of change of pixel values ​​of latent variables. Based on different frequency characteristics, latent variables can be classified into low-frequency latent variables and high-frequency latent variables.

[0039] For example, low-frequency regions in a target image, such as the sky, walls, and background, describe the overall structure or global content of the target image. These regions are typically large, and therefore, their pixel values ​​change slowly. The features of low-frequency regions in the latent space are represented as low-frequency latent variables. These variables often extract the overall or global content of the target image, containing the most salient information such as basic object information, scene information, or layout information. They are crucial for understanding the "big picture" of the target image; therefore, low-frequency latent variables can be called semantic latent variables. High-frequency regions in a target image, such as object edges, textures, and details, describe the local structure or detailed content of the target image. Therefore, their pixel values ​​change rapidly. The features of high-frequency regions in the latent space are represented as high-frequency latent variables. These variables often extract the local structure or detailed content of the target image, containing texture information such as edges and texture; they can be called texture latent variables.

[0040] Semantic latent variables are widely used in scenarios requiring a high level of abstraction, such as image content analysis and image retrieval. Texture latent variables, representing image information, have a significant impact on human visual perception of quality; however, due to their complexity and subtle differences, they are often difficult to fully capture using semantic latent variables.

[0041] Neural codecs in image compression typically have two main objectives: minimizing distortion between the original input target image and its reconstructed image; and reducing the bitrate for efficient data transmission or storage. Most existing neural codecs employ Gaussian decoders, primarily based on mean squared error (MSE) or mean absolute error (MAE) as loss functions, optimizing the model to reduce the difference between the reconstructed and original images. However, these metrics often fail to adequately capture the perceptual differences in image quality perceived by the human eye, especially at the texture and detail levels. Therefore, simply optimizing the bitrate and distortion may result in a blurry reconstructed image.

[0042] In the technical solutions of this application, different denoising methods are used to decode latent variables with different frequency characteristics. Exemplarily, the denoising decoding process of latent variables typically includes two processes: a blurring process and a denoising process. The blurring process transforms the latent variables into almost pure Gaussian noise over multiple time steps, and latent variables with different frequency characteristics can have different blurring rates and intensities. The denoising process in the frequency space can be understood as a deblurring process in the time domain. During the denoising process, different denoising rates and intensities can be executed for latent variables with different frequency characteristics (such as high-frequency texture latent variables and low-frequency speech latent variables), thereby learning the appropriate denoising method for latent variables with different frequency characteristics. This fully captures the differences in human perception of image quality, especially the differences in perception of high-frequency regions, thereby improving the overall perceived quality for the user while preserving image details and texture clarity.

[0043] It is understood that the embodiments of this application adopt an asymmetric encoding and decoding architecture, which allows different processing methods to be used for semantic latent variables and texture latent variables. That is, semantic latent variables undergo deeper blurring and denoising, while texture latent variables retain their complete information to ensure accurate reconstruction of image details. This can preserve the texture details of the original target image with minimal information loss during the encoding and decoding process, thereby effectively maximizing encoding and decoding efficiency without sacrificing image quality.

[0044] In one embodiment, in step S203, determining the denoising method of the latent variable based on its frequency characteristics and denoising and decoding the latent variable according to the denoising method may include: denoising and decoding the latent variable using a diffusion model, wherein the denoising method includes a denoising rate, and the diffusion model is used to schedule the denoising rate of the latent variable according to its frequency characteristics.

[0045] The diffusion model processes input data differently based on its direction or specific attributes, exhibiting varying responses or sensitivities to content with different orientations or structures. In this embodiment, the denoising rate can be understood as the diffusion rate within the diffusion model. During training, the diffusion model is subjected to inductive bias that distinguishes different frequency characteristics; that is, for latent variables with different frequency characteristics, the diffusion model schedules different diffusion rates to achieve different denoising rates. For example, semantic latent variables, as low-frequency latent variables, can be denoised earlier or faster during the denoising process to prioritize the reconstruction of the global information of the target image, helping the diffusion model better understand the general outline and semantic integrity of the target image. Texture latent variables, as high-frequency latent variables, may require slower diffusion to finely and gradually restore the texture and detail information in the target image.

[0046] Compared with traditional Gaussian decoders, the diffusion model in this application embodiment, as an isotropic model, can modify the decoding process based on traditional Gaussian decoders into a diffusion process. During the diffusion process, the diffusion model controls latent variables with different frequency characteristics (such as semantic latent variables and texture latent variables) to diffuse at different denoising rates, gradually reconstructing the target image. This allows for more flexible handling of image uncertainty and diversity, especially in detail reconstruction, where it can more accurately process complex signals and generate diverse image details. While ensuring the overall perceived quality for users, it generates clear and detailed reconstructed images.

[0047] In one implementation, the latent variables are denoised using a diffusion model, which includes: performing multi-round iterative denoising decoding on the latent variables using a diffusion model, wherein the diffusion model is used to infer the reconstructed image information of the latent variables in the next round based on probability information and the reconstructed image information of the latent variables in the current round in each round of iteration, the probability information including the distribution probability of the latent variables in the target image, and the probability information is used to determine the denoising rate.

[0048] In the diffusion model, the denoising process in the frequency space can be understood as the deblurring process in the time domain, which can be expressed as:

[0049]

[0050] Among them, f t It is the state of the reconstructed image information corresponding to the latent variables at time t; f t-1 This is the state of the reconstructed image information after denoising, i.e., the output of the next step; the conditional distribution of the diffusion model is a Gaussian distribution, μ t→t-1 It is the mean from time t to t-1. It is the square matrix of variance, representing the uncertainty in the Gaussian distribution; f x These are conditional features used to guide state updates during the denoising process.

[0051] The model parameters θ of the diffusion model can be obtained through training. By re-optimizing the above formula using the model parameters θ, the diffusion model P can be obtained. θ for:

[0052]

[0053] in,

[0054]

[0055] Among them, f ∈ It is the noise term added in each iteration; α t These are the adjustment parameters used in each iteration of the denoising process to control the scaling during denoising; σ t It is a noise-related parameter that defines the degree of noise impact at a given moment.

[0056] In this embodiment of the application, α t It is the control parameter in each iteration of the denoising process. This control parameter is derived by combining the probability information output by the entropy model. Therefore, the probability information can be used to determine the denoising rate.

[0057] The entropy model is responsible for modeling (determining) the probabilistic information z, which includes the distribution probabilities of each latent variable in the target image, such as the mean μ and variance σ among the latent variables. The distribution probabilities reflect the probability distribution of pixel values ​​appearing in the target image. As prior knowledge, the probabilistic information z provides rich statistical features for the entropy encoding and decoding processes. These statistical features, as prior inputs to the diffusion model, can be used to guide the diffusion model in selecting the denoising rate that best matches the image's statistical features during the denoising process. This optimizes the denoising direction at each step, controls the diffusion of reconstructed image information towards the original target image, and makes the generation of reconstructed image information more robust and accurate.

[0058] In one embodiment, the denoising method includes a denoising rate. In step S203, the denoising method of the latent variable is determined according to the frequency characteristics of the latent variable, and the latent variable is denoised and decoded according to the denoising method to obtain reconstructed image information. This includes: determining the denoising rate of the latent variable according to the frequency characteristics of the latent variable; and decoding the latent variable into reconstructed image information according to the denoising rate.

[0059] Among them, "determining the denoising rate of latent variables based on the frequency characteristics of latent variables" can include using a diffusion model to schedule the denoising rate of latent variables, modifying the traditional Gaussian decoder-based decoding process into a diffusion process. In this way, during the diffusion process, latent variables with different frequency characteristics (such as semantic latent variables and texture latent variables) are controlled to gradually reconstruct the image at different denoising rates, which can more effectively generate clear and detailed reconstructed images.

[0060] In one implementation, decoding latent variables into reconstructed image information includes: dividing the latent variables into multiple channel blocks, each channel block including at least one channel of the latent variables; and decoding each channel block sequentially in descending order of entropy information, wherein the decoding of the current channel block is based on the context information of the previously decoded channel blocks.

[0061] Latent variables can be divided into multiple channel blocks along the channel dimension, with each channel block containing at least one channel. This division can take the form of uneven grouping, meaning the number of channels in each channel block can be different. For example, a latent variable with M channels can be divided into 5 channel blocks: the first channel block contains 16 channels, such as channels 1 to 16 of the latent variable; the second channel block contains 16 channels, such as channels 17 to 32 of the latent variable; the third channel block contains 32 channels, such as channels 33 to 64 of the latent variable; the fourth channel block contains 64 channels, such as channels 65 to 128 of the latent variable; and the fifth channel block contains M-128 channels, such as channels 129 to M of the latent variable.

[0062] Entropy information is typically used to measure the amount of information or complexity of different channels within a latent variable. A higher entropy information in a channel indicates more complex data or more information contained within that channel; this is called a high-entropy channel. Conversely, a lower entropy information in a channel indicates simpler data or less information contained within that channel; this is called a low-entropy channel. Channel blocks are decoded sequentially according to their entropy information from highest to lowest. The decoding of the current channel block is based on the context information of already decoded channel blocks. This ensures that high-entropy channels with more information are decoded before low-entropy channels with less information, thereby improving the efficiency and quality of the entire decoding process.

[0063] In one implementation, a channel block includes calibrated and uncalibrated channels for latent variables. "Calibration" can also be called "anchoring." A channel block contains both calibrated and uncalibrated channels. The calibration method can be random, meaning the calibrated channels can be randomly selected, but this application embodiment is not limited to this.

[0064] Furthermore, the decoding of calibrated channels is based on the context information of the decoded channel blocks, while the decoding of non-calibrated channels is based on the context information of the decoded channel blocks and the context information of each decoded calibrated channel. For example, embodiments of this application provide a context understanding module that generates context information, which can be part of an entropy model. In each channel block, the context understanding module can model the distribution correlation of the channel dimension and the distribution correlation of the spatial dimension, respectively outputting channel context information and spatial context information (including local context information, latent variables, and global context information of the channel block).

[0065] The context understanding module employs a parallel, bidirectional spatial context generation architecture, capable of capturing the local and global spatial relationships of channel blocks. Specifically, the decoding of calibrated channels is based on prior information and the context information of already decoded channel blocks, while the decoding of non-calibrated channels is based on prior information, the context information of already decoded channel blocks, and the context information of each decoded calibrated channel.

[0066] Figure 3 This diagram illustrates the application of context information during the decoding process of the i-th channel block. It can be understood that the decoding process typically targets the quantized latent variables... Figure 3 In the diagram, the i-th channel block is represented as Figure 3 The left side represents the extraction of context information for the calibration channel. Figure 3 The right side represents the extraction of context information for non-calibrated channels.

[0067] The context understanding module can include multiple sub-modules, each extracting a type of context information. For example, the local context information sub-module ( Figure 3 The part marked "gl" is used to extract local context information, while the global context information submodule ( Figure 3 The module marked "gs" is used to extract global context information, and the channel context information submodule ( Figure 3 The module marked "gch" is used to extract channel context information, and the prior information processing module ( Figure 3 The symbol "hs" is used to process prior information.

[0068] (1) Process of extracting context information from the calibration channel:

[0069] The inputs include: zeros, representing the initial padding zeros used to initialize channel information; and channel context information. This represents the context information of all channels that have been decoded before the decoding of the i-th channel block; This represents hyperprior information, such as the probability information output by an entropy model.

[0070] The fully connected layer concatenates the channel context information extracted by the gch submodule (all channels that have been decoded before the i-th channel block is decoded) and the prior information processed by the hs submodule to obtain the concatenated information and outputs it to the distribution probability modeling module of the entropy model (marked as "gep" in the figure). The gep module processes and outputs the probability information, including the mean μ and variance σ in the Gaussian distribution parameters.

[0071] The output is the context information extracted after decoding the calibration channel. This refers to the local context information of the calibration channel.

[0072] (2) Context information extraction process for non-calibrated channels:

[0073] Inputs include: local context information of the calibration channel. Convolutional generation of context information after decoding the calibration channels; global context information Channel context information Prior information The output is the context information extracted after decoding the non-calibrated channel. This refers to the local context information of non-calibrated channels.

[0074] The local context information of the non-calibrated channel is then input into the global context information submodule to extract the global context information of the latent variables.

[0075] In one embodiment, the method of this application may further include: calculating local context information of the channel block and global context information of the latent variables; using the local context information and global context information to determine probability information, the probability information including the distribution probability of the latent variables in the target image, and the probability information being used for entropy encoding and entropy decoding.

[0076] The local context information includes the local context information of the calibrated channel and the local context information of the non-calibrated channel. The global context information of the potential channel is obtained based on the local context information of the non-calibrated channel. The local context information of the non-calibrated channel is obtained based on the local context information of the calibrated channel and the global context information of the potential channel, thus forming the entropy model structure of cyclic iterative channel-by-channel extraction.

[0077] This channel-by-channel extraction entropy model can fully utilize the channel context information of each channel, as well as the local and global context information of each channel block, thereby accurately modeling the spatial correlation and channel correlation in the latent space, facilitating the accurate capture of the distribution probability of latent variables, and providing accurate probability information.

[0078] In one implementation, calculating the local context information of the channel block and the global context information of the latent variables may include: calculating the local context information and the global context information based on the Laplace relative position encoding of the channel block, wherein the Laplace relative position encoding includes the relative position information of each element in the channel block and the receptive field adjustment parameter, the receptive field adjustment parameter being used to adjust the receptive field of the channel block.

[0079] In this embodiment, the local context information submodule and the global context information submodule are deployed as a network structure based on Laplacian relative position encoding and self-attention mechanism, which can be specifically represented as follows:

[0080] Atten(Q,K,V)=softmax(QK T ⊙m+P lap )V.

[0081] In this model, the query (Q) matrix, key (K) matrix, and value (V) matrix represent the inputs to the self-attention mechanism. The self-attention mechanism uses the query Q and key K to calculate attention weights and applies these weights to the value V to output the result. M represents the checkerboard masking matrix, used for positional bias in the self-attention calculation; ⊙ represents the element-wise multiplication operation; P... lap This represents the Laplace relative position code, a learned term for positional bias used to enhance the attention mechanism's perception of spatial location information. The Laplace relative position code can be represented as:

[0082]

[0083] Where A represents the receptive field adjustment parameter, which controls the amplitude of the position encoding; σ represents the receptive field adjustment parameter, which controls the range of the position encoding. |x| and |y| represent the relative positions of the latent variables in the latent space, used for calculating the position bias.

[0084] Taking a specific channel block as an example, this channel block contains a spatial window of size N*N (e.g., ...). Figure 4A (As shown). Creating the Laplace relative position encoding for this channel block can include: generating a two-dimensional relative position table, such as... Figure 4B As shown, each coordinate of the relative position is in the range [-N+1, N-1]; the relative position matrix is ​​derived for each small cell in the spatial window, as follows. Figure 4C As shown. The relative distance coordinates of the first small block are (0,0) (relative distance from its position, located at...). Figure 4B The top left corner of the orange box, and the relative distance (0,0) of the last small block is located at... Figure 4B The bottom right corner of the green box; flatten each relative position matrix and stack them together. The unfolded relative position matrix is ​​as follows: Figure 4C As shown; applying the two-dimensional Laplace function to each element of the relative position matrix generates a Laplace relative position code, as follows. Figure 4D As shown.

[0085] Based on this, the chessboard-based attention mechanism is optimized into a self-attention mechanism that utilizes receptive field perception. This is achieved by employing learnable Laplace-shaped relative position encoding to compute local and global contextual information. Consequently, each channel block has its own receptive field during self-attention computation, which dynamically adjusts according to changes in channel block entropy.

[0086] Traditional entropy models mostly employ simple statistical methods based on probability distributions, failing to fully extract local and global contextual information. This not only limits the compression efficiency of entropy models but also results in slow decoding speeds. The receptive field-aware self-attention mechanism, through learnable Laplace relative position encoding, dynamically adjusts the receptive field width across different channel blocks. In this way, the entropy model effectively balances the trade-off between compression efficiency and decoding speed. By leveraging the network's global awareness, it dynamically adapts to the characteristics of different data blocks, processing complex data structures more quickly and accurately, thus achieving rapid decoding. This performance improvement can significantly shorten data transmission and processing time, substantially reduce latency, and enhance system response speed and user experience. It is particularly suitable for applications requiring rapid processing of large amounts of data, such as streaming media playback and real-time communication.

[0087] For example, the receptive field adjustment parameter is a learnable parameter that can be obtained using machine learning algorithms, thereby enabling the Laplacian relative position encoding to better learn the entropy information of the channel blocks to adaptively adjust the learnable parameter of each channel cluster, thereby more accurately modeling the probability distribution of the latent representation.

[0088] Figure 5 This is a functional module diagram of an image processing method according to an embodiment of the present application, including an encoding / decoding module and an entropy model module. Figure 5 The encoding / decoding module and the entropy model module are laid out separately, but in reality, it is an end-to-end network structure, such as... Figure 6 As shown. Figure 7A This is a flowchart of the encoding method according to an embodiment of this application. Figure 7B This is a flowchart of a decoding method according to an embodiment of this application. The following is in conjunction with… Figure 5 , Figure 6 , Figure 7A and Figure 7B Examples of encoding and decoding methods in embodiments of this application are provided.

[0089] At the encoding end: the target image is input, and the encoder performs convolution, feature extraction, and representation on the target image to generate latent variable y. The quantization submodule of the encoding / decoding module can quantize the latent variable into Q(y). In the arithmetic encoding submodule, the quantized latent variable is arithmetically encoded (AE) into a bitstream. At the decoding end: the bitstream is input, and the bitstream is arithmetically decoded (AD) into quantized latent variables. The dequantization submodule performs dequantization on the quantized latent variables and outputs image features, such as latent variables. The diffusion model (deployed in the denoising and deblurring module) uses the probability information input from the entropy model to deblur and denoise the latent variables. The arithmetic decoding (decoer) submodule and the dequantization submodule of the image reconstruction module output the reconstructed image. The entropy model module includes a context understanding module and an image distribution inference module. The context understanding module calculates local and global context information through a receptive field-aware self-attention mechanism and Laplacian relative position encoding (attention calculation submodule and position encoding submodule), which helps the image distribution inference module to calculate the distribution probability of latent variables using local and global context information and output probability information. The probability information output by the image distribution inference module can be used for entropy encoding and entropy decoding in the entropy encoding submodule.

[0090] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide an image processing apparatus. This image processing apparatus may include: a bitstream acquisition module, used to acquire a bitstream, wherein the bitstream is obtained by entropy encoding of latent variables of a target image; an entropy decoding module, used to entropy decode the bitstream to obtain the latent variables; and a denoising decoding module, used to determine a denoising method for the latent variables based on their frequency characteristics, and to denoise and decode the latent variables according to the denoising method to obtain reconstructed image information, wherein the frequency characteristics of the latent variables are used to represent the rate of change of pixel values ​​of the latent variables, and the reconstructed image information is used to reconstruct the target image.

[0091] In one implementation, the denoising decoding module is specifically used to: perform denoising decoding on the latent variables using a diffusion model, wherein the denoising method includes a denoising rate, and the diffusion model is used to schedule the denoising rate of the latent variables according to the frequency characteristics of the latent variables.

[0092] In one implementation, the denoising decoding module is specifically used to: perform multi-round iterative denoising decoding on the latent variables using a diffusion model, wherein the diffusion model is used to infer the reconstructed image information of the latent variables in the next round based on probability information and the reconstructed image information of the latent variables in the current round in each iteration, wherein the probability information includes the distribution probability of the latent variables in the target image, and the probability information is used to determine the denoising rate.

[0093] In one embodiment, the denoising method includes a denoising rate, and the denoising decoding module is specifically used to: determine the denoising rate of the latent variable based on the frequency characteristics of the latent variable; and decode the latent variable into the reconstructed image information according to the denoising rate.

[0094] In one implementation, the denoising decoding module is specifically used to: divide the latent variable into multiple channel blocks, each channel block including at least one channel of the latent variable; and decode each channel block sequentially according to the entropy information from high to low, wherein the decoding of the current channel block is based on the context information of the previously decoded channel blocks.

[0095] In one implementation, the channel block includes calibrated and uncalibrated channels of the latent variables. The decoding of the calibrated channels is based on the context information of the decoded channel block, and the decoding of the uncalibrated channels is based on the context information of the decoded channel block and the context information of each of the decoded calibrated channels.

[0096] In one embodiment, the image processing apparatus may further include: a context information calculation module for calculating local context information of the channel block and global context information of the latent variable; and a probability information modeling module for determining probability information using the local context information and the global context information, wherein the probability information includes the distribution probability of the latent variable in the target image, and the probability information is used for the entropy encoding and the entropy decoding.

[0097] In one implementation, the context information calculation module is specifically used to: calculate the local context information and the global context information based on the Laplace relative position encoding of the channel block, wherein the Laplace relative position encoding includes the relative position information of each element in the channel block and the receptive field adjustment parameter, and the receptive field adjustment parameter is used to adjust the receptive field of the channel block.

[0098] In one implementation, the receptive field adjustment parameters are obtained through a machine learning algorithm.

[0099] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0100] Figure 8 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 8As shown, the electronic device includes a memory 801 and a processor 802. The memory 801 stores a computer program that can run on the processor 802. When the processor 802 executes the computer program, it implements the method described in the above embodiments. The number of memories 801 and processors 802 can be one or more. In a specific implementation, the electronic device may also include a communication interface 803 for communicating with external devices and performing data exchange and transmission.

[0101] In practical implementation, if the memory 801, processor 802, and communication interface 803 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0102] Optionally, in a specific implementation, if the memory 801, the processor 802, and the communication interface 803 are integrated on a single chip, then the memory 801, the processor 802, and the communication interface 803 can communicate with each other through an internal interface.

[0103] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0104] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0105] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0106] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0107] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0108] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0109] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0110] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0111] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0112] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0113] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0114] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0115] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0116] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, comprising: Obtain the bitstream, which is obtained by entropy encoding the latent variables of the target image; The latent variables are obtained by entropy decoding of the bitstream. The denoising method of the latent variable is determined based on the frequency characteristics of the latent variable, and the latent variable is denoised and decoded according to the denoising method to obtain reconstructed image information. The frequency characteristics of the latent variable are used to represent the pixel value change rate of the latent variable, and the reconstructed image information is used to reconstruct the target image.

2. The method according to claim 1, wherein, The denoising method for the latent variables is determined based on their frequency characteristics, and the latent variables are then denoised and decoded according to the denoising method, including: The latent variables are denoised and decoded using a diffusion model, wherein the denoising method includes a denoising rate, and the diffusion model is used to schedule the denoising rate of the latent variables according to the frequency characteristics of the latent variables.

3. The method according to claim 2, wherein, Denoising and decoding of the latent variables using a diffusion model includes: The latent variables are denoised and decoded using a diffusion model in multiple iterations. In each iteration, the diffusion model is used to infer the reconstructed image information of the latent variables in the next iteration based on probability information and the reconstructed image information of the latent variables in the current iteration. The probability information includes the distribution probability of the latent variables in the target image and is used to determine the denoising rate.

4. The method according to claim 1, wherein, The denoising method includes a denoising rate, determining the denoising method for the latent variables based on their frequency characteristics, and performing denoising decoding on the latent variables according to the denoising method to obtain reconstructed image information, including: The denoising rate of the latent variables is determined based on their frequency characteristics. According to the denoising rate, the latent variables are decoded into the reconstructed image information.

5. The method according to claim 4, wherein, Decoding the latent variables into the reconstructed image information includes: The latent variable is divided into multiple channel blocks, and each channel block includes at least one channel of the latent variable; Each channel block is decoded sequentially according to the entropy information from high to low. The decoding of the current channel block is based on the context information of the previously decoded channel blocks.

6. The method according to claim 5, wherein, The channel block includes calibrated and uncalibrated channels of the latent variables. The decoding of the calibrated channels is based on the context information of the decoded channel blocks, and the decoding of the uncalibrated channels is based on the context information of the decoded channel blocks and the context information of each decoded calibrated channel.

7. The method according to claim 5 or 6, further comprising: Calculate the local context information of the channel block and the global context information of the latent variables; The probability information is determined using the local context information and the global context information. The probability information includes the distribution probability of the latent variable in the target image. The probability information is used for the entropy encoding and the entropy decoding.

8. The method according to claim 7, wherein, Calculating the local context information of the channel block and the global context information of the latent variables includes: Based on the Laplace relative position encoding of the channel block, the local context information and the global context information are calculated. The Laplace relative position encoding includes the relative position information of each element in the channel block and the receptive field adjustment parameter, which is used to adjust the receptive field of the channel block.

9. The method according to claim 8, wherein, The receptive field adjustment parameters are obtained through machine learning algorithms.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 9.

11. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1 to 9.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images