Image decoding method and apparatus

Abstract

Provided in the embodiments of the present application are an image decoding method and apparatus. The method comprises: first, receiving a code stream; then, acquiring first image information on the basis of the code stream, and performing feature transformation on the first image information, so as to obtain a feature of the first image information; then, inputting second image information into a diffusion model, so as to obtain a target feature, wherein the feature of the first image information is used for adjusting an intermediate feature generated during the process of performing feature generation processing on the diffusion model, and the second image information comprises one of a preset noise image, an initial reconstructed image determined on the basis of the code stream, or an image obtained by adding noise on the basis of the initial reconstructed image; and then performing reconstruction processing on the target feature, so as to obtain a target reconstructed image. In this way, the subjective quality of a reconstructed image can be improved.

Description

Image decoding method and device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on September 28, 2023, with application number 202311292210.8 and application name “Image Decoding Method and Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention relates to the field of coding and decoding, and more particularly to a method and apparatus for decoding an image. Background Art

[0003] As video evolves from HD to UHD, people's expectations for video quality are growing. At the same time, HD video places increasing demands on bandwidth and storage. Consequently, the need for video encoding, or video processing, is becoming increasingly urgent due to cost controls on bandwidth, transmission latency, and storage.

[0004] Among them, the artificial intelligence (AI) video compression (or codec) algorithm is based on deep learning (for example, based on the variational auto-encoder (VAE)), and has better compression effect than traditional video compression technologies (such as H265 and H266).

[0005] However, the VAE-based image compression scheme also has certain defects, namely, the reconstructed image obtained by decoding is blurry or has pseudo-textures, and the subjective quality of the reconstructed image is relatively poor.

[0006] Summary of the Invention

[0007] In view of this, the present application provides an image decoding method and apparatus; the image decoding method can improve the subjective quality of the reconstructed image.

[0008] In a first aspect, an embodiment of the present application provides a decoding method, which includes: first, receiving a code stream; then, obtaining first image information based on the code stream; and performing feature transformation on the first image information to obtain features of the first image information; then, inputting the second image information into a diffusion model to obtain target features; wherein the features of the first image information are used to adjust intermediate features generated during feature generation processing of the diffusion model, and the second image information includes one of a preset noise image, an initial reconstructed image determined based on the code stream, or an image obtained by adding noise to the initial reconstructed image; thereafter, reconstructing the target features to obtain a target reconstructed image.

[0009] First, some existing technologies directly input the feature map of the image obtained from entropy decoding of the code stream into the image decoder for decoding to obtain a reconstructed image; however, the present application first determines the features of the first image information (including the feature map of the image) obtained from the code stream, then uses a diffusion model to perform feature generation processing on the second image information to obtain target features, and uses the features of the first image information to adjust the intermediate features generated during the feature generation process of the diffusion model, and then uses a decoder (which is different from the image decoder) to reconstruct the target features to obtain a reconstructed image (i.e., a target reconstructed image). Because the diffusion model learns general image information (such as texture information) during training, the diffusion model has a stronger generation capability than the image decoder of the prior art, and therefore requires less information during the feature generation process; thus, under the same subjective quality, the code stream of the present application carries less information. In other words, under the same code stream size, the subjective quality of the reconstructed image obtained by the decoding method of the present application is better.

[0010] Secondly, some prior art techniques use the initial reconstructed image as a control condition for intermediate features generated during the feature generation process of the diffusion model. In other words, the diffusion model of the prior art operates in the image domain. However, the present application uses the features of the first image information as a control condition for intermediate features generated during the feature generation process of the diffusion model. In other words, the diffusion model of the present application operates in the feature domain. The scale of the feature domain is smaller than that of the image domain. Therefore, the present application can reduce the computational complexity of the diffusion model and improve the efficiency of the decoding process. (Here, scale can be understood as size. By downsampling an image at multiple multiples, features at multiple scales can be obtained.)

[0011] Again, in the field of image super-resolution, some existing technologies use a stable diffusion model (SD) for super-resolution. Among them, the stable diffusion model may include: a time-aware feature transformer network (TAFT), a spatial feature transformation network (SFT), a U-NET, an encoder, and a decoder. In the super-resolution process, a low-resolution image is input into the encoder to obtain the first feature; the first feature is then input into the TAFT to obtain the second feature; then the second feature and a preset noise image are input into the generative model (including U-NET and SFT) to obtain the third feature; then, the third feature is input into the decoder to obtain a high-resolution image. Compared with the existing technology: First, the existing technology inputs the image into the Encoder, and then inputs the features output by the Encoder into the TAFT; while in the decoding process, the present application obtains the first image information from the code stream, and then determines the features of the first image information (for example, the first image information can be input into the first feature transformation model such as TAFT). That is to say, the present application does not use the Encoder in the decoding process. Secondly, in the prior art, the first feature and the third feature belong to the same domain (that is, the dimension of the first feature is the same as the dimension of the third feature, and the physical meaning of each dimension is the same; for example, the dimension of the first feature is m1*m2, the dimension of the third feature is m1*m2, and m1 and m2 are positive integers); in other words, the first feature matches the decoder, that is, only by inputting the first feature into the decoder, a reconstructed image can be obtained; while in the present application, the features of the first image information and the target features belong to different domains (that is, the dimensions of the features of the first image information and the dimensions of the target features are different, and the physical meaning of each dimension is different; for example, the dimensions of the features of the first image information are m3*m4, the dimensions of the target features are m5*m6, m3~m6 are positive integers, m3 is not equal to m5 or m4 is not equal to m6); in other words, the features of the first image information do not match the decoder, that is, only by inputting the features of the first image information into the decoder, a reconstructed image cannot be obtained.

[0012] Exemplarily, the diffusion model is a model for generating images / features, which can be implemented by a neural network. Exemplarily, the diffusion model may include a convolutional layer. In one possible manner, the diffusion model may include at least one convolutional layer and at least one activation layer. In one possible manner, the diffusion model may include at least one convolutional layer and at least one normalization layer. In one possible manner, the diffusion model may include at least one convolutional layer, at least one activation layer, and at least one normalization layer. For example, the network structure adopted in the denoising process (or feature generation processing process) of the diffusion model may be a network structure such as U-NET; it should be understood that the present application does not limit the network structure of the diffusion model.

[0013] Exemplarily, the intermediate features generated during the feature generation process of the diffusion model may refer to the features output by the input layer and the features output by the intermediate layer included in the processing of the diffusion model.

[0014] Exemplarily, the preset noise image can be obtained by sampling Gaussian noise.

[0015] Exemplarily, the first image information may be at least one of an image or a feature.

[0016] It should be noted that the second image information can be understood as input information of the diffusion model, and the features of the first image information can be understood as control information of the diffusion model.

[0017] It should be noted that the decoding method of the present application can be applied to decoding images / videos.

[0018] According to the first aspect, reconstructing the target features to obtain a target reconstructed image includes: reconstructing the first image information and the target features to obtain a target reconstructed image.

[0019] Since determining the features of the first image information and performing feature generation processing using a diffusion model will introduce errors and weaken the original features and detail features contained in the first image information, the first image information is more accurate than the target features. Furthermore, reconstructing the first image information and target features determined based on the bitstream can balance the detail richness and fidelity of the target reconstructed image.

[0020] According to the first aspect, or any implementation of the first aspect above, the first image information and the target features are reconstructed to obtain a target reconstructed image, including: inputting the first image information and the target features into a first decoding network to obtain a target reconstructed image.

[0021] Among them, the first decoding network can be implemented by a neural network, and the first decoding network can be a decoder; that is, the present application performs AI reconstruction of the first image information and target features; compared with traditional reconstruction methods, the present application can improve the subjective quality of the target reconstructed image obtained by decoding.

[0022] According to the first aspect, or any implementation of the first aspect above, performing feature transformation on the first image information to obtain features of the first image information includes: inputting the first image information and time information into a first feature transformation model to obtain features of the first image information.

[0023] On the one hand, the first feature transformation model may refer to a model for performing feature transformation, and the first feature transformation model may be implemented by a neural network and may include multiple convolutional layers; for example, the first feature transformation model is TAFT; in this way, using an AI network for feature transformation can improve the accuracy of the features of the obtained first image information, and thereby improve the subjective quality of the target reconstructed image.

[0024] On the other hand, since the diffusion model performs denoising step by step, that is, the denoising process of the diffusion model is associated with time, therefore, inputting time information into the first feature transformation model can enable the first feature transformation model to perceive time changes and generate features of the first image information corresponding to different time information, thereby more accurately guiding feature adjustment and improving the accuracy of target features, thereby improving the subjective quality of the target reconstructed image.

[0025] According to the first aspect, or any implementation of the first aspect above, the features of the first image information correspond to multiple time information, one time information in the multiple time information corresponds to one or more features of the first image information, and the scales of the multiple features of the first image information are different.

[0026] According to the first aspect, or any implementation method of the first aspect above, the second image information is input into the diffusion model to obtain the target feature, including: the tth time information and the second image information are input into the diffusion model to obtain the feature of the t-1th time information; wherein the initial value of t is T, and T is a positive integer; t is subtracted by 1, and the following steps are repeatedly performed until t is equal to 1: the tth time information and the feature of the tth time information are input into the diffusion model to obtain the feature of the t-1th time information and t is subtracted by 1; wherein the feature of the first image information corresponds to multiple time information, and the feature of the first image information corresponding to the tth time information is used to adjust the intermediate feature generated in the process of the diffusion model generating the feature of the t-1th time information; when t is equal to 1, the feature of the t-1th time information is the target feature.

[0027] The value of t in the t-th time information varies in different cycles.

[0028] According to the first aspect, or any implementation of the first aspect above, the method further includes: inputting the features of the first image information into a second feature transformation model to obtain feature adjustment parameters; and adjusting the intermediate features generated during the feature generation process of the diffusion model based on the feature adjustment parameters.

[0029] Exemplarily, the second feature transformation model can be a model for performing feature transformation, and the second feature transformation model can be implemented by a neural network. The second feature transformation module can include multiple blocks, each of which can include at least one convolutional layer. For example, the second feature model can be an SFT. In this way, the features of the first image information can be transformed into feature adjustment parameters that can directly adjust intermediate features generated during the feature generation process of the diffusion model.

[0030] According to the first aspect, or any implementation of the first aspect above, the feature adjustment parameters include a first feature adjustment parameter and a second feature adjustment parameter, and based on the feature adjustment parameters, the intermediate features generated during the feature generation processing of the diffusion model are adjusted, including: determining the product value between the first feature adjustment parameter and the intermediate features generated during the feature generation processing of the diffusion model; determining the sum value between the product value and the second feature adjustment parameter; and determining the adjusted intermediate features based on the sum value.

[0031] According to the first aspect, or any implementation of the first aspect above, determining the adjusted intermediate feature based on the sum value includes: adding the sum value to the intermediate feature generated during feature generation processing of the diffusion model to obtain the adjusted intermediate feature.

[0032] After feature adjustment is performed on the intermediate features generated in the feature generation process of the diffusion model, the original features and detail features contained in the intermediate features generated in the feature generation process of the diffusion model are weakened; then, the intermediate features before adjustment are added to the adjusted intermediate features, and the original features and detail features contained in the intermediate features before adjustment can be added to the adjusted intermediate features; in this way, the accuracy of the features input to the next network layer of the diffusion model can be guaranteed, and the feature accuracy of the obtained t-1th time information can be increased; thereby improving the subjective quality of the target reconstructed image.

[0033] According to the first aspect, or any implementation manner of the first aspect above, the first image information includes at least one of a feature map of the image, side information features of the image, an initial reconstructed image, or decoding features of the initial reconstructed image; wherein the feature map of the image and the side information features of the image are obtained by entropy decoding the code stream by the entropy decoding unit, the initial reconstructed image is obtained by decoding the feature map of the image by the second decoding network, and the decoding features of the initial reconstructed image are features generated in the process of decoding the feature map of the image by the second decoding network.

[0034] It should be noted that the first image information used for reconstructing the target reconstructed image by performing the reconstruction process with the target features may include at least one of the feature map of the image, the side information features of the image, or the decoding features of the initial reconstructed image.

[0035] According to the first aspect, or any implementation of the first aspect above, the code stream includes a first code stream and a second code stream, and obtaining the first image information based on the code stream includes:

[0036] Inputting the first code stream into an entropy decoding unit to obtain side information features of the image; using the side information features of the image as first image information; and / or

[0037] Inputting the first bitstream into an entropy decoding unit to obtain side information features of the image; inputting the side information features of the image and the second bitstream into the entropy decoding unit to obtain a feature map of the image; using the feature map of the image as the first image information; and / or

[0038] Inputting the first code stream into the entropy decoding unit to obtain side information features of the image; inputting the side information features of the image and the second code stream into the entropy decoding unit to obtain a feature map of the image; inputting the feature map of the image into the second decoding network to obtain an initial reconstructed image and / or decoding features of the initial reconstructed image; and using the initial reconstructed image and / or the decoding features of the initial reconstructed image as the first image information.

[0039] For example, after obtaining the image side information features, a probability distribution of a feature map of the image can be determined based on the image side information features. Then, based on the probability distribution of the image feature map, entropy decoding is performed on the second bitstream to obtain the image feature map. The probability distribution of the image feature map can be Gaussian distribution parameters, such as mean and variance.

[0040] In a second aspect, an embodiment of the present application provides an image decoding device, the image decoding device including an entropy decoding unit, a first feature transformation model, a diffusion model, and a first decoding network; wherein:

[0041] an entropy decoding unit, configured to perform entropy decoding on the received code stream to obtain first image information;

[0042] A first feature transformation model is used to perform feature transformation on the first image information to obtain features of the first image information;

[0043] a diffusion model configured to perform feature generation processing on the second image information to obtain target features; wherein the features of the first image information are used to adjust intermediate features generated during the feature generation processing performed by the diffusion model, and the second image information includes one of a preset noise image, an initial reconstructed image determined based on a bitstream, or an image obtained by adding noise to the initial reconstructed image;

[0044] The first decoding network is used to reconstruct the target features to obtain a target reconstructed image.

[0045] According to the second aspect, the first decoding network is specifically used to reconstruct the first image information and the target features to obtain a target reconstructed image.

[0046] According to the second aspect, or any implementation of the second aspect above, the image decoding device further includes a second feature transformation model,

[0047] The second feature transformation model is used to perform feature transformation on the features of the first image information to obtain feature adjustment parameters; wherein the feature adjustment parameters are used to adjust the intermediate features generated during the feature generation process of the diffusion model.

[0048] According to the second aspect, or any implementation of the above second aspect, the first image information includes at least one of a feature map of the image or a side information feature of the image, and the feature map of the image or the side information feature of the image is output information of the entropy decoding unit.

[0049] According to the second aspect, or any implementation of the second aspect above, the image decoding apparatus further includes a second decoding network,

[0050] The second decoding network is used to decode the feature map of the image;

[0051] In which, the feature map of the image is the output information of the entropy decoding unit, the first image information includes at least one of the initial reconstructed image and the decoding features of the initial reconstructed image, the initial reconstructed image is the output information of the second decoding network, and the decoding features of the initial reconstructed image are the features generated in the process of the second decoding network decoding the feature map of the image.

[0052] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0053] In a third aspect, the present application provides a device for decoding an image, the device for decoding an image comprising:

[0054] A receiving module, used for receiving code stream;

[0055] An information acquisition module, configured to acquire first image information based on a code stream;

[0056] a feature determination module, configured to perform feature transformation on the first image information to obtain features of the first image information;

[0057] a feature generation module, configured to input the second image information into the diffusion model to obtain target features; wherein the features of the first image information are used to adjust intermediate features generated during feature generation processing by the diffusion model, and the second image information includes one of a preset noise image, an initial reconstructed image determined based on a bitstream, or an image obtained by adding noise to the initial reconstructed image;

[0058] The reconstruction module is used to reconstruct the target features to obtain a target reconstructed image.

[0059] It should be understood that the above-mentioned device for decoding can be used to execute the decoding method in the first aspect or any possible implementation of the first aspect.

[0060] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0061] In a fourth aspect, the present application provides a coding method, which includes: acquiring an image; performing feature extraction on the image to obtain a feature map of the image; and performing entropy coding on the image to obtain a code stream.

[0062] The code stream may correspond to the second code stream in the first aspect or any implementation of the first aspect above.

[0063] According to the fourth aspect, the method also includes: performing probability estimation on the feature map of the image to obtain the probability distribution of the feature map of the image; performing entropy encoding on the image to obtain a code stream, including: performing entropy encoding on the feature map of the image based on the probability distribution of the feature map of the image to obtain the code stream.

[0064] According to the fourth aspect, or any implementation method of the fourth aspect above, probability estimation is performed on the feature map of the image to obtain the probability distribution of the feature map of the image, including: feature extraction is performed on the feature map of the image to obtain the side information features of the image; based on the side information features of the image, the probability distribution of the feature map of the image is determined.

[0065] According to the fourth aspect, or any implementation of the fourth aspect above, the method further includes: performing entropy coding on the side information features of the image. At this time, the first bitstream in the first aspect or any implementation of the first aspect above can be obtained.

[0066] In a fifth aspect, an embodiment of the present application provides a decoding device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the electronic device to execute the image decoding method of the first aspect or any possible implementation of the first aspect.

[0067] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0068] In a sixth aspect, an embodiment of the present application provides a decoding device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the electronic device to execute the image decoding method of the first aspect or any possible implementation of the first aspect.

[0069] The sixth aspect and any implementation of the sixth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0070] In the seventh aspect, an embodiment of the present application provides a chip comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the image decoding method in the first aspect or any possible implementation of the first aspect are executed.

[0071] The seventh aspect and any implementation of the seventh aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the seventh aspect and any implementation of the seventh aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0072] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the image decoding method in the first aspect or any possible implementation of the first aspect.

[0073] The eighth aspect and any implementation of the eighth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the eighth aspect and any implementation of the eighth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0074] In a ninth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the image decoding method in the first aspect or any possible implementation of the first aspect.

[0075] The ninth aspect and any implementation of the ninth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the ninth aspect and any implementation of the ninth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] FIG1A is a schematic diagram illustrating an exemplary application framework;

[0077] FIG1B is a schematic diagram of an exemplary storage framework;

[0078] FIG1C is a schematic diagram illustrating an exemplary transmission framework;

[0079] FIG2A is a schematic diagram of an exemplary compression frame;

[0080] FIG2B is a schematic diagram of an exemplary compression frame;

[0081] FIG2C is a schematic diagram of an exemplary training framework;

[0082] FIG2D is a schematic diagram of an exemplary training framework;

[0083] FIG2E is a schematic diagram illustrating an exemplary feature adjustment process;

[0084] FIG3A is a schematic diagram of an exemplary compression frame;

[0085] FIG3B is a schematic diagram of an exemplary compression frame;

[0086] FIG3C is a schematic diagram of an exemplary compression frame;

[0087] FIG3D is a schematic diagram of an exemplary compression frame;

[0088] FIG3E is a schematic diagram illustrating an exemplary encoding process;

[0089] FIG4 is a schematic diagram illustrating an exemplary decoding process;

[0090] FIG5A is a schematic diagram of an exemplary reconstructed image;

[0091] FIG5B is a schematic diagram of an exemplary reconstructed image;

[0092] FIG5C is a schematic diagram of an exemplary reconstructed image;

[0093] FIG5D is a schematic diagram of an exemplary reconstructed image;

[0094] FIG6 is a schematic diagram illustrating an exemplary apparatus for decoding;

[0095] FIG7 is a schematic structural diagram of an exemplary device. DETAILED DESCRIPTION

[0096] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0097] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0098] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0099] In the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0100] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0101] This application can be applied to all services of image / video transmission and image / video storage, such as terminal photography, terminal video, photo album, Huawei Cloud, live broadcast, etc., and this application does not impose any restrictions on this.

[0102] For example, the decoding method of the present application can be applied to image / video decoding. This application is described using an image as an example.

[0103] FIG1A is a schematic diagram illustrating an exemplary application framework.

[0104] Referring to Figure 1A , for example, a camera can capture an image to obtain an image. The AI ​​encoding module 100 then performs AI encoding on the image to obtain a bitstream. This bitstream can then be stored locally or transmitted to a remote device.

[0105] Continuing with reference to FIG. 1A , in one possible approach, after the code stream is stored locally, when an image needs to be played or edited, the AI ​​decoding module 200 may perform AI decoding on the code stream to obtain a reconstructed image; subsequently, the reconstructed image may be played or edited.

[0106] Continuing with reference to FIG. 1A , in one possible approach, after the remote device receives the code stream, when it needs to play or edit an image, the AI ​​decoding module 200 can perform AI decoding on the code stream to obtain a reconstructed image; subsequently, the reconstructed image can be played or edited.

[0107] FIG. 1B is a schematic diagram illustrating an exemplary storage framework.

[0108] 1B , exemplarily, the AI ​​encoding module 100 may include an AI encoding unit 110 and an entropy encoding unit 120, and the AI ​​decoding module 200 may include an AI decoding unit 210 and an entropy decoding unit 220; it should be understood that FIG1B is merely an example of the present application, and the AI ​​encoding module 100 and the AI ​​decoding module 200 may include more units than shown in FIG1B , and the present application does not limit this.

[0109] For example, the AI ​​encoding unit 110 and the AI ​​decoding unit 210 can be set in an embedded neural network processing unit (NPU) or a graphics processing unit (GPU). For example, the entropy encoding unit 120 and the entropy decoding unit 220 can be set in a central processing unit (CPU). For example, the file saving module and the file loading module can be set in the CPU.

[0110] Referring to Figure 1B , illustratively, after the camera captures an image, it may input the image into the AI ​​encoding unit 110. Next, the AI ​​encoding unit 110 may extract features from the image to obtain a feature map, and output the feature map to the entropy encoding unit 120. The entropy encoding unit 120 may then perform entropy encoding on the feature map to obtain a bitstream, which is then output to the file storage module for storage, resulting in a file.

[0111] Continuing with FIG. 1B , illustratively, when an image needs to be played or edited, the file loading module can load the file and then output the code stream in the file to the entropy decoding unit 220. The entropy decoding unit 220 can then entropy decode the feature map of the image from the code stream and output the feature map of the image to the AI ​​decoding unit 210. The AI ​​decoding unit 210 can then decode the feature map of the image to obtain a reconstructed image.

[0112] FIG1C is a schematic diagram illustrating an exemplary transmission framework.

[0113] 1C , illustratively, after the camera on the encoding end captures an image, it can input the image into the AI ​​encoding unit 110. Next, the AI ​​encoding unit 110 can extract features from the image to obtain a feature map, and output the feature map to the entropy encoding unit 120. The entropy encoding unit 120 can then perform entropy encoding on the feature map to obtain a bitstream, which is then sent to the server.

[0114] Continuing with FIG1C , illustratively, after receiving the bitstream sent by the server, the decoding end may input the bitstream into the entropy decoding unit 220. The entropy decoding unit 220 may then entropy decode the bitstream to obtain an image feature map and output the image feature map to the AI ​​decoding unit 210. The AI ​​decoding unit 210 may then decode the image feature map to obtain a reconstructed image.

[0115] It should be understood that the encoding end and the decoding end can be directly connected; in this way, the encoding end can directly send the code stream to the decoding end without forwarding through other devices (such as servers), and this application does not impose any restrictions on this.

[0116] Figure 2A is a schematic diagram of an exemplary compression frame. Figure 2A shows the compression frame of the present application.

[0117] 2A , illustratively, the AI ​​encoding unit 110 may include a coding network 111 and an entropy estimation module 112 ( 211 ). It should be understood that FIG. 2A is merely an example of the present application, and the AI ​​encoding unit 110 may include more models or modules, which is not limited in the present application.

[0118] 2A , illustratively, the AI ​​decoding unit 210 may include a first feature transformation model 212, a second feature transformation model 213, a diffusion model 214, a decoding network 215, and an entropy estimation module 112 (211). It should be understood that FIG. 2A is merely an example of the present application, and the AI ​​decoding unit 210 may include more or fewer models or modules, and the present application does not limit this.

[0119] Figure 2B is a schematic diagram of an exemplary compression frame. Figure 2B shows another compression frame of the present application.

[0120] 2B , illustratively, the AI ​​encoding unit 110 may include a coding network 111 and an entropy estimation module 112 ( 211 ). It should be understood that FIG. 2B is merely an example of the present application, and the AI ​​encoding unit 110 may include more models or modules, which is not limited in the present application.

[0121] 2B , illustratively, the AI ​​decoding unit 210 includes a first decoding network 216, a first feature transformation model 212, a second feature transformation model 213, a diffusion model 214, a second decoding network 215, and an entropy estimation module 112 (211). It should be understood that FIG. 2B is merely an example of the present application, and the AI ​​decoding unit 210 may include more or fewer models or modules, and the present application does not limit this.

[0122] Exemplarily, each model / network in Figure 2A and Figure 2B can be implemented by a neural network, such as a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a residual network, a neural network using a transformer model, or other neural networks, etc., and this application does not limit this.

[0123] For example, the work of each layer in a neural network can be expressed mathematically as To describe: From a physical perspective, the work of each layer in a neural network can be understood as completing the transformation from input space to output space (i.e., from the row space to the column space of a matrix) through five operations on the input space (a set of input vectors). These five operations include: 1. Dimensionality increase / decrease; 2. Zoom in / out; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are represented by Completed, operation 4 is completed by +b, and operation 5 is performed by a(). The word "space" is used here because the object being classified is not a single thing, but a category of things, and space refers to the collection of all individuals in this category. W is the weight vector, and each value in this vector represents the weight of a neuron in that layer of the neural network. This vector W determines the spatial transformation from the input space to the output space mentioned above. In other words, the weight W of each layer controls how the space is transformed.

[0124] Exemplarily, the diffusion model 214 is a model for generating images / features, and the diffusion model 214 may include a convolutional layer. In one possible manner, the diffusion model 214 may include at least one convolutional layer and at least one activation layer. In one possible manner, the diffusion model 214 may include at least one convolutional layer and at least one normalization layer. In one possible manner, the diffusion model 214 may include at least one convolutional layer, at least one activation layer, and at least one normalization layer. For example, the network structure adopted in the denoising process (or feature generation processing process) of the diffusion model 214 may be a network structure such as U-NET; it should be understood that the present application does not limit the network structure of the diffusion model 214.

[0125] Exemplarily, the first feature transformation model 212 and the second feature transformation model 213 may include convolutional layers.

[0126] In the following, the encoding network 111 is used as an image encoder (Image Compression Encoder, IC Encoder), the first decoding network 216 is used as an image decoder (Image Compression Decoder, IC Decoder), the network structure adopted by the first feature transformation model 212 is a time-aware feature transformation network (Time-Aware Feature Transformer, TAFT), the network structure adopted by the second feature transformation model 213 is a spatial feature transformation network (spatial feature transformations, SFT), the network structure adopted by the denoising process of the diffusion model 214 is U-NET, and the second decoding network 215 in Figure 2B (or the decoding network 215 in Figure 2A) is a decoder (Decoder) as an example for explanation.

[0127] For example, each of the aforementioned networks can be pre-trained so that the trained networks can be used for subsequent encoding and decoding. The purpose of training a neural network is to ultimately obtain the weight matrices of all layers of the trained neural network (a weight matrix formed by many layers of vectors W). Therefore, the training process of a neural network is essentially about learning how to control spatial transformations, and more specifically, learning the weight matrices.

[0128] FIG2C is a schematic diagram of an exemplary training framework.

[0129] 2C , illustratively, the IC Encoder and IC Decoder may include network layers such as convolutional layers, deconvolutional layers, and activation layers, which are not limited in this application.

[0130] For example, the training process for the IC Encoder and IC Decoder in FIG2C may be as follows:

[0131] First, the training image is input into the IC Encoder, which extracts features from the training image to obtain a feature map of the training image.

[0132] Next, the feature map of the training image is input into the entropy estimation module 112 (211) and the IC Decoder.

[0133] For example, the IC Decoder can decode the feature map of the training image to obtain a reconstructed image. Then, based on the training image and the reconstructed image, a distortion loss can be determined (the distortion loss can refer to the difference (e.g., error) between the training image and the reconstructed image, i.e., Loss1 in FIG2C , which can also be referred to as image reconstruction loss). For example, the mean square error (MSE) between the training image and the reconstructed image can be calculated as Loss1.

[0134] Furthermore, the entropy estimation module 112 (211) can determine the bit rate loss (ie, the estimated bit rate of the encoded training image), namely Loss2 in FIG. 2C , based on the feature map of the training image.

[0135] Next, the final loss Loss for training the IC Encoder and IC Decoder in FIG2C can be determined based on Loss1 and Loss2; the following formula (1) can be used:

[0136] Loss=αLoss1+Loss2 (1)

[0137] Wherein, α in formula (1) is the weight coefficient of the distortion loss, which can be set according to the specific requirements and is not limited in this application.

[0138] It should be understood that other losses (for example, perceptual loss (i.e., learned perceptual image patch similarity (LPIPS))) can also be introduced, and other losses can be combined with the above-mentioned Loss1 and Loss2 (such as weighted) to obtain the final loss Loss for training the IC Encoder and IC Decoder in Figure 2C. This application does not impose any restrictions on this.

[0139] Afterwards, the IC Encoder and IC Decoder in FIG2C can be trained with the goal of minimizing the Loss in formula (1) until the first preset condition is met. The first preset condition can be set as required, for example, the number of training times reaches the first preset number; for example, the difference in Loss obtained from two adjacent trainings is less than the first preset difference; for example, the Loss is less than the first threshold, etc. This application does not impose any restrictions on this. The first preset number, the first preset difference, and the first threshold can be set as required, and this application does not impose any restrictions on this.

[0140] FIG2D is a schematic diagram of an exemplary training framework.

[0141] For example, the TAFT, SFT, U-NET, and Encoder in FIG2D belong to an image generation model (e.g., a stable diffusion model (SD), which may include TAFT, SFT, U-NET, Encoder, and Decoder). Completing the training of the image generation model also completes the training of TAFT, SFT, U-NET, Encoder, and Decoder. For details, please refer to the training process for the image generation model in the prior art and will not be repeated here.

[0142] It should be noted that this application only optimizes and trains the TAFT and SFT obtained by training the image generation model, and does not train the U-NET and Encoder obtained by training the image generation model.

[0143] Exemplarily, a TAFT may include n (n is a positive integer) downsampling layers (wherein the downsampling layers may be convolutional layers). In FIG2D , n=3, i.e., the TAFT shown in FIG2D includes three downsampling layers; wherein, a white rectangle in the TAFT in FIG2D represents a downsampling layer. Exemplarily, the three downsampling layers included in the TAFT in FIG2D have different downsampling multiples, and accordingly, the three downsampling layers of the TAFT output features of three scales (i.e., output features of three sizes).

[0144] For example, the SFT may include 2n blocks, and a single block may include at least one convolutional layer. In FIG2D , n=3, i.e., the SFT shown in FIG2D includes 6 blocks; wherein, a black rectangle in the SFT of FIG2D represents a block.

[0145] It should be noted that in this embodiment, U-NET is a U-shaped structure, and the output information of SFT is used to adjust the intermediate features generated during the feature generation process of U-NET; therefore, SFT can also be a U-shaped structure, as shown in Figure 2D, the left half includes 3 blocks and the right half includes 3 blocks.

[0146] 2D , illustratively, the training process of TAFT and SFT may be as follows:

[0147] Exemplarily, t in FIG2D is time information, and the value of t ranges from 0 to T, where T is a positive integer.

[0148] On the one hand, the training image is input into the IC Encoder (the IC Encoder trained according to the above description) to obtain the feature map of the training image For example, if the size of the training image is 3×512×512 (where “×” does not represent multiplication, but represents the size or dimension of the training image, “3” represents the number of channels of the training image, the first “512” represents the length of the training image, and the second “512” represents the width of the training image), then the feature map of the training image is The size of the training image may be 192 × 32 × 32. It should be understood that the present application does not impose any restrictions on the size of the training image and the feature map of the training image.

[0149] On the other hand, the training image can be input into the Encoder to obtain image Z0 (that is, the state (or image) when time information t=0 or the state (or image) of the 0th time information). Then, Z0 is denoised to obtain image Z1 (that is, the state (or image) when time information t=1 or the state (or image) of the 1st time information). Subsequently, Z1 is denoised to obtain image Z2 (that is, the state (or image) when time information t=2 or the state (or image) of the 2nd time information); ...; and so on. By gradually denoising, image Z can be obtained. T-1 and image Z T (That is, the state (or image) at time information t=T or the state (or image) of the T-th time information.) This process is also the noise adding process of the diffusion model 214 .

[0150] Exemplarily, for each time information, random Gaussian noise can be sampled to obtain preset noise ε~N(0,1) (N(0,1) is Gaussian distribution, ε~N(0,1) indicates that the preset noise conforms to Gaussian distribution); based on the preset noise sampled from the current time information, the image of the previous time information is denoised to obtain the image of the current time information; wherein, the specific noise addition method can refer to the description in the prior art and will not be repeated here. The size of the sampled preset noise is the same as the size of Z0. For example, the size of the training image is 3×512×512, and the size of Z0 output after Encoder processing can be 4×64×64, then the size of the sampled preset noise can be 4×64×64.

[0151] For example, a time information can be randomly selected from 1 to T as the value of t, and the feature map of the training image And time information t is input into TAFT to obtain feature F t 1 、F t 2 and F t 3 For example, t = T, then the feature map of the training image And time information T is input to TAFT to get F T 1 、F T 2 and F T 3 .

[0152] Then, TAFT can F T 1 、F T 2 and F T 3 Input into SFT to obtain feature adjustment parameters: the first feature adjustment parameter and the second feature adjustment parameter. For example, TAFT can input the features of a scale into a Block in the left half of the SFT and a Block in the right half of the SFT respectively. For the convenience of description, the three convolutional layers in TAFT can be referred to from top to bottom as: the first convolutional layer, the second convolutional layer and the third convolutional layer; the blocks in the left half of the SFT can be referred to from left to right as: the first block, the second block and the third block; the blocks in the right half of the SFT can be referred to from right to left as: the first block, the second block and the third block. For example, the first convolutional layer of TAFT inputs the feature F t 1 , respectively input the first block of the left half of SFT and the first block of the right half of SFT; the second convolution layer of TAFT takes the feature F t 2 , respectively input the second block of the left half of SFT and the second block of the right half of SFT; the third convolution layer of TAFT takes the feature F t 3 , input the 3rd Block in the left half of SFT and the 3rd Block in the right half of SFT respectively.

[0153] For example, for each Block of the SFT, the first feature adjustment parameter and the second feature adjustment parameter may be determined according to the following formula (2):

[0154] A n ,B n =M θ n (F t n ) (2)

[0155] Among them, A n is the first feature adjustment parameter corresponding to the nth Block in SFT, B nM is the second feature adjustment parameter corresponding to the nth Block in SFT. θ n It is the network parameter of the nth Block in SFT.

[0156] For example, referring to formula (2), assuming t=T, n is 1, 2 and 3, then F T 1 Input to the first Block in the left half of SFT and the first Block in the right half of SFT (the parameters of these two Blocks are M θ 1 ), and obtain the corresponding first feature adjustment parameter A 1 and the second characteristic adjustment parameter B 1 ; F T 2 Input to the second block of the left half of SFT and the second block of the right half of SFT (the parameters of these two blocks are M θ 2 ), and obtain the corresponding first feature adjustment parameter A 2 and the second characteristic adjustment parameter B 2 ; F T 3 Input to the third Block in the left half of SFT and the third Block in the right half of SFT (the parameters of these two Blocks are M θ 3 ), and obtain the corresponding first feature adjustment parameter A 3 and the second characteristic adjustment parameter B 3 .

[0157] Then, the state at time t can be determined by referring to the following formula (3):

[0158] Among them, φ t is a hyperparameter of the diffusion model 214.

[0159] And to determine the state at time t-1, refer to the following formula (4):

[0160] Among them, φ t-1 is a hyperparameter of the diffusion model 214.

[0161] Then, randomly select a time information (such as T) from 1 to T and Z t Input into U-NET and get (That is, the predicted state (or image) of the T-1th time information). For example, when the time information t=T, Z T Input into U-NET and get )

[0162] FIG2E is a schematic diagram illustrating an exemplary feature adjustment process.

[0163] Exemplarily, an adjustment module is provided between the second feature transformation model 213 and the diffusion model 214 in FIG. 2A and FIG. 2B ; wherein, the adjustment module can be implemented by an algorithm, and the adjustment module can be used to execute feature adjustment parameters based on the output of the second feature transformation model 213, and adjust the intermediate features generated during the feature generation process of the diffusion model 214.

[0164] 2E , exemplarily, the intermediate features output by the network layer 1 of the U-NET and the feature adjustment parameters output by a Block of the SFT can be input into the adjustment module; the adjustment module adjusts the intermediate features output by the network layer 1 of the U-NET based on the feature adjustment parameters output by the SFT, obtains the adjusted intermediate features, and inputs the adjusted intermediate features into the network layer 2 of the U-NET.

[0165] In one possible approach, the adjustment module can adjust the intermediate features generated during the feature generation process of U-NET according to the following formula (5):

[0166] in, The intermediate features generated during the feature generation process of U-NET (for example, the intermediate features output by the network layer 1 of U-NET in Figure 2E), is the adjusted intermediate feature (e.g., the feature input to network layer 2 of U-NET in Figure 2E).

[0167] Among them, after the intermediate features generated in the process of U-NET feature generation are adjusted, the original features and detail features contained in the intermediate features generated in the process of U-NET feature generation are weakened; therefore, compared with C n =A n In terms of n =(A n +1) retains the intermediate features before adjustment in the adjusted intermediate features, that is, the original features and detail features contained in the intermediate features before adjustment are added to the adjusted intermediate features; in this way, the accuracy of the features input to the next network layer can be guaranteed, thereby increasing the accuracy of the state of the obtained t-1th time information.

[0168] It should be noted that the output of U-NET is the predicted noise (that is, the noise predicted by U-NET) (ε θis a general term for the network parameters of TATF, SFT and U-NET). Subsequently, the noise addition principle and predicted noise can be used to determine Refer to the following formulas (6) and (7):

[0169] Among them, φ t ,φ t-1 is a hyperparameter of the diffusion model 214.

[0170] For example, it can be minimized As the target, back propagation is performed for SFT and TAFT, and the network parameters of SFT and TAFT are adjusted. In order to reduce the amount of calculation, the loss function can be changed from Convert to Loss = ||ε-ε θ (Z t ,t,y)||; Then, minimize Loss=||ε-ε θ (Z t ,t,y)|| is the target, and back propagation is performed on SFT and TAFT to adjust the network parameters of SFT and TAFT.

[0171] In this way, the SFT and TAFT can be trained using one training image. Then, more training images can be used to train the SFT and TAFT in this way until the second preset condition is met. The second preset condition can be set as required, for example, Loss = ||ε-ε θ (Z t ,t,y)|| is less than a second threshold; for another example, the number of training times for SFT and TAFT reaches a second predicted number; for another example, the difference between the losses obtained from two adjacent training sessions is less than a second preset difference, etc.; this application is not limited to this. The second preset number, the second preset difference, and the second threshold can be set as required and are not limited to this application.

[0172] After completing the training of SFT and TAFT, SFT, TAFT, U-NET and Decoder (Decoder belongs to the above-mentioned image generation model and is obtained by training the above-mentioned image generation model. This application can optimize the training of the Decoder or not, as will be explained later) can be used for decoding during the decoding process.

[0173] The following describes the image decoding process based on the compression framework shown in FIG. 3A to FIG. 3D .

[0174] In order to facilitate the description of the decoding process, the image encoding process can be described with reference to FIG3E .

[0175] 3E , illustratively, the entropy estimation module 112 (211) can be implemented by a super-prior network, which can include a super-encoding network 1121, a probability estimation model 1122, and a super-decoding network 1123. It should be understood that FIG3E is only an example of the present application, and the super-prior network of the present application can include more or fewer networks or models, and the present application does not limit this. In addition, the entropy estimation module 112 (211) can also be implemented by other networks, and the present application does not limit this.

[0176] Illustratively, the super encoding network 1121 and the super decoding network 1123 may include network layers such as convolutional layers, deconvolutional layers, and activation layers, which is not limited in this application.

[0177] Exemplarily, the entropy coding unit 120 in FIG. 3A to FIG. 3D may include the entropy coding unit A1 ( 1201 ) and the entropy coding unit A2 ( 1202 ) in FIG. 3E .

[0178] Exemplarily, the encoding process of an image may be as follows:

[0179] First, the image is input into IC Encoder, which extracts features from the image to obtain the feature map of the image. And the feature map of the image Input to the super coding network 1121 and the entropy coding unit A1 (1201).

[0180] Then, the super coding network 1121 can process the feature map of the image Processing is performed to obtain side information features of the image; and the side information features of the image are input into the entropy coding unit A2 (1202) and the probability estimation model 1122.

[0181] Exemplarily, the probability estimation model 1122 can be a factorized entropy model; the factorized entropy model can determine the probability distribution of the side information features of the image (i.e., probability distribution 1) based on the side information features of the image, and output the probability distribution 1 to the entropy encoding unit A2 (1202) and the entropy decoding unit 300.

[0182] Afterwards, the entropy coding unit A2 (1202) may perform entropy coding (eg, asymmetric numeral systems (ANS)) on the side information features of the image according to the probability distribution 1 to obtain a first bitstream.

[0183] Subsequently, the entropy decoding unit 300 may perform entropy decoding on the first code stream according to the probability distribution 1 to obtain the side information features of the image; and output the side information features of the image to the super decoding network 1123.

[0184] Then, the super decoding network 1123 can determine the probability distribution of the feature map of the image (i.e., probability distribution 2; for example, probability distribution 2 is a Gaussian probability distribution, and probability distribution 2 may include a mean and a variance) based on the side information features of the image; and output probability distribution 2 to the entropy coding unit A1 (1201).

[0185] Exemplarily, the entropy coding unit A1 (1201) may perform entropy coding on the feature map of the image according to probability distribution 2 to obtain a second code stream.

[0186] That is, the code streams in FIG. 3A to FIG. 3D may include a first code stream and a second code stream.

[0187] The decoding process is described below.

[0188] FIG4 is a schematic diagram illustrating an exemplary decoding process.

[0189] S401, receiving a code stream.

[0190] Exemplarily, the decoding end receives a code stream, wherein the code stream may include a first code stream and a second code stream, wherein the first code stream may be a code stream obtained by entropy encoding side information features of an image, and the second code stream may be a code stream obtained by entropy encoding a feature map of the image.

[0191] S402: Acquire first image information based on the code stream.

[0192] Referring to FIG3A or FIG3C , for example, in one possible approach, entropy decoding can be performed on the first bitstream to obtain image side information features, and the image side information features can be used as the first image information. Specifically, the decoding end can also include the probability estimation model 1122 and the super decoding network 1123 shown in FIG3E ; the probability estimation model 1122 can determine the probability distribution of the image side information features based on the first bitstream, and output the probability distribution of the image side information features to the entropy decoding unit 220; the entropy decoding unit 220 can perform entropy decoding (e.g., asymmetric coefficient entropy decoding) on ​​the first bitstream based on the probability distribution of the image side information features to obtain the image side information features.

[0193] 3A or 3C , illustratively, in one possible approach, the first bitstream can be entropy decoded to obtain image side information features; the second bitstream can be entropy decoded based on the image side information features to obtain an image feature map; and the image feature map is used as the first image information. The method for obtaining the image side information features can refer to the above description and will not be repeated here. Next, the image side information features can be output to the super decoding network 1123 to obtain a probability distribution of the image feature map (for example, the mean μ and variance σ of a Gaussian probability distribution); and the probability distribution of the image feature map is output to the entropy decoding unit 220. The entropy decoding unit 220 can entropy decode the second bitstream (for example, asymmetric coefficient entropy decoding) based on the probability distribution of the image feature map to obtain the image feature map.

[0194] 3B or 3D , illustratively, in one possible approach, the first bitstream can be entropy decoded to obtain image side information features; the second bitstream can be entropy decoded based on the image side information features to obtain an image feature map; the image feature map can be decoded to obtain an initial reconstructed image and / or decoded features of the initial reconstructed image; and the initial reconstructed image and / or decoded features of the initial reconstructed image can be used as the first image information. The method for obtaining the image side information features and the image feature map can refer to the above description and will not be repeated here. Exemplarily, the image feature map can be input into an IC Decoder, which decodes the image feature map and outputs the initial reconstructed image. Features are generated during the process of the IC Decoder decoding the image feature map; in this case, the features generated during the process of the IC Decoder decoding the image feature map can be referred to as decoded features of the initial reconstructed image. It should be understood that during the process of the IC Decoder decoding the image feature map, only some steps can be performed (wherein, performing all steps can output the initial reconstructed image), and in this case, decoded features of the initial reconstructed image can also be obtained.

[0195] It should be noted that, in the present application, at least one of the feature map of the image, the side information feature of the image, the initial reconstructed image, or the decoding feature of the initial reconstructed image may be used as the first image information.

[0196] S403: Perform feature transformation on the first image information to obtain features of the first image information.

[0197] Exemplarily, when the first image information includes at least two of the image feature map, the image side information feature, the initial reconstructed image or the decoding feature of the initial reconstructed image, the at least two types of information can be fused (such as weighted calculation) and input into the TAFT; or the at least two types of information can be input into the TAFT as at least two input information channels, and this application does not impose any restrictions on this.

[0198] 3A to 3D , the first image information and time information t may be input into the TAFT, which performs feature transformation on the first image information and outputs features of the first image information.

[0199] Based on the description of the above training process, the value range of the time information t can be 0 to T; in the order of T to 1, one time information can be selected each time (that is, T is selected for the first time, T-1 is selected for the second time, T-2 is selected for the third time, and so on, until t=1) and the first image information can be input into the TAFT to obtain the features of the first image information corresponding to the time information output by the TAFT. Among them, a single time information corresponds to one or more features of the first image information, and the scales of the multiple features of the first image information corresponding to a single time information are different. For example, in any of the schematic diagrams in Figures 3A to 3D, a single time information corresponds to 3 features of the first image information, and the scales of these 3 features are different.

[0200] S404: Input the second image information into the diffusion model to obtain target features; wherein the features of the first image information are used to adjust intermediate features generated during feature generation processing of the diffusion model, and the second image information includes one of a preset noise image, an initial reconstructed image determined based on a bitstream, or an image obtained by adding noise to the initial reconstructed image.

[0201] Exemplarily, the second image information can be determined, the second image information and the time information t can be input into the diffusion model 214, and the features of the first image information can be input into the second feature transformation model 213; the diffusion model 214 performs feature generation processing on the second image information based on the time information, and the second feature transformation model 213 performs feature transformation on the features of the first image information, and outputs feature adjustment parameters (including first feature adjustment parameters and second feature adjustment parameters); and in the process of feature generation processing by the diffusion model 214, the adjustment module uses the feature adjustment parameters to adjust the intermediate features generated in the process of feature generation processing by the diffusion model 214; thereafter, the diffusion model 214 continues to perform feature generation processing based on the adjusted features, and outputs the target features.

[0202] For example, the network structure of the denoising process of the diffusion model 214 is U-NET, and the second feature transformation model 213 is SFT as an example for explanation: Initially, T can be selected as the value of the time information t and input into the U-NET, and the second image information can be input into the U-NET, and the U-NET performs feature generation processing on the second image information; at the same time, in the process of U-NET performing feature generation processing on the second image information, the feature adjustment parameter determined based on the feature of the first image information corresponding to the moment t=T (that is, the feature of the first image information corresponding to the T-th time information) is used to adjust the intermediate features generated in the feature generation process of the U-NET to obtain the adjusted features (the specific adjustment method can refer to the above formula (2) and formula (5), which will not be repeated here). Afterwards, the U-NET continues to perform feature generation processing based on the adjusted features and outputs the predicted noise of the T-1th time information; then, based on the predicted noise of the T-1th time information and the above formula (6) and formula (7), the features of the T-1th time information (that is, the features in Figures 3A, 3B, 3C and 3D) are determined. That is, the feature of the predicted T-1th time information).

[0203] Next, t is subtracted by 1, and the following steps are performed cyclically until t is equal to 1: the t-th time information and the features of the t-th time information are input into U-NET, and U-NET performs feature generation processing on the features of the t-th time information; at the same time, in the process of U-NET performing feature generation processing on the features of the t-th time information, the feature adjustment parameters determined based on the features of the first image information corresponding to the time t (i.e., the features of the first image information corresponding to the t-th time information) are used to adjust the intermediate features generated in the feature generation process of U-NET to obtain the adjusted features. After that, U-NET continues to perform feature generation processing based on the adjusted features and outputs the predicted noise of the t-1-th time information; then, based on the predicted noise of the t-1-th time information and the above formulas (6) and (7), the features of the t-1-th time information are determined and t is subtracted by 1. In this way, after T-1 cycles, when t is equal to 1, the features of the t-1-th time information are the target features (i.e., the features in Figures 3A, 3B, 3C, and 3D). That is, the feature of the predicted 0th time information).

[0204] S405: reconstruct the target features to obtain a target reconstructed image.

[0205] 3A or 3B , for example, in one possible approach, the target features can be input into a decoder, which then reconstructs the target features to obtain a target reconstructed image. In this case, the present application does not need to train the decoder obtained by training the image generation model.

[0206] 3C or 3D , exemplarily, in one possible approach, the feature map of the image in the first image information, the side information features of the image, and at least one of the decoding features of the initial reconstructed image, as well as the target features can be input into the Decoder to obtain the target reconstructed image. In this case, the present application can train the Decoder obtained by training the image generation model. Specifically, the training image information (including the feature map of the training image, the side information features of the training image, the initial reconstructed image of the training image, and the decoding features of the initial reconstructed image of the training image) can be determined, and then S403 and S404 can be executed to determine the corresponding features (referred to as the fourth features for ease of description); the fourth features, as well as at least one of the feature map of the training image, the side information features of the training image, and the initial reconstructed image of the training image, are input into the Decoder to obtain the target reconstructed image of the training image; then, the training image and the target reconstructed image of the training image can be compared to determine the loss value; and the Decoder is back-propagated based on the loss value.

[0207] For example, in one possible approach, at least one of the image feature map, image side information features, and decoded features of the initial reconstructed image in the first image information can be input into a third feature transformation model to obtain feature 1; feature 1 and the target feature are input into a fusion model to obtain feature 2; and feature 2 is input into a decoder to obtain the target reconstructed image. In this case, the present application does not need to train the decoder obtained by training the image generation model; only the third feature transformation model and the fusion model need to be trained. For details, please refer to the decoder training process, which will not be repeated here.

[0208] It should be noted that S401 to S405 are steps in the decoding process.

[0209] Since TAFT, SFT, and U-NET processing introduce errors and weaken the original features and detail features contained in the first image information, the first image information is more accurate than the target features. Furthermore, inputting the first image information into the decoder can balance the detail richness and fidelity of the target reconstructed image.

[0210] First, some prior art techniques directly input the feature map of an image obtained from entropy decoding of a bitstream into an image decoder for decoding to obtain a reconstructed image. However, the present application first determines the features of the first image information (including the image feature map) obtained from the bitstream, then uses a diffusion model 214 to perform feature generation processing on the second image information to obtain target features, and uses the features of the first image information to adjust the intermediate features generated during the feature generation process of the diffusion model. The decoder (which is different from the image decoder) then reconstructs the target features to obtain a reconstructed image (i.e., a target reconstructed image). Because the diffusion model 214 learns general image information (such as texture information) during training, the diffusion model 214 has a stronger generation capability than the image decoder of the prior art, and therefore requires less information during the reconstruction process. Consequently, under the same subjective quality, the bitstream of the present application carries less information. In other words, under the same bitstream size, the reconstructed image obtained by the decoding method of the present application has better subjective quality.

[0211] Secondly, some prior art techniques use the initial reconstructed image as a control condition for adjusting the intermediate features generated during the feature generation process of diffusion model 214. In other words, the diffusion model 214 of the prior art operates in the image domain. However, the present application uses the features of the first image information as a control condition for adjusting the intermediate features generated during the feature generation process of diffusion model 214. In other words, the diffusion model 214 of the present application operates in the feature domain. The scale of the feature domain is smaller than that of the image domain. Therefore, the present application can reduce the computational complexity of diffusion model 214 and improve the efficiency of the decoding process. (Here, scale can be understood as size. By downsampling an image at multiple multiples, features at multiple scales can be obtained.)

[0212] Again, in the field of image super-resolution, some existing technologies use a stable diffusion model (SD) for super-resolution. Among them, the stable diffusion model may include: a time-aware feature transformer network (TAFT), a spatial feature transformation network (SFT), a U-NET, an encoder, and a decoder. In the super-resolution process, a low-resolution image is input into the encoder to obtain the first feature; the first feature is then input into the TAFT to obtain the second feature; then the second feature and a preset noise image are input into the generative model (including U-NET and SFT) to obtain the third feature; then, the third feature is input into the decoder to obtain a high-resolution image. Compared with the existing technology: first, the existing technology inputs the image into the Encoder, and then inputs the features output by the Encoder into the TAFT; while in the decoding process, the present application obtains the first image information from the code stream and then determines the features of the first image information (for example, the first image information can be input into the first feature transformation model 212 such as the TAFT). That is to say, the present application does not use the Encoder in the decoding process. Secondly, in the prior art, the first feature and the third feature belong to the same domain (that is, the dimension of the first feature is the same as the dimension of the third feature, and the physical meaning of each dimension is the same; for example, the dimension of the first feature is m1*m2, the dimension of the third feature is m1*m2, and m1 and m2 are positive integers); in other words, the first feature matches the decoder, that is, only by inputting the first feature into the decoder, a reconstructed image can be obtained; while in the present application, the features of the first image information and the target features belong to different domains (that is, the dimensions of the features of the first image information and the dimensions of the target features are different, and the physical meaning of each dimension is different; for example, the dimensions of the features of the first image information are m3*m4, the dimensions of the target features are m5*m6, m3~m6 are positive integers, m3 is not equal to m5 or m4 is not equal to m6); in other words, the features of the first image information do not match the decoder, that is, only by inputting the features of the first image information into the decoder, a reconstructed image cannot be obtained.

[0213] 5A and 5B are schematic diagrams illustrating exemplary reconstructed images.

[0214] Figure 5A is a reconstructed image obtained using the decoding method of the prior art, and Figure 5B is a reconstructed image (i.e., the target reconstructed image) obtained using the decoding method of the present application. Comparing Figures 5A and 5B, Figure 5B has finer textures and clearer details, i.e., Figure 5B has better subjective quality.

[0215] 5C and 5D are schematic diagrams illustrating exemplary reconstructed images.

[0216] Figure 5C is a reconstructed image obtained using the decoding method of the prior art, and Figure 5D is a reconstructed image (i.e., the target reconstructed image) obtained using the decoding method of the present application. Comparing Figure 5C and Figure 5D, Figure 5D has finer texture and clearer details, i.e., Figure 5D has better subjective quality.

[0217] Figure 6 is a schematic diagram of an exemplary apparatus for decoding an image. The apparatus for decoding an image can be used to perform the method of the aforementioned embodiment. Therefore, the beneficial effects achieved by the apparatus can refer to the beneficial effects of the corresponding method provided above and are not further described here.

[0218] 6 , the apparatus for decoding an image may include:

[0219] Receiving module 601, used for receiving code stream;

[0220] An information acquisition module 602 is configured to acquire first image information based on a code stream;

[0221] The feature determination module 603 is used to perform feature transformation on the first image information to obtain features of the first image information;

[0222] A feature generation module 604 is configured to input the second image information into the diffusion model 214 to obtain target features. The features of the first image information are used to adjust intermediate features generated during feature generation processing by the diffusion model 214. The second image information includes one of a preset noise image, an initial reconstructed image determined based on a bitstream, or an image obtained by adding noise to the initial reconstructed image.

[0223] The reconstruction module 605 is used to reconstruct the target features to obtain a target reconstructed image.

[0224] Exemplarily, the reconstruction module 605 is configured to perform reconstruction processing on the first image information and target features to obtain a target reconstructed image.

[0225] Exemplarily, the reconstruction module 605 is specifically configured to input the first image information and the target feature into the first decoding network 216 to obtain a target reconstructed image.

[0226] Exemplarily, the feature determination module 603 is specifically configured to determine the features of the first image information, including: inputting the first image information and time information into the first feature transformation model 212 to obtain the features of the first image information.

[0227] Exemplarily, a feature of the first image information corresponds to a plurality of time information, one piece of time information in the plurality of time information corresponds to one or more features of the first image information, and the scales of the plurality of features of the first image information are different.

[0228] Exemplarily, the feature generation module 604 is specifically used to input the t-th time information and the second image information into the diffusion model 214 to obtain the features of the t-1-th time information; wherein, the initial value of t is T, and T is a positive integer; t is subtracted by 1, and the following steps are performed in a loop until t is equal to 1: the t-th time information and the features of the t-th time information are input into the diffusion model 214 to obtain the features of the t-1-th time information and t is subtracted by 1; wherein, the features of the first image information correspond to multiple time information, and the features of the first image information corresponding to the t-th time information are used to adjust the intermediate features generated in the process of the diffusion model 214 generating the features of the t-1-th time information; when t is equal to 1, the features of the t-1-th time information are the target features.

[0229] Exemplarily, the apparatus for decoding an image further includes a feature adjustment module:

[0230] The feature adjustment module is used to input the features of the first image information into the second feature transformation model 213 to obtain feature adjustment parameters; based on the feature adjustment parameters, adjust the intermediate features generated during the feature generation process of the diffusion model 214.

[0231] Exemplarily, the feature adjustment module is specifically used for feature adjustment parameters including a first feature adjustment parameter and a second feature adjustment parameter. Based on the feature adjustment parameters, the intermediate features generated during the feature generation process of the diffusion model 214 are adjusted, including: determining the product value between the first feature adjustment parameter and the intermediate features generated during the feature generation process of the diffusion model 214; determining the sum value between the product value and the second feature adjustment parameter; and determining the adjusted intermediate features based on the sum value.

[0232] Exemplarily, the feature adjustment module is specifically configured to add the sum value to the intermediate feature generated during the feature generation process performed by the diffusion model 214 to obtain the adjusted intermediate feature.

[0233] Exemplarily, the first image information includes at least one of a feature map of the image, a side information feature of the image, an initial reconstructed image, or a decoded feature of the initial reconstructed image;

[0234] Among them, the feature map of the image and the side information features of the image are obtained by entropy decoding the code stream by the entropy decoding unit 220, and the initial reconstructed image is obtained by decoding the feature map of the image by the second decoding network 215. The decoding features of the initial reconstructed image are the features generated in the process of decoding the feature map of the image by the second decoding network 215.

[0235] Exemplarily, the information acquisition module 602 is specifically configured to input the first code stream into the entropy decoding unit 220 to obtain the side information features of the image; use the side information features of the image as the first image information; and / or

[0236] Inputting the first bitstream into the entropy decoding unit 220 to obtain side information features of the image; inputting the side information features of the image and the second bitstream into the entropy decoding unit 220 to obtain a feature map of the image; using the feature map of the image as the first image information; and / or

[0237] The first code stream is input into the entropy decoding unit 220 to obtain side information features of the image; the side information features of the image and the second code stream are input into the entropy decoding unit 220 to obtain a feature map of the image; the feature map of the image is input into the second decoding network 215 to obtain an initial reconstructed image and / or decoding features of the initial reconstructed image; the initial reconstructed image and / or the decoding features of the initial reconstructed image are used as the first image information.

[0238] In an example, FIG7 shows a schematic block diagram of an apparatus 700 according to an embodiment of the present application. The apparatus 700 may include: a processor 701 and a transceiver / transceiver pin 702 , and optionally, a memory 703 .

[0239] The various components of the device 700 are coupled together via a bus 704, wherein the bus 704 includes, in addition to a data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the various buses are collectively referred to as bus 704 in the figure.

[0240] Optionally, the memory 703 may be used to store instructions in the aforementioned method embodiment. The processor 701 may be used to execute the instructions in the memory 703 and control the receiving pin to receive a signal and control the transmitting pin to send a signal.

[0241] The apparatus 700 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0242] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0243] The present application also provides a chip including one or more interface circuits and one or more processors. The one or more processors receive or send data via the one or more interface circuits. When the one or more processors execute computer instructions, the steps of the above-mentioned related methods are implemented. The interface circuit is a transceiver / transceiver pin 702.

[0244] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0245] This embodiment further provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer executes the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0246] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the methods in the above-mentioned method embodiments.

[0247] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0248] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0249] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0250] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0251] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0252] Any content of each embodiment of this application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0253] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0254] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disc (CD-ROM) or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0255] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0256] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A method for decoding an image, characterized in that: The method comprises: Receive code stream; Acquire first image information based on the code stream; Performing feature transformation on the first image information to obtain features of the first image information; Inputting the second image information into the diffusion model to obtain the target features; wherein the second image information includes one of a preset noise image, an initial reconstructed image determined based on the bitstream, or an image obtained by adding noise to the initial reconstructed image, and the features of the first image information are used to adjust the intermediate features generated in the feature generation process of the diffusion model; The target features are reconstructed to obtain a target reconstructed image.

2. The method according to claim 1, characterized in that The step of reconstructing the target features to obtain a target reconstructed image includes: The first image information and the target features are reconstructed to obtain the target reconstructed image.

3. The method according to claim 2, characterized in that The reconstructing the first image information and the target feature to obtain the target reconstructed image includes: The first image information and the target features are input into a first decoding network to obtain the target reconstructed image.

4. The method according to any one of claims 1 to 3, characterized in that: The performing feature transformation on the first image information to obtain features of the first image information includes: The first image information and time information are input into a first feature transformation model to obtain features of the first image information.

5. The method according to any one of claims 1 to 4, characterized in that: The features of the first image information correspond to a plurality of time information, one time information among the plurality of time information corresponds to one or more features of the first image information, and the scales of the plurality of features of the first image information are different.

6. The method according to claim 5, characterized in that The step of inputting the second image information into the diffusion model to obtain the target feature includes: Inputting the t-th time information and the second image information into the diffusion model to obtain the features of the t-1-th time information; wherein the initial value of t is T, and T is a positive integer; Subtract 1 from t and repeat the following steps until t is equal to 1: Input the t-th time information and the feature of the t-th time information into the diffusion model, obtain the feature of the t-1-th time information and reduce t by 1; Among them, the features of the first image information correspond to multiple time information, and the features of the first image information corresponding to the t-th time information are used to adjust the intermediate features generated in the process of generating the features of the t-1-th time information by the diffusion model; when t is equal to 1, the features of the t-1-th time information are the target features.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Inputting the features of the first image information into the second feature transformation model to obtain feature adjustment parameters; Based on the feature adjustment parameter, the intermediate features generated during the feature generation process of the diffusion model are adjusted.

8. The method according to claim 7, characterized in that The feature adjustment parameters include a first feature adjustment parameter and a second feature adjustment parameter, and adjusting the intermediate features generated in the feature generation process of the diffusion model based on the feature adjustment parameters includes: Determine a product value between the first feature adjustment parameter and an intermediate feature generated in a feature generation process performed by the diffusion model; determining a sum value between the product value and the second feature adjustment parameter; Based on the sum value, an adjusted intermediate feature is determined.

9. The method according to claim 8, characterized in that The step of determining the adjusted intermediate feature based on the sum value comprises: The sum value is added to the intermediate feature generated in the feature generation process of the diffusion model to obtain the adjusted intermediate feature.

10. The method according to any one of claims 1 to 9, characterized in that: The first image information includes at least one of a feature map of the image, a side information feature of the image, the initial reconstructed image, or a decoded feature of the initial reconstructed image; Among them, the feature map of the image and the side information features of the image are obtained by entropy decoding the code stream by an entropy decoding unit, the initial reconstructed image is obtained by decoding the feature map of the image by a second decoding network, and the decoding features of the initial reconstructed image are features generated in the process of decoding the feature map of the image by the second decoding network.

11. The method according to any one of claims 1 to 10, characterized in that: The code stream includes a first code stream and a second code stream, and the acquiring the first image information based on the code stream includes: Inputting the first bitstream into an entropy decoding unit to obtain side information features of the image; using the side information features of the image as the first image information; and / or Inputting the first bitstream into the entropy decoding unit to obtain side information features of the image; inputting the side information features of the image and the second bitstream into the entropy decoding unit to obtain a feature map of the image; using the feature map of the image as the first image information; and / or Inputting the first code stream into the entropy decoding unit to obtain side information features of the image; inputting the side information features of the image and the second code stream into the entropy decoding unit to obtain a feature map of the image; inputting the feature map of the image into a second decoding network to obtain the initial reconstructed image and / or the decoding features of the initial reconstructed image; and using the initial reconstructed image and / or the decoding features of the initial reconstructed image as the first image information.

12. An image decoding device, characterized in that: The image decoding device comprises an entropy decoding unit, a first feature transformation model, a diffusion model and a first decoding network; wherein: The entropy decoding unit is used to perform entropy decoding on the received code stream to obtain the first image information; The first feature transformation model is used to perform feature transformation on the first image information to obtain features of the first image information; The diffusion model is used to perform feature generation processing on the second image information to obtain target features; wherein the second image information includes one of a preset noise image, an initial reconstructed image determined based on the bitstream, or an image obtained by adding noise to the initial reconstructed image, and the features of the first image information are used to adjust intermediate features generated in the feature generation processing of the diffusion model; The first decoding network is used to reconstruct the target features to obtain a target reconstructed image.

13. The device according to claim 12, characterized in that The first decoding network is specifically used to reconstruct the first image information and the target features to obtain the target reconstructed image.

14. The device according to claim 12 or 13, characterized in that The image decoding device also includes a second feature transformation model, The second feature transformation model is used to perform feature transformation on the features of the first image information to obtain feature adjustment parameters; wherein the feature adjustment parameters are used to adjust the intermediate features generated during the feature generation process of the diffusion model.

15. The device according to claim 12 or 13, characterized in that The first image information includes at least one of a feature map of the image or a side information feature of the image, and the feature map of the image or the side information feature of the image is output information of the entropy decoding unit.

16. The device according to any one of claims 12 to 14, characterized in that The image decoding device further comprises a second decoding network, The second decoding network is used to decode the feature map of the image; The feature map of the image is the output information of the entropy decoding unit, the first image information includes at least one of the initial reconstructed image and the decoding feature of the initial reconstructed image, the initial reconstructed image is the output information of the second decoding network, and the initial The decoded features of the initially reconstructed image are features generated in the process of the second decoding network decoding the feature map of the image.

17. A device for decoding an image, characterized in that: The device for decoding an image comprises: A receiving module, used for receiving a code stream; An information acquisition module, used for acquiring first image information based on the code stream; a feature determination module, configured to perform feature transformation on the first image information to obtain features of the first image information; a feature generation module, configured to input the second image information into the diffusion model to obtain target features; wherein the second image information includes one of a preset noise image, an initial reconstructed image determined based on the bitstream, or an image obtained by adding noise to the initial reconstructed image, and the features of the first image information are used to adjust intermediate features generated during the feature generation process of the diffusion model; The reconstruction module is used to reconstruct the target features to obtain a target reconstructed image.

18. A decoding device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the decoding device executes the image decoding method according to any one of claims 1 to 11.

19. A decoding device, characterized in that: It comprises one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the method as claimed in any one of claims 1 to 11 are executed.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the image decoding method according to any one of claims 1 to 11.

21. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a computer or a processor, the steps of the method according to any one of claims 1 to 11 are performed.

Images