Image decoding method and device

Abstract

The embodiment of the invention provides an image decoding method and device. The method comprises the following steps: firstly, receiving a code stream; then, acquiring first image information based on the code stream; performing feature transformation on the first image information to obtain features of the first image information; then, inputting the second image information into a diffusion model to obtain a target feature; wherein the features of the first image information are used for adjusting intermediate features generated in the feature generation processing process of the diffusion model, and the second image information comprises one of a preset noise image, an initial reconstruction image determined based on a code stream or an image obtained by adding noise to the initial reconstruction image; and then, performing reconstruction processing on the target features to obtain a target reconstruction image. In this way, the subjective quality of the reconstructed image can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of coding and decoding, and in particular to a method and device for decoding an image. Background Art

[0002] As videos evolve from high-definition to ultra-high-definition, people have higher and higher requirements for video quality. At the same time, high-definition videos have higher and higher requirements for bandwidth and storage. Accordingly, due to cost control considerations for bandwidth, transmission delay, and storage, the demand for video encoding, i.e., video processing, has become more and more urgent.

[0003] Among them, the artificial intelligence (AI) video compression (or codec) algorithm is based on deep learning (for example, based on the variational auto-encoder (VAE)), and has better compression effect than traditional video compression technologies (such as H265 and H266).

[0004] However, the VAE-based image compression scheme also has certain defects, namely, the reconstructed image obtained by decoding is blurry or has pseudo textures, and the subjective quality of the reconstructed image is relatively poor. Summary of the invention

[0005] In view of this, the present application provides an image decoding method and device; the image decoding method can improve the subjective quality of the reconstructed image.

[0006] In a first aspect, an embodiment of the present application provides a decoding method, which includes: first, receiving a code stream; then, obtaining first image information based on the code stream; and performing feature transformation on the first image information to obtain features of the first image information; then, inputting the second image information into a diffusion model to obtain target features; wherein the features of the first image information are used to adjust intermediate features generated during feature generation processing of the diffusion model, and the second image information includes a preset noise image, an initial reconstructed image determined based on the code stream, or an image obtained by adding noise to the initial reconstructed image; thereafter, reconstructing the target features to obtain a target reconstructed image.

[0007] First, some prior art technologies directly input the feature map of the image obtained from the bitstream entropy decoding into the image decoder for decoding to obtain the reconstructed image; while the present application first determines the features of the first image information (including the feature map of the image) obtained from the bitstream, then uses the diffusion model to perform feature generation processing on the second image information to obtain the target features and uses the features of the first image information to adjust the intermediate features generated in the feature generation processing of the diffusion model, and then uses the decoder (the decoder is different from the image decoder) to reconstruct the target features to obtain the reconstructed image (i.e., the target reconstructed image). Since the diffusion model learns the general information of the image (such as texture information) during the training process, the diffusion model has a stronger generation capability than the image decoder of the prior art, so the diffusion model requires less information during the feature generation processing; and thus, under the same subjective quality, the bitstream of the present application carries less information. In other words, under the same bitstream size, the subjective quality of the reconstructed image obtained by the decoding method of the present application is better.

[0008] Secondly, some prior art uses the initial reconstructed image as a control condition for the intermediate features generated in the feature generation process of the adjustment diffusion model, that is, the diffusion model of the prior art acts on the image domain; while the present application uses the features of the first image information as a control condition for the intermediate features generated in the feature generation process of the adjustment diffusion model, that is, the diffusion model of the present application acts on the feature domain; the scale of the feature domain is smaller than the scale of the image domain, so the present application can reduce the computational complexity of the diffusion model and improve the efficiency of the decoding process. (Wherein, scale, i.e., size, can be understood as size, and features of multiple scales can be obtained by downsampling an image by multiple multiples.)

[0009] Again, in the field of image super-resolution, some existing technologies use a stable diffusion model (SD) for super-resolution. Among them, the stable diffusion model may include: a time-aware feature transformer network (TAFT), a spatial feature transformation network (SFT), U-NET, an encoder, and a decoder. In the super-resolution process, a low-resolution image is input into the encoder to obtain a first feature; the first feature is then input into the TAFT to obtain a second feature; then the second feature and a preset noise image are input into a generation model (including U-NET and SFT) to obtain a third feature; then, the third feature is input into the decoder to obtain a high-resolution image. Compared with the prior art: first, the prior art inputs the image into the Encoder, and then inputs the features of the Encoder output into the TAFT; while in the decoding process, the present application obtains the first image information from the code stream and then determines the features of the first image information (for example, the first image information can be input into the first feature transformation model such as TAFT), that is, the present application does not use the Encoder in the decoding process. Secondly, in the prior art, the first feature and the third feature belong to the same domain (that is, the dimension of the first feature is the same as the dimension of the third feature, and the physical meaning of each dimension is the same; for example, the dimension of the first feature is m1*m2, the dimension of the third feature is m1*m2, and m1 and m2 are positive integers); in other words, the first feature matches the Decoder, that is, only by inputting the first feature into the Decoder, a reconstructed image can be obtained; while in the present application, the feature of the first image information and the target feature belong to different domains (that is, the dimension of the feature of the first image information and the dimension of the target feature are different, and the physical meaning of each dimension is different; for example, the dimension of the feature of the first image information is m3*m4, the dimension of the target feature is m5*m6, m3~m6 are positive integers, m3 is not equal to m5 or m4 is not equal to m6); in other words, the feature of the first image information does not match the Decoder, that is, only by inputting the feature of the first image information into the Decoder, a reconstructed image cannot be obtained.

[0010] Exemplarily, a diffusion model is a model for generating images / features, which can be implemented by a neural network. Exemplarily, the diffusion model may include a convolutional layer. In one possible manner, the diffusion model may include at least one convolutional layer and at least one activation layer. In one possible manner, the diffusion model may include at least one convolutional layer and at least one normalization layer. In one possible manner, the diffusion model may include at least one convolutional layer, at least one activation layer, and at least one normalization layer. For example, the network structure adopted in the denoising process (or feature generation process) of the diffusion model may be a network structure such as U-NET; it should be understood that the present application does not limit the network structure of the diffusion model.

[0011] Exemplarily, the intermediate features generated during the feature generation process of the diffusion model may refer to the features of the input layer output and the features of the intermediate layer output included in the processing of the diffusion model.

[0012] Exemplarily, the preset noise image can be obtained by sampling Gaussian noise.

[0013] Exemplarily, the first image information may be at least one of an image or a feature.

[0014] It should be noted that the second image information can be understood as input information of the diffusion model, and the features of the first image information can be understood as control information of the diffusion model.

[0015] It should be noted that the decoding method of the present application can be applied to decoding images / videos.

[0016] According to the first aspect, reconstructing the target features to obtain a target reconstructed image includes: reconstructing the first image information and the target features to obtain a target reconstructed image.

[0017] Since the determination of the features of the first image information and the feature generation process of the diffusion model will introduce errors and weaken the original features and detail features contained in the first image information, the first image information is more accurate than the target features. Therefore, the reconstruction of the first image information and the target features determined based on the code stream can balance the detail richness and fidelity of the target reconstructed image.

[0018] According to the first aspect, or any implementation of the first aspect above, the first image information and the target feature are reconstructed to obtain a target reconstructed image, including: inputting the first image information and the target feature into a first decoding network to obtain a target reconstructed image.

[0019] Among them, the first decoding network can be implemented by a neural network, and the first decoding network can be a Decoder; that is, the present application performs AI reconstruction on the first image information and target features; compared with traditional reconstruction methods, the present application can improve the subjective quality of the target reconstructed image obtained by decoding.

[0020] According to the first aspect, or any implementation of the first aspect above, feature transformation is performed on the first image information to obtain features of the first image information, including: inputting the first image information and time information into a first feature transformation model to obtain features of the first image information.

[0021] On the one hand, the first feature transformation model may refer to a model used for feature transformation. The first feature transformation model may be implemented by a neural network and may include multiple convolutional layers. For example, the first feature transformation model is TAFT. In this way, the use of an AI network for feature transformation may improve the accuracy of the features of the obtained first image information, thereby improving the subjective quality of the target reconstructed image.

[0022] On the other hand, since the diffusion model performs denoising step by step, that is, the denoising process of the diffusion model is associated with time, therefore, inputting the time information into the first feature transformation model can enable the first feature transformation model to perceive the time change and generate the features of the first image information corresponding to different time information, thereby more accurately guiding the feature adjustment and improving the accuracy of the target features, thereby improving the subjective quality of the target reconstructed image.

[0023] According to the first aspect, or any implementation of the first aspect above, the features of the first image information correspond to multiple time information, one time information in the multiple time information corresponds to one or more features of the first image information, and the scales of the multiple features of the first image information are different.

[0024] According to the first aspect, or any implementation method of the first aspect above, the second image information is input into the diffusion model to obtain the target feature, including: the tth time information and the second image information are input into the diffusion model to obtain the feature of the t-1th time information; wherein the initial value of t is T, and T is a positive integer; t is subtracted by 1, and the following steps are repeatedly performed until t is equal to 1: the tth time information and the feature of the tth time information are input into the diffusion model to obtain the feature of the t-1th time information and t is subtracted by 1; wherein the feature of the first image information corresponds to multiple time information, and the feature of the first image information corresponding to the tth time information is used to adjust the intermediate feature generated in the process of the diffusion model generating the feature of the t-1th time information; when t is equal to 1, the feature of the t-1th time information is the target feature.

[0025] The value of t in the t-th time information varies in different cycles.

[0026] According to the first aspect, or any implementation of the first aspect above, the method also includes: inputting the features of the first image information into a second feature transformation model to obtain feature adjustment parameters; based on the feature adjustment parameters, adjusting the intermediate features generated during the feature generation process of the diffusion model.

[0027] Exemplarily, the second feature transformation model may be a model for performing feature transformation, the second feature transformation model may be implemented by a neural network, the second feature transformation module may include multiple Blocks, each Block may include at least one convolutional layer, for example, the second feature model may be an SFT. In this way, the features of the first image information may be transformed into feature adjustment parameters that can directly adjust the intermediate features generated during the feature generation process of the diffusion model.

[0028] According to the first aspect, or any implementation method of the first aspect above, the feature adjustment parameter includes a first feature adjustment parameter and a second feature adjustment parameter, and based on the feature adjustment parameter, the intermediate features generated during the feature generation process of the diffusion model are adjusted, including: determining the product value between the first feature adjustment parameter and the intermediate features generated during the feature generation process of the diffusion model; determining the sum value between the product value and the second feature adjustment parameter; based on the sum value, determining the adjusted intermediate features.

[0029] According to the first aspect, or any implementation of the first aspect above, determining the adjusted intermediate feature based on the sum value includes: adding the sum value to the intermediate feature generated during the feature generation process of the diffusion model to obtain the adjusted intermediate feature.

[0030] After feature adjustment is performed on the intermediate features generated in the feature generation process of the diffusion model, the original features and detail features contained in the intermediate features generated in the feature generation process of the diffusion model are weakened; and then the intermediate features before the adjustment are added to the adjusted intermediate features, and the original features and detail features contained in the intermediate features before the adjustment can be added to the adjusted intermediate features; in this way, the accuracy of the features input to the next network layer of the diffusion model can be guaranteed, and then the feature accuracy of the obtained t-1th time information can be increased; thereby improving the subjective quality of the target reconstructed image.

[0031] According to the first aspect, or any implementation manner of the first aspect above, the first image information includes at least one of a feature map of the image, a side information feature of the image, an initial reconstructed image, or a decoding feature of the initial reconstructed image; wherein the feature map of the image and the side information feature of the image are obtained by entropy decoding the bitstream by an entropy decoding unit, the initial reconstructed image is obtained by decoding the feature map of the image by a second decoding network, and the decoding feature of the initial reconstructed image is a feature generated in the process of decoding the feature map of the image by the second decoding network.

[0032] It should be noted that the first image information used for reconstructing the target reconstructed image with the target features may include at least one of a feature map of the image, a side information feature of the image, or a decoded feature of the initial reconstructed image.

[0033] According to the first aspect, or any implementation of the first aspect, the code stream includes a first code stream and a second code stream, and obtaining the first image information based on the code stream includes:

[0034] Inputting the first code stream into an entropy decoding unit to obtain side information features of the image; using the side information features of the image as first image information; and / or

[0035] Inputting the first bitstream into an entropy decoding unit to obtain side information features of the image; inputting the side information features of the image and the second bitstream into the entropy decoding unit to obtain a feature map of the image; using the feature map of the image as the first image information; and / or

[0036] The first bitstream is input into an entropy decoding unit to obtain side information features of the image; the side information features of the image and the second bitstream are input into the entropy decoding unit to obtain a feature map of the image; the feature map of the image is input into a second decoding network to obtain an initial reconstructed image and / or a decoding feature of the initial reconstructed image; the initial reconstructed image and / or the decoding feature of the initial reconstructed image is used as the first image information.

[0037] Exemplarily, after obtaining the side information features of the image, the probability distribution of the feature map of the image can be determined based on the side information features of the image; then, based on the probability distribution of the feature map of the image, the second bitstream is entropy decoded to obtain the feature map of the image. The probability distribution of the feature map of the image can be a Gaussian distribution parameter, such as a mean and a variance.

[0038] In a second aspect, an embodiment of the present application provides an image decoding device, the image decoding device comprising an entropy decoding unit, a first feature transformation model, a diffusion model and a first decoding network; wherein:

[0039] An entropy decoding unit, configured to perform entropy decoding on the received code stream to obtain first image information;

[0040] A first feature transformation model is used to perform feature transformation on the first image information to obtain features of the first image information;

[0041] A diffusion model is used to perform feature generation processing on the second image information to obtain target features; wherein the features of the first image information are used to adjust intermediate features generated during the feature generation processing of the diffusion model, and the second image information includes one of a preset noise image, an initial reconstructed image determined based on a bitstream, or an image obtained by adding noise to the initial reconstructed image;

[0042] The first decoding network is used to reconstruct the target features to obtain a target reconstructed image.

[0043] According to the second aspect, the first decoding network is specifically used to reconstruct the first image information and the target features to obtain a target reconstructed image.

[0044] According to the second aspect, or any implementation of the second aspect above, the image decoding device further includes a second feature transformation model,

[0045] The second feature transformation model is used to perform feature transformation on the features of the first image information to obtain feature adjustment parameters; wherein the feature adjustment parameters are used to adjust the intermediate features generated during the feature generation process of the diffusion model.

[0046] According to the second aspect, or any implementation of the second aspect above, the first image information includes at least one of a feature map of the image or a side information feature of the image, and the feature map of the image or the side information feature of the image is output information of an entropy decoding unit.

[0047] According to the second aspect, or any implementation of the second aspect above, the image decoding device further includes a second decoding network,

[0048] A second decoding network is used to decode the feature map of the image;

[0049] Among them, the feature map of the image is the output information of the entropy decoding unit, the first image information includes at least one of the initial reconstructed image and the decoding features of the initial reconstructed image, the initial reconstructed image is the output information of the second decoding network, and the decoding features of the initial reconstructed image are the features generated in the process of the second decoding network decoding the feature map of the image.

[0050] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0051] In a third aspect, the present application provides a device for decoding an image, the device for decoding an image comprising:

[0052] A receiving module, used for receiving a code stream;

[0053] An information acquisition module, used for acquiring first image information based on a code stream;

[0054] A feature determination module, used for performing feature transformation on the first image information to obtain features of the first image information;

[0055] a feature generation module, configured to input the second image information into the diffusion model to obtain target features; wherein the features of the first image information are used to adjust intermediate features generated during the feature generation process of the diffusion model, and the second image information includes one of a preset noise image, an initial reconstructed image determined based on a bitstream, or an image obtained by adding noise to the initial reconstructed image;

[0056] The reconstruction module is used to reconstruct the target features to obtain a target reconstructed image.

[0057] It should be understood that the above-mentioned device for decoding can be used to execute the decoding method in the first aspect or any possible implementation of the first aspect.

[0058] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0059] In a fourth aspect, the present application provides a coding method, which includes: acquiring an image; performing feature extraction on the image to obtain a feature map of the image; and performing entropy coding on the image to obtain a code stream.

[0060] The code stream may correspond to the second code stream in the first aspect or any implementation of the first aspect above.

[0061] According to the fourth aspect, the method also includes: performing probability estimation on the feature map of the image to obtain the probability distribution of the feature map of the image; performing entropy encoding on the image to obtain a code stream, including: performing entropy encoding on the feature map of the image based on the probability distribution of the feature map of the image to obtain a code stream.

[0062] According to the fourth aspect, or any implementation of the fourth aspect, performing probability estimation on a feature map of an image to obtain a probability distribution of the feature map of the image includes:

[0063] Feature extraction is performed on the feature map of the image to obtain the side information features of the image; based on the side information features of the image, the probability distribution of the feature map of the image is determined.

[0064] According to the fourth aspect, or any implementation of the fourth aspect above, the method further includes: performing entropy coding on the side information features of the image. At this time, the first bit stream in the first aspect or any implementation of the first aspect above can be obtained.

[0065] In a fifth aspect, an embodiment of the present application provides a decoding device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the electronic device to execute the image decoding method in the first aspect or any possible implementation of the first aspect.

[0066] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0067] In a sixth aspect, an embodiment of the present application provides a decoding device, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the electronic device to execute the image decoding method in the first aspect or any possible implementation of the first aspect.

[0068] The sixth aspect and any implementation of the sixth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0069] In the seventh aspect, an embodiment of the present application provides a chip, comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the image decoding method in the first aspect or any possible implementation of the first aspect are executed.

[0070] The seventh aspect and any implementation of the seventh aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the seventh aspect and any implementation of the seventh aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0071] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the image decoding method in the first aspect or any possible implementation of the first aspect.

[0072] The eighth aspect and any implementation of the eighth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the eighth aspect and any implementation of the eighth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0073] In a ninth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the image decoding method in the first aspect or any possible implementation of the first aspect.

[0074] The ninth aspect and any implementation of the ninth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the ninth aspect and any implementation of the ninth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1A is a schematic diagram of an application framework shown as an example;

[0076] Figure 1B A schematic diagram of a storage framework is shown as an example;

[0077] Figure 1C is a schematic diagram of an exemplary transmission framework;

[0078] Figure 2A is a schematic diagram of a compression framework shown as an example;

[0079] Figure 2B is a schematic diagram of a compression framework shown as an example;

[0080] Figure 2C is a schematic diagram of an exemplary training framework;

[0081] Figure 2D is a schematic diagram of an exemplary training framework;

[0082] Figure 2E A schematic diagram of a characteristic adjustment process is shown as an example;

[0083] Figure 3Ais a schematic diagram of a compression framework shown as an example;

[0084] Figure 3B is a schematic diagram of a compression framework shown as an example;

[0085] Figure 3C is a schematic diagram of a compression framework shown as an example;

[0086] Figure 3D is a schematic diagram of a compression framework shown as an example;

[0087] Figure 3E is a schematic diagram of an exemplary encoding process;

[0088] Figure 4 is a schematic diagram of an exemplary decoding process;

[0089] Figure 5A is a schematic diagram of a reconstructed image shown as an example;

[0090] Figure 5B is a schematic diagram of a reconstructed image shown as an example;

[0091] Figure 5C is a schematic diagram of a reconstructed image shown as an example;

[0092] Figure 5D is a schematic diagram of a reconstructed image shown as an example;

[0093] Figure 6 is a schematic diagram of an exemplary device for decoding;

[0094] Figure 7 Schematic diagram of the structure of the device shown as an example. DETAILED DESCRIPTION

[0095] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0096] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0097] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0098] In the embodiments of the present application, the words "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0099] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0100] This application can be applied to all services of image / video transmission and image / video storage, such as terminal photography, terminal video, photo album, Huawei Cloud, live broadcast, etc., and this application does not impose any restrictions on this.

[0101] Exemplarily, the decoding method of the present application can be applied to decode an image / video. The present application is described by taking an image as an example.

[0102] Figure 1A The following is a schematic diagram of an application framework shown as an example.

[0103] Reference Figure 1A For example, the camera can capture an image to obtain an image; then, the AI ​​encoding module 100 can perform AI encoding on the image to obtain a code stream. Afterwards, the code stream can be stored locally or transmitted to a remote device.

[0104] Continue to refer to Figure 1A In one possible manner, after the code stream is stored locally, when an image needs to be played or edited, the AI ​​decoding module 200 can perform AI decoding on the code stream to obtain a reconstructed image; subsequently, the reconstructed image can be played or edited.

[0105] Continue to refer to Figure 1A In one possible manner, after the remote device receives the code stream, when it needs to play or edit an image, the AI ​​decoding module 200 can perform AI decoding on the code stream to obtain a reconstructed image; then, the reconstructed image can be played or edited.

[0106] Figure 1BA schematic diagram of a storage framework is shown as an example.

[0107] Reference Figure 1B , exemplarily, the AI ​​encoding module 100 may include an AI encoding unit 110 and an entropy encoding unit 120, and the AI ​​decoding module 200 may include an AI decoding unit 210 and an entropy decoding unit 220; it should be understood that, Figure 1B This is only an example of the present application. The AI ​​encoding module 100 and the AI ​​decoding module 200 may include Figure 1B More units are shown, but the present application is not limited thereto.

[0108] Exemplarily, the AI ​​encoding unit 110 and the AI ​​decoding unit 210 may be arranged in an embedded neural network processor (Neural network Processing Unit, NPU) or a graphics processor (Graphics Processing Unit, GPU). Exemplarily, the entropy encoding unit 120 and the entropy decoding unit 220 may be arranged in a central processing unit (Central Processing Unit, CPU). Exemplarily, the file saving module and the file loading module may be arranged in a CPU.

[0109] Reference Figure 1B For example, after the camera captures an image, the image can be input to the AI ​​coding unit 110. Then, the AI ​​coding unit 110 can extract features from the image to obtain a feature map of the image, and output the feature map of the image to the entropy coding unit 120. After that, the entropy coding unit 120 can entropy encode the feature map of the image to obtain a code stream; then, the code stream is output to the file saving module for saving to obtain a file.

[0110] Continue to refer to Figure 1B For example, when an image needs to be played or edited, the file loading module can load the file and then output the code stream in the file to the entropy decoding unit 220. Then, the entropy decoding unit 220 can entropy decode the feature map of the image from the code stream and output the feature map of the image to the AI ​​decoding unit 210. Then, the AI ​​decoding unit 210 can decode the feature map of the image to obtain a reconstructed image.

[0111] Figure 1C FIG. 1 is a schematic diagram of an exemplary transmission framework.

[0112] Reference Figure 1CFor example, after the camera at the encoding end captures an image, the image can be input to the AI ​​encoding unit 110. Then, the AI ​​encoding unit 110 can extract features from the image to obtain a feature map of the image, and output the feature map of the image to the entropy encoding unit 120. Afterwards, the entropy encoding unit 120 can entropy encode the feature map of the image to obtain a bitstream, and then send the bitstream to the server.

[0113] Continue to refer to Figure 1C For example, after receiving the bitstream sent by the server, the decoding end may input the bitstream into the entropy decoding unit 220. Then, the entropy decoding unit 220 may entropy decode the feature map of the image from the bitstream, and output the feature map of the image to the AI ​​decoding unit 210. Then, the AI ​​decoding unit 210 may decode the feature map of the image to obtain a reconstructed image.

[0114] It should be understood that the encoding end and the decoding end can be directly connected; in this way, the encoding end can directly send the code stream to the decoding end without forwarding through other devices (such as a server), and this application does not impose any restrictions on this.

[0115] Figure 2A A schematic diagram of a compression framework is shown as an example. Figure 2A Shown is a compressed framework of the present application.

[0116] Reference Figure 2A , exemplarily, the AI ​​encoding unit 110 may include an encoding network 111 and an entropy estimation module 112 (211). It should be understood that, Figure 2A This is only an example of the present application. The AI ​​encoding unit 110 may include more models or modules, and the present application does not limit this.

[0117] Continue to refer to Figure 2A , exemplarily, the AI ​​decoding unit 210 may include a first feature transformation model 212, a second feature transformation model 213, a diffusion model 214, a decoding network 215 and an entropy estimation module 112 (211). It should be understood that, Figure 2A This is only an example of the present application. The AI ​​decoding unit 210 may include more or fewer models or modules, and the present application does not limit this.

[0118] Figure 2B A schematic diagram of a compression framework is shown as an example. Figure 2B Shown is another compressed framework of the present application.

[0119] Reference Figure 2B , exemplarily, the AI ​​encoding unit 110 may include an encoding network 111 and an entropy estimation module 112 (211). It should be understood that, Figure 2BThis is only an example of the present application. The AI ​​encoding unit 110 may include more models or modules, and the present application does not limit this.

[0120] Continue to refer to Figure 2B , exemplarily, the AI ​​decoding unit 210 includes a first decoding network 216, a first feature transformation model 212, a second feature transformation model 213, a diffusion model 214, a second decoding network 215 and an entropy estimation module 112 (211). It should be understood that, Figure 2B This is only an example of the present application. The AI ​​decoding unit 210 may include more or fewer models or modules, and the present application does not limit this.

[0121] For example, Figure 2A and Figure 2B Each model / network in can be implemented by a neural network, such as deep neural networks (DNN), convolutional neural networks (CNN), recurrent neural networks (RNN), residual networks, neural networks using transformer models or other neural networks, etc., and this application does not impose any restrictions on this.

[0122] For example, the work of each layer in a neural network can be expressed mathematically as To describe: From a physical perspective, the work of each layer in a neural network can be understood as completing the transformation from input space to output space (i.e., from row space to column space of a matrix) through five operations on the input space (a set of input vectors). These five operations include: 1. Dimensionality increase / decrease; 2. Enlargement / reduction; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are represented by Completed, operation 4 is completed by +b, and operation 5 is implemented by a(). The word "space" is used here because the object being classified is not a single thing, but a class of things, and space refers to the collection of all individuals of this class of things. Among them, W is a weight vector, and each value in the vector represents the weight value of a neuron in this layer of neural network. The vector W determines the spatial transformation from the input space to the output space above, that is, the weight W of each layer controls how to transform the space.

[0123] Exemplarily, the diffusion model 214 is a model for generating images / features, and the diffusion model 214 may include a convolutional layer. In one possible manner, the diffusion model 214 may include at least one convolutional layer and at least one activation layer. In one possible manner, the diffusion model 214 may include at least one convolutional layer and at least one normalization layer. In one possible manner, the diffusion model 214 may include at least one convolutional layer, at least one activation layer, and at least one normalization layer. For example, the network structure adopted in the denoising process (or feature generation process) of the diffusion model 214 may be a network structure such as U-NET; it should be understood that the present application does not limit the network structure of the diffusion model 214.

[0124] Exemplarily, the first feature transformation model 212 and the second feature transformation model 213 may include convolutional layers.

[0125] In the following, the encoding network 111 is an image encoder (Image Compression Encoder, IC Encoder), the first decoding network 216 is an image decoder (Image Compression Decoder, IC Decoder), the network structure adopted by the first feature transformation model 212 is a time-aware feature transformation network (Time-Aware Feature Transformer, TAFT), the network structure adopted by the second feature transformation model 213 is a spatial feature transformation network (spatial feature transformations, SFT), and the network structure adopted by the denoising process of the diffusion model 214 is U-NET. Figure 2B The second decoding network 215 (or Figure 2A The decoding network 215 in FIG. 1 is taken as a decoder for explanation.

[0126] For example, the above networks can be trained in advance so that the trained networks can be used for encoding and decoding later. The purpose of training the neural network is to finally obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by many layers of vectors W). Therefore, the training process of the neural network is essentially to learn how to control spatial transformation, and more specifically to learn the weight matrix.

[0127] Figure 2C A schematic diagram of an exemplary training framework is shown.

[0128] Reference Figure 2C , ,illustratively, ,IC Encoder and IC Decoder may ,include network layers such as convolution layer, deconvolution layer, activation layer, and other ,layers, and the present application is

[0129] For example, for Figure 2C The training process of IC Encoder and IC Decoder can be as follows:

[0130] First, the training image is input into the IC Encoder, and the IC Encoder extracts features of the training image to obtain a feature map of the training image.

[0131] Next, the feature map of the training image is input into the entropy estimation module 112 (211) and the IC Decoder.

[0132] For example, the IC Decoder may decode the feature map of the training image to obtain a reconstructed image. Then, based on the training image and the reconstructed image, a distortion loss may be determined (the distortion loss may refer to the difference (such as error) between the training image and the reconstructed image, i.e. Figure 2C The Loss1 in , can also be called image reconstruction loss). For example, the mean square error (MSE) between the training image and the reconstructed image can be calculated as Loss1.

[0133] Furthermore, the entropy estimation module 112 (211) can determine the bit rate loss (that is, the estimated bit rate of the encoded training image) based on the feature map of the training image, that is, Figure 2C Loss2 in.

[0134] Next, we can determine the training Figure 2C The final loss Loss of IC Encoder and IC Decoder can be referred to the following formula (1):

[0135] Loss = α Loss1 + Loss2 (1)

[0136] Among them, α in formula (1) is the weight coefficient of distortion loss, which can be set according to requirements and is not limited in this application.

[0137] It should be understood that other losses (e.g., perceptual loss (i.e., learned perceptual image patch similarity (LPIPS))) can also be introduced, and other losses can be combined with the above Loss1 and Loss2 (e.g., weighted) to obtain the training loss. Figure 2C The final loss Loss of IC Encoder and IC Decoder is not limited in this application.

[0138] After that, we can minimize the Loss in formula (1) as the goal. Figure 2C The IC Encoder and IC Decoder in the training are trained until the first preset condition is met. The first preset condition can be set as required, for example, the number of training times reaches the first preset number of times; for example, the difference in Loss obtained from two adjacent trainings is less than the first preset difference; for example, Loss is less than the first threshold, etc.; this application does not impose any restrictions on this. The first preset number of times, the first preset difference and the first threshold can be set as required, and this application does not impose any restrictions on this.

[0139] Figure 2D A schematic diagram of an exemplary training framework is shown.

[0140] For example, Figure 2D TAFT, SFT, U-NET and Encoder belong to the image generation model (such as the stable diffusion model (SD), the image diffusion model may include: TAFT, SFT, U-NET, Encoder and Decoder). Among them, the training of the image generation model is completed, and the training of TAFT, SFT, U-NET, Encoder and Decoder is also completed; the specific training process of the image generation model can be referred to in the prior art, which will not be repeated here.

[0141] It should be noted that this application only optimizes and trains TAFT and SFT obtained by training the image generation model, and does not train U-NET and Encoder obtained by training the image generation model.

[0142] Exemplarily, TAFT may include n (n is a positive integer) downsampling layers (wherein the downsampling layers may be convolutional layers). Figure 2D n = 3, that is Figure 2D The TAFT shown in FIG. 1 includes three downsampling layers; wherein, Figure 2D A white rectangle in TAFT represents a downsampling layer. For example, Figure 2D The three downsampling layers included in TAFT have different downsampling multiples. Correspondingly, the three downsampling layers of TAFT output features of three scales (that is, output features of three sizes).

[0143] Exemplarily, the SFT may include 2n blocks, and a single block may include at least one convolutional layer. Figure 2D n = 3, that is Figure 2D The SFT shown in FIG. 1 includes 6 blocks; among them, Figure 2D A black rectangle in the SFT represents a Block.

[0144] It should be noted that, in this embodiment, U-NET is a U-shaped structure, and the output information of SFT is used to adjust the intermediate features generated in the feature generation process of U-NET; therefore, SFT can also be a U-shaped structure, such as Figure 2D As shown, the left half includes 3 Blocks and the right half includes 3 Blocks.

[0145] Reference Figure 2D , exemplarily, the training process of TAFT and SFT can be as follows:

[0146] For example, Figure 2D The t in it is the time information, the value of t ranges from 0 to T, and T is a positive integer.

[0147] On the one hand, the training image is input into the IC Encoder (the IC Encoder trained according to the above description) to obtain the feature map of the training image For example, if the size of the training image is 3×512×512 (where “×” does not represent multiplication, but represents the size or dimension of the training image, “3” represents the number of channels of the training image, the first “512” represents the length of the training image, and the second “512” represents the width of the training image), then the feature map of the training image is The size of may be 192×32×32. It should be understood that the present application does not limit the size of the training image and the feature map of the training image.

[0148] On the other hand, the training image can be input into the Encoder to obtain the image Z 0 (That is, the state (or image) at time information t=0 or the state (or image) at the 0th time information). Next, 0 Add noise and get image Z 1 (That is, the state (or image) when the time information is t=1 or the state (or image) of the first time information). 1 Add noise and get image Z 2 (that is, the state (or image) when the time information t=2 or the state (or image) of the second time information); ...; and so on, by gradually adding noise, the image Z can be obtained. T-1 and image Z T (That is, the state (or image) at the time information t=T or the state (or image) of the Tth time information.) This process is also the noise adding process of the diffusion model 214 .

[0149] For example, for each time information, a random Gaussian noise can be sampled to obtain a preset noise ε~N(0,1) (N(0,1) is a Gaussian distribution, and ε~N(0,1) indicates that the preset noise conforms to the Gaussian distribution); based on the preset noise sampled from the current time information, the image of the previous time information is denoised to obtain the image of the current time information; the specific denoising method can refer to the description in the prior art and will not be repeated here. The size of the sampled preset noise is similar to Z 0 For example, the size of the training image is 3×512×512, and the Z output by the Encoder is 0 The size of may be 4×64×64, and the size of the preset noise obtained by sampling may be 4×64×64.

[0150] For example, a time information can be randomly selected from 1 to T as the value of t, and the feature map of the training image and time information t are input into TAFT to obtain the characteristic and For example, t = T, then the feature map of the training image and time information T are input to TAFT, and we get and

[0151] Then, TAFT can and Input into SFT to obtain feature adjustment parameters: the first feature adjustment parameter and the second feature adjustment parameter. Exemplarily, TAFT can input the features of a scale into a Block in the left half of SFT and a Block in the right half of SFT respectively. For the convenience of description, the three convolutional layers in TAFT can be referred to from top to bottom as: the first convolutional layer, the second convolutional layer, and the third convolutional layer; the blocks in the left half of SFT are referred to from left to right as: the first Block, the second Block, and the third Block; the blocks in the right half of SFT are referred to from right to left as: the first Block, the second Block, and the third Block. Exemplarily, the first convolutional layer of TAFT inputs the feature Input the first block of the left half of SFT and the first block of the right half of SFT respectively; the second convolution layer of TAFT converts the features The second block in the left half of SFT and the second block in the right half of SFT are input respectively; the third convolution layer of TAFT converts the features Input the third block in the left half of the SFT and the third block in the right half of the SFT respectively.

[0152] For example, for each Block of the SFT, the first feature adjustment parameter and the second feature adjustment parameter may be determined according to the following formula (2):

[0153]

[0154] Among them, A n is the first feature adjustment parameter corresponding to the nth Block in SFT, B n is the second feature adjustment parameter corresponding to the nth Block in SFT, M θ n It is the network parameter of the nth Block in SFT.

[0155] For example, referring to formula (2), assuming that t=T, and n is 1, 2, and 3, then F T 1 Input to the first block of the left half of SFT and the first block of the right half of SFT (the parameters of these two blocks are M θ 1 ), and obtain the corresponding first feature adjustment parameter A 1 and the second characteristic adjustment parameter B 1 ; F T 2 Input to the second block of the left half of SFT and the second block of the right half of SFT (the parameters of these two blocks are M θ 2 ), and obtain the corresponding first feature adjustment parameter A 2 and the second characteristic adjustment parameter B 2 ; F T 3 Input to the third block of the left half of SFT and the third block of the right half of SFT (the parameters of these two blocks are M θ 3 ), and obtain the corresponding first feature adjustment parameter A 3 and the second characteristic adjustment parameter B 3 .

[0156] Then, the state at time t can be determined by referring to the following formula (3):

[0157]

[0158] Among them, φ t is a hyperparameter of the diffusion model 214.

[0159] And to determine the state at time t-1, refer to the following formula (4):

[0160]

[0161] Among them, φ t-1 is a hyperparameter of the diffusion model 214.

[0162] Then, randomly select a time information (such as T) from 1 to T and Z t Input into U-NET and get (That is, the predicted state (or image) of the T-1th time information). For example, when the time information t=T, Z T Input into U-NET and get )

[0163] Figure 2E The figure is a schematic diagram of an exemplary feature adjustment process.

[0164] For example, Figure 2A and Figure 2B An adjustment module is provided between the second feature transformation model 213 and the diffusion model 214; wherein the adjustment module can be implemented by an algorithm, and the adjustment module can be used to execute feature adjustment parameters based on the output of the second feature transformation model 213, and adjust the intermediate features generated during the feature generation process of the diffusion model 214.

[0165] Reference Figure 2E For example, the intermediate features output by network layer 1 of U-NET and the feature adjustment parameters output by a Block of SFT can be input into the adjustment module; the adjustment module adjusts the intermediate features output by network layer 1 of U-NET based on the feature adjustment parameters output by SFT, obtains the adjusted intermediate features and inputs the adjusted intermediate features into network layer 2 of U-NET.

[0166] In one possible approach, the adjustment module can adjust the intermediate features generated during the feature generation process of U-NET according to the following formula (5):

[0167]

[0168] in, The intermediate features generated during the feature generation process of U-NET (for example, Figure 2E The intermediate features output by layer 1 of U-NET in is the adjusted intermediate feature (for example, input to Figure 2E Characteristics of network layer 2 of U-NET in ).

[0169] Among them, after the intermediate features generated in the process of U-NET feature generation are adjusted, the original features and detail features contained in the intermediate features generated in the process of U-NET feature generation are weakened; therefore, compared with C n =A n For C n =(A n +1) retains the intermediate features before adjustment in the intermediate features after adjustment, that is, the original features and detail features contained in the intermediate features before adjustment are added to the intermediate features after adjustment; in this way, the accuracy of the features input to the next network layer can be guaranteed, thereby increasing the accuracy of the state of the obtained t-1th time information.

[0170] It should be noted that the output of U-NET is the predicted noise (that is, the noise predicted by U-NET) (ε θ is a general term for the network parameters of TATF, SFT and U-NET). Subsequently, the noise can be determined based on the noise addition principle and predicted noise. Please refer to the following formula (6) and formula (7):

[0171]

[0172]

[0173] Among them, φ t ,φ t-1 is a hyperparameter of the diffusion model 214.

[0174] For example, it can be minimized As the target, back propagation is performed for SFT and TAFT, and the network parameters of SFT and TAFT are adjusted. In order to reduce the amount of calculation, the loss function can be changed from Convert to Loss = ||ε-ε θ (Z t ,t,y)||; then, minimize Loss=||ε-ε θ (Z t ,t,y)|| is taken as the target, and back propagation is performed on SFT and TAFT to adjust the network parameters of SFT and TAFT.

[0175] In this way, the SFT and TAFT can be trained using one training image. Then, more training images can be used to train the SFT and TAFT in this way until the second preset condition is met. The second preset condition can be set as required, for example, Loss = ||ε-ε θ (Z t,t,y)|| is less than the second threshold; for example, the number of trainings for SFT and TAFT reaches the second prediction number; for example, the difference between the Losses obtained from two adjacent trainings is less than the second preset difference, etc.; this application does not limit this. Among them, the second preset number, the second preset difference and the second threshold can be set as required, and this application does not limit this.

[0176] After completing the training of SFT and TAFT, SFT, TAFT, U-NET and Decoder (Decoder belongs to the above-mentioned image generation model, which is obtained by training the above-mentioned image generation model. This application may optimize the training of Decoder or may not optimize the training of Decoder, as will be described later) can be used for decoding during the decoding process.

[0177] The following Figure 3A to Figure 3D The image decoding process is described based on the compression framework shown.

[0178] In order to facilitate the description of the decoding process, please refer to Figure 3E , the image encoding process is explained.

[0179] Reference Figure 3E , exemplarily, the entropy estimation module 112 (211) may be implemented by a super priori network, which may include a super encoding network 1121, a probability estimation model 1122, and a super decoding network 1123. It should be understood that, Figure 3E This is only an example of the present application. The super-prior network of the present application may include more or less networks or models, and the present application does not limit this. In addition, the entropy estimation module 112 (211) may also be implemented by other networks, and the present application does not limit this.

[0180] Exemplarily, the super encoding network 1121 and the super decoding network 1123 may include network layers such as convolutional layers, deconvolutional layers, and activation layers, which is not limited in the present application.

[0181] For example, Figure 3A to Figure 3D The entropy coding unit 120 may include Figure 3E The entropy coding unit A1 (1201) and the entropy coding unit A2 (1202) in it.

[0182] Exemplarily, the encoding process of the image may be as follows:

[0183] First, the image is input into IC Encoder, which extracts features from the image to obtain the feature map of the image. And the feature map of the image Input to the super coding network 1121 and the entropy coding unit A1 (1201).

[0184] Next, the super coding network 1121 can process the feature map of the image Processing is performed to obtain the side information features of the image; and the side information features of the image are input into the entropy coding unit A2 (1202) and the probability estimation model 1122.

[0185] Exemplarily, the probability estimation model 1122 can be a factorized entropy model; the factorized entropy model can determine the probability distribution of the side information features of the image (i.e., probability distribution 1) based on the side information features of the image, and output the probability distribution 1 to the entropy encoding unit A2 (1202) and the entropy decoding unit 300.

[0186] Afterwards, the entropy coding unit A2 (1202) may perform entropy coding (eg, asymmetric numerical systems (ANS)) on the side information features of the image according to the probability distribution 1 to obtain a first bitstream.

[0187] Subsequently, the entropy decoding unit 300 may perform entropy decoding on the first bitstream according to the probability distribution 1 to obtain the side information features of the image; and output the side information features of the image to the super decoding network 1123.

[0188] Then, the super decoding network 1123 can determine the probability distribution of the feature map of the image (i.e., probability distribution 2; for example, probability distribution 2 is a Gaussian probability distribution, and probability distribution 2 may include a mean and a variance) based on the side information features of the image; and output probability distribution 2 to the entropy coding unit A1 (1201).

[0189] Exemplarily, the entropy coding unit A1 (1201) may perform entropy coding on the feature map of the image according to probability distribution 2 to obtain a second code stream.

[0190] That is to say, Figure 3A to Figure 3D The code stream in may include a first code stream and a second code stream.

[0191] The decoding process is described below.

[0192] Figure 4 FIG. 4 is a schematic diagram showing an exemplary decoding process.

[0193] S401, receiving a code stream.

[0194] Exemplarily, the decoding end receives a bitstream, wherein the bitstream may include a first bitstream and a second bitstream, the first bitstream may refer to a bitstream obtained by entropy encoding the side information features of the image, and the second bitstream may refer to a bitstream obtained by entropy encoding the feature map of the image.

[0195] S402: Acquire first image information based on the code stream.

[0196] Reference Figure 3A or Figure 3C , exemplarily, in one possible manner, the first bitstream may be entropy decoded to obtain the side information features of the image; and the side information features of the image are used as the first image information. Specifically, the decoding end may also include a reference Figure 3E The probability estimation model 1122 and the super decoding network 1123 in the image processing unit 1100; the rate estimation model 1122 can determine the probability distribution of the side information features of the image according to the first bitstream, and output the probability distribution of the side information features of the image to the entropy decoding unit 220; the entropy decoding unit 220 can perform entropy decoding (for example, asymmetric coefficient entropy decoding) on ​​the first bitstream based on the probability distribution of the side information features of the image to obtain the side information features of the image.

[0197] Reference Figure 3A or Figure 3C , exemplarily, in one possible manner, the first bitstream may be entropy decoded to obtain the side information features of the image; the second bitstream may be entropy decoded based on the side information features of the image to obtain the feature map of the image; and the feature map of the image may be used as the first image information. The manner of obtaining the side information features of the image may refer to the above description and will not be repeated here. Then, the side information features of the image may be output to the super decoding network 1123 to obtain the probability distribution of the feature map of the image (for example, the mean μ and variance σ of the Gaussian probability distribution); and the probability distribution of the feature map of the image may be output to the entropy decoding unit 220. The entropy decoding unit 220 may entropy decode the second bitstream (for example, asymmetric coefficient entropy decoding) according to the probability distribution of the feature map of the image to obtain the feature map of the image.

[0198] Reference Figure 3B or Figure 3D, exemplarily, in one possible manner, the first code stream may be entropy decoded to obtain the side information features of the image; the second code stream may be entropy decoded based on the side information features of the image to obtain the feature map of the image; the feature map of the image may be decoded to obtain the initial reconstructed image and / or the decoding features of the initial reconstructed image; the initial reconstructed image and / or the decoding features of the initial reconstructed image may be used as the first image information. Wherein, the manner of obtaining the side information features of the image and the feature map of the image may refer to the above description, and will not be repeated here. Exemplarily, the feature map of the image may be input into the IC Decoder, and the IC Decoder may decode the feature map of the image and output the initial reconstructed image. Wherein, features may be generated in the process of the IC Decoder decoding the feature map of the image; at this time, the features generated in the process of the IC Decoder decoding the feature map of the image may be referred to as the decoding features of the initial reconstructed image. It should be understood that in the process of the IC Decoder decoding the feature map of the image, only some steps may be executed (wherein, the initial reconstructed image may be output by executing all the steps), and at this time, the decoding features of the initial reconstructed image may also be obtained.

[0199] It should be noted that, in the present application, at least one of a feature map of an image, a side information feature of an image, an initial reconstructed image, or a decoded feature of an initial reconstructed image may be used as the first image information.

[0200] S403: Perform feature transformation on the first image information to obtain features of the first image information.

[0201] Exemplarily, when the first image information includes at least two of the feature map of the image, the side information feature of the image, the initial reconstructed image or the decoding feature of the initial reconstructed image, the at least two types of information can be fused (such as weighted calculation) and input into the TAFT; or the at least two types of information can be input into the TAFT as at least two paths of input information, and the present application does not impose any restrictions on this.

[0202] Reference Figure 3A to Figure 3D According to any of the schematic diagrams in the figure, the first image information and the time information t can be input into TAFT, and TAFT performs feature transformation on the first image information and outputs the features of the first image information.

[0203] Based on the description of the above training process, the value range of the time information t can be 0 to T; in the order of T to 1, one time information can be selected each time (that is, T is selected for the first time, T-1 is selected for the second time, T-2 is selected for the third time, and so on, until t=1) and the first image information can be input into TAFT to obtain the features of the first image information corresponding to the time information output by TAFT. Among them, a single time information corresponds to one or more features of the first image information, and the scales of the multiple features of the first image information corresponding to a single time information are different. For example, Figure 3A to Figure 3D In any of the schematic diagrams, a single piece of time information corresponds to three features of the first image information, and the scales of these three features are different.

[0204] S404, inputting the second image information into the diffusion model to obtain the target features; wherein the features of the first image information are used to adjust the intermediate features generated during the feature generation process of the diffusion model, and the second image information includes a preset noise image, an initial reconstructed image determined based on a bitstream, or an image obtained by adding noise to the initial reconstructed image.

[0205] Exemplarily, the second image information can be determined, the second image information and the time information t can be input into the diffusion model 214, and the features of the first image information can be input into the second feature transformation model 213; the diffusion model 214 performs feature generation processing on the second image information based on the time information, and the second feature transformation model 213 performs feature transformation on the features of the first image information, and outputs feature adjustment parameters (including first feature adjustment parameters and second feature adjustment parameters); and in the process of feature generation processing by the diffusion model 214, the adjustment module uses the feature adjustment parameters to adjust the intermediate features generated in the process of feature generation processing by the diffusion model 214; thereafter, the diffusion model 214 continues to perform feature generation processing based on the adjusted features, and outputs the target features.

[0206] Exemplarily, the network structure of the denoising process of the diffusion model 214 is U-NET, and the second feature transformation model 213 is SFT as an example for explanation: Initially, T can be selected as the value of the time information t and input into the U-NET, and the second image information can be input into the U-NET, and the U-NET performs feature generation processing on the second image information; at the same time, in the process of U-NET performing feature generation processing on the second image information, the feature adjustment parameters determined based on the features of the first image information corresponding to the moment t=T (that is, the features of the first image information corresponding to the T-th time information) are used to adjust the intermediate features generated in the feature generation process of the U-NET to obtain the adjusted features (the specific adjustment method can refer to the above formula (2) and formula (5), which will not be repeated here). Afterwards, the U-NET continues to perform feature generation processing based on the adjusted features and outputs the predicted noise of the T-1th time information; then, based on the predicted noise of the T-1th time information and the above formula (6) and formula (7), the features of the T-1th time information (that is, Figure 3A , Figure 3B , Figure 3C and Figure 3D Z T-1 , which is the feature of the predicted T-1th time information).

[0207] Next, subtract 1 from t, and loop through the following steps until t equals 1: input the t-th time information and the features of the t-th time information into U-NET, and let U-NET perform feature generation processing on the features of the t-th time information; at the same time, in the process of U-NET performing feature generation processing on the features of the t-th time information, use the feature adjustment parameters determined based on the features of the first image information corresponding to time t (i.e., the features of the first image information corresponding to the t-th time information) to adjust the intermediate features generated in the feature generation process of U-NET to obtain the adjusted features. Afterwards, U-NET continues to perform feature generation processing based on the adjusted features and outputs the predicted noise of the t-1th time information; then, based on the predicted noise of the t-1th time information and the above formulas (6) and (7), determine the features of the t-1th time information and subtract 1 from t. In this way, after T-1 cycles, when t equals 1, the features of the t-1th time information are the target features (i.e., Figure 3A , Figure 3B , Figure 3C and Figure 3D In That is, the feature of the predicted 0th time information).

[0208] S405, reconstructing the target features to obtain a target reconstructed image.

[0209] Reference Figure 3A or Figure 3B For example, in one possible approach, the target features can be input into a decoder, and the decoder reconstructs the target features to obtain a target reconstructed image. In this case, the present application does not need to train the decoder obtained by training the image generation model.

[0210] Reference Figure 3C or Figure 3D , exemplarily, in one possible manner, the feature map of the image in the first image information, the side information feature of the image, at least one of the decoding features of the initial reconstructed image, and the target feature can be input into the Decoder to obtain the target reconstructed image. In this case, the present application can train the Decoder obtained by training the image generation model. Specifically, the training image information (including the feature map of the training image, the side information feature of the training image, the initial reconstructed image of the training image, and the decoding features of the initial reconstructed image of the training image) can be determined, and then S403 and S404 can be executed to determine the corresponding features (referred to as the fourth features for ease of description); the fourth features, as well as the feature map of the training image, the side information feature of the training image, and at least one of the initial reconstructed image of the training image, are input into the Decoder to obtain the target reconstructed image of the training image; then, the training image and the target reconstructed image of the training image can be compared to determine the loss value; the Decoder is back-propagated based on the loss value.

[0211] Exemplarily, in one possible method, at least one of the feature map of the image in the first image information, the side information feature of the image, and the decoded feature of the initial reconstructed image can be input into the third feature transformation model to obtain feature 1; feature 1 and the target feature are input into the fusion model to obtain feature 2; feature 2 is input into the decoder to obtain the target reconstructed image. In this case, the present application does not need to train the decoder obtained by training the image generation model; only the third feature transformation model and the fusion model need to be trained. For details, please refer to the training process of the decoder, which will not be repeated here.

[0212] It should be noted that S401 to S405 are steps in the decoding process.

[0213] Since the processing of TAFT, SFT and U-NET will introduce errors and weaken the original features and detail features contained in the first image information, the first image information is more accurate than the target features. Therefore, the first image information is also input into the Decoder to balance the detail richness and fidelity of the target reconstructed image.

[0214] First, some prior art technologies directly input the feature map of the image obtained from the bitstream entropy decoding into the image decoder for decoding to obtain the reconstructed image; while the present application first determines the features of the first image information (including the feature map of the image) obtained from the bitstream, then uses the diffusion model 214 to perform feature generation processing on the second image information to obtain the target features and uses the features of the first image information to adjust the intermediate features generated in the feature generation processing of the diffusion model, and then uses the decoder (the decoder is different from the image decoder) to reconstruct the target features to obtain the reconstructed image (i.e., the target reconstructed image). Since the diffusion model 214 learns the general information of the image (such as texture information) during the training process, the diffusion model 214 has a stronger generation capability than the image decoder of the prior art, so the diffusion model 214 requires less information during the reconstruction process; and thus, under the same subjective quality, the bitstream of the present application carries less information. In other words, under the same bitstream size, the subjective quality of the reconstructed image obtained by the decoding method of the present application is better.

[0215] Secondly, some prior art uses the initial reconstructed image as a control condition for the intermediate features generated in the feature generation process of the diffusion model 214, that is, the diffusion model 214 of the prior art acts on the image domain; while the present application uses the features of the first image information as a control condition for the intermediate features generated in the feature generation process of the diffusion model 214, that is, the diffusion model 214 of the present application acts on the feature domain; the scale of the feature domain is smaller than the scale of the image domain, so the present application can reduce the computational complexity of the diffusion model 214 and improve the efficiency of the decoding process. (Wherein, scale, i.e., scale, can be understood as size, and features of multiple scales can be obtained by downsampling an image by multiple multiples.)

[0216] Again, in the field of image super-resolution, some existing technologies use a stable diffusion model (SD) for super-resolution. Among them, the stable diffusion model may include: a time-aware feature transformer network (TAFT), a spatial feature transformation network (SFT), U-NET, an encoder, and a decoder. In the super-resolution process, a low-resolution image is input into the encoder to obtain a first feature; the first feature is then input into the TAFT to obtain a second feature; then the second feature and a preset noise image are input into a generation model (including U-NET and SFT) to obtain a third feature; then, the third feature is input into the decoder to obtain a high-resolution image. Compared with the prior art: first, the prior art inputs the image into the Encoder, and then inputs the features of the Encoder output into the TAFT; while in the decoding process, the present application obtains the first image information from the code stream and then determines the features of the first image information (for example, the first image information can be input into the first feature transformation model 212 such as TAFT), that is, the present application does not use the Encoder in the decoding process. Secondly, in the prior art, the first feature and the third feature belong to the same domain (that is, the dimension of the first feature is the same as the dimension of the third feature, and the physical meaning of each dimension is the same; for example, the dimension of the first feature is m1*m2, the dimension of the third feature is m1*m2, and m1 and m2 are positive integers); in other words, the first feature matches the Decoder, that is, only by inputting the first feature into the Decoder, a reconstructed image can be obtained; while in the present application, the feature of the first image information and the target feature belong to different domains (that is, the dimension of the feature of the first image information and the dimension of the target feature are different, and the physical meaning of each dimension is different; for example, the dimension of the feature of the first image information is m3*m4, the dimension of the target feature is m5*m6, m3~m6 are positive integers, m3 is not equal to m5 or m4 is not equal to m6); in other words, the feature of the first image information does not match the Decoder, that is, only by inputting the feature of the first image information into the Decoder, a reconstructed image cannot be obtained.

[0217] Figure 5A and Figure 5B FIG. 1 is a schematic diagram of an exemplary reconstructed image.

[0218] Figure 5A is a reconstructed image obtained by using the decoding method of the prior art, Figure 5B is the reconstructed image (i.e., the target reconstructed image) obtained by using the decoding method of the present application. Figure 5A and Figure 5B , Figure 5B The texture is finer and the details are clearer, that is Figure 5B The subjective quality is better.

[0219] Figure 5C and Figure 5D FIG. 1 is a schematic diagram of an exemplary reconstructed image.

[0220] Figure 5C is a reconstructed image obtained by using the decoding method of the prior art, Figure 5D is the reconstructed image (i.e., the target reconstructed image) obtained by using the decoding method of the present application. Figure 5C and Figure 5D , Figure 5D The texture is finer and the details are clearer, that is Figure 5D The subjective quality is better.

[0221] Figure 6 The schematic diagram of an exemplary apparatus for decoding an image is shown in FIG. The apparatus for decoding an image can be used to execute the method of the aforementioned embodiment, and therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, which will not be described in detail here.

[0222] Reference Figure 6 , the apparatus for decoding an image may include:

[0223] Receiving module 601, used for receiving code stream;

[0224] The information acquisition module 602 is used to acquire first image information based on a code stream;

[0225] The feature determination module 603 is used to perform feature transformation on the first image information to obtain features of the first image information;

[0226] The feature generation module 604 is used to input the second image information into the diffusion model 214 to obtain the target feature; wherein the feature of the first image information is used to adjust the intermediate feature generated in the feature generation process of the diffusion model 214, and the second image information includes one of a preset noise image, an initial reconstructed image determined based on a bit stream, or an image obtained by adding noise to the initial reconstructed image;

[0227] The reconstruction module 605 is used to reconstruct the target features to obtain a target reconstructed image.

[0228] Exemplarily, the reconstruction module 605 is used to reconstruct the first image information and the target features to obtain a target reconstructed image.

[0229] Exemplarily, the reconstruction module 605 is specifically configured to input the first image information and the target feature into the first decoding network 216 to obtain a target reconstructed image.

[0230] Exemplarily, the feature determination module 603 is specifically used to determine the features of the first image information, including: inputting the first image information and time information into the first feature transformation model 212 to obtain the features of the first image information.

[0231] Exemplarily, a feature of the first image information corresponds to a plurality of time information, one time information in the plurality of time information corresponds to one or more features of the first image information, and the scales of the plurality of features of the first image information are different.

[0232] Exemplarily, the feature generation module 604 is specifically used to input the tth time information and the second image information into the diffusion model 214 to obtain the feature of the t-1th time information; wherein, the initial value of t is T, and T is a positive integer; t is subtracted by 1, and the following steps are performed repeatedly until t is equal to 1: the tth time information and the feature of the tth time information are input into the diffusion model 214 to obtain the feature of the t-1th time information and t is subtracted by 1; wherein, the feature of the first image information corresponds to multiple time information, and the feature of the first image information corresponding to the tth time information is used to adjust the intermediate feature generated in the process of the diffusion model 214 generating the feature of the t-1th time information; when t is equal to 1, the feature of the t-1th time information is the target feature.

[0233] Exemplarily, the apparatus for decoding an image further includes a feature adjustment module:

[0234] The feature adjustment module is used to input the features of the first image information into the second feature transformation model 213 to obtain feature adjustment parameters; based on the feature adjustment parameters, the intermediate features generated during the feature generation process of the diffusion model 214 are adjusted.

[0235] Exemplarily, the feature adjustment module is specifically used for feature adjustment parameters including a first feature adjustment parameter and a second feature adjustment parameter. Based on the feature adjustment parameters, the intermediate features generated during the feature generation process of the diffusion model 214 are adjusted, including: determining the product value between the first feature adjustment parameter and the intermediate features generated during the feature generation process of the diffusion model 214; determining the sum value between the product value and the second feature adjustment parameter; and determining the adjusted intermediate features based on the sum value.

[0236] Exemplarily, the feature adjustment module is specifically configured to add the sum value to the intermediate feature generated during the feature generation process performed by the diffusion model 214 to obtain the adjusted intermediate feature.

[0237] Exemplarily, the first image information includes at least one of a feature map of the image, a side information feature of the image, an initial reconstructed image, or a decoded feature of the initial reconstructed image;

[0238] Among them, the feature map of the image and the side information features of the image are obtained by entropy decoding the code stream by the entropy decoding unit 220, and the initial reconstructed image is obtained by decoding the feature map of the image by the second decoding network 215. The decoding features of the initial reconstructed image are the features generated in the process of decoding the feature map of the image by the second decoding network 215.

[0239] Exemplarily, the information acquisition module 602 is specifically configured to input the first bitstream into the entropy decoding unit 220 to obtain the side information features of the image; use the side information features of the image as the first image information; and / or

[0240] Inputting the first bitstream into the entropy decoding unit 220 to obtain the side information features of the image; inputting the side information features of the image and the second bitstream into the entropy decoding unit 220 to obtain a feature map of the image; using the feature map of the image as the first image information; and / or

[0241] The first code stream is input into the entropy decoding unit 220 to obtain the side information features of the image; the side information features of the image and the second code stream are input into the entropy decoding unit 220 to obtain the feature map of the image; the feature map of the image is input into the second decoding network 215 to obtain the initial reconstructed image and / or the decoding features of the initial reconstructed image; the initial reconstructed image and / or the decoding features of the initial reconstructed image are used as the first image information.

[0242] In one example, Figure 7 A schematic block diagram of a device 700 according to an embodiment of the present application is shown. The device 700 may include: a processor 701 and a transceiver / transceiver pin 702 , and optionally, a memory 703 .

[0243] The components of the device 700 are coupled together via a bus 704, wherein the bus 704 includes a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, all buses are referred to as bus 704 in the figure.

[0244] Optionally, the memory 703 may be used to store instructions in the aforementioned method embodiment. The processor 701 may be used to execute instructions in the memory 703, and control the receiving pin to receive a signal, and control the sending pin to send a signal.

[0245] The apparatus 700 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0246] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0247] The embodiment of the present application also provides a chip, including one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the above-mentioned related method steps implement the steps of the method in the above-mentioned embodiment. The interface circuit is a transceiver / transceiver pin 702.

[0248] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0249] This embodiment further provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer executes the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0250] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor may execute the computer-executable instructions stored in the memory so that the chip executes the methods in the above-mentioned method embodiments.

[0251] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0252] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0253] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0254] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0255] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0256] Any content of each embodiment of the present application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of the present application.

[0257] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0258] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (ReadOnly Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disk (CD-ROM) or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0259] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0260] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for decoding an image, characterized in that: The method comprises: Receive code stream; Acquire first image information based on the code stream; Performing feature transformation on the first image information to obtain features of the first image information; Inputting the second image information into the diffusion model to obtain the target features; wherein the second image information includes one of a preset noise image, an initial reconstructed image determined based on the bitstream, or an image obtained by adding noise to the initial reconstructed image, and the features of the first image information are used to adjust the intermediate features generated in the feature generation process of the diffusion model; The target features are reconstructed to obtain a target reconstructed image.

2. The method according to claim 1, characterized in that The step of reconstructing the target features to obtain a target reconstructed image includes: The first image information and the target features are reconstructed to obtain the target reconstructed image.

3. The method according to claim 2, characterized in that The reconstructing the first image information and the target feature to obtain the target reconstructed image includes: The first image information and the target features are input into a first decoding network to obtain the target reconstructed image.

4. The method according to any one of claims 1 to 3, characterized in that: The performing feature transformation on the first image information to obtain features of the first image information includes: The first image information and time information are input into a first feature transformation model to obtain features of the first image information.

5. The method according to any one of claims 1 to 4, characterized in that: The features of the first image information correspond to a plurality of time information, one time information among the plurality of time information corresponds to one or more features of the first image information, and the scales of the plurality of features of the first image information are different.

6. The method according to claim 5, characterized in that The step of inputting the second image information into the diffusion model to obtain the target feature includes: Inputting the t-th time information and the second image information into the diffusion model to obtain the features of the t-1-th time information; wherein the initial value of t is T, and T is a positive integer; Subtract 1 from t and repeat the following steps until t is equal to 1: Input the t-th time information and the feature of the t-th time information into the diffusion model, obtain the feature of the t-1-th time information and reduce t by 1; Among them, the features of the first image information correspond to multiple time information, and the features of the first image information corresponding to the t-th time information are used to adjust the intermediate features generated in the process of generating the features of the t-1-th time information by the diffusion model; when t is equal to 1, the features of the t-1-th time information are the target features.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Inputting the features of the first image information into the second feature transformation model to obtain feature adjustment parameters; Based on the feature adjustment parameter, the intermediate features generated during the feature generation process of the diffusion model are adjusted.

8. The method according to claim 7, characterized in that The feature adjustment parameters include a first feature adjustment parameter and a second feature adjustment parameter, and adjusting the intermediate features generated in the feature generation process of the diffusion model based on the feature adjustment parameters includes: Determine a product value between the first feature adjustment parameter and an intermediate feature generated in a feature generation process performed by the diffusion model; determining a sum value between the product value and the second feature adjustment parameter; Based on the sum value, an adjusted intermediate feature is determined.

9. The method according to claim 8, characterized in that The step of determining the adjusted intermediate feature based on the sum value comprises: The sum value is added to the intermediate feature generated in the feature generation process of the diffusion model to obtain the adjusted intermediate feature.

10. The method according to any one of claims 1 to 9, characterized in that: The first image information includes at least one of a feature map of the image, a side information feature of the image, the initial reconstructed image, or a decoded feature of the initial reconstructed image; Among them, the feature map of the image and the side information features of the image are obtained by entropy decoding the code stream by an entropy decoding unit, the initial reconstructed image is obtained by decoding the feature map of the image by a second decoding network, and the decoding features of the initial reconstructed image are features generated in the process of decoding the feature map of the image by the second decoding network.

11. The method according to any one of claims 1 to 10, characterized in that: The code stream includes a first code stream and a second code stream, and the acquiring the first image information based on the code stream includes: Inputting the first bitstream into an entropy decoding unit to obtain side information features of the image; using the side information features of the image as the first image information; and / or Inputting the first bitstream into the entropy decoding unit to obtain side information features of the image; inputting the side information features of the image and the second bitstream into the entropy decoding unit to obtain a feature map of the image; using the feature map of the image as the first image information; and / or Inputting the first code stream into the entropy decoding unit to obtain side information features of the image; inputting the side information features of the image and the second code stream into the entropy decoding unit to obtain a feature map of the image; inputting the feature map of the image into a second decoding network to obtain the initial reconstructed image and / or the decoding features of the initial reconstructed image; and using the initial reconstructed image and / or the decoding features of the initial reconstructed image as the first image information.

12. An image decoding device, characterized in that: The image decoding device comprises an entropy decoding unit, a first feature transformation model, a diffusion model and a first decoding network; wherein: The entropy decoding unit is used to perform entropy decoding on the received code stream to obtain the first image information; The first feature transformation model is used to perform feature transformation on the first image information to obtain features of the first image information; The diffusion model is used to perform feature generation processing on the second image information to obtain target features; wherein the second image information includes one of a preset noise image, an initial reconstructed image determined based on the bitstream, or an image obtained by adding noise to the initial reconstructed image, and the features of the first image information are used to adjust intermediate features generated in the feature generation processing of the diffusion model; The first decoding network is used to reconstruct the target features to obtain a target reconstructed image.

13. The device according to claim 12, characterized in that The first decoding network is specifically used to reconstruct the first image information and the target features to obtain the target reconstructed image.

14. The device according to claim 12 or 13, characterized in that The image decoding device also includes a second feature transformation model, The second feature transformation model is used to perform feature transformation on the features of the first image information to obtain feature adjustment parameters; wherein the feature adjustment parameters are used to adjust the intermediate features generated during the feature generation process of the diffusion model.

15. The device according to claim 12 or 13, characterized in that The first image information includes at least one of a feature map of the image or a side information feature of the image, and the feature map of the image or the side information feature of the image is output information of the entropy decoding unit.

16. The device according to any one of claims 12 to 14, characterized in that The image decoding device further comprises a second decoding network, The second decoding network is used to decode the feature map of the image; Among them, the feature map of the image is the output information of the entropy decoding unit, the first image information includes at least one of the initial reconstructed image and the decoding features of the initial reconstructed image, the initial reconstructed image is the output information of the second decoding network, and the decoding features of the initial reconstructed image are the features generated in the process of the second decoding network decoding the feature map of the image.

17. A device for decoding an image, characterized in that: The device for decoding comprises: A receiving module, used for receiving a code stream; An information acquisition module, used for acquiring first image information based on the code stream; a feature determination module, configured to perform feature transformation on the first image information to obtain features of the first image information; a feature generation module, configured to input the second image information into the diffusion model to obtain target features; wherein the second image information includes one of a preset noise image, an initial reconstructed image determined based on the bitstream, or an image obtained by adding noise to the initial reconstructed image, and the features of the first image information are used to adjust intermediate features generated during the feature generation process of the diffusion model; The reconstruction module is used to reconstruct the target features to obtain a target reconstructed image.

18. A decoding device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the decoding device executes the image decoding method according to any one of claims 1 to 11.

19. A decoding device, characterized in that: It comprises one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the method as claimed in any one of claims 1 to 11 are executed.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the image decoding method according to any one of claims 1 to 11.

21. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a computer or a processor, the steps of the method according to any one of claims 1 to 11 are performed.

Images