Image inpainting method, model training method and related apparatuses

Abstract

Embodiments of the present application disclose an image repairing method, a model training method and related devices, wherein the image repairing method comprises: obtaining an initial image; inputting the initial image into a convolution repairing model, obtaining intermediate features generated by the convolution repairing model in a pre-repairing process of the initial image, and obtaining an output image obtained after the convolution repairing model completes the pre-repairing process; performing semantic feature coding on the initial image through a first encoder to obtain first semantic features; determining second reference features according to the intermediate features; performing semantic feature coding on the output image through a second encoder to obtain third semantic features; generating repairing image features based on the first semantic features, the second semantic features and the third semantic features through a diffusion model; and the repairing image features are used to obtain a repaired image through feature decoding. The method can improve the image repairing effect.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image restoration method, a model training method, and related apparatus. Background Technology

[0002] In practical applications, images are often subject to degradation issues such as noise, blurring, and compression due to factors such as acquisition and storage conditions. In order to improve the viewing experience and facilitate subsequent image-based tasks, it is necessary to repair such low-quality images to improve their image clarity.

[0003] In related technologies, diffusion models, which have powerful image generation capabilities, can be used to repair images. However, current techniques for image repair using diffusion models generally suffer from unstable image repair results. Specifically, the repaired image generated by the diffusion model is prone to artifacts and may not be semantically consistent with the original image to be repaired. Summary of the Invention

[0004] This application provides an image restoration method, a model training method, and related apparatus, which can improve the image restoration effect.

[0005] A first aspect of this application provides an image restoration method, the method comprising:

[0006] Get the initial image;

[0007] The initial image is input into the convolutional inpainting model to obtain the intermediate features generated by the convolutional inpainting model during the pre-inpainting process of the initial image, and to obtain the output image obtained by the convolutional inpainting model after completing the pre-inpainting process.

[0008] The initial image is semantically encoded using a first encoder to obtain first semantic features;

[0009] Based on the intermediate features, a second semantic feature is determined; the second semantic feature is used to characterize the semantic information extracted by the convolutional inpainting model during the pre-inpainting process of the initial image.

[0010] The output image is semantically encoded using a second encoder to obtain a third semantic feature;

[0011] Using a diffusion model, repaired image features are generated based on the first semantic feature, the second semantic feature, and the third semantic feature; the repaired image features are used to obtain the repaired image through feature decoding.

[0012] A second aspect of this application provides a model training method, the method comprising:

[0013] Obtain the ground truth image; perform degradation processing on the ground truth image to obtain a degraded image;

[0014] The degraded image is input into the convolutional restoration model to obtain the convolutional intermediate features generated by the convolutional restoration model during the pre-restoration process of the degraded image, and to obtain the convolutional restored image obtained by the convolutional restoration model after completing the pre-restoration process.

[0015] In the ground truth image and the convolutional repair image, select the labeled repair image corresponding to the degraded image; project the labeled repair image into the latent space to obtain labeled image features; add labeled noise features to the labeled image features to obtain training noise features;

[0016] The degraded image is semantically encoded using a first encoder to obtain first training semantic features;

[0017] Based on the convolutional intermediate features, a second training semantic feature is determined; the second training semantic feature is used to characterize the semantic information extracted by the convolutional restoration model during the pre-restoration process of the degraded image.

[0018] The convolutional repair image is semantically encoded using a second encoder to obtain a third training semantic feature;

[0019] Using the diffusion model to be trained, based on the first training semantic features, the second training semantic features, and the third training semantic features, the predicted noise features in the training noise features are determined;

[0020] The diffusion model is trained based on the difference between the predicted noise features and the labeled noise features.

[0021] A third aspect of this application provides an image restoration apparatus, the apparatus comprising:

[0022] The first acquisition module is used to acquire the initial image;

[0023] The first convolutional inpainting module is used to input the initial image into the convolutional inpainting model, obtain the intermediate features generated by the convolutional inpainting model during the pre-inpainting process of the initial image, and obtain the output image obtained by the convolutional inpainting model after completing the pre-inpainting process.

[0024] The first feature preprocessing module is used to encode the semantic features of the initial image using a first encoder to obtain a first semantic feature; determine a second semantic feature based on the intermediate features, wherein the second semantic feature is used to characterize the semantic information extracted by the convolutional inpainting model during the prepainting process of the initial image; and encode the semantic features of the output image using a second encoder to obtain a third semantic feature.

[0025] The first diffusion repair module is used to generate repaired image features based on the first semantic feature, the second semantic feature, and the third semantic feature using a diffusion model; the repaired image features are used to obtain the repaired image through feature decoding.

[0026] A fourth aspect of this application provides a model training apparatus, the apparatus comprising:

[0027] The second acquisition module is used to acquire the ground truth image; and to perform degradation processing on the ground truth image to obtain a degraded image;

[0028] The second convolutional inpainting module is used to input the degraded image into the convolutional inpainting model, obtain the convolutional intermediate features generated by the convolutional inpainting model during the pre-inpainting process of the degraded image, and obtain the convolutional inpainted image obtained by the convolutional inpainting model after completing the pre-inpainting process.

[0029] The annotation module is used to select the annotated restoration image corresponding to the degraded image from the ground truth image and the convolutional restoration image; project the annotated restoration image into the latent space to obtain the annotated image features; and add annotation noise features to the annotated image features to obtain the training noise features.

[0030] The second feature preprocessing module is used to encode the semantic features of the degraded image through the first encoder to obtain the first training semantic features; determine the second training reference features based on the convolution intermediate features, the second training semantic features being used to characterize the semantic information extracted by the convolutional restoration model during the pre-restoration process of the degraded image; and encode the semantic features of the convolutional restored image through the second encoder to obtain the third training semantic features.

[0031] The noise prediction module is used to determine the predicted noise features in the training noise features based on the first training semantic features, the second training semantic features, and the third training semantic features using the diffusion model to be trained.

[0032] The model training module is used to train the diffusion model based on the difference between the predicted noise features and the labeled noise features.

[0033] A fifth aspect of this application provides a computer device, the device comprising a processor and a memory:

[0034] The memory is used to store computer programs;

[0035] The processor is configured to perform, according to the computer program, the steps of the image restoration method as described in the first aspect above, or the steps of the model training method as described in the second aspect above.

[0036] The sixth aspect of this application provides a computer-readable storage medium for storing a computer program for performing the steps of the image restoration method described in the first aspect above, or for performing the steps of the model training method described in the second aspect above.

[0037] A seventh aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the image restoration method described in the first aspect, or the steps of the model training method described in the second aspect.

[0038] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0039] This application provides an image inpainting method that innovatively couples a convolutional inpainting model with a diffusion model to collaboratively perform image inpainting tasks. The method includes: acquiring an initial image; inputting the initial image into a convolutional inpainting model; acquiring intermediate features generated by the convolutional inpainting model during pre-inpainting processing of the initial image; and acquiring the output image obtained after the convolutional inpainting model completes the pre-inpainting processing. Next, the initial image is semantically encoded using a first encoder to obtain a first semantic feature, which represents the original semantic information of the initial image. Based on intermediate features, a second semantic feature is determined. Since the convolutional inpainting model extracts multi-dimensional features from the initial image during the inpainting process, the intermediate features generated in this process can fully and meticulously represent the global and local semantic information of the initial image. Correspondingly, the second semantic feature determined based on this intermediate feature can also fully and meticulously represent the global and local semantic information extracted by the convolutional inpainting model during the pre-inpainting process of the initial image. The output image is then semantically encoded using a second encoder to obtain a third semantic feature. Because the convolutional inpainting model has high stability in image inpainting tasks, the output image obtained after inpainting using the convolutional inpainting model is also relatively stable. This output image has a high degree of semantic consistency with the initial image and has higher image quality than the initial image. Correspondingly, the third semantic feature obtained by encoding the output image can represent the semantic information of the higher-quality output image. Furthermore, based on the first, second, and third semantic features determined above, a restoration image feature is generated using a diffusion model. This restoration image feature is used to obtain the restored image through feature decoding. In other words, the diffusion model is controlled by the first, second, and third semantic features to perform relevant processing. This allows the diffusion model to utilize information from the original initial image, intermediate features generated by a highly stable convolutional restoration model, and information from the output image obtained by the convolutional restoration model after restoring the initial image. This assists the diffusion model in generating corresponding restoration image features that indicate the restored image. This combines the high stability of the convolutional restoration model with the powerful image generation capability of the diffusion model (capable of generating rich image details), ensuring that the final restored image contains rich image details and is stable, thus improving the image restoration effect. Attached Figure Description

[0040] Figure 1 This is a schematic diagram illustrating an application scenario of the image restoration method provided in the embodiments of this application;

[0041] Figure 2 A schematic flowchart illustrating the image restoration method provided in this application embodiment;

[0042] Figure 3aA schematic diagram illustrating the selection of intermediate features provided in an embodiment of this application;

[0043] Figure 3b This is a schematic diagram illustrating the working principle of the second encoder provided in an embodiment of this application;

[0044] Figure 4 This is a schematic diagram of the control network structure provided in an embodiment of this application;

[0045] Figure 5 This is a schematic diagram illustrating the working principle of the image restoration method provided in the embodiments of this application;

[0046] Figure 6 This is a schematic diagram of the structure of the control sub-block provided in an embodiment of this application;

[0047] Figure 7 This is a flowchart illustrating an image restoration method provided in this application when applied to a video restoration task.

[0048] Figure 8 This is a schematic diagram of the structure of the convolutional repair model with a multi-input single-output structure provided in the embodiments of this application;

[0049] Figure 9 A flowchart illustrating another image restoration method provided in this application when applied to a video restoration task;

[0050] Figure 10 This is a schematic diagram of the structure of the convolutional repair model with a multi-input multi-output structure provided in the embodiments of this application;

[0051] Figure 11 A flowchart illustrating the training process of the convolutional insulation model provided in this application embodiment;

[0052] Figure 12 A flowchart illustrating the training process of the second encoder provided in this application embodiment;

[0053] Figure 13 A schematic flowchart illustrating the model training method provided in this application embodiment;

[0054] Figure 14 This is a schematic diagram of the image restoration apparatus provided in the embodiments of this application;

[0055] Figure 15 This is a schematic diagram of the structure of the model training device provided in the embodiments of this application;

[0056] Figure 16 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application;

[0057] Figure 17 This is a schematic diagram of the server structure provided in an embodiment of this application. Detailed Implementation

[0058] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0059] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0060] The image restoration method and model training method provided in this application can be executed by a computer device, which can be a terminal device or a server. The terminal device includes, but is not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, and aircraft. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server.

[0061] It should be noted that the information, data and signals involved in the embodiments of this application are all authorized by the relevant parties or fully authorized by all parties, and the collection, use and processing of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0062] To facilitate understanding of the image restoration method provided in this application embodiment, the following example uses a server as the execution subject of the image restoration method to illustrate its application scenarios.

[0063] See Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of the image restoration method provided in the embodiments of this application. For example... Figure 1As shown, this application scenario includes a terminal device 110 and a server 120. The terminal device 110 and the server 120 can communicate directly or indirectly through a wired network or a wireless network.

[0064] The terminal device 110 runs a target application with image restoration function. The user can upload an initial image through the input page provided by the target application so that the server 120 can obtain the initial image from the terminal device 110. The server 120 is used to execute the image restoration method provided in the embodiments of this application, and to perform image restoration on the initial image based on the convolutional restoration model and the diffusion model.

[0065] Specifically, server 120 can first input the initial image into the convolutional inpainting model, obtain the intermediate features generated by the convolutional inpainting model during the pre-inpainting process of the initial image, and obtain the output image obtained after the convolutional inpainting model completes the pre-inpainting process.

[0066] Subsequently, server 120 can encode the semantic features of the initial image using a first encoder to obtain a first semantic feature, which can represent the original semantic information of the initial image. Based on intermediate features, a second semantic feature is determined. Since the convolutional inpainting model extracts multi-dimensional features during the inpainting process, the intermediate features generated during this process can fully and meticulously represent the global and local semantic information of the initial image. Correspondingly, the second semantic feature determined based on this intermediate feature can also fully and meticulously represent the global and local semantic information extracted by the convolutional inpainting model during the pre-inpainting process of the initial image. The output image is then encoded using a second encoder to obtain a third semantic feature. Because the convolutional inpainting model has high stability in image inpainting tasks, the output image obtained after inpainting using the convolutional inpainting model is also relatively stable. This output image has a high degree of semantic consistency with the initial image and has higher image quality than the initial image. Accordingly, the third semantic feature obtained by encoding the output image can represent the semantic information of the higher-quality output image.

[0067] Furthermore, server 120 can generate restored image features based on the first, second, and third semantic features determined above using a diffusion model. These restored image features are used to obtain the restored image through feature decoding. Specifically, the diffusion model can refer to the first, second, and third semantic features to denoise the initial noise features, obtaining restored image features. The decoder then performs feature decoding on the restored image features to obtain the restored image corresponding to the initial image. That is, by using the first, second, and third semantic features to control the diffusion model to perform denoising, the information of the original initial image, the intermediate features generated by the highly stable convolutional restoration model, and the information of the output image obtained by the convolutional restoration model in repairing the initial image are used to assist the diffusion model in generating the corresponding restored image. This combines the high stability of the convolutional restoration model with the powerful image generation capability (capable of generating rich image details) of the diffusion model, ensuring that the final restored image contains rich image details and is stable, thus improving the image restoration effect.

[0068] Finally, the server 120 can send the obtained repaired image to the terminal device 110 so that the terminal device 110 can render and display the repaired image.

[0069] It should be understood that Figure 1 The application scenarios shown are merely examples. In practical applications, the image restoration method provided in this application embodiment can also be applied to other scenarios. No limitations are made here on the application scenarios of the image restoration method provided in this application embodiment.

[0070] The image restoration method provided in this application will be described in detail below through method embodiments.

[0071] See Figure 2 , Figure 2 This is a schematic flowchart illustrating the image restoration method provided in an embodiment of this application. For ease of description, the following description uses a server as the execution subject of this image restoration method. Figure 2 As shown, the image restoration method includes the following steps:

[0072] S201: Obtain the initial image.

[0073] The initial image refers to the object to be restored in an image restoration method. The initial image is usually a low-quality (LQ) image. For example, the initial image can be an image containing degradation problems such as noise, blur, and compression.

[0074] It should be noted that the initial image can be an independent image, such as a photograph, or an image from the video resource to be repaired, such as one or more frames from the video resource to be repaired. This application does not specifically limit the type of the initial image.

[0075] As an example, the server can obtain the initial image from the terminal device. For instance, the terminal device may be running a target application with image restoration capabilities. The user can input the initial image through an input page in the target application, and the terminal device can then send the user-inputted initial image to the server, allowing the server to obtain the initial image. Alternatively, the server can obtain the initial image from a database. For example, the database may store several low-quality historical images, and the server can retrieve any one or more historical images from the database as initial images to restore the retrieved historical images. This application does not specifically limit the method of obtaining the initial image.

[0076] S202: Input the initial image into the convolutional inpainting model, obtain the intermediate features generated by the convolutional inpainting model during the pre-inpainting process of the initial image, and obtain the output image obtained after the convolutional inpainting model completes the pre-inpainting process.

[0077] Convolutional inpainting models refer to convolutional neural network models used to perform image inpainting tasks. They primarily extract global and / or local features from the input image through convolution operations to generate a corresponding inpainted image based on the extracted features. As an example, the model structures used in convolutional inpainting models include, but are not limited to, those described below, such as LeNet-5, U-shaped Network (UNet), and AlexNet. This application does not specifically limit the model structure used in convolutional inpainting models.

[0078] Taking the UNet model structure used in the convolutional inpainting model as an example, this model structure includes multiple encoding blocks, intermediate blocks, and multiple decoding blocks. There is a one-to-one correspondence between the encoding and decoding blocks, and these blocks are connected via skip connections for information propagation and gradient backpropagation. The encoding blocks include convolutional and pooling layers. Through the encoding blocks, the size of the initial image's feature map is gradually reduced, and its channel number is increased to extract global semantic features that represent global feature information, as well as local semantic features that represent local feature information. The intermediate blocks serve as the feature transfer channel between the last encoding block and the first decoding block. They include convolutional layers that perform convolution operations on the features output by the last encoding block. The decoding blocks include convolutional and upsampling layers to gradually restore the size of the feature map to the size of the initial image and generate an output image of the same size as the initial image.

[0079] In this embodiment, the initial image can be input into a convolutional inpainting model for pre-inpainting processing to obtain an output image after the convolutional inpainting model completes the pre-inpainting processing. The output image is the output result obtained by the convolutional inpainting model after pre-inpainting the initial image. The output image has the same semantics as the initial image and has higher quality than the initial image. However, in some cases, the output image may have the problem of insufficient image details, such as some areas of the output image lacking detailed content. However, compared with the initial image, the output image still has richer details.

[0080] Intermediate features refer to image features (i.e., feature maps) generated by the convolutional inpainting model during the pre-inpainting process. They are used to indicate the global semantic information (i.e., the content included in the entire initial image) and / or local semantic information (i.e., the content included in the local regions of the initial image). Taking the convolutional inpainting model using the UNet model structure as an example, intermediate features can be image features output by any one or more coding blocks, intermediate features can be image features output by intermediate blocks, and intermediate features can be image features output by any one or more decoding blocks. In this regard, this application does not specifically limit the generation stage of intermediate features.

[0081] Taking the UNet model structure used in the convolutional inpainting model as an example, the inpainting process can include: inputting the initial image into the convolutional inpainting model, downsampling the initial image step by step through multiple encoding blocks to gradually extract deep features and reduce its spatial resolution. Then, the feature map output from the last encoding block can be input into an intermediate block, which performs convolution processing on the feature map to extract deep features. Next, the feature map output from the intermediate block can be input into the connected decoding block for upsampling, restoring the feature map extracted in the encoding block stage to the size of the initial image, and gradually generating the output image. Simultaneously, during the inpainting process, through skip connections between the decoding and encoding blocks, the feature map output from the encoding block can be directly transmitted to the corresponding decoding block, achieving cross-layer transfer and fusion of feature maps. This ensures that relevant information from the encoding process is fused during decoding, avoiding the loss of image information, and thus ensuring that the output image can reflect the details and semantic information of the initial image, resulting in a stable inpainting effect.

[0082] It should be understood that convolutional inpainting models with different model structures can perform image inpainting based on different working mechanisms, and the embodiments of this application do not specifically limit the working mechanism of the convolutional inpainting model.

[0083] In one possible implementation, "obtaining intermediate features generated by the convolutional inpainting model during the pre-inpainting process of the initial image" in S202 above may include:

[0084] The resolution of the input noise characteristics of the diffusion model is determined as the reference resolution;

[0085] Obtain intermediate features with reference resolution generated by the convolutional inpainting model during the pre-inpainting process.

[0086] The reference resolution refers to the resolution of the noise features input to the diffusion model. In this embodiment, the reference resolution can be used as a reference condition for selecting intermediate features, that is, it must be ensured that the selected intermediate features have the reference resolution. The noise feature refers to the object being processed when the diffusion model performs denoising. For example, the noise feature can be random Gaussian noise features in the latent space.

[0087] Specifically, based on the reference resolution, image features with the reference resolution can be selected from the image features generated during the pre-inpainting process of the convolutional inpainting model as intermediate features. Taking the UNet model structure used in the convolutional inpainting model as an example, the resolution of the image features output by each coding block, intermediate block, and decoding block in the convolutional inpainting model can be determined. Then, image features with the aforementioned reference resolution can be selected from the image features output by each coding block, intermediate block, and decoding block as intermediate features. It should be understood that when the convolutional inpainting model is a UNet structure, there will be an image feature output by a coding block with the reference resolution, and there will also be an image feature output by a decoding block with the reference resolution. In this case, the image feature output by the decoding block with the reference resolution can be selected as an intermediate feature. The reason for this operation is that, compared with the image features output by the coding block, the image features output by the decoding block usually contain richer information. Using the image feature output by the decoding block with the reference resolution as an intermediate feature is beneficial for the diffusion model to refer to richer information in the subsequent image inpainting process.

[0088] Therefore, by using the above method, the resolution of the input noise features of the diffusion model is used as the reference resolution. From the image features generated by the convolutional inpainting model during the inpainting process, features with the reference resolution are selected as intermediate features. This ensures that the selected intermediate features are in the same feature space as the noise features processed by the diffusion model, avoiding additional operations on the intermediate features and thus saving computational resources.

[0089] In another possible implementation, "obtaining intermediate features generated by the convolutional inpainting model during the pre-inpainting process of the initial image" in S202 above may include:

[0090] The convolutional inpainting model generates multiple intermediate features during the pre-inpainting process; these intermediate features correspond to various resolutions.

[0091] During the pre-inpainting process of the convolutional inpainting model on the image to be inpainted, multiple image features are generated. Since the inpainting process of the convolutional inpainting model includes an encoding stage and a decoding stage, the encoding stage downsamples the image to be inpainted to gradually reduce its spatial resolution, while the decoding stage upsamples the features of the image to be inpainted to gradually restore its spatial resolution. Therefore, multiple image features with different resolutions are generated during the inpainting process.

[0092] As an example, see Figure 3a , Figure 3a This is a schematic diagram illustrating the selection of intermediate features as provided in an embodiment of this application. For example... Figure 3aAs shown in (1), multiple image features of different resolutions can be selected from the image features output by the coding blocks of the convolutional inpainting model, such as directly selecting the image features output by each coding block and using them all as intermediate features. Or, as shown in (1), Figure 3a As shown in (2), multiple image features of different resolutions can also be selected from the image features output by the decoding blocks of the convolutional inpainting model, such as directly selecting the image features output by each decoding block and using them all as intermediate features. Or, as shown in (2), Figure 3a As shown in (3), multiple image features of different resolutions can also be selected from the image features output by the coding block, the image features output by the intermediate block, and the image features output by the decoding block. For example, the image features output by each coding block, the image features output by the intermediate block, and the image features output by each decoding block can be directly selected and used as intermediate features.

[0093] Therefore, by using the above method, multiple intermediate features can be obtained from the pre-inpainting process of the convolutional inpainting model to obtain more comprehensive global and / or local semantic information. Accordingly, the semantic features used to provide reference information for the diffusion model to perform denoising processing can be determined based on multiple intermediate features, which can also ensure that the information provided by the semantic features is more comprehensive.

[0094] Therefore, through the above repair process, the intermediate features generated by the convolutional repair model during the pre-repair process can be determined, and the output image obtained after the convolutional repair model completes the pre-repair process can be obtained.

[0095] S203: The initial image is semantically encoded using the first encoder to obtain the first semantic feature.

[0096] Before performing image inpainting based on the diffusion model, the reference information used by the diffusion model to perform the image inpainting task needs to be processed. That is, the initial image, intermediate features and output image in S202 are processed separately to obtain the semantic features that the diffusion model can refer to when performing the image inpainting task.

[0097] The initial image can be processed as follows: The initial image is semantically encoded using a first encoder to obtain the latent space features corresponding to the initial image, i.e., the first semantic features. These first semantic features characterize the semantic information of the initial image, indicating its contents. In other words, the first semantic features are features that embody the original image information of the initial image, representing key information in the initial image with low-dimensional features. Furthermore, to adapt to the feature dimension processing requirements of the diffusion model, these first semantic features are latent space features.

[0098] The first encoder projects the initial image into the latent space. For example, the first encoder can be a Variational Auto Encoder (VAE), a Vector Quantized Generative Adversarial Network Encoder (VQ-GAN Encoder), etc. This application does not specifically limit the structure of the first encoder. Encoding semantic features of the initial image using the first encoder means mapping the initial image into the latent space. The latent space is a low-dimensional feature space. Mapping the initial image into the latent space allows the image information of the initial image to be represented with a smaller dimension compared to the original data. By reducing the dimensionality, unimportant information in the original data can be filtered out, retaining only the key information.

[0099] In one possible implementation, the step of "encoding the semantic features of the initial image using the first encoder to obtain the first semantic features" in S203 above may include:

[0100] The initial image is mapped to the latent space by a vector quantization encoder based on generative adversarial networks to obtain the first semantic features; the latent space is the feature space of the input noise features of the diffusion model.

[0101] The VQ-GAN-based encoder transforms the initial image into a latent space representation, and then discretizes this latent space representation into a set of fixed vectors through a vector quantization step. The VQ-GAN encoder maps the initial image to the latent space, and the vector quantization layer discretizes the latent space features into a set of fixed vectors, yielding the first semantic feature. The vector quantization layer contains a codebook, where each vector represents a possible latent space representation. The latent space features output by the encoder are compared with the latent space representations in the codebook, allowing the closest latent space representation to be selected as the discretized first semantic feature.

[0102] The latent space is the feature space of the input noise features of the diffusion model. Mapping the initial image to the latent space aims to adapt it to the feature space of the diffusion model, facilitating image inpainting based on the first semantic features. It's important to note that the input noise features of the diffusion model are essentially image features carrying noise information. Typically, the diffusion model needs to iteratively perform multiple rounds of denoising. During the first round of denoising, the input noise features are the initial noise features, which are the feature representation of the initial noisy image (an image composed of random Gaussian noise) in the latent space. During subsequent rounds of denoising, the input noise features have already removed some noise information compared to the initial noise features. As the number of denoising rounds increases, the amount of noise removed gradually increases.

[0103] Therefore, based on the above method, it can be seen that in this application, the initial image can be mapped into the latent space using a VQ-GAN encoder to obtain the first semantic features used as reference information for the diffusion model. Since the VQ-GAN encoder combines the advantages of vector quantization and generative adversarial networks and introduces a discrete structure during the encoding process, it can improve the encoding quality.

[0104] S204: Determine the second semantic feature based on the intermediate features; the second semantic feature is used to characterize the semantic information extracted by the convolutional inpainting model during the pre-inpainting process of the initial image.

[0105] Intermediate features can be processed in the following way: A second semantic feature is determined based on the intermediate features. For example, when the resolution of the intermediate features is the same as the resolution of the input noise features of the diffusion model, the intermediate features can be directly used as the second semantic feature. Alternatively, when the resolution of the intermediate features is different from the resolution of the input noise features of the diffusion model, the obtained intermediate features can be transformed to obtain the second semantic feature. Since the convolutional inpainting model extracts global and local semantic information from multiple different dimensions when processing the image to be repaired, the intermediate features generated by the convolutional inpainting model during the pre-repair process of the initial image can reflect the global and local semantic information of the initial image. Correspondingly, the second semantic feature determined based on the intermediate features is a feature used to reflect the global and local semantic information in the initial image, and it can represent the global and local semantic information in the initial image with low-dimensional features.

[0106] In one possible implementation, when the resolution of the acquired intermediate feature is the same as the resolution of the input noise feature of the diffusion model (i.e., the reference resolution), the intermediate feature can be directly used as the second semantic feature, so that the diffusion model can directly perform image inpainting based on the second semantic feature with the same resolution, thereby saving computational resources.

[0107] In one possible implementation, when the resolution of the acquired intermediate features is not exactly the same as the resolution of the input noise features of the diffusion model, the second semantic feature can be determined based on the acquired intermediate features through the following processing method:

[0108] The resolution of the input noise characteristics of the diffusion model is determined as the reference resolution;

[0109] If multiple intermediate features are obtained, and there are intermediate features whose resolution is not equal to the reference resolution, the resolution of each obtained intermediate feature is adjusted to the reference resolution, and the intermediate features with the reference resolution are fused to obtain the second semantic feature.

[0110] After obtaining multiple intermediate features with different resolutions, the resolution of the input noise features of the diffusion model can be used as the reference resolution. The resolution of the input noise features of the diffusion model is preset, which makes it easy to adjust the resolution of multiple intermediate features based on the reference resolution, so that the resolution of the adjusted intermediate features is the reference resolution, thus making them suitable for the diffusion model.

[0111] In practice, the resolution of intermediate features whose resolution is not equal to the reference resolution can be adjusted first so that the resolution of such intermediate features is equal to the reference resolution; then, the intermediate features with reference resolution can be fused.

[0112] As an example, the resolution of intermediate features whose resolution is not equal to the reference resolution can be adjusted using methods such as upsampling, downsampling, and interpolation until the resolution of each intermediate feature is the same as the reference resolution. For instance, when the resolution of an intermediate feature is less than the reference resolution, upsampling can be used to adjust the resolution of the intermediate feature to the reference resolution; when the resolution of an intermediate feature is greater than the reference resolution, downsampling can be used to adjust the resolution of the intermediate feature to the reference resolution.

[0113] Having obtained each intermediate feature with a reference resolution, the intermediate features with the reference resolution can be fused. For example, the elements in each intermediate feature can be added bit by bit (that is, the elements in the same position in each intermediate feature are added together), or the elements in the same position in each intermediate feature can be weighted and summed bit by bit (that is, the elements in the same position in each intermediate feature are weighted and summed together) to achieve the fusion of the intermediate features with reference resolution, and the fused intermediate features can be used as the second semantic features.

[0114] It should be noted that the above-described fusion process for multiple intermediate features can be performed in an intermediate feature fusion layer. This intermediate feature fusion layer can be deployed outside the diffusion model and connected to the second control network in the diffusion model used to process the second semantic feature. That is, multiple intermediate features obtained from the convolutional inpainting model can be input into this intermediate feature fusion layer. This layer performs the above-described fusion process to fuse the multiple intermediate features, obtaining the second semantic feature, which is then provided to the second control network in the diffusion model.

[0115] Therefore, by using the above method, the resolution of each intermediate feature can be adjusted to the reference resolution, and the intermediate features with the reference resolution can be fused to obtain a second semantic feature that can characterize the rich global semantic information and / or local semantic information in the initial image. Based on the second semantic feature, the diffusion model can be assisted in performing denoising processing, which can facilitate the diffusion model to refer to the global information and / or local information in the image to be repaired more fully and meticulously.

[0116] S205: The output image is semantically encoded by the second encoder to obtain the third semantic feature.

[0117] The output image of the convolutional inpainting model can be processed as follows: The output image is semantically encoded using a second encoder to obtain a third semantic feature. The second encoder encodes the output image to convert it into image features with the same resolution as the input features in the diffusion model, i.e., the third semantic feature. This third semantic feature characterizes the semantic information of the output image, indicating its contents. In other words, it represents the image information of a higher-quality output image compared to the initial image, and it can characterize key information in the output image with low-dimensional features. Furthermore, since the output image is obtained by repairing the initial image using the convolutional inpainting model, it has higher image quality (e.g., higher resolution and clarity) and richer image details (e.g., when the initial image has missing information, the convolutional inpainting model repairs the missing information, supplementing it in the output image). Therefore, the third semantic feature characterizing the semantic information of the output image has richer semantic information than the first semantic feature characterizing the semantic information of the initial image. Encoding the output image with features through the second encoder means mapping the output image to the latent space to obtain the third semantic features.

[0118] In one possible implementation, the second encoder includes an image encoding structure and a feature adaptation structure in a multimodal model; the multimodal model is used to map images and text to the same feature space through feature encoding; the "encoding semantic features of the output image through the second encoder to obtain third semantic features" in S203 above may include:

[0119] The output image is processed using an image coding structure to obtain its features;

[0120] Through the feature adaptation structure, at least one third semantic feature is generated based on the output image features; the resolution of the at least one third semantic feature matches the input feature resolution of at least one processing block in the denoising network of the diffusion model, wherein the processing block is an intermediate block in the denoising network and at least one of the various decoding blocks included in the decoding structure of the denoising network.

[0121] A multimodal model refers to a neural network model that enhances the model's ability to understand and represent input images and text through deep feature extraction and fusion. Multimodal models are used to map images and text to the same feature space through feature encoding. Specifically, a multimodal model can be a Contrastive Language-Image Pretraining (CLIP) model, which includes an image encoding structure and a text encoding structure. The image encoding structure is used to encode features of the input image, and the text encoding structure is used to encode features of the input text. Through these image and text encoding structures, images and text can be encoded into the same feature space. In this embodiment, the image encoding structure from the aforementioned multimodal model is applied to the second encoder.

[0122] Since the resolution of the image features output by the image coding structure in the multimodal model often does not match the resolution of the semantic features required by the diffusion model, this application embodiment additionally sets a feature adaptation structure in the second encoder. This feature adaptation structure is used to adjust the resolution of the image features output by the image coding structure to the resolution of the denoising network suitable for the diffusion model, that is, to adjust the image features output by the image coding structure to the resolution of the semantic features that can be directly applied by the denoising network.

[0123] For reference Figure 3b , Figure 3b This is a schematic diagram of the working principle of the second encoder provided in the embodiments of this application. The second encoder includes an image encoding structure and a feature adaptation structure. The output image is input into the image encoding structure. The image encoding structure can perform image encoding processing on the output image to compress the output image from the pixel space to a specific feature space. That is, the feature representation of the output image can be characterized by a smaller dimension than the original dimension to obtain the output image features. The output image features can represent the key image information in the output image with low-dimensional features.

[0124] Then, the output image features can be input to... Figure 3b The feature adaptation structure in the model adjusts the resolution. Based on the output image features, at least one third semantic feature can be generated through the feature adaptation structure. The resolution of the generated at least one third semantic feature matches the input feature resolution of at least one processing block in the denoising network of the diffusion model. The processing block can be an intermediate block in the denoising network or at least one of the various decoding blocks included in the decoding structure.

[0125] It should be understood that when the third semantic feature needs to be input into the intermediate block of the denoising network of the diffusion model, the resolution of the output image feature is adjusted by the feature adaptation structure, which can generate an image feature with the same resolution as the input feature of the intermediate block. This image feature is the third semantic feature. When the third semantic feature needs to be input into at least one decoding block included in the decoding structure of the denoising network, the resolution of the output image feature is adjusted by the feature adaptation structure, which can generate an image feature with the same resolution as the input feature of at least one decoding block. This at least one image feature is the third semantic feature.

[0126] Therefore, as can be seen from the above method, by using the second encoder composed of an image coding structure and a feature adaptation structure, at least one third semantic feature can be generated based on the output image of the convolutional inpainting model. The resolution of the third semantic feature is the same as the input feature resolution of at least one processing block in the denoising network of the diffusion model. This allows at least one processing block in the denoising network to directly refer to the third semantic feature for image inpainting, avoiding additional adjustments to the third semantic feature and thus saving computational resources.

[0127] S206: Using a diffusion model, based on the first semantic feature, the second semantic feature, and the third semantic feature, the repaired image features are obtained; the repaired image features are used to obtain the repaired image through feature decoding.

[0128] A diffusion model is an image generation model based on a diffusion process. It can inpaint an initial image based on reference information to obtain a restored image. Specifically, the diffusion model can denoise a random noise feature and, under the influence of reference information, generate restored image features corresponding to the reference information. These restored image features can then characterize the restored image. Common diffusion models include, but are not limited to, stable diffusion models. This application does not limit itself to a specific diffusion model.

[0129] After determining the first, second, and third semantic features, a diffusion model can be used to denoise the initial noise features based on these features, resulting in the repaired image features. The repaired image features refer to the output of the reverse diffusion process (denoising) using the diffusion model; these features represent the final, repaired image's characteristics in the latent space.

[0130] In one possible implementation, the diffusion model includes a first control network, a second control network, and a denoising network. The phrase "generating repaired image features based on the first semantic feature, the second semantic feature, and the third semantic feature using the diffusion model" in step S204 includes:

[0131] The first semantic feature is processed by the first control network to obtain multiple first control features; the multiple first control features correspond one-to-one with multiple control resolutions, including the input feature resolution of the intermediate block in the denoising network and the input feature resolution of each of the multiple decoding blocks in the decoding structure of the denoising network.

[0132] The second semantic features are processed by the second control network to obtain multiple second control features; the multiple second control features correspond one-to-one with multiple control resolutions.

[0133] The initial noise features are denoised using a denoising network. Based on the first control features, the second control features, and the third semantic features, the initial noise features are denoised in N rounds to obtain the repaired image features. N is an integer greater than 1.

[0134] The control network in the diffusion model is a neural network structure that controls the diffusion model to perform denoising by introducing additional control conditions. The control network is used to provide control conditions for the diffusion model, so that the diffusion model generates a restored image with stable restoration effect based on the control conditions. The control network preserves the quality and capability of the diffusion model by locking the parameters of the denoising network and copying its encoding layer.

[0135] For reference Figure 4 , Figure 4 This is a schematic diagram of the control network structure provided in an embodiment of this application. The model structure of the control network is derived from the encoding structure (obtained by copying the encoding structure in the denoising network), i.e. Figure 4 The control network consists of the encoding structure on the right and the corresponding zero convolutional layers. During the training of the control network, the weights of the original denoising network are locked, and only the weights of the encoding structure and zero convolutional layers in the control network are updated. This ensures that the restoration effect of the diffusion model itself is not affected. The aforementioned zero convolutional layers can be 1*1 convolutional layers with weights and biases initialized to 0. Therefore, in the early stages of training, the control network will not affect the restoration effect of the diffusion model. As the training progresses, the control network can learn to gradually adjust the original restoration process of the diffusion model, making the restored image gradually approach the newly introduced control conditions.

[0136] In this embodiment, a first control network and a second control network are deployed in the diffusion model. The first control network is used to incorporate information carried by a first semantic feature to control the diffusion model in performing denoising processing; that is, it incorporates image information from the original initial image to control the diffusion model in performing denoising processing. The second control network is used to incorporate information carried by a second semantic feature to control the diffusion model in performing denoising processing; that is, it incorporates information carried by intermediate features generated by the convolutional inpainting model during the pre-inpainting process to control the diffusion model in performing denoising processing.

[0137] The denoising network in the diffusion model performs denoising during the backdiffusion process. Specifically, it progressively reduces noise in the initial noise features to recover a noise-free restored image corresponding to the initial image; more specifically, it recovers the feature representation of the restored image in the latent space. The denoising network includes an encoding structure, intermediate blocks, and a decoding structure. The encoding structure consists of multiple sequentially connected encoding blocks. Figure 4 In the denoising network, the encoding structure consists of four blocks: A, B, C, and D. Each block has a different input feature resolution and is used to progressively downsample the input features to extract higher-level features. Intermediate blocks, located between the encoding and decoding structures, are processing blocks that connect them. The input and output features of intermediate blocks have the same size. These blocks process the higher-level features output by the encoding structure, further extracting and integrating information. Specifically, they can reference information provided by the first, second, and third control features to process the features output by the encoding structure, incorporating information from these features during processing. The decoding structure consists of multiple sequentially connected decoding blocks. Figure 4 In this structure, the decoding structure consists of D decoding blocks, C decoding blocks, B decoding blocks, and A decoding blocks. Each decoding block has a different input feature resolution, which is the same as the output feature resolution of its corresponding encoding block. The decoding structure is used to progressively upsample high-level features to restore the resolution of the original input. For example, the denoising network model structure can adopt the UNet model structure; however, this application does not specifically limit the model structure used by the denoising network.

[0138] It should be noted that, in the embodiments of this application, although both the convolutional inpainting model and the denoising network can adopt the Unet structure, the processing blocks (including encoding blocks, intermediate blocks and decoding blocks) in the convolutional inpainting model and the denoising network perform different operations. The processing blocks in the convolutional inpainting model are used to perform convolution operations, while the processing blocks in the diffusion model are used to perform attention operations.

[0139] For reference Figure 5 , Figure 5This is a schematic diagram illustrating the working principle of the image restoration method provided in this application embodiment. The first control network is used to convert the first semantic feature into a first control feature for controlling the denoising network to perform denoising processing. The first control feature refers to the output result obtained by the first control network after processing the first semantic feature. The first control feature is used to indicate the original image semantic information of the initial image in the denoising process, so as to influence the denoising process through the image semantic information.

[0140] Processing the first semantic feature through the first control network means adjusting the first semantic feature to a first control feature adapted to the resolution of different processing blocks in the denoising network. In other words, processing the first semantic feature through the first control network yields multiple first control features used to control different processing blocks in the denoising network. These multiple first control features correspond one-to-one with multiple control resolutions; each first control feature has a control resolution, and the control resolution corresponding to the first control feature is the input feature resolution of the processing block to which the first control feature is applied. Since the first control feature controls which processing block performs denoising processing, its resolution should be the same as the input feature resolution of that processing block. Specifically, the aforementioned multiple control resolutions are used to indicate the input feature resolution of multiple processing blocks in the denoising network. These resolutions can include the input feature resolution of intermediate blocks in the denoising network and the input feature resolution of each decoding block within the decoding structure of the denoising network. The input feature resolution of an intermediate block in the denoising network refers to the resolution of the input features of the intermediate block; the decoding structure includes multiple decoding blocks, and the input feature resolution of each decoding block refers to the resolution of the input features of the decoding block.

[0141] The second control network is used to convert the second semantic feature into a second control feature for controlling the denoising network to perform denoising processing. The second control feature refers to the output result obtained by the second control network after processing the second semantic feature. The second control feature is used to indicate the global semantic information and / or local semantic information in the initial image in the denoising process, so as to influence the denoising process through the global semantic information and / or local semantic information.

[0142] Processing the second semantic features through a second control network means adjusting the second semantic features to adapt them to the resolutions of different processing blocks in the denoising network. In other words, processing the second semantic features through the second control network yields multiple second control features used to control different processing blocks in the denoising network. These multiple second control features correspond one-to-one with multiple control resolutions; each second control feature has a specific control resolution, which is the input feature resolution of the processing block to which it is applied. Since the second control feature controls which processing block performs denoising processing, its resolution should be the same as the input feature resolution of that processing block.

[0143] It should be noted that the first control feature obtained through the first control network can be input only into an intermediate block or any one of the decoding blocks of the denoising network, or it can be input into an intermediate block and multiple decoding blocks simultaneously. Similarly, the second control feature obtained through the second control network can also be input only into an intermediate block or any one of the decoding blocks of the denoising network, or it can be input into an intermediate block and multiple decoding blocks simultaneously. This application does not specifically limit the output paths of the first and second control features. Furthermore, the third semantic feature can be input only into an intermediate block, or it can be input into an intermediate block and multiple decoding blocks in the decoding structure simultaneously. Alternatively, the third semantic feature can be input into multiple encoding blocks, intermediate blocks, and multiple decoding blocks in the decoding structure simultaneously. Or, the third semantic feature can be input into multiple encoding blocks and intermediate blocks in the encoding structure. This application does not specifically limit the output path of the third semantic feature.

[0144] After determining the first and second control features, a denoising network can be used to refine the initial noise feature Z based on each of the first and second control features and the third semantic feature. t The denoising process is performed iteratively for N rounds to obtain the repaired image features Z0, where N is an integer greater than 1.

[0145] In one possible implementation, the i-th round of denoising in the above N rounds of denoising processing includes:

[0146] The current noise features are processed by the coding structure in the denoising network to obtain the coded noise features;

[0147] By using the intermediate block, the encoded noise features are processed with reference to the first control feature, the second control feature, and the third semantic feature to obtain the intermediate noise features;

[0148] By using the decoding structure, intermediate noise features are processed with reference to the first control feature, the second control feature, and the third semantic feature to obtain the predicted noise features; and the predicted noise features are removed from the current noise features to obtain the denoised features for the i-th round of denoising.

[0149] Where i is an integer greater than or equal to 1 and less than or equal to N; when i equals 1, the current noise feature is the initial noise feature; when i is greater than 1, the current noise feature is the denoising feature of the (i-1)th round of denoising processing; when i equals N, the denoising feature of the i-th round of denoising processing is the restored image feature.

[0150] Taking the UNet model structure as an example, the denoising network includes an encoding structure, intermediate blocks, and a decoding structure. The encoding structure includes multiple encoding blocks, and the decoding structure includes multiple decoding blocks. Specifically, the current noise features can be denoised using the encoding structure, intermediate blocks, and decoding structure of the denoising network to obtain denoised features. The current noise features refer to the processing object of the i-th round of denoising, that is, the noise features input to the diffusion model in the i-th round of denoising; the denoised features refer to the output result of the i-th round of denoising, that is, the image features obtained after removing the predicted noise features from the current noise features of the i-th round.

[0151] Specifically, the denoising process can include: encoding the current noise features using the coding structure in the denoising network, that is, progressively extracting deep features from the current noise features through the coding blocks included in the coding structure of the denoising network. In this process, the spatial dimension of the current noise features can be gradually reduced, ultimately yielding coded noise features. These coded noise features refer to the output of the coding structure in the denoising network, specifically the features output by the last coding block in the coding structure. The coded noise features are used to represent the current noise features with low-dimensional features.

[0152] After obtaining the encoded noise features, intermediate noise features can be obtained by processing the encoded noise features with reference to the first control features, second control features, and third semantic features through intermediate blocks in the denoising network. For example, feature fusion can be performed on the first control features, second control features, third semantic features, and encoded noise features, such as concatenating the first control features, second control features, third semantic features, and encoded noise features along their respective channel dimensions to obtain fused features. Then, further processing can be performed on the fused features to extract deeper features, resulting in intermediate noise features. These intermediate noise features refer to the output of the intermediate block.

[0153] After obtaining the intermediate noise features, the intermediate noise features can be processed by the decoding structure in the denoising network, referring to the first control features, the second control features, and the third semantic features, to obtain the predicted noise features. For example, when each decoding block in the decoding structure processes the noise, the first control features, the second control features, the third semantic features, and the intermediate noise features can be fused first, such as concatenating the first control features, the second control features, the third semantic features, and the intermediate noise features in the corresponding channel dimensions to obtain the fused features. Then, the fused features can be further processed to expand the low-dimensional feature representation into a high-dimensional feature representation. The last decoding block in the decoding structure will output the predicted noise features of the i-th round. The predicted noise features refer to the output results of the decoding structure, which are used to indicate the noise contained in the current noise features predicted in the i-th round of denoising processing.

[0154] After determining the predicted noise feature, it can be removed from the current noise feature to obtain the denoised feature for the i-th round of denoising. Here, i is an integer greater than or equal to 1 and less than or equal to N, and so on. The above denoising process is iteratively executed in the denoising network to obtain the restored image feature. That is, when i equals N, the denoised feature for the i-th round of denoising is the restored image feature.

[0155] Therefore, based on the i-th round of denoising in the above N rounds of denoising, it can be seen that, based on the first control feature, the second control feature, and the third semantic feature, the predicted noise feature contained in the current noise feature can be determined in each round of denoising. Then, the predicted noise feature is removed from the current noise feature to obtain the corresponding denoised feature. By iteratively executing the above denoising process, the final restored image features can be gradually determined. It is evident that introducing the first control feature, the second control feature, and the third semantic feature during the denoising process enables the denoising network to generate rich details in the initial image and learn the stable restoration effect of the convolutional restoration network, thereby generating restored image features with rich details and stable restoration effect.

[0156] In one possible implementation, both the intermediate block and the decoding block in the denoising network include control sub-blocks. These control sub-blocks are used to perform feature processing with reference to a first control feature, a second control feature, and a third semantic feature. That is, the intermediate block and the decoding block implement their respective processing procedures based on their included control sub-blocks. Specifically, the intermediate block processes the encoded noise features based on the first control feature, the second control feature, and the third semantic feature through its included control sub-blocks, enabling the intermediate block to dynamically focus on different feature information in the initial image according to the aforementioned reference information, thereby improving the restoration effect of the restored image. Correspondingly, the decoding block processes intermediate noise features based on the first control feature, the second control feature, and the third semantic feature through its included control sub-blocks, enabling the decoding block to dynamically focus on different feature information in the initial image according to the aforementioned reference information, thereby further improving the restoration effect of the restored image.

[0157] The aforementioned control sub-block includes: a first attention layer that performs attention operations based on a first control feature and a second control feature; a second attention layer that performs attention operations based on the output features of the first attention layer and the first control feature; a third attention layer that performs self-attention operations based on the output features of the second attention layer; and a fourth attention layer that performs attention operations based on the output features of the third attention layer and a third semantic feature.

[0158] For reference Figure 6 , Figure 6 This is a schematic diagram of the structure of a control sub-block provided in an embodiment of this application. The control sub-block is a structure based on an attention mechanism. The first attention layer refers to the attention layer that focuses on the first control feature and the second control feature in the control sub-block. Through the first attention layer, the processing block (intermediate block or decoding block) to which the control sub-block belongs can focus on the first control feature and the second control feature. For the first attention layer, it can use the first control feature as a query (Query, Q) and the second control feature as a key (Key, K) and a value (Value, V) to perform attention operations based on the first control feature and the second control feature. Specifically, the process of performing attention operations based on the first control feature and the second control feature can include: performing a linear transformation on the first control feature and the second control feature to obtain feature matrices as Q, K, and V, respectively. Then, the similarity between Q and K can be calculated based on the feature matrix of Q and the feature matrix of K to obtain the attention score. After that, the attention score can be normalized by applying the activation function softmax function to obtain the attention weight corresponding to V. Finally, the feature matrix of V can be weighted and summed according to the attention weight corresponding to V to obtain the output feature of the first attention layer.

[0159] The second attention layer refers to the attention layer that focuses on the first control feature and the output feature of the first attention layer within the control sub-block. Through the second attention layer, the processing block to which the control sub-block belongs can focus on the output feature and the first control feature of the first attention layer. For the second attention layer, it can add the output feature of the previous processing block to the output feature of the first attention layer, using the added feature as Q, and the first control feature as K and V, to perform attention operations based on the added feature and the first control feature. Specifically, the process of performing attention operations based on the output feature and the first control feature of the first attention layer can include: performing a linear transformation on the added feature and the first control feature to obtain feature matrices as Q, K, and V, respectively. Then, the dot product between Q and K can be calculated based on the feature matrices of Q and K to obtain the attention score. Afterwards, the attention score can be normalized using the softmax activation function to obtain the attention weight corresponding to V. Finally, the feature matrix of V can be weighted and summed according to the attention weight corresponding to V to obtain the output feature of the second attention layer.

[0160] The third attention layer refers to the attention layer that performs self-attention operations based on the output features of the second attention layer in the control sub-block. In other words, the third attention layer is a self-attention layer. Through the third attention layer, the processing block to which the control sub-block belongs can pay attention to the output features of the second attention layer. For the third attention layer, it can add the features that are Q in the second attention layer to the output features of the second attention layer, and use the added features as Q, K and V in the third attention layer, and perform self-attention operations with Q, K and V.

[0161] The fourth attention layer refers to the attention layer that focuses on the output features and third semantic features of the third attention layer in the control sub-block. Through the fourth attention layer, the processing block to which the control sub-block belongs can focus on the output features and third semantic features of the third attention layer. For the fourth attention layer, it can add the input features and output features of the third attention layer, and use the added feature as Q. It can use the third semantic features as K and V to perform attention operations based on the output features and third semantic features of the third attention layer.

[0162] It should be noted that when the third semantic feature is only input to the intermediate block as reference information, the fourth attention layer in the control sub-block of the decoding block performs attention operations based on the output features of the third attention layer and the text features corresponding to the blank text.

[0163] As can be seen, both the intermediate block and the decoding block in the denoising network include control sub-blocks. Each control sub-block is composed of an attention mechanism, and each control sub-block includes an attention layer that performs attention operations based on different reference information. This can increase the attention of the denoising network to each reference information, that is, enhance the influence of each reference information on the denoising process, thereby improving the denoising capability of the denoising network and making the features retained after denoising more closely resemble the image features indicated by the reference information.

[0164] Therefore, by using the above method, in the denoising process of the diffusion model, initial image features can be added as the first control condition, and intermediate features used to characterize the global semantic information and / or local semantic information of the initial image can be added as the second control condition, so that the repaired image output by the diffusion model is closer to the above control conditions. At the same time, the initial noise features can be iteratively denoised N times by combining the third semantic features with stable repair effect, so as to obtain repaired image features with rich details and stable repair effect.

[0165] Finally, the restored image can be obtained by decoding the features of the restored image using a decoder. Specifically, the decoder converts the feature representation of the restored image in the latent space into its pixel space, thus yielding the restored image corresponding to the initial image. Here, the decoder refers to the structure used to convert the feature representation in the latent space into its pixel space. Specifically, it can be a decoder corresponding to the first encoder. For example, when the first encoder is a Vector Quantized Generative Adversarial Network Encoder (VQ-GAN Encoder), the decoder can be a corresponding Vector Quantized Generative Adversarial Network Decoder (VQ-GAN Decoder). This application does not specifically limit the structure of the decoder.

[0166] The image inpainting method provided in this application innovatively couples a convolutional inpainting model with a diffusion model to collaboratively perform image inpainting tasks. The method includes: acquiring an initial image; inputting the initial image into a convolutional inpainting model; acquiring intermediate features generated by the convolutional inpainting model during pre-inpainting processing of the initial image; and acquiring the output image obtained after the convolutional inpainting model completes the pre-inpainting processing. Subsequently, the initial image is feature-encoded by the first encoder to obtain the first semantic feature, which can represent the original semantic information of the initial image. Based on the intermediate features, the second semantic feature is determined. Since the convolutional inpainting model extracts features from the initial image in multiple dimensions during the inpainting process, the intermediate features generated in this process can fully and meticulously represent the global and local semantic information of the initial image. Correspondingly, the second semantic feature determined based on the intermediate features can also fully and meticulously represent the global and local semantic information extracted by the convolutional inpainting model during the pre-inpainting process of the initial image. The output image is feature-encoded by the second encoder to obtain the third semantic feature. Since the convolutional inpainting model has high stability in image inpainting tasks, the output image obtained after inpainting by the convolutional inpainting model is also relatively stable. The output image is highly consistent with the semantics of the initial image and has higher image quality than the initial image. Correspondingly, the third semantic feature obtained by encoding the output image can represent the semantic information of the higher quality output image. Furthermore, based on the first, second, and third semantic features determined above, a restoration image feature is generated using a diffusion model. This restoration image feature is used to obtain the restored image through feature decoding. In other words, the diffusion model is controlled by the first, second, and third semantic features to perform relevant processing. This allows the diffusion model to utilize information from the original initial image, intermediate features generated by a highly stable convolutional restoration model, and information from the output image obtained by the convolutional restoration model after restoring the initial image. This assists the diffusion model in generating corresponding restoration image features that indicate the restored image. This combines the high stability of the convolutional restoration model with the powerful image generation capability of the diffusion model (capable of generating rich image details), ensuring that the final restored image contains rich image details and is stable, thus improving the image restoration effect.

[0167] In one possible implementation, the image restoration method provided in this application can be used to perform a video restoration task, that is, to restore an initial image sequence composed of multiple initial images. See also Figure 7 , Figure 7 This is a flowchart illustrating the application of the image restoration method provided in this embodiment to a video restoration task. Figure 7 As shown, the method includes the following steps:

[0168] S701: Obtain an initial image sequence consisting of multiple initial images.

[0169] The initial image sequence includes multiple sequentially arranged initial images, which can be video frames from the video to be repaired. These initial images can be arranged according to their chronological order within the video. The video to be repaired refers to a video resource containing low-quality video frames.

[0170] As an example, the server can obtain the video to be repaired from the terminal device. For instance, the terminal device may be running a target application with video repair capabilities. The user can input the video to be repaired through the video input page of the target application. Accordingly, the terminal device can send the user-inputted video to the server so that the server can obtain the video to be repaired. Then, the server can extract several video frames from the video to be repaired as initial images. For example, several video frames can be extracted as initial images according to a preset frame interval, and the extracted initial images can be arranged in chronological order to obtain an initial image sequence.

[0171] As another example, the server can also retrieve the video to be repaired from a database. For instance, if the database stores several historical videos with low image quality, the server can retrieve any one or more of these historical videos as the videos to be repaired. Then, the server can extract several video frames from the retrieved videos as initial images, and arrange these extracted initial images in chronological order to obtain an initial image sequence. This application does not specifically limit the method of obtaining the initial image sequence.

[0172] S702: Input the initial image sequence into the convolutional inpainting model, obtain the comprehensive inter-frame fusion features generated by the convolutional inpainting model during the pre-inpainting process of the initial image sequence, and use them as intermediate features; and obtain the output image obtained after the convolutional inpainting model completes the pre-inpainting process, the output image corresponding to the initial image located in the middle position in the initial image sequence.

[0173] In this embodiment, the convolutional inpainting model is used to perform image inpainting tasks based on image sequences. Specifically, the convolutional inpainting model can be a multi-input single-output (MIMO) structure, meaning the input data of the convolutional inpainting model is an initial image sequence composed of multiple initial images, and the output data of the convolutional inpainting model is the output image corresponding to one of the initial images in the initial image sequence. Specifically, the output image can correspond to the initial image located in the middle position of the initial image sequence; that is, the output image is the inpainting result obtained by the convolutional inpainting model inpainting the initial image located in the middle position of the initial image sequence. When the above-mentioned convolutional inpainting model adopts a multi-input single-output structure similar to the Video Restoration with Enhanced Deformable Convolutional Networks (EDVR) framework, this embodiment of the application does not impose any limitations on the model structure of the convolutional inpainting model.

[0174] As an example, see Figure 8 , Figure 8This is a schematic diagram of a convolutional inpainting model with a multi-input single-output structure provided in an embodiment of this application. The convolutional inpainting model includes: a downsampling structure, used to reduce the resolution of image features corresponding to the initial image; and a predeblur module, used to pre-deblur the image features output by the downsampling structure. The Concatenated Deformable Alignment Module (PCD Align Module) aligns image frames at different time steps to capture inter-frame motion information. Specifically, it aligns the image features output by the pre-deblurring module at the feature level. This module can align image features from different frames at different resolutions to capture motion information at different scales. It uses an image pyramid to downsample each input frame to different resolution levels, aligning layer by layer from high to low resolution. Then, it progressively refines the alignment at different levels, using the alignment result of each layer as input to the next, improving alignment accuracy layer by layer from coarse to fine. Furthermore, it can handle complex, non-linear motion by introducing deformable convolutions, allowing convolutional kernels to sample features at irregular locations, enhancing the network's flexibility and adaptability to different motions. After aligning the image features from multiple initial images using the PCD Align Module, aligned features are obtained. The Temporal Spatial Attention (TSA) Fusion module effectively fuses temporal and spatial information to generate high-quality super-resolution images. Specifically, this module selectively utilizes temporal series information through a temporal attention mechanism to enhance the contribution of important frames. This is achieved by learning the importance weights of different frames and dynamically adjusting their contributions to enhance the effective fusion of temporal information. Furthermore, a spatial attention mechanism enhances key details of spatial information, highlighting important regions and features while ignoring background and unimportant details. The Reconstruction Module reconstructs the fused features into a high-resolution image. This is achieved through a series of convolutional layers that progressively restore the extracted and fused features to a super-resolution image. This structure is typically simple yet effective, ensuring high quality and clarity in the generated image. The Upsampling structure restores the resolution of the image features output by the Reconstruction Module.

[0175] Specifically, the initial image sequence can be input into a convolutional inpainting model for pre-inpainting processing. The input data of the convolutional inpainting model can have the size of [B, t, 3, H, W], where B represents the batch number, t indicates the number of initial images included in each batch, i.e., the number of initial images included in the initial image sequence, 3 represents the number of channels of the initial image, H represents the height of the initial image, and W represents the width of the initial image.

[0176] The TSA Fusion module in the convolutional inpainting model can fuse the image features corresponding to multiple initial images. For example, the image features corresponding to multiple initial images can be added according to their respective channels to obtain the fused image features, which can then be used as the comprehensive inter-frame fusion feature. This comprehensive inter-frame fusion feature refers to the fusion result obtained after fusing the image features corresponding to multiple initial images contained in the initial image sequence. This comprehensive inter-frame fusion feature is used to characterize the key image information in the multiple initial images contained in the initial image sequence. The dimensions of this comprehensive inter-frame fusion feature are [B, C, H, W], where C represents the number of channels of the comprehensive inter-frame fusion feature, B represents the batch number, H represents the height of the comprehensive inter-frame fusion feature, and W represents the width of the comprehensive inter-frame fusion feature.

[0177] After performing pre-inpainting processing on the input initial image sequence, the convolutional inpainting model produces an output image. This output image represents the inpainted result of the initial image located in the middle of the initial image sequence. For example, if the initial image sequence contains 5 initial images, the final output image of the convolutional inpainting model should be the inpainted result of the 3rd initial image in the sequence. The output data of the convolutional inpainting model has dimensions [B, 3, H, W], where B represents the batch number, 3 represents the number of channels in the output image, H represents the height of the output image, and W represents the width of the output image.

[0178] In video restoration tasks, the aforementioned integrated inter-frame fusion features can be used as intermediate features to determine the second semantic features to be provided to the diffusion model. At the same time, the output image obtained after the convolutional restoration model completes the pre-restoration process can also be obtained to determine the third semantic features to be provided to the diffusion model.

[0179] S703: The first encoder encodes the semantic features of the initial image located in the middle position of the initial image sequence to obtain the first semantic feature; the second semantic feature is determined based on the middle feature; the third semantic feature is obtained by encoding the semantic features of the output image through the second encoder.

[0180] When the output image obtained by the convolutional inpainting model corresponds to the initial image located in the middle of the initial image sequence, the first semantic feature can be determined based on the initial image located in the middle of the initial image sequence. Specifically, the semantic features of the initial image located in the middle of the initial image sequence are encoded by the first encoder to obtain the latent space features corresponding to the initial image, i.e., the first semantic feature.

[0181] The determination methods for the second and third semantic features and Figure 2 The implementation process of step S203 in the illustrated embodiment is similar and will not be repeated here.

[0182] S704: Using a diffusion model, a repaired image feature sequence is generated based on the first semantic feature, the second semantic feature, and the third semantic feature; the repaired image feature sequence is used to obtain the repaired image sequence through feature decoding.

[0183] When the diffusion model is used to perform video restoration tasks, it can denoise the initial noise feature sequence based on the first, second, and third semantic features to obtain the restored image feature sequence. The initial noise feature sequence is a sequence composed of multiple initial noise features. The restored image feature sequence is the output result obtained through the reverse diffusion process (i.e., denoising) of the diffusion model, used to characterize the feature representation of the restored image sequence corresponding to the initial image sequence in the latent space. The restored image sequence is a sequence composed of the restored images corresponding to each initial image in the initial image sequence, arranged sequentially.

[0184] Among them, the above-mentioned noise reduction process is related to Figure 2 The implementation process of step S204 in the illustrated embodiment is similar and will not be repeated here.

[0185] After obtaining the feature sequence of the restored image, the feature sequence of the restored image can be decoded by the decoder to obtain the restored image sequence corresponding to the initial image sequence. That is, the feature representation of the restored image sequence in the latent space can be converted into the feature representation of the restored image sequence in the pixel space by the decoder, so as to obtain the restored video corresponding to the video to be restored.

[0186] Therefore, the above method can be used to repair videos. When the convolutional inpainting model uses a multi-input single-output structure for pre-repair processing, the comprehensive inter-frame fusion feature, which integrates the image features of multiple initial images during the pre-repair process, can be used as an intermediate feature to enhance the diffusion model's ability to learn image details. Furthermore, the repaired image corresponding to the initial image located in the middle of the initial image sequence can be used as the output image obtained after the convolutional inpainting model completes the pre-repair processing, ensuring that the output image can more comprehensively reflect the image feature information represented by the initial image sequence. Subsequently, based on the aforementioned initial image sequence, intermediate features, and output image, the first semantic feature, second semantic feature, and third semantic feature can be determined respectively. These features are then used to control the diffusion model to perform denoising processing. This achieves the use of information from the original video to be repaired, the comprehensive inter-frame fusion feature generated by the highly stable convolutional inpainting model, and the information from the output image obtained by the convolutional inpainting model after pre-repairing the video to be repaired, to assist the diffusion model in generating the corresponding repaired video. This results in a final repaired video containing rich image details and being stable, thus achieving video repair and improving the video repair effect.

[0187] In another possible implementation, this embodiment of the application can also be used to perform video restoration tasks. In this embodiment, the convolutional restoration model can output multiple output images. See also Figure 9 , Figure 9 A flowchart illustrating another image restoration method provided in this application when applied to a video restoration task, as shown below. Figure 9 As shown, the method includes the following steps:

[0188] S901: Obtain an initial image sequence consisting of multiple initial images.

[0189] The method of obtaining the initial image sequence in this step is the same as... Figure 7 The implementation process of step S701 in the illustrated embodiment is similar and will not be repeated here.

[0190] S902: Input the initial image sequence into the convolutional inpainting model, obtain the inter-frame fusion features generated by the convolutional inpainting model for each initial image during the pre-inpainting process of the initial image sequence, and form an intermediate feature sequence; and obtain the output image sequence obtained after the convolutional inpainting model completes the pre-inpainting process. The output image sequence includes multiple output images, and each output image corresponds one-to-one with multiple initial images.

[0191] When the convolutional inpainting model adopts a multiple-input multiple-output structure similar to the video super-resolution algorithm (basicvsr), the convolutional inpainting model can output multiple output images. That is, when the input of the convolutional inpainting model is an initial image sequence composed of multiple initial images, the output of the convolutional inpainting model is the output image corresponding to each initial image in the initial image sequence.

[0192] As an example, see Figure 10 , Figure 10 This is a schematic diagram of another convolutional inpainting model provided in an embodiment of this application, taking an initial image sequence that includes multiple initial images of the same resolution as an example. Figure 10 The convolutional repair model includes: a propagation module, which employs a bidirectional recurrent mechanism and is divided into a forward branch F. b and backward branch F f Forward branch F b The backward branch F is used to fuse the image features of the current initial image with the image features of the initial images preceding it. f This is used to fuse the image features of the current initial image with the image features of the subsequent initial images, through the forward branch F. b and backward branch F f All information from the initial image sequence can be incorporated into the subsequent fusion module to maximize the collection and utilization of information from the initial image sequence; the fusion module (i.e. Figure 6 The U module is used to perform feature fusion on the output features of the propagation module; the upsampling structure is used to transform the output features of the fusion module into a high-resolution layer output image.

[0193] Specifically, the initial image sequence can be input into a convolutional inpainting model for inpainting processing. The dimensions of the image features input to this convolutional inpainting model can be [B, t, 3, H, W]. Here, B represents the batch number, t indicates the number of initial images included in each batch (i.e., the number of initial images in the initial image sequence), 3 represents the number of channels in the initial image, H represents the height of the initial image, and W represents the width of the initial image. When inpainting the initial image sequence using the convolutional inpainting model, for each initial image, the forward branch F in the propagation module of the convolutional inpainting model can be used for inpainting. b The system collects the image features of the initial image itself and fuses them with the image features of other initial images in the initial image sequence that are arranged before the initial image to obtain the forward fusion features corresponding to the initial image. At the same time, the system can collect the image features of the initial image itself through the backward branch in the propagation module and fuse them with the image features of other initial images in the initial image sequence that are arranged after the initial image to obtain the backward fusion features corresponding to the initial image.

[0194] After obtaining the forward and backward fusion features corresponding to each initial image, for each initial image, the U module in the convolutional inpainting model can be used to fuse the forward and backward fusion features corresponding to that initial image. For example, the forward and backward fusion features can be added according to their corresponding channels to obtain the fused image features corresponding to that initial image. These fused image features are the inter-frame fusion features corresponding to that initial image. The inter-frame fusion features refer to the result obtained by fusing the image features of the initial image itself with the image features of the adjacent initial images. These inter-frame fusion features are used to characterize the key image information in the adjacent initial images in the initial image sequence.

[0195] Furthermore, the inter-frame fusion features corresponding to each initial image can be used as intermediate features, and these intermediate features can be combined to form an intermediate feature sequence. This intermediate feature sequence can serve as the basis for generating the second semantic feature, providing reference information for subsequent denoising processing by the diffusion model. The size of this intermediate feature sequence is [B, t, C, h, w], where C represents the number of output channels of the inter-frame fusion feature, t represents the number of initial images included in each batch, B represents the batch number, h represents the height of the inter-frame fusion feature, and w represents the width of the inter-frame fusion feature.

[0196] Next, the inter-frame fusion features corresponding to each initial image can be input into the upsampling structure to restore the resolution, obtaining the output image corresponding to each initial image. This yields the output image sequence obtained after the convolutional inpainting model completes the pre-inpainting process. This output image sequence includes multiple sequentially arranged output images, each corresponding one-to-one with a specific initial image in the initial image sequence. The order of the output images matches the order of their respective initial images within the initial image sequence. The size of this output image sequence is [B, t, 3, H, W], where B represents the batch number, t represents the number of initial images in each batch, 3 represents the number of channels in the output image, H represents the height of the output image, and W represents the width of the output image.

[0197] S903: Adjust the size of the initial image sequence, intermediate feature sequence, and output image sequence respectively, so that the adjusted initial image sequence, adjusted intermediate feature sequence, and adjusted output image sequence are all adapted to the input data size of the diffusion model.

[0198] Because the sizes of the initial image sequence, intermediate feature sequence, and output image sequence do not match the size of the input data of the diffusion model, the diffusion model cannot directly reference the semantic features determined based on these sequences. Therefore, it is necessary to adjust the sizes of these sequences separately. For example, the reshape function can be used to adjust the sizes of the initial image sequence, intermediate feature sequence, and output image sequence to ensure that the adjusted sequences are compatible with the input data size of the diffusion model. The reshape function adjusts the image size by changing the image dimensions and / or the number of channels while keeping the total number of pixels in the image constant.

[0199] For example, the input data of a diffusion model is usually four-dimensional, such as [b', c', h', w'], while the initial image sequence, intermediate feature sequence, and output image sequence are all five-dimensional, such as the initial image sequence being [B, t, 3, H, W], the intermediate feature sequence being [B, t, C, h, w], and the output image sequence being [B, t, 3, H, W]. The diffusion model cannot process this kind of five-dimensional data. Therefore, it is necessary to adjust the size of the initial image sequence, intermediate feature sequence, and output image sequence respectively. Specifically, the size of the initial image sequence can be adjusted to [B*t, 3, H, W], the size of the intermediate feature sequence can be adjusted to [B*t, C, h, w], and the size of the output image sequence can be adjusted to [B*t, 3, H, W].

[0200] S904: The first semantic feature is obtained by encoding the semantic features of the adjusted initial image sequence through the first encoder; the second semantic feature is determined based on the adjusted intermediate feature sequence; the third semantic feature is obtained by encoding the semantic features of the adjusted output image sequence through the second encoder.

[0201] After obtaining the adjusted initial image sequence, the first encoder can be used to encode the features of the adjusted initial image sequence to obtain the first semantic features. The first encoder encodes the features of each initial image in the adjusted initial image sequence to obtain the latent space features corresponding to each initial image. The latent space features corresponding to each initial image are combined to obtain the encoded feature sequence corresponding to the initial image sequence. The encoded feature sequence corresponding to the initial image sequence can then be used as the first semantic features.

[0202] After obtaining the adjusted intermediate feature sequence, the second semantic feature can be determined based on the adjusted intermediate feature sequence. For example, when the resolution of the adjusted intermediate feature sequence is the same as the resolution of the input noise feature of the diffusion model, the intermediate feature sequence can be directly used as the second semantic feature. Or, for example, when multiple adjusted intermediate feature sequences are obtained, and the resolution of the multiple adjusted intermediate feature sequences is not exactly the same as the resolution of the input noise feature of the diffusion model, the multiple adjusted intermediate feature sequences with different resolutions can be adjusted to a reference resolution, and then these multiple intermediate feature sequences with reference resolutions can be fused to obtain the second semantic feature.

[0203] After obtaining the adjusted output image sequence, the second encoder can be used to encode the features of the adjusted output image sequence to obtain the third semantic feature. The second encoder encodes the features of each output image in the adjusted output image sequence to obtain the output image features corresponding to each output image. The output image features corresponding to each output image are combined to obtain the output image feature sequence corresponding to the output image sequence. The output image feature sequence corresponding to the output image sequence can then be used as the third semantic feature.

[0204] Therefore, after determining the intermediate feature sequence and the output image sequence, the initial image sequence, intermediate feature sequence, and output image sequence can be resized using the above method, so that the sizes of the resized initial image sequence, resized intermediate feature sequence, and resized output image sequence are all adapted to the input data size of the diffusion model. Furthermore, the first semantic feature, the second semantic feature, and the third semantic feature can be determined directly based on the resized resized initial image sequence, resized intermediate feature sequence, and resized output image sequence, respectively. This allows the diffusion model to directly perform image inpainting based on the resized first semantic feature, second semantic feature, and third semantic feature, thus achieving strong coupling between the diffusion model and the convolutional inpainting model with a multi-input multi-output structure.

[0205] S905: Using a diffusion model, based on the first semantic feature, the second semantic feature, and the third semantic feature, a repaired image feature sequence is obtained; the repaired image feature sequence is used to obtain the repaired image sequence through feature decoding.

[0206] The method for determining the restored image sequence corresponding to the initial image sequence in this step is as follows: Figure 7 The implementation process of step S704 in the illustrated embodiment is similar and will not be repeated here.

[0207] Therefore, the above method can be used to control the diffusion model to perform denoising processing using the first, second, and third semantic features determined above, thereby achieving strong coupling between the diffusion model and the multi-input multi-output convolutional inpainting model, realizing the video inpainting task. Furthermore, by using the information of the original video to be repaired, the intermediate feature sequence composed of the inter-frame fusion features generated by the highly stable convolutional inpainting model, and the output image sequence output by the convolutional inpainting model, the diffusion model can be controlled to generate the repaired video corresponding to the video to be repaired, so that the final generated repaired video contains rich image details and is stable, thereby improving the video inpainting effect.

[0208] Therefore, as shown above, when the convolutional inpainting model adopts a multi-input multi-output structure, the inter-frame fusion features used to characterize key image features in adjacent initial images during the inpainting process can be used as intermediate features. Furthermore, based on the inter-frame fusion features corresponding to each initial image in the initial image sequence, an intermediate feature sequence can be determined. This allows the diffusion model to focus on detailed image information in adjacent initial images when performing image inpainting based on the intermediate feature sequence. Moreover, an output image sequence composed of output images obtained after inpainting each initial image can be obtained, enabling the diffusion model to perform image restoration on each output image in the output image sequence. This allows the diffusion model to inpaint video resources, improving the stability of the diffusion model's inpainting process and enhancing the video inpainting effect.

[0209] In one possible implementation, for the convolutional repair model used in S202 above, this application embodiment provides a model training method, which can be referred to... Figure 11 , Figure 11 The flowchart illustrating the training process of the convolutional insulation model provided in this application embodiment shows that the convolutional insulation model can be trained through the following steps:

[0210] S1101: Perform degradation processing on the ground truth image to obtain the degradation image corresponding to the ground truth image.

[0211] A ground truth image (GTimage) refers to a clear, high-quality image. Degradation processing is used to reduce image quality; for example, degradation processing can include noise addition, blurring, and compression. Noise addition involves adding various types of noise to the ground truth image to simulate noise interference that may occur during acquisition, transmission, and processing. The noise added during noise addition can be Gaussian noise, etc., and this application does not specifically limit the type of noise added. Blurring refers to smoothing the ground truth image using certain algorithms or filters to reduce its sharpness and detail. Compression involves encoding and compressing the ground truth image to reduce its data size for easier storage and transmission; however, compression may reduce the quality of the ground truth image.

[0212] In practical applications, before training a convolutional inpainting model, the server needs to obtain training samples for this model, i.e., degraded images corresponding to the ground truth image. As an example, the aforementioned degradation processing method can be used to degrade the ground truth image, reducing its quality and obtaining a degraded image corresponding to the ground truth image. The degraded image corresponding to the ground truth image refers to the result of degradation processing on the ground truth image; this degraded image is the low-quality image corresponding to the ground truth image.

[0213] S1102: Input the degraded image into the convolutional inpainting model to be trained, and obtain the training output image obtained after the convolutional inpainting model has completed the pre-inpainting processing of the degraded image.

[0214] After obtaining the degraded image corresponding to the ground truth image, the server can input the degraded image into the convolutional inpainting model to be trained. This convolutional inpainting model performs a convolution operation (i.e., pre-inpainting) on ​​the degraded image and outputs a training output image corresponding to that degraded image. This training output image refers to the output result of the convolutional inpainting model to be trained, representing the predicted inpainted image after the trained convolutional inpainting model performs pre-inpainting on the degraded image.

[0215] The model structure and working principle of the convolutional repair model to be trained are the same as those of the convolutional repair model in S202 above. For details, please refer to the relevant content above, which will not be repeated here.

[0216] S1103: Train a convolutional inpainting model based on the difference between the training output image and the ground truth image.

[0217] After obtaining the training output image, the difference between the training output image and the ground truth image can be calculated using a loss function. Based on the calculated loss value, the convolutional inpainting model can be trained.

[0218] As an example, the loss function used in the training process of the convolutional inpainting model can be the L1 function. The L1 function is used to measure the sum of the absolute differences between the ground truth image and the training output image. Specifically, refer to Formula 1.

[0219] L convIR =||I GT -convIR(I LQ )||1Formula 1

[0220] Wherein, L in Formula 1 above convIR This represents the difference between the training output image calculated using the L1 function and the ground truth image; that is, the loss value calculated using the L1 function. GT convIR(I) represents the pixel values ​​of the ground truth image. LQ ) represents the pixel value of the training output image obtained through the convolutional inpainting model. This application does not specifically limit the loss function used during the training of the convolutional inpainting model.

[0221] After calculating the loss value based on the loss function, the convolutional repair model can be trained based on this loss value. That is, with the goal of reducing the loss value, the repair performance of the convolutional repair model is optimized by continuously adjusting the parameters of the convolutional repair model, so as to improve the accuracy and stability of the convolutional repair model.

[0222] In addition, other ground truth images can be degraded to obtain degraded images as test samples. The trained convolutional inpainting model is then tested using these degraded test images. By comparing the output image of the convolutional inpainting model with the corresponding ground truth image in the test samples, the accuracy and stability of the convolutional inpainting model are evaluated. When the accuracy of the convolutional inpainting model is lower than a preset threshold, further adjustments and optimizations can be made, such as adjusting the model parameters, changing the model structure, or adopting other optimization strategies, to improve the inpainting performance.

[0223] It should be understood that in practical applications, a training termination condition can be preset. When the training of the convolutional repair model reaches this termination condition, the training of the convolutional repair model can be considered to have ended. This termination condition could be, for example, that the number of training epochs of the convolutional repair model reaches a preset threshold; or, for example, that the performance of the convolutional repair model is tested and found to have reached a preset performance standard (such as reaching a preset repair accuracy); or, for example, that the performance of the convolutional repair model is tested and found to no longer significantly improve with further training. This application does not impose any limitations on this training termination condition.

[0224] Therefore, the degraded image corresponding to the ground truth image can be used as a training sample through the above method. The convolutional inpainting model can be trained based on the training steps and the training sample. Since the training sample is obtained directly from the ground truth image through degradation processing, the cost of obtaining the training sample can be reduced, which is conducive to obtaining a large number of training samples to train the convolutional inpainting model. Training the convolutional inpainting model based on a large number of training samples can improve the inpainting effect of the convolutional inpainting model.

[0225] In one possible implementation, regarding the use of a second encoder in S203 above, this application embodiment also provides a model training method. The second encoder includes an image encoding structure and a feature adaptation structure in a multimodal model; the multimodal model is a pre-trained model, which is used to map images and text to the same feature space through feature encoding; see reference... Figure 12 , Figure 12 This is a flowchart illustrating the training process of the second encoder provided in an embodiment of this application. The second encoder can be trained through the following steps:

[0226] S1201: Obtain image-text sample pairs and their corresponding annotation results; the image-text sample pairs include training images and training text, and the annotation results indicate the matching relationship between the training images and training text.

[0227] The image-text sample pair includes a training image and training text, which refer to the training data of the second encoder. The image-text sample pair also has a corresponding annotation result, which is used to indicate the matching relationship between the training image and the training text. For example, if the content indicated by the training image in the image-text sample pair matches the content expressed by the training text, the annotation result can be 1, while if the content indicated by the training image in the image-text sample pair does not match the content expressed by the training text, the annotation result can be 0.

[0228] For example, image-text sample pairs and their corresponding annotation results are stored in a database, and the server can retrieve image-text sample pairs and their corresponding annotation results from the database.

[0229] S1202: Semantic feature encoding is performed on the training image through the second encoder to obtain training image features; text feature encoding is performed on the training text through the text encoding structure in the multimodal model to obtain training text features, and the size of the training text features is adjusted so that the size of the adjusted training text features is the same as the size of the training image features.

[0230] The multimodal model is a pre-trained model, for example, it can be a pre-trained CLIP model. The second encoder includes the image encoding structure and feature adaptation structure from the multimodal model. Specifically, when using the second encoder to encode training images, the semantic features of the training images can be encoded first using the image encoding structure in the second encoder. Then, the encoding result output by the feature adaptation structure in the second encoder is adjusted to obtain the training image features. This image encoding structure can be the image encoder in the CLIP model. The training image features refer to the output result obtained after feature encoding of the training images using the second encoder. These training image features can represent the image information in the training images with low-dimensional features.

[0231] Furthermore, text features can be encoded from the training text using a text encoding structure within a multimodal model to obtain training text features. For example, this text encoding structure can be the text encoder in the CLIP model. Training text features refer to the output obtained after encoding the training text using the text encoding structure; these training text features are used to represent the semantic information of the training text.

[0232] After determining the training text features, these features can be rearranged. For example, if the training text features are 1*n dimensional and the training image features are h*w dimensional, the first w features in the 1*n dimensional feature set can be used as the first row of features, the w+1 to 2w features as the second row of features, and so on, until h*w dimensional features are obtained. If n is less than h*w, subsequent features can be padded with 0; if n is greater than h*w, subsequent features are discarded. By adjusting the size of the training text features in this way, the size of the adjusted training text features can be made the same as the size of the training image features, thereby aligning the training text features and training image features to the same feature space.

[0233] S1203: Based on the feature distance between the training image features and the adjusted training text features, as well as the annotation results, train the feature adaptation structure in the second encoder.

[0234] Finally, the feature adaptation structure in the second encoder can be trained based on the feature distance between the training image features and the adjusted training text features, as well as the annotation results. During this process, the parameters of the image encoding structure in the second encoder are frozen. During training, a contrastive learning mechanism can be used to minimize the feature distance between the corresponding training image features and the adjusted training text features. By continuously adjusting the parameters of the feature adaptation structure, the performance of the feature adaptation structure in the second encoder can be optimized.

[0235] It should be noted that the reason for keeping the parameters of the image coding structure frozen and only adjusting the parameters of the feature adaptation structure when training the second encoder is that the image coding structure in the second encoder is the image coding structure in the pre-trained multimodal model. This multimodal model has been trained with a large number of training samples during the pre-training process. Therefore, the image coding structure therein has superior image coding performance. Accordingly, in order to reduce the training resources required in this embodiment, the image coding structure with superior performance can be used directly, and only the feature adaptation structure can be adjusted. This makes the second encoder composed of the image coding structure and the feature adaptation structure suitable for encoding the output image and obtaining a third reference feature that can provide reference information for the diffusion model.

[0236] It should be understood that in practical applications, a training termination condition can be preset. When this condition is met, the training of the feature adaptation structure can be considered complete. This termination condition could be, for example, reaching a preset threshold for the number of training iterations of the feature adaptation structure; or, for example, testing the overall performance of the second encoder and finding that its performance meets a preset performance standard (such as achieving a preset matching accuracy); or, for example, testing the performance of the second encoder and finding that its performance no longer significantly improves with training. This application does not impose any limitations on the training termination condition.

[0237] Therefore, based on the above training steps and the image-text sample pairs and their corresponding annotation results, the final second encoder can be obtained. Since the second encoder ensures that the size of the training text features is the same as the size of the training image features during the training process, and the feature adaptation structure can learn the matching relationship between the training text and the training image, the trained second encoder can generate a third reference feature that is adapted to the size of the diffusion model input data while ensuring that the input data and output data have the same semantic information, thus improving the training effect of the second encoder.

[0238] Regarding the diffusion model used in S204 above, this application embodiment also provides a model training method. (Refer to...) Figure 13 , Figure 13This is a flowchart illustrating the model training method provided in an embodiment of this application. For ease of description, the following description uses a server as the execution entity of the model training method.

[0239] like Figure 13 As shown, the model training method includes the following steps:

[0240] S1301: Obtain the ground truth image; perform degradation processing on the ground truth image to obtain a degraded image.

[0241] As an example, the server can retrieve a ground truth image from a database. After retrieving the ground truth image, it can be degraded using degradation processing methods to obtain a degraded image. Degradation processing can include, but is not limited to, noise reduction, blurring, and compression. This application does not specifically limit the method used to degrade the ground truth image.

[0242] S1302: Input the degraded image into the convolutional inpainting model, obtain the intermediate convolutional features generated by the convolutional inpainting model during the pre-inpainting process of the degraded image, and obtain the convolutional inpainted image obtained after the convolutional inpainting model completes the pre-inpainting process.

[0243] In this embodiment, the degraded image can be input into a convolutional inpainting model for pre-inpainting processing to obtain a convolutionally inpainted image after the model completes the pre-inpainting process. This convolutionally inpainted image is the output result of the convolutional inpainting model after pre-inpainting the degraded image. Simultaneously, the image features generated by the convolutional inpainting model during the pre-inpainting process can be used as intermediate convolutional features. These intermediate convolutional features are used to indicate the global semantic information and / or local semantic information of the degraded image. Taking the UNet model structure sampled by the convolutional inpainting model as an example, the intermediate convolutional features can be image features output by any one or more encoders, intermediate convolutional features can be image features output by intermediate blocks, or image features output by any one or more decoders. This application does not specifically limit the generation stage of the intermediate convolutional features.

[0244] It should be understood that the convolutional image restoration in this step is related to... Figure 2 Similar to the output image in S202 of the illustrated embodiment, the convolutional intermediate features in this step are... Figure 2 The intermediate features in S202 of the illustrated embodiment are similar. You can refer to the description of the output image and intermediate features in S202 to understand the convolutional image restoration and convolutional intermediate features in this step.

[0245] S1303: Select the labeled restoration image corresponding to the degraded image from the ground truth image and the convolutional restoration image; project the labeled restoration image into the latent space to obtain the labeled image features; add labeled noise features to the labeled image features to obtain the training noise features.

[0246] In the process of selecting a labeled restoration image from both the ground truth image and the convolutional restoration image, a predetermined probability can be used to choose the image corresponding to the degraded image. The labeled restoration image refers to a high-quality image that provides annotation information during model training. It can be either the ground truth image corresponding to the degraded image or a convolutional restoration image obtained by the convolutional restoration model based on the degraded image. For example, if the predetermined probability for both the ground truth image and the convolutional restoration image is 50%, then when selecting a labeled restoration image for a certain degraded image, the labeled restoration image corresponding to that degraded image can be selected with equal probability from both the ground truth image and the convolutional restoration image.

[0247] Since the convolutional inpainting model outputs an inpainted image obtained by processing intermediate convolutional features through a subsequent decoding structure, these intermediate features are strongly correlated with the inpainted image. The training process of the diffusion model can be understood as making the noise features predicted by the diffusion model approximate the labeled noise features, thereby making the features of the denoised inpainted image approximate the labeled image features. Therefore, selecting the inpainted image output by the convolutional inpainting model as the labeled inpainted image with a preset probability allows the features of the inpainted image output by the diffusion model to approximate the image features of the inpainted image with a preset probability. Furthermore, to make the output inpainted image features approximate the image features of the inpainted image, the diffusion model will more fully utilize the reference features determined based on the intermediate convolutional features. Thus, the diffusion model can better learn how to use the intermediate convolutional features during training.

[0248] After identifying the labeled and restored image corresponding to the degraded image, an encoder can be used to project the labeled and restored image into the latent space to obtain the labeled image features. For example, a VQ-GAN encoder can be used to project the labeled and restored image into the latent space to obtain the feature representation of the labeled and restored image in the latent space, i.e., the labeled image features.

[0249] Then, labeled noise features can be added to the labeled image features to obtain training noise features. These labeled noise features are noise features added to the labeled image features and will be recorded for use as labeled data during subsequent model training. For example, after determining the labeled image features, forward diffusion (i.e., noise addition) can be iteratively performed on the labeled image features, that is, noise features are iteratively added to the labeled image features to obtain training noise features. During this process, each added noise feature is recorded as the labeled noise feature.

[0250] S1304: The degraded image is semantically encoded by the first encoder to obtain the first training semantic features; the second training semantic features are determined based on the convolution intermediate features. The second training semantic features are used to characterize the semantic information extracted by the convolutional restoration model during the pre-restoration process of the degraded image; the convolutional restoration image is semantically encoded by the second encoder to obtain the third training semantic features.

[0251] After identifying the degraded image, a first encoder can be used to encode its semantic features, projecting the degraded image into a latent space to obtain the latent space features corresponding to the degraded image, i.e., the first trained semantic features. These first trained semantic features are used to represent the original semantic information in the degraded image; they can characterize key information in the degraded image with low-dimensional features. These first trained semantic features are... Figure 2 The first semantic feature in S203 of the illustrated embodiment is similar, and its specific determination method can be found in [reference needed]. Figure 2 The relevant content in the illustrated embodiment.

[0252] The second training semantic feature is determined based on the intermediate features of the convolution. For example, when the resolution of the intermediate features of the convolution is the same as the resolution of the input noise features of the diffusion model, the intermediate features of the convolution can be directly used as the second training semantic feature. Alternatively, when the resolution of the intermediate features of the convolution is different from the resolution of the input noise features of the diffusion model, the second training semantic feature can be obtained by performing a corresponding resolution transformation on the acquired intermediate features of the convolution. The second training semantic feature is used to represent the global and local semantic information in the degraded image. It can characterize the global and local semantic information in the degraded image with low-dimensional features. This second training semantic feature is related to... Figure 2 The second semantic feature in S203 of the illustrated embodiment is similar, and its specific determination method can be found in [reference needed]. Figure 2 The relevant content in the illustrated embodiment.

[0253] As an example, when the resolution of the acquired convolutional intermediate features is the same as the resolution of the training noise features processed by the diffusion model, the convolutional intermediate features can be directly used as the second training semantic features.

[0254] As another example, when the resolution of the acquired convolutional intermediate features is different from the resolution of the training noise features processed by the diffusion model, the resolution of the training noise features can be used as the reference resolution, and the resolution of each acquired convolutional intermediate feature can be adjusted to the reference resolution. For example, the resolution of each acquired convolutional intermediate feature can be adjusted to the reference resolution through upsampling, downsampling, and interpolation, and the convolutional intermediate features with the reference resolution can be fused to obtain the second training semantic features.

[0255] The convolutional restoration image is feature-encoded using a second encoder to map it into a latent space, resulting in a third trained semantic feature. This third trained semantic feature is used to represent the image information of the higher-quality degraded image. It can characterize key information in the higher-quality degraded image with low-dimensional features. This third trained semantic feature is related to... Figure 2 The third semantic feature in S203 of the illustrated embodiment is similar, and its specific determination method can be found in [reference needed]. Figure 2 The relevant content in the illustrated embodiment.

[0256] S1305: Using the diffusion model to be trained, based on the first training semantic features, the second training semantic features, and the third training semantic features, determine the predicted noise features in the training noise features.

[0257] After determining the first, second, and third training semantic features, the diffusion model to be trained can be used to denoise the training noise features based on these features to identify the predicted noise features. The predicted noise features refer to the noise features added during the denoising process, predicted by the diffusion model.

[0258] The diffusion model to be trained includes a first control network, a second control network, and a denoising network. The first control network is used to incorporate information carried by the first training semantic features to control the diffusion model to perform denoising processing; that is, it incorporates the semantic information of the original degraded image to control the diffusion model to perform denoising processing. The second control network is used to incorporate information carried by the second training semantic features to control the diffusion model to perform denoising processing; that is, it incorporates the information carried by the intermediate convolutional features generated by the convolutional inpainting model during the pre-inpainting process to control the diffusion model to perform denoising processing.

[0259] During training, the first training semantic features can be adjusted to adapt to the resolution of different processing blocks in the denoising network through the first control network. That is, by processing the first training semantic features through the first control network, multiple first training control features can be obtained to control different processing blocks in the denoising network. The first training control feature refers to the output result obtained by the first control network after processing the first training semantic features. The first training control feature is used to indicate the first control condition of the diffusion model; that is, the first training control feature is used to indicate the original image information of the degraded image in the denoising process, so as to influence the denoising process through this image information.

[0260] Multiple first training control features correspond one-to-one with multiple control resolutions. That is, each first training control feature has a control resolution. The control resolution corresponding to the first training control feature is the input feature resolution of the processing block to which the first training control feature is applied. The first training control feature is used to control which processing block performs denoising processing. Therefore, the resolution of the first training control feature should be the same as the resolution of the input feature of the processing block.

[0261] The second training semantic features are adjusted by a second control network to adapt to the resolution of different processing blocks in the denoising network. In other words, by processing the second training semantic features through the second control network, multiple second training control features can be obtained to control different processing blocks in the denoising network. The second training control features refer to the output results obtained after the second control network processes the second training semantic features. These second training control features are used to indicate the second control conditions of the diffusion model; that is, they are used to indicate the global and local semantic information in the degraded image during denoising, so as to influence the denoising process through this global and local semantic information.

[0262] Multiple second training control features correspond one-to-one with multiple control resolutions. That is, each second training control feature has a control resolution. The control resolution corresponding to the second training control feature is the input feature resolution of the processing block to which the second training control feature is applied. The second training control feature is used to control which processing block performs denoising processing. Therefore, the resolution of the second training control feature should be the same as the resolution of the input feature of the processing block.

[0263] After determining the first and second training control features, a denoising network can iteratively perform N rounds of denoising on the training noise features based on each of the first and second training control features and the third training semantic feature, obtaining the corresponding predicted noise feature in each round. The N rounds of denoising during training are identical to those in the image inpainting method described above, except that the first control feature is replaced with the first training control feature, the second control feature with the second training control feature, and the third semantic feature with the third training semantic feature; these details will not be elaborated further here.

[0264] S1306: Train a diffusion model based on the difference between predicted noise features and labeled noise features.

[0265] It should be noted that during the noise addition process, a labeled noise feature is added in each round of noise addition. Correspondingly, during the denoising process, a predicted noise feature is obtained in each round of denoising. When calculating the difference between the predicted noise feature and the labeled noise feature, for the predicted noise feature obtained in each round of denoising, it is necessary to find the corresponding labeled noise feature of that round. For example, the predicted noise feature obtained in the first round of denoising corresponds to the labeled noise feature added in the last round of noise addition. Then, the loss value is calculated based on the corresponding predicted noise feature and labeled noise feature.

[0266] After determining the predicted noise features, the difference between the corresponding predicted noise features and the labeled noise features can be calculated using a loss function. Based on the calculated loss value, the diffusion model can be trained.

[0267] As an example, the loss function in the diffusion model can be referenced from Equation 2.

[0268] L DF =||∈-∈ θ (Z t ,t,c,Z lp f lp )||2Formula 2

[0269] Wherein, L in Formula 2 above DF This represents the difference between the predicted noise features calculated using the loss function and the labeled noise features; that is, the loss value calculated using the loss function. ∈ represents the labeled noise features, ∈ θ Z represents the predicted noise characteristics. t Z represents the training noise features, t represents the denoising round corresponding to the training noise features, i.e., the diffusion step in the process of obtaining training noise features based on labeled image features. lp f represents the first training control feature. lp Let represent the second training control feature, and c represent the third training semantic feature. This application does not specifically limit the loss function used during the training of the diffusion model.

[0270] After calculating the loss value based on the loss function, the diffusion model can be trained based on this loss value. That is, with the goal of reducing the loss value, the repair performance of the diffusion model is optimized by continuously adjusting the parameters of the diffusion model, so as to improve the accuracy and stability of the diffusion model.

[0271] In one possible implementation, the diffusion model includes a first control network, a second control network, and a denoising network. The denoising network includes an encoding structure composed of multiple encoding blocks, an intermediate block, and a decoding structure composed of multiple decoding blocks. The intermediate block and the decoding block include a control sub-block for feature processing with reference to the first training semantic features, the second training semantic features, and the third training semantic features (the structure of which can be found in [reference]). Figure 6 The phrase "training a diffusion model based on the difference between predicted noise features and labeled noise features" in S1306 above can include:

[0272] Based on the difference between predicted noise features and labeled noise features, the first control network, the second control network, and a partial attention layer in the control sub-block of the diffusion model are trained; the partial attention layer includes a first attention layer that performs attention operations based on the first training semantic features and the second training semantic features, and a second attention layer that performs attention operations based on the output features of the first attention layer and the first training semantic features.

[0273] Since the predicted noise features identified above are obtained by denoising the training noise features based on the first training control features, the second training control features, and the third training semantic features, the loss value calculated based on the loss function between the corresponding predicted noise features and the labeled noise features reflects the accuracy of the denoising process performed by the diffusion model based on the first training control features, the second training control features, and the third training semantic features. It should be understood that the larger the loss value, the lower the accuracy of the denoising process performed by the diffusion model, indicating poorer reference performance for the first training control features, the second training control features, and the third training semantic features; conversely, the smaller the loss value, the higher the accuracy of the denoising process performed by the diffusion model, indicating better reference performance for the first training control features, the second training control features, and the third training semantic features.

[0274] It should be noted that before training, the weight parameters of the first control network, the second control network, and the denoising network can be initialized to improve training efficiency. For example, the first and second control networks can be preloaded with the parameters of the pre-trained denoising network, specifically the parameters of the encoding structure in the denoising network. For a control sub-block, the second attention layer can be initialized using the pre-trained parameters of the third attention layer, while the first attention layer can be randomly initialized.

[0275] Finally, based on this loss value, the first control network in the diffusion model can be trained, i.e., with the goal of reducing the loss value, backward computation is performed to calculate the gradient of the first training control feature output by the first control network, and the weights in the first control network are adjusted according to the gradient to optimize the accuracy of the denoising network in image restoration based on the first training control feature output by the first control network. Based on this loss value, the second control network in the diffusion model can be trained, i.e., with the goal of reducing the loss value, backward computation is performed to calculate the gradient of the second training control feature output by the second control network, and the weights in the second control network are adjusted according to the gradient to optimize the accuracy of the denoising network in image restoration based on the second training control feature output by the second control network. Based on this loss value, some attention layers in each control sub-block are trained. These attention layers can include a first attention layer that performs attention operations based on the first and second training semantic features, and a second attention layer that performs attention operations based on the output features of the first attention layer and the first training semantic features. Based on this loss value, the first and second attention layers in each control sub-block can be trained, i.e., with the goal of reducing the loss value, by continuously adjusting the parameters in the first and second attention layers in each control sub-block to optimize the restoration performance of the diffusion model.

[0276] Therefore, through the above method, it can be seen that during the training process of the diffusion model, only the control sub-blocks of the first control network, the second control network, and the denoising network that participate in the fusion training of semantic features can be trained. This can save the computational resources of the diffusion model while ensuring its repair capability.

[0277] It should be understood that in practical applications, a training termination condition can be preset. When the training of the diffusion model reaches this termination condition, the training of the diffusion model can be considered to have ended. This termination condition could be, for example, the number of training epochs for the diffusion model reaching a preset threshold, or it could be that the performance of the diffusion model has been tested and found to have reached a preset performance standard (such as reaching a preset repair accuracy), or it could be that the performance of the diffusion model no longer significantly improves with further training. This application does not impose any limitations on the training termination condition in this embodiment.

[0278] Therefore, based on the above training steps, a final diffusion model for image restoration can be obtained. Since the diffusion model combines the first training control feature for representing the original semantic information of the degraded image, the second training control value feature for representing the global and local semantic information of the degraded image, and the third training semantic feature for representing the semantic information of the higher quality degraded image during the training process, the diffusion model combines the high stability of the convolutional restoration model with the powerful image generation capability of the diffusion model during the training process. As a result, the trained diffusion model can generate stable images with rich image details, which alleviates the artifacts and semantic inconsistencies that may be caused when the diffusion model performs image restoration tasks, and improves the restoration effect when the diffusion model performs image restoration tasks.

[0279] Based on the image restoration method provided in the preceding embodiments, this application also provides an image restoration apparatus. The following, in conjunction with... Figure 14 To explain, Figure 14 This is a schematic diagram of the structure of the image restoration apparatus 1400 provided in the embodiments of this application. The apparatus includes:

[0280] The first acquisition module 1401 is used to acquire the initial image;

[0281] The first convolutional inpainting module 1402 is used to input the initial image into the convolutional inpainting model, obtain the intermediate features generated by the convolutional inpainting model during the pre-inpainting process of the initial image, and obtain the output image obtained by the convolutional inpainting model after completing the pre-inpainting process.

[0282] The first feature preprocessing module 1403 is used to encode the semantic features of the initial image using a first encoder to obtain a first semantic feature; determine a second semantic feature based on the intermediate features; the second semantic feature is used to characterize the semantic information extracted by the convolutional inpainting model during the prepainting process of the initial image; and encode the semantic features of the output image using a second encoder to obtain a third semantic feature.

[0283] The first diffusion repair module 1404 is used to generate repaired image features based on the first semantic feature, the second semantic feature and the third semantic feature through a diffusion model; the repaired image features are used to obtain the repaired image through feature decoding.

[0284] Optionally, the diffusion model includes a first control network, a second control network, and a denoising network; the first diffusion repair module 1404 includes:

[0285] The first processing unit is configured to process the first semantic feature through the first control network to obtain a plurality of first control features; the plurality of first control features correspond one-to-one with a plurality of control resolutions, the plurality of control resolutions including the input feature resolution of the intermediate block in the denoising network and the input feature resolution of each of the plurality of decoding blocks in the decoding structure of the denoising network;

[0286] The second processing unit is used to process the second semantic feature through the second control network to obtain multiple second control features; the multiple second control features correspond one-to-one with the multiple control resolutions.

[0287] The first denoising processing unit is configured to perform N rounds of denoising processing on the initial noise features through the denoising network, based on each of the first control features, each of the second control features and the third semantic features, to obtain the repaired image features; where N is an integer greater than 1.

[0288] Optionally, the i-th round of denoising in the N rounds of denoising in the first denoising processing unit includes the following units:

[0289] The third processing unit is used to process the current noise features through the coding structure in the denoising network to obtain coded noise features;

[0290] The fourth processing unit is used to process the encoded noise feature through the intermediate block, referring to the first control feature, the second control feature and the third semantic feature, to obtain the intermediate noise feature;

[0291] The fifth processing unit is configured to process the intermediate noise features using the decoding structure, referring to the first control feature, the second control feature, and the third semantic feature, to obtain the predicted noise features; and to remove the predicted noise features from the current noise features to obtain the denoised features of the i-th round of denoising.

[0292] Wherein, i is an integer greater than or equal to 1 and less than or equal to N; when i equals 1, the current noise feature is the initial noise feature; when i is greater than 1, the current noise feature is the denoising feature of the (i-1)th round of denoising processing; when i equals N, the denoising feature of the i-th round of denoising processing is the repaired image feature.

[0293] Optionally, both the intermediate block and the decoding block include a control sub-block; the control sub-block is used to perform feature processing with reference to the first control feature, the second control feature, and the third semantic feature;

[0294] The control sub-block includes: a first attention layer that performs attention operations based on the first control feature and the second control feature; a second attention layer that performs attention operations based on the output feature of the first attention layer and the first control feature; a third attention layer that performs self-attention operations based on the output feature of the second attention layer; and a fourth attention layer that performs attention operations based on the output feature of the third attention layer and the third semantic feature.

[0295] Optionally, the first feature preprocessing module 1403 includes:

[0296] The mapping unit is used to map the initial image to the latent space through a vector quantization encoder based on a generative adversarial network to obtain the first semantic feature; the latent space is the feature space of the input noise features of the diffusion model.

[0297] Optionally, the first convolutional repair module 1402 includes:

[0298] The first determining unit is used to determine the resolution of the input noise features of the diffusion model as a reference resolution;

[0299] The first acquisition unit is used to acquire intermediate features with the reference resolution generated by the convolutional inpainting model during the pre-inpainting process;

[0300] Correspondingly, the first feature preprocessing module 1403 includes:

[0301] The second determining unit is used to use the intermediate feature as the second semantic feature.

[0302] Optionally, the first convolutional repair module 1402 includes:

[0303] The second acquisition unit is used to acquire multiple intermediate features generated by the convolutional insulation model during the pre-insulation process; the multiple intermediate features correspond to multiple resolutions.

[0304] Correspondingly, the first feature preprocessing module 1403 includes:

[0305] The third determining unit is used to determine the resolution of the input noise features of the diffusion model as a reference resolution;

[0306] The fusion processing unit is configured to, when acquiring the plurality of intermediate features, and when there is an intermediate feature among the plurality of intermediate features whose resolution is not equal to the reference resolution, adjust the resolution of each acquired intermediate feature to the reference resolution, and perform fusion processing on each intermediate feature having the reference resolution to obtain the second semantic feature.

[0307] Optionally, the second encoder includes an image encoding structure and a feature adaptation structure in a multimodal model; the multimodal model is used to map images and text to the same feature space through feature encoding;

[0308] The first feature preprocessing module 1403 includes:

[0309] The sixth processing unit is used to process the output image through the image coding structure to obtain output image features;

[0310] A generation unit is configured to generate at least one third semantic feature based on the output image features through the feature adaptation structure; the resolution of the at least one third semantic feature matches the input feature resolution of at least one processing block in the denoising network of the diffusion model, wherein the processing block is an intermediate block in the denoising network or at least one of the decoding blocks included in the decoding structure of the denoising network.

[0311] Optionally, the first acquisition module 1401 includes:

[0312] The third acquisition unit is used to acquire an initial image sequence composed of multiple initial images;

[0313] Correspondingly, the first convolutional repair module 1402 includes:

[0314] The first repair processing unit is used to input the initial image sequence into the convolutional repair model, obtain the comprehensive inter-frame fusion features generated by the convolutional repair model during the pre-repair processing of the initial image sequence, and use them as the intermediate features; and obtain the output image obtained by the convolutional repair model after completing the pre-repair processing, wherein the output image corresponds to the initial image located in the middle position in the initial image sequence.

[0315] Correspondingly, the first feature preprocessing module 1403 includes:

[0316] The first semantic feature is obtained by encoding the semantic features of the initial image located in the middle position of the initial image sequence using the first encoder.

[0317] Optionally, the first acquisition module 1401 includes:

[0318] The fourth acquisition unit is used to acquire an initial image sequence composed of multiple initial images;

[0319] Correspondingly, the first convolutional repair module 1402 includes:

[0320] The second repair processing unit is used to input the initial image sequence into the convolutional repair model, obtain the inter-frame fusion features generated by the convolutional repair model for each initial image during the pre-repair processing of the initial image sequence, and form an intermediate feature sequence; and obtain the output image sequence obtained by the convolutional repair model after completing the pre-repair processing, wherein the output image sequence includes multiple output images, and the multiple output images correspond one-to-one with the multiple initial images.

[0321] Optionally, the first feature preprocessing module 1403 is further configured to:

[0322] The initial image sequence, the intermediate feature sequence, and the output image sequence are resized so that the resized initial image sequence, the resized intermediate feature sequence, and the resized output image sequence are all adapted to the input data size of the diffusion model; wherein, the resized initial image sequence is used to generate the first semantic feature, the resized intermediate feature sequence is used to generate the second semantic feature, and the resized output image sequence is used to generate the third semantic feature.

[0323] Initial image semantic features. Optionally, the convolutional inpainting model is trained using the following units:

[0324] A degradation processing unit is used to perform degradation processing on the ground truth image to obtain a degradation image corresponding to the ground truth image;

[0325] The third restoration processing unit is used to input the degraded image into the convolutional restoration model to be trained, and obtain the training output image obtained by the convolutional restoration model after completing the pre-restoration processing of the degraded image;

[0326] The first training unit is used to train the convolutional inpainting model based on the difference between the training output image and the ground truth image.

[0327] Optionally, the second encoder includes an image encoding structure and a feature adaptation structure in a multimodal model; the multimodal model is a pre-trained model used to map images and text to the same feature space through feature encoding; the second encoder is trained using the following units:

[0328] The fifth acquisition unit is used to acquire image-text sample pairs and their corresponding annotation results; the image-text sample pairs include training images and training text, and the annotation results indicate the matching relationship between the training images and the training text;

[0329] The second feature encoding unit is used to perform semantic feature encoding on the training image through the second encoder to obtain training image features; to perform text feature encoding on the training text through the text encoding structure in the multimodal model to obtain training text features; and to adjust the size of the training text features so that the size of the adjusted training text features is the same as the size of the training image features.

[0330] The second training unit is used to train the feature adaptation structure in the second encoder based on the feature distance between the training image features and the adjusted training text features, as well as the annotation results.

[0331] Based on the model training method provided in the preceding embodiments, this application also provides a corresponding model training apparatus. The following, in conjunction with... Figure 15 To explain, Figure 15 This is a schematic diagram of the structure of the model training device 1500 provided in the embodiments of this application. The device includes:

[0332] The second acquisition module 1501 is used to acquire a ground truth image; and to perform degradation processing on the ground truth image to obtain a degraded image;

[0333] The second convolutional inpainting module 1502 is used to input the degraded image into the convolutional inpainting model, obtain the convolutional intermediate features generated by the convolutional inpainting model during the pre-inpainting process of the degraded image, and obtain the convolutional inpainted image obtained by the convolutional inpainting model after completing the pre-inpainting process.

[0334] The annotation module 1503 is used to select the annotated restoration image corresponding to the degraded image from the ground truth image and the convolutional restoration image; project the annotated restoration image into the latent space to obtain the annotated image features; and add annotation noise features to the annotated image features to obtain training noise features.

[0335] The second feature preprocessing module 1504 is used to encode the degraded image with semantic features through the first encoder to obtain a first training semantic feature; determine a second training semantic feature based on the convolution intermediate features, wherein the second training semantic feature is used to characterize the semantic information extracted by the convolutional restoration model during the pre-restoration process of the degraded image; and encode the convolutional restored image with semantic features through the second encoder to obtain a third training semantic feature.

[0336] The noise prediction module 1505 is used to determine the predicted noise features in the training noise features based on the first training semantic features, the second training semantic features, and the third training semantic features using the diffusion model to be trained.

[0337] The model training module 1506 is used to train the diffusion model based on the difference between the predicted noise features and the labeled noise features.

[0338] Optionally, the diffusion model includes a first control network, a second control network, and a denoising network. The denoising network includes an encoding structure composed of multiple encoding blocks, an intermediate block, and a decoding structure composed of multiple decoding blocks. The intermediate block and the decoding block include control sub-blocks for performing feature processing with reference to the first training semantic features, the second training semantic features, and the third training semantic features.

[0339] The model training module 1506 includes:

[0340] The third training unit is used to train the first control network, the second control network, and a portion of the attention layer in the control sub-block of the diffusion model based on the difference between the predicted noise features and the labeled noise features; the portion of the attention layer includes a first attention layer that performs attention operations based on the first training semantic features and the second training semantic features, and a second attention layer that performs attention operations based on the output features of the first attention layer and the first training semantic features.

[0341] This application also provides a computer device, which may specifically be a terminal device or a server. The terminal device and server provided in this application will be described below from the perspective of hardware implementation.

[0342] See Figure 16 , Figure 16 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. For example... Figure 16 As shown, for ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. The terminal can be any terminal device including mobile phones, tablets, personal digital assistants (PDAs), point-of-sale (POS) terminals, in-vehicle computers, etc. Taking a computer as an example:

[0343] Figure 16 This is a block diagram illustrating a portion of the structure of a computer associated with the terminal provided in an embodiment of this application. (Reference) Figure 16The computer includes: a radio frequency (RF) circuit 1210, a memory 1220, an input unit 1230 (including a touch panel 1231 and other input devices 1232), a display unit 1240 (including a display panel 1241), a sensor 1250, an audio circuit 1260 (connected to a speaker 1261 and a microphone 1262), a wireless fidelity (WiFi) module 1270, a processor 1280, and a power supply 1290, etc. Those skilled in the art will understand that... Figure 16 The computer architecture shown does not constitute a limitation on the computer and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0344] The memory 1220 can be used to store software programs and modules. The processor 1280 executes various computer functions and data processing by running the software programs and modules stored in the memory 1220. The memory 1220 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer (such as audio data, telephone directory, etc.). In addition, the memory 1220 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0345] The processor 1280 is the control center of the computer, connecting various parts of the computer through various interfaces and lines. It performs various computer functions and processes data by running or executing software programs and / or modules stored in the memory 1220, and by calling data stored in the memory 1220. Optionally, the processor 1280 may include one or more processing units; preferably, the processor 1280 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 1280.

[0346] In the embodiments of this application, the processor 1280 included in the terminal is used to execute the steps in the image restoration method described in the foregoing embodiments, or to execute the steps in the model training method described in the foregoing embodiments.

[0347] See Figure 17 , Figure 17This is a schematic diagram of the structure of a server 1300 provided in an embodiment of this application. The server 1300 can vary significantly due to different configurations or performance, and may include one or more central processing units (CPUs) 1322 (e.g., one or more processors) and memory 1332, and one or more storage media 1330 (e.g., one or more mass storage devices) for storing application programs 1342 or data 1344. The memory 1332 and storage media 1330 can be temporary or persistent storage. The program stored in the storage media 1330 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the CPU 1322 may be configured to communicate with the storage media 1330 and execute the series of instruction operations stored in the storage media 1330 on the server 1300.

[0348] Server 1300 may also include one or more power supplies 1326, one or more wired or wireless network interfaces 1350, one or more input / output interfaces 1358, and / or one or more operating systems, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0349] The steps performed by the server in the above embodiments can be based on this Figure 17 The server structure shown.

[0350] The CPU 1322 is used to execute the steps in the image inpainting method described in the foregoing embodiments, or to execute the steps in the model training method described in the foregoing embodiments.

[0351] This application also provides a computer-readable storage medium for storing a computer program that performs steps in the image restoration methods described in the foregoing embodiments, or performs steps in the model training methods described in the foregoing embodiments.

[0352] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform steps in the image inpainting methods described in the foregoing embodiments, or steps in the model training methods described in the foregoing embodiments.

[0353] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0354] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0355] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0356] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0357] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing computer programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0358] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0359] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0360] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image restoration method, characterized in that, The method includes: Get the initial image; The initial image is input into the convolutional inpainting model to obtain the intermediate features generated by the convolutional inpainting model during the pre-inpainting process of the initial image, and to obtain the output image obtained by the convolutional inpainting model after completing the pre-inpainting process. The initial image is semantically encoded using a first encoder to obtain first semantic features; Based on the intermediate features, a second semantic feature is determined, which is used to characterize the semantic information extracted by the convolutional inpainting model during the pre-inpainting process of the initial image. The output image is semantically encoded using a second encoder to obtain a third semantic feature; Using a diffusion model, repaired image features are generated based on the first semantic feature, the second semantic feature, and the third semantic feature. These repaired image features are used to obtain the repaired image through feature decoding.

2. The method according to claim 1, characterized in that, The diffusion model includes a first control network, a second control network, and a denoising network; the step of generating repaired image features based on the first semantic feature, the second semantic feature, and the third semantic feature using the diffusion model includes: The first semantic feature is processed by the first control network to obtain multiple first control features; the multiple first control features correspond one-to-one with multiple control resolutions, and the multiple control resolutions include the input feature resolution of the intermediate block in the denoising network and the input feature resolution of each of the multiple decoding blocks in the decoding structure of the denoising network; The second semantic feature is processed by the second control network to obtain multiple second control features; the multiple second control features correspond one-to-one with the multiple control resolutions. The denoising network iteratively performs N rounds of denoising processing on the initial noise features based on each of the first control features, each of the second control features, and the third semantic features to obtain the repaired image features; where N is an integer greater than 1.

3. The method according to claim 2, characterized in that, The i-th round of noise reduction in the N-round noise reduction process includes: The current noise features are processed by the coding structure in the denoising network to obtain the coded noise features; The intermediate noise features are obtained by processing the encoded noise features with reference to the first control feature, the second control feature, and the third semantic feature through the intermediate block; Using the decoding structure, the intermediate noise features are processed with reference to the first control feature, the second control feature, and the third semantic feature to obtain the predicted noise features; and the predicted noise features are removed from the current noise features to obtain the denoised features of the i-th round of denoising. Wherein, i is an integer greater than or equal to 1 and less than or equal to N; when i equals 1, the current noise feature is the initial noise feature; when i is greater than 1, the current noise feature is the denoising feature of the (i-1)th round of denoising processing; when i equals N, the denoising feature of the i-th round of denoising processing is the repaired image feature.

4. The method according to claim 3, characterized in that, Both the intermediate block and the decoding block include a control sub-block; the control sub-block is used to perform feature processing with reference to the first control feature, the second control feature, and the third semantic feature; The control sub-block includes: a first attention layer that performs attention operations based on the first control feature and the second control feature; a second attention layer that performs attention operations based on the output feature of the first attention layer and the first control feature; a third attention layer that performs self-attention operations based on the output feature of the second attention layer; and a fourth attention layer that performs attention operations based on the output feature of the third attention layer and the third semantic feature.

5. The method according to any one of claims 1 to 4, characterized in that, The step of encoding semantic features of the initial image using a first encoder to obtain first semantic features includes: The initial image is mapped to the latent space using a vector quantization encoder based on a generative adversarial network to obtain the first semantic feature; the latent space is the feature space of the input noise features of the diffusion model.

6. The method according to any one of claims 1 to 5, characterized in that, The process of obtaining intermediate features generated by the convolutional inpainting model during the pre-inpainting process of the initial image includes: The resolution of the input noise features of the diffusion model is determined as the reference resolution; Obtain the intermediate features with the reference resolution generated by the convolutional inpainting model during the pre-inpainting process; The step of determining the second semantic feature based on the intermediate feature includes: The intermediate feature is used as the second semantic feature.

7. The method according to any one of claims 1 to 5, characterized in that, The process of obtaining intermediate features generated by the convolutional inpainting model during the pre-inpainting process of the initial image includes: The convolutional inpainting model generates multiple intermediate features during the pre-inpainting process; these intermediate features correspond to multiple resolutions. The step of determining the second semantic feature based on the intermediate feature includes: The resolution of the input noise features of the diffusion model is determined as the reference resolution; If, when the plurality of intermediate features are acquired, and there is an intermediate feature among the plurality of intermediate features whose resolution is not equal to the reference resolution, the resolution of each acquired intermediate feature is adjusted to the reference resolution, and the intermediate features with the reference resolution are fused to obtain the second semantic feature.

8. The method according to any one of claims 1 to 7, characterized in that, The second encoder includes an image encoding structure and a feature adaptation structure in a multimodal model; the multimodal model is used to map images and text to the same feature space through feature encoding; The step of encoding semantic features in the output image using a second encoder to obtain a third semantic feature includes: The output image is processed by the image coding structure to obtain the output image features; The feature adaptation structure generates at least one third semantic feature based on the output image features; the resolution of the at least one third semantic feature matches the input feature resolution of at least one processing block in the denoising network of the diffusion model, wherein the processing block is an intermediate block in the denoising network or at least one of the decoding blocks included in the decoding structure of the denoising network.

9. The method according to any one of claims 1 to 8, characterized in that, The acquisition of the initial image includes: Obtain an initial image sequence consisting of multiple initial images; The step of inputting the initial image into the convolutional inpainting model, obtaining the intermediate features generated by the convolutional inpainting model during the pre-inpainting process of the initial image, and obtaining the output image obtained by the convolutional inpainting model after completing the pre-inpainting process includes: The initial image sequence is input into the convolutional inpainting model to obtain the comprehensive inter-frame fusion features generated by the convolutional inpainting model during the pre-inpainting process of the initial image sequence, which are used as the intermediate features; and the output image obtained by the convolutional inpainting model after completing the pre-inpainting process is obtained, wherein the output image corresponds to the initial image located in the middle position in the initial image sequence. The step of encoding semantic features of the initial image using a first encoder to obtain first semantic features includes: The first semantic feature is obtained by encoding the semantic features of the initial image located in the middle position of the initial image sequence using the first encoder.

10. The method according to any one of claims 1 to 8, characterized in that, The process of acquiring the image to be repaired includes: Obtain an initial image sequence consisting of multiple initial images; The step of inputting the initial image into the convolutional inpainting model, obtaining the intermediate features generated by the convolutional inpainting model during the pre-inpainting process of the initial image, and obtaining the output image obtained by the convolutional inpainting model after completing the pre-inpainting process includes: The initial image sequence is input into the convolutional inpainting model to obtain the inter-frame fusion features generated by the convolutional inpainting model for each initial image during the pre-inpainting process of the initial image sequence, forming an intermediate feature sequence; and the output image sequence obtained by the convolutional inpainting model after completing the pre-inpainting process is obtained, the output image sequence including multiple output images, the multiple output images corresponding one-to-one with multiple initial images.

11. The method according to claim 10, characterized in that, The method further includes: The initial image sequence, the intermediate feature sequence, and the output image sequence are resized so that the resized initial image sequence, the resized intermediate feature sequence, and the resized output image sequence are all adapted to the input data size of the diffusion model. The adjusted initial image sequence is used to generate the first semantic feature, the adjusted intermediate feature sequence is used to generate the second semantic feature, and the adjusted output image sequence is used to generate the third semantic feature.

12. The method according to any one of claims 1 to 11, characterized in that, The convolutional repair model is trained in the following way: The ground truth image is degraded to obtain the degraded image corresponding to the ground truth image; The degraded image is input into the convolutional inpainting model to be trained, and the training output image obtained by the convolutional inpainting model after pre-inpainting the degraded image is obtained. The convolutional inpainting model is trained based on the difference between the training output image and the ground truth image.

13. The method according to any one of claims 1 to 12, characterized in that, The second encoder includes an image encoding structure and a feature adaptation structure in a multimodal model; the multimodal model is a pre-trained model used to map images and text to the same feature space through feature encoding; the second encoder is trained in the following manner: Obtain image-text sample pairs and their corresponding annotation results; the image-text sample pairs include training images and training text, and the annotation results indicate the matching relationship between the training images and the training text. The training image is semantically encoded using the second encoder to obtain training image features; the training text is text-encoded using the text encoding structure in the multimodal model to obtain training text features, and the size of the training text features is adjusted so that the size of the adjusted training text features is the same as the size of the training image features. Based on the feature distance between the training image features and the adjusted training text features, and the annotation results, the feature adaptation structure in the second encoder is trained.

14. A model training method, characterized in that, The method includes: Obtain the ground truth image; perform degradation processing on the ground truth image to obtain a degraded image; The degraded image is input into the convolutional restoration model to obtain the convolutional intermediate features generated by the convolutional restoration model during the pre-restoration process of the degraded image, and to obtain the convolutional restored image obtained by the convolutional restoration model after completing the pre-restoration process. In the ground truth image and the convolutional repair image, select the labeled repair image corresponding to the degraded image; project the labeled repair image into the latent space to obtain labeled image features; add labeled noise features to the labeled image features to obtain training noise features; The degraded image is semantically encoded using a first encoder to obtain first training semantic features; Based on the convolutional intermediate features, a second training semantic feature is determined; the second training semantic feature is used to characterize the semantic information extracted by the convolutional restoration model during the pre-restoration process of the degraded image. The convolutional repair image is semantically encoded using a second encoder to obtain a third training semantic feature. Using the diffusion model to be trained, based on the first training semantic features, the second training semantic features, and the third training semantic features, the predicted noise features in the training noise features are determined; The diffusion model is trained based on the difference between the predicted noise features and the labeled noise features.

15. The method according to claim 14, characterized in that, The diffusion model includes a first control network, a second control network, and a denoising network. The denoising network includes an encoding structure composed of multiple encoding blocks, an intermediate block, and a decoding structure composed of multiple decoding blocks. The intermediate block and the decoding block include control sub-blocks for performing feature processing with reference to the first training semantic features, the second training semantic features, and the third training semantic features. The step of training the diffusion model based on the difference between the predicted noise features and the labeled noise features includes: Based on the difference between the predicted noise features and the labeled noise features, the first control network, the second control network, and a partial attention layer in the control sub-block of the diffusion model are trained; the partial attention layer includes a first attention layer that performs attention operations based on the first training semantic features and the second training semantic features, and a second attention layer that performs attention operations based on the output features of the first attention layer and the first training semantic features.

16. An image restoration device, characterized in that, The device includes: The first acquisition module is used to acquire the initial image; The first convolutional inpainting module is used to input the initial image into the convolutional inpainting model to obtain intermediate features generated by the convolutional inpainting model during the pre-inpainting process of the initial image, and to obtain the output image obtained by the convolutional inpainting model after completing the pre-inpainting process. The first feature preprocessing module is used to encode the semantic features of the initial image using a first encoder to obtain a first semantic feature; determine a second semantic feature based on the intermediate features, wherein the second semantic feature is used to characterize the semantic information extracted by the convolutional inpainting model during the prepainting process of the initial image; and encode the semantic features of the output image using a second encoder to obtain a third semantic feature. The first diffusion repair module is used to generate repaired image features based on the first semantic feature, the second semantic feature, and the third semantic feature using a diffusion model; the repaired image features are used to obtain the repaired image through feature decoding.

17. A model training device, characterized in that, The device includes: The second acquisition module is used to acquire the ground truth image; and to perform degradation processing on the ground truth image to obtain a degraded image; The second convolutional inpainting module is used to input the degraded image into the convolutional inpainting model, obtain the convolutional intermediate features generated by the convolutional inpainting model during the pre-inpainting process of the degraded image, and obtain the convolutional inpainted image obtained by the convolutional inpainting model after completing the pre-inpainting process. The annotation module is used to select the annotated restoration image corresponding to the degraded image from the ground truth image and the convolutional restoration image; project the annotated restoration image into the latent space to obtain the annotated image features; and add annotation noise features to the annotated image features to obtain the training noise features. The second feature preprocessing module is used to encode the semantic features of the degraded image using the first encoder to obtain the first training semantic features; determine the second training semantic features based on the convolutional intermediate features; the second training semantic features are used to characterize the semantic information extracted by the convolutional restoration model during the pre-restoration process of the degraded image; and encode the semantic features of the convolutional restored image using the second encoder to obtain the third training semantic features. The noise prediction module is used to determine the predicted noise features in the training noise features based on the first training semantic features, the second training semantic features, and the third training semantic features using the diffusion model to be trained. The model training module is used to train the diffusion model based on the difference between the predicted noise features and the labeled noise features.

18. A computer device, characterized in that, The device includes a processor and a memory; The memory is used to store computer programs; The processor is configured to execute, according to the computer program, the image restoration method of any one of claims 1 to 13, or the model training method of any one of claims 14 to 15.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when executed by an electronic device, implements the image restoration method of any one of claims 1 to 13, or the model training method of any one of claims 14 to 15.

20. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image restoration method according to any one of claims 1 to 13, or the model training method according to any one of claims 14 to 15.

Images