An image inpainting method, device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide an image repairing method and device, electronic equipment and storage medium, and relate to the technical field of image processing. The method comprises: obtaining a target image to be repaired; calling a pre-trained initial repairing network to perform image degeneration removal processing on the target image to obtain a degeneration removal result of the target image; performing repairing optimization on the degeneration removal result according to a predetermined repairing optimization mode to obtain an image repairing result of the target image; the predetermined repairing optimization mode comprises: processing the degeneration removal result based on a diffusion network, and taking a prediction result related to a first task and a prediction result related to a second task as content to be fused, and performing fusion processing through a pre-trained image fusion network; the first task is used for an optimized prediction result, and the second task is used for outputting a prediction result retaining identity information. Through the present solution, image repairing can be performed while improving image quality and retaining identity information.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image restoration method, apparatus, electronic device, and storage medium. Background Technology

[0002] In some scenarios, due to factors such as long-distance shooting and equipment limitations, the acquired images are often characterized by a small proportion of human faces and low overall resolution; in order to improve image quality (such as image clarity), these images need to be restored.

[0003] Currently, image restoration is usually performed using image restoration models. These models can be trained using ground truth images (such as high-quality images) and simulated images corresponding to the ground truth images (e.g., low-quality images obtained by simulating degradation using ground truth images). In other words, the simulated images are used as input to the image restoration model, and the corresponding ground truth images are used for supervision, thereby training the image restoration model.

[0004] However, the degree of anomaly in the image to be repaired and the shooting mode are uncontrollable. This can lead to the image obtained by repairing the image based on the trained image repair model, which may produce image anomalies or the identity information may differ significantly from the identity information in the image to be repaired.

[0005] Therefore, how to balance improving image quality with preserving identity information during image restoration is an urgent problem to be solved. Summary of the Invention

[0006] The purpose of this application is to provide an image restoration method, apparatus, electronic device, and storage medium to simultaneously improve image quality and preserve identity information during image restoration. The specific technical solution is as follows:

[0007] In a first aspect, embodiments of this application provide an image restoration method, the method comprising:

[0008] Obtain the target image to be repaired;

[0009] A pre-trained initial insulation network is invoked to perform image de-degradation processing on the target image, thereby obtaining the de-degradation result of the target image;

[0010] The de-degradation result is repaired and optimized according to the predetermined repair and optimization method to obtain the image restoration result of the target image;

[0011] The predetermined repair and optimization methods include:

[0012] The de-degradation result is processed based on a pre-trained diffusion network used to perform the first and second tasks, and the prediction results for the first task and the prediction results for the second task obtained by the diffusion network when processing the de-degradation result of the target image are used as the content to be fused, and fusion processing is performed by a pre-trained image fusion network.

[0013] The first task is to perform quality optimization on the image input to the diffusion network to output an optimized prediction result, and the second task is to output a prediction result that retains the identity information in the input image on the image input to the diffusion network.

[0014] Secondly, embodiments of this application provide an image restoration apparatus, the apparatus comprising:

[0015] The acquisition module is used to acquire the target image to be repaired;

[0016] The calling module is used to call the pre-trained initial instigation network to perform image de-degradation processing on the target image and obtain the de-degradation result of the target image;

[0017] The repair module is used to repair and optimize the de-degradation result according to a predetermined repair and optimization method to obtain the image repair result of the target image;

[0018] The predetermined repair and optimization methods include:

[0019] The de-degradation result is processed based on a pre-trained diffusion network used to perform the first and second tasks, and the prediction results for the first task and the prediction results for the second task obtained by the diffusion network when processing the de-degradation result of the target image are used as the content to be fused, and fusion processing is performed by a pre-trained image fusion network.

[0020] The first task is to perform quality optimization on the image input to the diffusion network to output an optimized prediction result, and the second task is to output a prediction result that retains the identity information in the input image on the image input to the diffusion network.

[0021] Thirdly, embodiments of this application provide an electronic device, including:

[0022] Memory, used to store computer programs;

[0023] A processor, when executing a program stored in memory, implements any of the described image restoration methods.

[0024] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the image restoration methods described above.

[0025] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the image restoration methods described above.

[0026] Beneficial effects of the embodiments in this application:

[0027] The image restoration method provided in this application, after acquiring the target image to be restored, first performs de-degradation processing on the target image through an initial restoration network; then, through a first task of a diffusion network, it optimizes the image quality of the de-degradation result to output an optimized prediction result for the first task; and then, through a second task of the diffusion network, it outputs a prediction result for the second task that retains identity information based on the de-degradation result; finally, it performs fusion processing on the target image through a pre-trained image fusion network to obtain the image restoration result. As can be seen, the image restoration process in this application first uses an initial restoration network to perform de-degradation processing on the target image for preliminary restoration, then uses a diffusion network to perform multi-task processing (first task and second task) to accommodate different restoration needs, and finally, based on the prediction result of the first task with optimized image quality and the prediction result of the second task with retained identity information, it uses a fusion network to perform fusion processing to obtain the image restoration result, thus completing the restoration process. Therefore, this application can simultaneously improve image quality and retain identity information during image restoration.

[0028] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0030] Figure 1 A schematic flowchart illustrating an image restoration method provided in an embodiment of this application;

[0031] Figure 2 This is a schematic diagram of the structure of each network used in an image restoration method provided in an embodiment of this application;

[0032] Figure 3aA schematic diagram illustrating the process of training the initial repair network as provided in an embodiment of this application;

[0033] Figure 3b This is a schematic diagram of the process for de-degradation processing using an initial repair network, provided in an embodiment of this application.

[0034] Figure 4 A schematic flowchart of the discriminator for the initial repair network provided in the embodiments of this application;

[0035] Figure 5 A schematic diagram of the diffusion network for distillation learning provided in an embodiment of this application;

[0036] Figure 6 A schematic diagram of the performer network of the reinforcement learning network provided in the embodiments of this application;

[0037] Figure 7 A flowchart illustrating the critic network of the reinforcement learning network provided in this application embodiment;

[0038] Figure 8 A schematic diagram illustrating the process of image inpainting using a reinforcement learning network provided in an embodiment of this application;

[0039] Figure 9 A schematic diagram of the training process of the initial repair network provided in the embodiments of this application;

[0040] Figure 10 A schematic diagram of the training process of the diffusion network provided in the embodiments of this application;

[0041] Figure 11 This application provides a quick overview of the training process for the image fusion network provided in the embodiments of this application.

[0042] Figure 12 This is a schematic diagram of the structure of an image restoration device provided in an embodiment of this application;

[0043] Figure 13 This is a block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0045] In related technologies, the degradation requirements (or restoration requirements) of the image to be restored may differ significantly from those of the training data used to train the image restoration model. If a trained image restoration model is used, it can typically maintain the position and structural relationships of facial features and other parts of the person in the image without abnormalities. However, the person's identity information or the content of the image (such as facial expressions, detailed textures, etc.) may be modified, meaning that the identity information of the person in the restored image differs significantly from that in the image to be restored. In short, maintaining the normality of the image and preserving its identity information present a contradiction during image restoration.

[0046] Based on this, embodiments of this application provide an image restoration method, apparatus, electronic device, and storage medium to simultaneously improve image quality and preserve identity information during image restoration.

[0047] The following section first introduces an image restoration method provided by an embodiment of this application.

[0048] The image restoration method provided in this application can be applied to electronic devices, such as mobile phones and computers; this application does not limit the specific form of the electronic device. Furthermore, the electronic device can invoke pre-trained initial restoration network, diffusion network, and image fusion network. For example, these networks can be deployed locally on the electronic device or on other devices communicating with the electronic device to achieve image restoration. Moreover, the electronic devices used in the training process of the initial restoration network, diffusion network, and image fusion network can be the same as or different from the electronic devices used in the image restoration process; this application does not limit this.

[0049] Furthermore, the image restoration method provided in this application can be applied to image restoration scenarios, such as: for images where the proportion of a person's face is small, the overall resolution is low, or the clarity is low, in order to improve the image quality, this application does not limit the application to such scenarios. The image restoration involved in this application is not limited to the restoration of images containing human objects (also referred to as human targets). For example, it is also reasonable to pre-train various networks using training data on animal objects, and then use these networks to perform image restoration on animal objects.

[0050] An image restoration method provided in this application includes:

[0051] Obtain the target image to be repaired;

[0052] The pre-trained initial instigation network is invoked to perform image de-degradation processing on the target image, and the de-degradation result of the target image is obtained.

[0053] The de-degradation result is repaired and optimized according to the predetermined repair and optimization method to obtain the image restoration result of the target image;

[0054] The planned repair and optimization methods include:

[0055] The de-degradation results are processed based on a pre-trained diffusion network used to perform the first and second tasks. The prediction results for the first task and the prediction results for the second task obtained by the diffusion network when processing the de-degradation results of the target image are used as the content to be fused, and fusion processing is performed by a pre-trained image fusion network.

[0056] The first task is to optimize the quality of the image input to the diffusion network to output an optimized prediction result. The second task is to output a prediction result that retains the identity information in the input image, based on the image input to the diffusion network.

[0057] The image restoration method provided in this application, after acquiring the target image to be restored, first performs de-degradation processing on the target image through an initial restoration network; then, through a first task of a diffusion network, it optimizes the image quality of the de-degradation result to output an optimized prediction result for the first task; and then, through a second task of the diffusion network, it outputs a prediction result for the second task that retains identity information based on the de-degradation result; finally, it performs fusion processing on the target image through a pre-trained image fusion network to obtain the image restoration result. As can be seen, the image restoration process in this application first uses an initial restoration network to perform de-degradation processing on the target image for preliminary restoration, then uses a diffusion network to perform multi-task processing (first task and second task) to accommodate different restoration needs, and finally, based on the prediction result of the first task with optimized image quality and the prediction result of the second task with retained identity information, it uses a fusion network to perform fusion processing to obtain the image restoration result, thus completing the restoration process. Therefore, this application can simultaneously improve image quality and retain identity information during image restoration.

[0058] The following description, in conjunction with the accompanying drawings, provides an exemplary method for image restoration according to an embodiment of this application.

[0059] like Figure 1 As shown in the embodiments of this application, an image restoration method may include the following steps:

[0060] S101: Obtain the target image to be repaired;

[0061] When performing image restoration, the target image to be restored can be obtained first. The target image can be understood as the image that requires restoration. It can be a real image or a simulated image (an image generated through algorithms and data transformations rather than actual physical acquisition). Furthermore, the target image can be a low-quality image with a small proportion of the face, low overall resolution, and / or low sharpness. A low-quality image can be understood as one whose quality score is below a predetermined threshold. For example, by scoring the image based on dimensions such as the proportion of the face, resolution, and sharpness, if the image score is below the first threshold (e.g., 40), it is considered a low-quality image; and if the image score is above the second threshold (e.g., 70), it is considered a high-quality image.

[0062] For example, in one implementation, the target image can be acquired by means of a camera, an image database, etc., and the image database is a database that at least contains the image to be repaired (it may also contain high-quality images with high clarity and high resolution, etc.), and the images acquired by the camera at least contain images that require image repair (such as low-resolution images, etc.). It should be noted that this application does not specifically limit the method of acquiring the target image to be repaired.

[0063] S102: Call the pre-trained initial insulation network to perform image de-degradation processing on the target image and obtain the de-degradation result of the target image;

[0064] After obtaining the target image, this application first calls a pre-trained initial restoration network to perform de-degradation processing on the target image, obtaining the de-degradation result of the target image. The initial restoration network can be trained using ground truth images and their corresponding simulated images. The process of the initial restoration network performing image de-degradation processing on the target image can be understood as a process of completing the structure of the target image. Furthermore, the initial restoration network used for de-degradation processing can also be referred to as a de-degradation network or a de-degradation module.

[0065] For example, the de-degradation result of the target image can be a de-degradation image of the target image. The de-degradation image can be further processed to obtain an image restoration result that balances improving image quality and preserving identity information.

[0066] For example, in one implementation, the initial inpainting network is trained based on a first sample image and a ground truth image of the first sample image, wherein the ground truth image of any sample image is: the image that represents the expected inpainting result of that sample image;

[0067] The first sample image is the image obtained after image degradation processing of a predetermined initial image. The ground truth image of the first sample image is the image obtained by low-pass filtering the image spectrum of the initial image according to the target frequency domain threshold set for the first sample image.

[0068] Accordingly, a pre-trained initial instigation network is invoked to perform image de-degradation processing on the target image, resulting in the de-degradation result of the target image, including:

[0069] Based on a specified frequency domain threshold, the initial repair network is invoked to perform image de-degradation processing on the target image based on the specified frequency domain threshold, and the de-degradation result of the target image is obtained.

[0070] The initial inpainting network can be trained based on the first sample image and its ground truth image. The ground truth image of any sample image can be used as the expected inpainting result for that sample image. Furthermore, since it is difficult to obtain the real low-quality image (the image to be inpainted) and the corresponding high-quality image (the image of the expected inpainting result for the image to be inpainted), the first sample image can be an image obtained by simulating degradation through a high-quality initial image. For any image, the simulated degradation process can be performed using the following formula: ;in, To simulate the first sample image after degradation, The initial image for simulated degradation processing is denoted by ⨂k, which represents the blur kernel convolution of the image, and ↓_s represents downsampling by a factor of s. σ represents Gaussian noise with a standard deviation of σ, and jpeg-q represents JPEG compression with a compression ratio of q.

[0071] Furthermore, the ground truth image of the first sample image can be the image obtained by low-pass filtering the image spectrum of the initial image according to the target frequency domain threshold set for the first sample image. That is, the initial image (which can be understood as a high-quality image, such as an image with a resolution greater than a certain threshold) can be used to obtain the first sample image through the above-mentioned simulated degradation processing. This application also sets a frequency domain threshold to limit the level of detail of the image. The image obtained after limiting the level of detail of the initial image through the target frequency domain threshold is used as the ground truth image of the first sample image. This can avoid the situation where the initial inpainting network trained directly uses the initial image as the ground truth image of the first sample image, resulting in image anomalies or significant differences in identity information when performing image inpainting.

[0072] For example, the ground truth image of the first sample image can be determined according to the following formula: Where th represents the target frequency domain threshold, This is the ground truth image of the first sample image, and lowpass represents the low-pass filtering process.

[0073] The ground truth image obtained by low-pass filtering the image spectrum of the initial image using a target frequency domain threshold is an image with finer restrictions after being applied by the target frequency domain threshold. The image quality of this ground truth image is higher than that of the first sample image. Furthermore, the identity information of this ground truth image is consistent with that of the first sample image. Thus, the trained initial inpainting network can perform de-degradation processing on the target image using a specified frequency domain threshold (the specified frequency domain threshold can be set according to the actual situation). The de-degradation result of the target image is a de-degradation result that balances reducing image anomalies and preserving identity information.

[0074] It is important to emphasize that each initial image can have a target frequency domain threshold set. The target frequency domain thresholds for multiple initial images can have different values. The initial inpainting network is trained using each first sample image and its ground truth image. The trained initial inpainting network can perceive the level of detail required for any frequency domain threshold and perform de-degradation processing according to the perceived level of detail. Furthermore, for an initial image, the target frequency domain threshold used in the generated ground truth image is the same frequency domain threshold used during model training based on that initial image. The terms "target frequency domain threshold" and "specified frequency domain threshold" are merely used to distinguish between the frequency domain thresholds used during model training and those used during inference, and do not have any limiting meaning. The frequency domain thresholds that the initial inpainting network can perceive can be normalized frequency domain thresholds, meaning the range of the frequency domain threshold values ​​is (0,1).

[0075] Furthermore, the training method for the initial repair network will be described in detail in subsequent embodiments, and will not be repeated here.

[0076] S103: According to the predetermined repair and optimization method, the de-degradation result is repaired and optimized to obtain the image repair result of the target image;

[0077] The planned repair and optimization methods include:

[0078] The de-degradation results are processed based on a pre-trained diffusion network used to perform the first and second tasks. The prediction results for the first task and the prediction results for the second task obtained by the diffusion network when processing the de-degradation results of the target image are used as the content to be fused, and fusion processing is performed by a pre-trained image fusion network.

[0079] The first task is to optimize the quality of the image input to the diffusion network to output an optimized prediction result. The second task is to output a prediction result that retains the identity information in the input image, based on the image input to the diffusion network.

[0080] To further improve image quality and preserve identity information during image restoration, after obtaining the de-degradation result, a pre-trained diffusion network can be used to process the de-degradation result to obtain a quality-optimized prediction result for the first task and a prediction result for the second task that preserves identity information. These results are then fused using a pre-trained image fusion network to obtain the image restoration result of the target image.

[0081] The diffusion network in this application can output quality-optimized prediction results (i.e., prediction results focusing on improving image quality) or prediction results that retain identity information (i.e., prediction results focusing on retaining identity information) for different input content. The specific construction of the input content and the training method will be described in detail in subsequent embodiments and will not be repeated here. When calling the diffusion network, input content for the first task and input content for the second task can be constructed separately and input into the diffusion network sequentially to obtain prediction results for the first task and the second task, respectively; or, input content for the first task and input content for the second task can be input into the diffusion network simultaneously to obtain prediction results for the first task and the second task, respectively.

[0082] For example, in one implementation, the de-degradation result is repaired and optimized according to a predetermined repair and optimization method to obtain the image restoration result of the target image, including:

[0083] Taking the first time step as the current time step, based on the intermediate results of the current time step and the de-degradation results of the target image, the input content of the diffusion network for the current time step regarding the first task and the input content regarding the second task are constructed. The constructed input content is input into the diffusion network to obtain the prediction results of the first task and the second task for the current time step.

[0084] Based on the prediction results of the first task and the second task at the current time step, the image fusion network is invoked so that the image fusion network performs result fusion processing on the prediction results of the first task and the second task at the current time step to obtain the fusion result at the current time step.

[0085] If the current time step is the last time step, the restored image of the current time step is decoded based on the fusion result of the current time step, and the restored image of the current time step is used as the image restoration result of the target image; otherwise, based on the fusion result of the current time step, the intermediate result of the next time step is calculated, and the next time step is used as the new current time step. The prediction results of the first task and the prediction results of the second task based on the current time step are returned, and the steps of the image fusion network are invoked.

[0086] Since the diffusion network includes forward denoising and backward denoising processes, and these processes involve multiple time steps (e.g., denoising from t=0 to t=max, and denoising from t=max to t=0), when repairing and optimizing the de-degradation result, the pure noise is first initialized, i.e., the de-degradation result at t=max is the result with pure noise added. Then, the first time step (t=max) is used as the current time step. Based on the intermediate results of the current time step and the de-degradation result of the target image, the input content of the diffusion network at the current time step regarding the first task and the input content regarding the second task are constructed. The input content of the tasks is fed into the diffusion network to obtain the prediction results of the first task and the second task at the current time step. Then, the image fusion network can be invoked to fuse the prediction results of the first and second tasks at the current time step to obtain the fusion result. If the current time step is the last time step, the fusion result can be decoded into the restored image at the current time step to obtain the image restoration result of the target image. For example, this can be done using a decoder VAEDEC (a decoder for Variational Autoencoder, VAE). For the diffusion network, the intermediate result at the current time step can be the diffusion intermediate result, and the intermediate result is initialized with pure noise; that is, the intermediate result at the time step t=max is an intermediate result with added pure noise.

[0087] If the current time step is the last time step, it means that the diffusion network has completed denoising, and the repaired image obtained at the current time step can be used as the image repair result of the target image; otherwise, it means that it needs to continue iterating to the next time step. Then, based on the fusion result of the current time step, the intermediate result of the next time step is calculated, and the next time step is used as the new current time step. Based on the intermediate result of the new current time step, the diffusion network and the image fusion network are called respectively.

[0088] For example, the intermediate result of the next time step after the current time step can be calculated using the following formula: ;

[0089] in, This is an intermediate result for the current time step. This is the fusion result at the current time step. This is an intermediate result for the next time step. The time parameter represents the next time step. The time parameter represented by the current time step. This can be understood as the time difference between two adjacent time steps.

[0090] It should be noted that the intermediate result at each time step is the denoised data state corresponding to that step, which achieves a smooth transition and accurate restoration of the data distribution through progressive noise control. Furthermore, the network parameters of the diffusion network can be understood as variables that can automatically adjust their values ​​based on training data to fit the mapping relationship between the input content and the prediction result; they are the core carriers of the diffusion network's "learning" ability.

[0091] In the technical solution of this application, all operations such as acquisition, storage, use, processing, transmission, provision and disclosure of target images, input content, repaired images, image repair results, any sample images, ground truth images, and initial images are carried out with the user's authorization.

[0092] It should be noted that the facial description information, facial features, identity information, etc. in this embodiment are not specific to any particular user and do not reflect the personal information of any particular user.

[0093] The image restoration method provided in this application, after acquiring the target image to be restored, first performs de-degradation processing on the target image through an initial restoration network; then, through a first task of a diffusion network, it optimizes the image quality of the de-degradation result to output an optimized prediction result for the first task; and then, through a second task of the diffusion network, it outputs a prediction result for the second task that retains identity information based on the de-degradation result; finally, it performs fusion processing on the target image through a pre-trained image fusion network to obtain the image restoration result. As can be seen, the image restoration process in this application first uses an initial restoration network to perform de-degradation processing on the target image for preliminary restoration, then uses a diffusion network to perform multi-task processing (first task and second task) to accommodate different restoration needs, and finally, based on the prediction result of the first task with optimized image quality and the prediction result of the second task with retained identity information, it uses a fusion network to perform fusion processing to obtain the image restoration result, thus completing the restoration process. Therefore, this application can simultaneously improve image quality and retain identity information during image restoration.

[0094] Alternatively, in another embodiment of this application, such as Figure 9As shown, the initial training methods for the repair network include:

[0095] A1: Input the vector features of the first sample image and the target frequency domain threshold into the initial repair network, so that the initial repair network uses the vector features of the target frequency domain threshold as the first reference prompt to perform image de-degradation processing on the first sample image, and obtain the de-degradation result of the first sample image;

[0096] A2: Based on the de-degradation result of the first sample image and the ground truth image of the first sample image, calculate the loss of the initial instigation network;

[0097] A3: In response to the loss characterization of the initial repair network indicating that the initial repair network has not converged, the network parameters of the initial repair network are adjusted;

[0098] Accordingly, based on a specified frequency domain threshold, the initial repair network is invoked to perform image de-degradation processing on the target image based on the specified frequency domain threshold, obtaining the de-degradation result of the target image, including:

[0099] The target image and the vector features of a specified frequency domain threshold are input into the initial inpainting network, so that the initial inpainting network uses the vector features of the specified frequency domain threshold as the first reference prompt to perform image de-degradation processing on the target image, and obtain the de-degradation result of the target image.

[0100] like Figure 3a As shown, during the training of the initial restoration network, for each first sample image, the vector features of the first sample image and the target frequency domain threshold (which can be obtained by encoding the target frequency domain threshold) can be input into the initial restoration network. The initial restoration network uses the vector features of the target frequency domain threshold as the first reference cue to perform image de-degradation processing on the first sample image, obtaining the de-degradation result of the first sample image. Furthermore, using the ground truth image of the first sample image obtained by low-pass filtering the initial image through the target frequency domain threshold, and the de-degradation result of the first sample image, the loss of the initial restoration network is calculated. If the calculated loss of the initial restoration network indicates that the initial restoration network has not converged, such as when the loss is greater than a certain threshold, it means that the initial restoration network needs to continue training. The network parameters of the initial restoration network can be adjusted, and the vector features of the first sample image and the target frequency domain threshold can be re-input into the initial restoration network.

[0101] It should be noted that each first sample image and the target frequency domain threshold set for that first sample image can be input into the initial restoration network in the same way to train the initial restoration network through each first sample image. Furthermore, the initial restoration network can learn the level of detail represented by each frequency domain threshold and perform de-degradation processing on the first sample image.

[0102] Correspondingly, such as Figure 3b As shown, during the image restoration process, when performing de-degradation processing through the trained initial restoration network, the target image and the vector features of a specified frequency domain threshold can be input into the initial restoration network. The initial restoration network can use the vector features of the specified frequency domain threshold as the first reference prompt to perform de-degradation processing on the target image and obtain the de-degradation result of the target image. That is, when the initial restoration network performs de-degradation processing on the target image, it limits the fineness of the de-degradation result by specifying the frequency domain threshold.

[0103] As can be seen, when training the initial restoration network, the initial restoration network can use the vector features of the target frequency domain threshold as a first reference cue to limit the fineness of the de-degradation result obtained by performing de-degradation processing on the first sample image. In this way, when the target image is de-degraded by the trained initial restoration network, the fineness of the de-degradation result of the target image can be limited according to the specified frequency domain threshold. That is, the de-degradation result of the target image is a de-degradation result with fineness limited according to the specified frequency domain threshold, which takes into account both reducing image anomalies and preserving identity information.

[0104] For example, in one implementation, the initial repair network includes a generator and a discriminator;

[0105] The vector features of the first sample image and the target frequency domain threshold are input into the initial inpainting network. This allows the initial inpainting network to use the vector features of the target frequency domain threshold as a first reference cue to perform image de-degradation processing on the first sample image, resulting in the de-degradation result of the first sample image, including:

[0106] The vectors of the first sample image and the target frequency domain threshold are input into the generator in the initial repair network, so that the generator in the initial repair network uses the vector features of the target frequency domain threshold as the first reference prompt to perform image de-degradation processing on the first sample image, and obtain the de-degradation result of the first sample image.

[0107] The initial training methods for the repair network also include:

[0108] Based on the de-degradation result of the first sample image and the ground truth image of the first sample image, the input content of the discriminator in the initial repair network is constructed;

[0109] The input content of the constructed discriminator is input into the discriminator so that the discriminator can perform discrimination processing on the received input content and obtain the discrimination result of the first sample image;

[0110] Based on the de-degradation result of the first sample image and the ground truth image of the first sample image, the loss of the initial inpainting network is calculated, including:

[0111] Based on the difference between the de-degradation result of the first sample image and the ground truth image of the first sample image, the loss is calculated to obtain the first type of loss;

[0112] Based on the discrimination results of the first sample image, the loss is calculated to obtain the second type of loss;

[0113] The loss of the initial repair network is determined based on the first and second type losses of the initial repair network.

[0114] The initial repair network in this application may include a generator and a discriminator. The generator can be understood as a generator with frequency domain constraints (i.e., frequency domain thresholds), and the discriminator is a frequency domain-aware discriminator for the facial region, which focuses on the frequency domain features of the facial region to ensure the authenticity of the de-degradation results generated by the generator.

[0115] The generator is used to generate the de-degradation result, i.e., the vector of the first sample image and the target frequency domain threshold. It can be input into the generator of the initial inpainting network. The generator uses the vector features of the target frequency domain threshold as a first reference cue to perform de-degradation processing on the first sample image, obtaining the de-degradation result of the first sample image. For example, the functionality that the generator can achieve is represented by this formula: ;in, The de-degradation result for the first sample image. For generator, This refers to the vector features obtained after encoding the target frequency domain threshold.

[0116] The discriminator is used to play against the generator, driving the generator to output realistic de-degradation results. Specifically, during the training of the initial repair network, the input content of the discriminator can be constructed based on the de-degradation result of the first sample image and the ground truth image of the first sample image, and then input into the discriminator. The discriminator can perform discrimination processing on the input content to obtain the discrimination result of the first sample image. This discrimination result can characterize the authenticity of the de-degradation result generated by the generator, for example, it can include a score of authenticity of the de-degradation result generated by the generator, etc.

[0117] Furthermore, when calculating the loss of the initial repair network, the loss can be calculated based on the difference between the de-degradation result of the first sample image and the ground truth image to obtain the first type of loss; and the loss can be calculated based on the discrimination result of the first sample image to obtain the second type of loss (the second type of loss can be understood as the loss of the discriminator); and the loss of the initial repair network can be determined based on the first type of loss and the second type of loss, for example: directly adding the first type of loss and the second type of loss to obtain the loss of the initial repair network, or fusing the first type of loss and the second type of loss according to the pre-set weight coefficients to obtain the loss of the initial repair network, etc.

[0118] It should be noted that the first type of loss may include: the mean squared error loss between the de-degradation result of the first sample image and the ground truth image, and / or, perceptual loss; for example, the first type of loss = ;in, lpips represents the mean squared error loss between the de-degradation result of the first sample image and the ground truth image, and represents the perceptual loss.

[0119] The initial inpainting network in this application may include a generator and a discriminator. The generator generates de-degradation results, and the discriminator acts adversarially against the generator, driving it to output realistic de-degradation results. Furthermore, during the training of the initial inpainting network, loss calculation can be performed not only based on the difference between the de-degradation results of the first sample image and the ground truth image, but also by combining the discrimination results of the first sample image output by the discriminator. Thus, the trained initial inpainting network can not only constrain the fineness of the de-degradation results by combining frequency domain thresholds, but also constrain the generator through the discriminator, further ensuring that realistic de-degradation results can be generated.

[0120] For example, in one implementation, the input content of the discriminator in the initial instigation network is constructed based on the de-degradation result of the first sample image and the ground truth image of the first sample image, including:

[0121] Based on the learnable parameters set for the discriminator, the target frequency domain threshold is processed to obtain the prior features corresponding to the target frequency domain threshold.

[0122] The prior features corresponding to the target frequency domain threshold and the facial region features of the de-degradation result of the first sample image are concatenated to obtain the first concatenated feature corresponding to the first sample image. The prior features corresponding to the target frequency domain threshold and the facial region features of the ground image of the first sample image are concatenated to obtain the second concatenated feature corresponding to the first sample image.

[0123] The input content of the constructed discriminator is input into the discriminator so that the discriminator performs discrimination processing on the received input content to obtain the discrimination result of the first sample image, including:

[0124] The first stitching feature corresponding to the first sample image is input into the discriminator so that the discriminator performs discrimination processing on the received first stitching feature to obtain the discrimination result of the first sample image with respect to the first stitching feature;

[0125] The second stitching feature of the first sample image is input into the discriminator so that the discriminator can perform discrimination processing on the received second stitching feature to obtain the discrimination result of the first sample image with respect to the second stitching feature;

[0126] Based on the discrimination results of the first sample image, loss is calculated to obtain the second type of loss, including:

[0127] Based on the discrimination results of the first sample image regarding the first stitching feature and the discrimination results of the first sample image regarding the second stitching feature, loss is calculated to obtain the second type of loss.

[0128] It should be noted that the discriminator can contain two branches: an encoding branch and a face branch; the input to the encoding branch can be the prior features corresponding to the target frequency domain threshold; the input to the face branch is the face region features of the de-degradation result of the first sample image or the face region features of the ground truth image of the first sample image.

[0129] When constructing the input content for the discriminator, the target frequency domain threshold can first be processed according to the learnable parameters set for the discriminator to obtain the prior features corresponding to the target frequency domain threshold. These prior features can belong to the input content of the discriminator's coding branch; for example, such as Figure 4 As shown, the learnable parameters set for the discriminator can be a set of learnable parameter tokens with a size of [T, N], where T is the number of learnable parameters, and each learnable parameter has a dimension of N, where N is a hyperparameter. For the target frequency domain threshold, the vector feature emb(th) of the target frequency domain threshold is projected onto the space of T and then subjected to softmax processing (an operation that makes the sum of all elements in the group equal to 1 after the operation). This softmax processing is used as the weight of each learnable parameter, and the summation of each learnable parameter is performed to obtain the prior features corresponding to the target frequency domain threshold. Specifically, this can be represented by the following formula: ;in, The prior features corresponding to the target frequency domain threshold. Represents linear operations (i.e., projection). Let be the i-th learnable parameter.

[0130] Next, the prior features corresponding to the target frequency domain threshold and the facial region features of the de-degradation result of the first sample image are sequentially concatenated to obtain the first concatenated feature; and the prior features corresponding to the target frequency domain threshold and the facial region features of the ground truth image of the first sample image are sequentially concatenated to obtain the second concatenated feature. Both the first and second concatenated features can be considered as input content to the discriminator; wherein, Figure 4The diagram only shows the concatenation and discrimination process of prior features and facial region features. The facial region features of the de-degradation result of the first sample image and the facial region features of the ground truth image of the first sample image can both be concatenated with the prior features corresponding to the target frequency domain threshold in a similar manner to obtain concatenated features. Either the first concatenated feature or the second concatenated feature contains the prior features and the corresponding facial region features. After any concatenated feature is input into the discriminator, the corresponding discrimination result can be obtained. That is, by performing discrimination processing on the first concatenated feature by the discriminator, the discrimination result of the first concatenated feature is obtained. And by performing discrimination processing on the second concatenated feature by the discriminator, the discrimination result of the second concatenated feature is obtained.

[0131] Among them, the facial region features of either the de-degradation result of the first sample image or the facial region features of the ground truth image of the first sample image can be extracted using a facial recognition model. The extracted features are then expanded into one-dimensional features. The specific implementation process can be represented by the following formula:

[0132] ;

[0133] in, The facial region features are either the de-degradation result of the first sample image or the facial region features of the ground truth image of the first sample image. The facial region is either the de-degradation result of the first sample image or the facial region of the ground truth image of the first sample image; A facial recognition model is used to extract facial region features by inputting any facial region into the model. Represents positional embedding; flat is short for flatten operation, used to convert a multidimensional tensor into a one-dimensional vector.

[0134] When performing discrimination processing by a discriminator, a first splicing feature can be input into the discriminator, which performs discrimination processing to obtain a discrimination result of the first sample image regarding the first splicing feature; and a second splicing feature can be input into the discriminator, which performs discrimination processing to obtain a discrimination result of the first sample image regarding the second splicing feature. Specifically, the first and second splicing features can be input into the discriminator separately for discrimination processing, yielding discrimination results regarding the first and second splicing features respectively. Figure 4 This only shows the process of obtaining a certain splicing feature through sequence splicing and then judging it using a discriminator.

[0135] Then, the second type of loss can be calculated based on the discrimination results of the first splicing feature and the second splicing feature.

[0136] In this application, the prior features corresponding to the target frequency domain threshold can be determined first. Then, the facial region features of the de-degradation result of the first sample image obtained by the generator are concatenated with the prior features, and the facial region features of the ground truth image of the first sample image are concatenated with the prior features. These concatenations are then judged by a discriminator. After that, the second loss is calculated based on the judgment results of the two concatenated features. It can be seen that the second loss is calculated by combining the target frequency domain threshold, the de-degradation result, and the ground truth image. It can combine the prior features corresponding to the target frequency domain threshold to more accurately describe the difference between the de-degradation result of the first sample image obtained by the generator and the ground truth image, thereby ensuring the accuracy of the trained initial repair network.

[0137] For example, in one implementation, a loss is calculated based on the discrimination result of the first sample image regarding the first stitching feature and the discrimination result of the first sample image regarding the second stitching feature to obtain a second type of loss, including:

[0138] From the discrimination results of the first splicing features of the first sample image, extract the result content of the prior features corresponding to the target frequency domain threshold, and score the first discrimination score of the first sample image based on the extracted result content.

[0139] From the discrimination results of the first sample image regarding the second stitching features, extract the result content of the prior features corresponding to the target frequency domain threshold, and score based on the extracted result content to obtain the second discrimination score of the first sample image;

[0140] Based on the first discriminant score of the first sample image, determine the adversarial loss;

[0141] The discriminator loss is determined based on the first and second discriminant scores of the first sample image.

[0142] like Figure 4 As shown, the discrimination result for the first or second splicing feature includes a discrimination score for the prior feature corresponding to the target frequency domain threshold (which can be understood as the result corresponding to the coding branch of the discriminator), and other results for the facial region features (which can be understood as the result corresponding to the facial branch of the discriminator). For example, a first discrimination score can be extracted from the discrimination result of the first splicing feature; and a second discrimination score can be extracted from the discrimination result of the second splicing feature, i.e., the discrimination result of the discriminator is split and recombined to obtain the discrimination score, while other results in the discrimination result are not used. The process of obtaining the first or second discrimination score can be represented by the following formula: ;

[0143] in, It can be either the first or second discrimination score; It is a non-linear activation function; For discriminator; cat represents concatenation. The first or second splicing feature obtained by characterizing the sequence splicing; The number of features in the prior features.

[0144] Then, the adversarial loss can be determined based on the first discriminant score; and the discriminator loss can be determined based on the first discriminant score and the second discriminant score; for example, the adversarial loss and the discriminator loss can be determined according to the following formula:

[0145] ;

[0146] ;

[0147] in, To combat the losses; For discriminator loss; The first discrimination score; This is the second discrimination score.

[0148] Furthermore, the loss of the initial repair network can be the fusion of the first type of loss and the second type of loss, which can be directly added together or fused according to a pre-set weight coefficient; for example, in one implementation, the loss of the initial repair network can be:

[0149] ;

[0150] in, This represents the losses incurred during the initial network repair.

[0151] It should be emphasized that the formulas involved in the training process of the initial repair network described above are merely examples and should not be construed as limiting the scope of this application.

[0152] As can be seen, the loss of the initial restoration network in this application, which combines mean squared error loss, perceptual loss, adversarial loss and discriminator loss, can more comprehensively describe the de-degradation capability of the initial restoration network, thereby enabling the training of an initial restoration network with higher accuracy that can both improve image quality and preserve identity information.

[0153] In addition, the initial restoration network in this application includes a frequency-domain-limited generator and a frequency-domain-aware discriminator for facial regions. The generator degrades the image's detail by limiting the frequency domain threshold, thereby effectively reducing the generation of illusions and anomalies. The discriminator can improve the model's perception of facial-specific information in different frequency domains. During the training process of the initial restoration network, identity filtering by the frequency domain threshold limits the predictive component in the identity information generation of the second task, reducing identity changes caused by identity information prediction.

[0154] Alternatively, in another embodiment of this application, such as Figure 10 As shown, the training methods for diffusion networks include:

[0155] B1: The second sample image is processed using the trained initial repair network to obtain the de-degradation result of the second sample image;

[0156] B2: According to the predetermined processing method, the de-degradation result of the second sample image is processed by a diffusion network to predict the first task and the second task, so as to obtain the prediction results of the first task and the prediction results of the second task at each time step.

[0157] B3: Based on the prediction results for the first task at each time step and the ground truth image of the second sample image, calculate the loss of the diffusion network for the first task; and based on the prediction results for the second task at each time step and the ground truth image of the second sample image, calculate the loss of the diffusion network for the second task.

[0158] B4: Based on the loss of the diffusion network with respect to the first task and the loss with respect to the second task, determine whether the diffusion network has converged. If it has not converged, adjust the network parameters of the diffusion network.

[0159] The predetermined processing method includes: for each time step, based on the intermediate results of that time step, the learnable parameters set for the diffusion network, the de-degradation results of the second sample image, and the set second reference cue, constructing the input content required for the first task at that time step, and inputting the constructed input content into the diffusion network to perform the first task;

[0160] Based on the intermediate results of this time step, the learnable parameters set for the diffusion network, the depth map of the de-degradation result of the second sample image, and the identity features and facial description information of the objects in the de-degradation result of the second sample image, the input content required for the second task at this time step is constructed, and the constructed input content is input into the diffusion network to perform the second task.

[0161] In this application, the initial repair network, the diffusion network, and the image fusion network can be trained sequentially. That is, after the initial repair network is trained, the diffusion network can be trained using the trained initial repair network, and after the initial repair network and the diffusion network are trained, the image fusion network can be trained using the trained initial repair network and the diffusion network. For the diffusion network, the second sample image is processed through the trained initial repair network to obtain the de-degradation result of the second sample image. Then, according to a predetermined processing method, the diffusion network performs prediction processing on the de-degradation result of the second sample image for the first and second tasks, obtaining the prediction results for the first and second tasks at each time step. Based on the ground truth image of the second sample image and the prediction results for the first and second tasks at each time step, the loss of the diffusion model with respect to the first task and the loss with respect to the second task are calculated respectively. The convergence of the diffusion network can be determined by the losses with respect to the first and second tasks (e.g., if the loss with respect to the first task is less than the set loss threshold for the first task, and the loss with respect to the second task is less than the set loss threshold for the second task, then the diffusion network has converged; otherwise, the diffusion network has not converged). If it has not converged, the network parameters of the diffusion network are adjusted, and the diffusion network is iteratively trained again.

[0162] It is important to emphasize that the second sample image and the first sample image can be the same or different. The first sample image is used to train the initial repair network, and the second sample image is used to train the diffusion network. The first, second, and subsequent third images are only used to distinguish the images used to train different networks and do not have any limiting significance.

[0163] The pre-defined processing method can be understood as follows: For the first task, the input content required for each time step of the first task is constructed and fed into the diffusion network to execute the first task; for the second task, the input content required for each time step of the second task is constructed and fed into the diffusion network to execute the second task. The input content required for the first task at any time step may include: the intermediate result of that time step, the learnable parameters set for the diffusion network, the de-degradation result of the second sample image, and the second reference cue (the second reference cue may be: the frequency domain threshold used by the trained initial inpainting network to de-degrade the second sample image).

[0164] For example, the process of a diffusion network performing the first task at any given time step can be represented by the following formula: ;

[0165] in, This is the prediction result for the first task at this time step; This is an intermediate result for this time step; The de-degradation result for the second sample image; Let be the expansion length of the intermediate results at any time step. Used to characterize multidimensional arrays (i.e., DIT) m2 The output content is sliced; rec represents the rearrangement of one-dimensional features according to the order of the original feature map; The first part is a diffusion network; the second reference cue is used to help the diffusion network understand the level of detail in the initial repair network's de-degradation of the second sample image, but it is not shown here. A specific set of learnable parameters is used as input to the diffusion model to learn prior content about the face, which helps the diffusion network perform distillation learning to optimize the facial region; the initial... It can be set randomly and can be updated during the training of the diffusion model. It can be understood as an implicit prior parameter.

[0166] The input required for the second task at any time step may include: the intermediate results of that time step, the learnable parameters set for the diffusion network, the depth map of the de-degradation result of the second sample image, and the identity features of the objects in the de-degradation result of the second sample image (extracted by a facial recognition model from the objects in the de-degradation result of the second sample image) and facial description information (such as facial attributes, which may include: facial pose, whether a mask is worn, items such as eyes, whether the eye area is occluded, etc.). The identity features and facial description information can be extracted from the de-degradation result of the second sample image, and the specific extraction method can be similar to that of existing technologies, which will not be elaborated here.

[0167] For example, the process of the diffusion network performing the second task at any time step can be represented by the following formula: ;

[0168] in, This is the prediction result for the second task at this time step; This is the depth map of the de-degradation result of the second sample image. Facial description information of the object in the de-degradation result of the second sample image; The identity features and facial description information (including expressions, facial feature visibility, etc.) of the object in the de-degradation result of the second sample image are encoded from text. For example, if a person is smiling and their right eye is obscured, the encoded text would be (to be converted to English): "This person is smiling, their right eye is not visible, and their identity feature is [ID]". The [ID] position is directly filled by the identity feature ID. Furthermore, all text used as input in this application can be converted to English before being input.

[0169] The diffusion network in this application only outputs corresponding prediction results when the input content is different. That is, if the input content is the input content of the first task, the output is the prediction result of the first task (i.e., the prediction result after quality optimization); if the input content is the input content of the second task, the output is the prediction result of the second task (i.e., the prediction result that retains the person's identity information). The specific diffusion process of the diffusion network is not concerned.

[0170] As can be seen, the diffusion network in this application can achieve both quality optimization and identity preservation. Specifically, it can output different prediction results through different input content. During the training of the diffusion model, different input content and different prediction results can be constructed to calculate the loss. Furthermore, if the diffusion network does not converge, the network parameters of the diffusion network can be adjusted so that the diffusion model's ability to optimize quality and preserve identity meets the requirements, thereby balancing the improvement of image quality and the preservation of identity information.

[0171] For example, in one implementation, the loss of the diffusion network with respect to the first task is calculated based on the prediction results for the first task at each time step and the ground truth image of the second sample image, including:

[0172] For each time step, the loss for the first task is calculated based on the prediction result for the first task at that time step, the intermediate result for that time step, the network parameters set by the diffusion network, and the latent space features of the ground truth image of the second sample image.

[0173] Based on the loss of the first task at each time step, determine the loss of the diffusion network for the first task;

[0174] And / or,

[0175] Based on the prediction results for the second task at each time step and the ground truth image of the second sample image, the loss of the diffusion network for the second task is calculated, including:

[0176] For each time step, the loss for the second task is calculated based on the prediction result for the second task at that time step, the intermediate result for that time step, the network parameters set by the diffusion network, and the latent space features of the reference image corresponding to the second sample image.

[0177] Based on the loss of the second task at each time step, determine the loss of the diffusion network with respect to the second task;

[0178] The methods for obtaining the reference image corresponding to the second sample image include:

[0179] For each of the multiple frequency domain thresholds set, the vector features of the second sample image and the frequency domain threshold are input into the trained initial repair network to obtain the de-degradation result under that frequency domain threshold.

[0180] From the de-degradation results obtained at each frequency domain threshold, select an image that satisfies the similarity condition between the de-degradation result of the second sample image and the ground truth image of the second sample image, and obtain the reference image corresponding to the second sample image.

[0181] Regarding the loss for the first task, the loss for the first task at each time step can be calculated separately, and the loss of the diffusion network for the first task can be determined by combining the losses for each time step. Specifically, for each time step, the loss for the first task at that time step can be calculated based on the prediction result for the first task at that time step, the intermediate result at that time step, the network parameters set by the diffusion network, and the latent space features of the ground truth image of the second sample image.

[0182] For example, the calculation of the loss at any time step with respect to the first task may include: ;in, For the time step regarding the loss of the first mission; This is an intermediate result for this time step; This is the time parameter for that time step (for example, this time parameter can be consistent with the network parameters of stable diffusion3). This is the prediction result for the first task at this time step; To obtain the latent space features of the ground truth image of the second sample image, , The ground truth image of the second sample image is VAEENC, which is the encoder (i.e., VAE Encoder, the encoder in variational autoencoder).

[0183] Regarding the loss for the second task, the loss for the second task at each time step can be calculated separately, and the loss of the diffusion network for the second task can be determined by combining the losses for each time step. Specifically, for each time step, the loss for the second task at that time step can be calculated based on the prediction result for the second task at that time step, the intermediate results at that time step, the network parameters set by the diffusion network, and the latent space features of the reference image corresponding to the second sample image.

[0184] The reference image corresponding to the second sample image can be obtained based on a frequency domain threshold. For example, multiple frequency domain thresholds can be preset. The vector features of each frequency domain threshold and the second sample image are input into the trained initial restoration network to obtain the de-degradation result at that frequency domain threshold, and then the de-degradation result at each frequency domain threshold is obtained. Then, from the obtained de-degradation results at each frequency domain threshold, an image is selected that satisfies the similarity condition between the de-degradation result of the second sample image and the ground truth image of the second sample image (e.g., the similarity to the de-degradation result of the second sample image is greater than a first similarity threshold, and the similarity to the ground truth image of the second sample image is greater than a second similarity threshold; the first and second similarity thresholds can be set as needed, and their relative magnitudes are not limited). This results in a reference image corresponding to the second sample image. Using the reference image of the second sample image, the frequency domain thresholds for the initial restoration network can be applied to the diffusion network, i.e., a fine-grained constraint is also applied to the diffusion network, enabling it to improve image quality and preserve identity information during prediction.

[0185] For example, the calculation of the loss of the second task at any time step may include: ;in, For the time step regarding the loss of the second task; This represents the prediction result for the second task at this time step; The latent space features of the reference image corresponding to the second sample image are used to obtain the reference image corresponding to the second sample image. Its latent space features can be obtained through the VAEENC encoder.

[0186] Latent space features refer to the low-dimensional, continuous, and structured vector representations mapped from high-dimensional original image data by the encoder. The latent space is a "compressed feature space" of the original image, with a dimension much lower than the original image. Furthermore, the distance between vector features is highly correlated with semantic similarity, making it the core carrier for achieving efficient image generation, editing, and retrieval.

[0187] Furthermore, each time step in this application can be a sampled time step, and the number of time steps can be one or more. Each time step can be obtained by sampling the time steps from t=max to t=0, such as random sampling or uniform sampling. The loss of the first task and the loss of the second task of the sampled time steps are used to determine whether the diffusion network has converged.

[0188] In this application, the loss of the diffusion network for the first task can be determined based on the loss for the first task at each time step; and the loss of the diffusion network for the second task can be determined based on the loss for the second task at each time step; furthermore, by using the reference image corresponding to the second sample image, the frequency domain threshold of the initial repair network is applied to the diffusion network to constrain the diffusion network, reducing the illusions that the diffusion network may produce when generating prediction results (for example, if the hair of a person covers the eyes in the de-degradation result of the second sample image, then in the prediction results of the first task and the second task generated by the diffusion network, the hair of the person must still cover the eyes to avoid the change of the person's identity information caused by the environment generated by predicting the eye area), thereby enabling the prediction to both improve image quality and preserve identity information.

[0189] For example, in one implementation, the diffusion network includes a backbone network;

[0190] Training methods for diffusion networks also include:

[0191] For each time step, obtain the output data obtained by the backbone network processing at least one input content at that time step; wherein, at least one input content is at least one of the input content required for the first task and the input content required for the second task;

[0192] For the obtained output data, the data corresponding to the learnable parameters of the diffusion network is extracted to obtain the facial output data, and the data corresponding to the intermediate results at that time step is extracted to obtain the backbone output data; the obtained facial output data is linearly processed to obtain the mean and variance at that time step, and the obtained backbone output data is converted into the intermediate image at that time step; and, based on the obtained variance and mean, the loss for the facial output is calculated, and based on the obtained variance, mean, and intermediate image, the loss for the backbone network is calculated.

[0193] Based on the loss of the diffusion network for the first task and the loss for the second task, determine whether the diffusion network has converged, including:

[0194] Based on the loss of the diffusion network for the first task, the loss for the second task, and the calculated loss for the backbone network and the loss for the facial output, it is determined whether the diffusion network has converged.

[0195] To further ensure that the diffusion network can both improve image quality and preserve identity information during prediction, such as Figure 5As shown, the diffusion network in this application can include a backbone network, and the output of the diffusion network can include backbone output data and facial output data. For each time step, the output data obtained by the backbone network processing at least one input content at that time step can be acquired first. In this way, the facial output data corresponding to the learnable parameters and the backbone output data corresponding to the intermediate results of the current time step can be obtained from the output data. To facilitate the calculation of loss, the facial output data can also be linearly processed to obtain the mean and variance of that time step, and the backbone output data can be converted into the intermediate image of that time step. Thus, the loss of the facial output can be calculated based on the variance and mean, and the loss of the backbone network can be calculated based on the variance, mean, and intermediate image.

[0196] It should be noted that the backbone output data may include the prediction results of the first task and / or the prediction results of the second task (the backbone output may include different prediction results depending on the different input content). The facial output data is... The corresponding output is used to help the diffusion network perform distillation learning to optimize the facial region; that is, the input and output data of the diffusion network have the same dimension. The diffusion network can be a DiffusionTransformer (DIT).

[0197] like Figure 5 As shown, for each time step, the input conditions of the first task, the intermediate results of the current time step, and the learnable parameters can be concatenated, and the concatenated result is input into the diffusion network to obtain the output result (or output data). The output result can include a first type of result, a second type of result, and a third type of result. The first type of result is the output corresponding to the intermediate results, which contains the backbone output data. The second type of result is the output corresponding to the learnable parameters, which contains facial output data. The third type of result is the output corresponding to the input conditions. The output results can be split and recombined to obtain the backbone output data (the prediction result of the first task) and facial output data of the current time step. Similarly, for the second task, the backbone output data (the prediction result of the second task) and facial output data of the current time step can be obtained in a similar manner. Among them, the third type of result is the output corresponding to the input conditions of the first task (i.e., the output other than the output corresponding to the intermediate results and the output corresponding to the learnable parameters). The third type of result is not used; instead, the backbone output data and facial output data of the current time step are obtained using the first type of result and the second type of result. The input conditions for the first task can be understood as including the de-degradation result of the second sample image and the target frequency domain threshold (or its textual prompt); the input conditions for the second task can be understood as including the depth map, identity features, and facial description information (or its textual prompt) extracted from the de-degradation result of the second sample image.

[0198] It should be noted that the so-called input content in this embodiment refers to the input content of a certain network, a certain module, or a certain device as a whole. For example, the input content of the first task may include the input conditions of the first task, the intermediate results of the current time step, and the learnable parameters, which are then concatenated and input into the diffusion network as a whole.

[0199] Accordingly, when determining whether the diffusion network has converged, convergence can be determined based on the losses for the first task, the second task, the backbone network, and the facial output. For example, if the losses for the first task, the second task, the backbone network, and the facial output are all less than their respective loss thresholds, the diffusion network is considered converged; otherwise, it is considered non-converged. Furthermore, the losses for the first task, the second task, the backbone network, and the facial output can be calculated for each time step obtained from the sampling.

[0200] For example, in one implementation, the input of the diffusion network... A one-dimensional token sequence: ;in, is a set of learnable parameters for the diffusion network; cond is the input condition for the first task or the input condition for the second task; the unfolded length of the intermediate result at any time step is . , The quantity is After passing through the diffusion network, the intermediate results are split and reassembled based on the positions of the intermediate results, learnable parameters, and input conditions to extract the intermediate results. The corresponding first-class results are recombined into a two-dimensional feature map to obtain the backbone output data: ;in, The main output data is either the prediction result of the first task or the prediction result of the second task.

[0201] After extracting and rearranging the facial output data, a linear layer is used for linear processing to calculate the mean and variance.

[0202] Facial output data includes:

[0203] ;

[0204] Facial output data can be broken down into And U (for example: dividing the facial output into two equal parts, the first part being the facial output) The latter part is the variance U); then, the backbone output data in the output of the diffusion network is converted into the intermediate image at that time step. and facial output images : ; ;in, This represents the average value of the facial output. This is the intermediate image at this time step; This refers to the time parameter corresponding to the current time step t; This indicates the operation of extracting the corresponding area of ​​the face.

[0205] The loss of facial output can be calculated using the following formula:

[0206] ;

[0207] The output loss of the main branch can be calculated using the following formula:

[0208] ;

[0209] The target loss of the diffusion network is: ;

[0210] in, Loss of facial output; The loss is due to the output of the main branch; The target loss for the diffusion network is the sum of the loss over the backbone network and the loss over the face output. The latent space features of the ground truth image of the second sample image; Characterize gradient truncation.

[0211] The diffusion network in this application may include a backbone network, and the output may include backbone output and facial output. Distillation learning is performed on the backbone output using the facial output, and the loss of the facial output is a regression loss based on uncertainty, constrained by... Towards Closer. The loss of the backbone output is combined with the loss from distillation learning. (Let the facial regions in the overall prediction result learn facial output based on confidence) and mean squared error loss. This further ensures that the diffusion network can both improve image quality and preserve identity information for prediction.

[0212] In this embodiment, the facial output of the diffusion network performs uncertainty regression learning on the backbone output. Self-distillation of the diffusion network can be achieved without using additional modules or networks. The learned softened labels serve as teachers for the backbone output, guiding the backbone to perform distillation learning, which can improve the learning effect of the backbone output on facial regions. The facial output during the inference process can output the face-aligned result without key points. At the same time, this result is also fused with the backbone output in the diffusion network through attention, thereby achieving the effect of extracting face-aligned information without using key points, helping the backbone output to better predict facial regions.

[0213] Optionally, in another embodiment of this application, the image fusion network is a reinforcement learning network, which includes a performer network and a critic network;

[0214] like Figure 11 As shown, the methods for training image fusion networks include:

[0215] C1: Obtain the third sample image;

[0216] C2: Using the first time step as the current time step, based on the intermediate results of the current time step and the third sample image, construct the input content of the current time step of the trained diffusion network, input the constructed input content into the diffusion network, and obtain the prediction result of the current time step corresponding to the third sample image.

[0217] C3: Based on the prediction results of the current time step corresponding to the third sample image, call the performer network to generate the fusion strategy for the current time step corresponding to the third sample image;

[0218] C4: Based on the generated fusion strategy, perform result fusion processing on the prediction results of the current time step corresponding to the third sample image to obtain the fusion result of the current time step corresponding to the third sample image;

[0219] C5: Based on the fusion result of the current time step corresponding to the third sample image, decode to obtain the repaired image of the current time step corresponding to the third sample image, and calculate the reward value of the repaired image of the current time step; and, based on the fusion strategy of the current time step corresponding to the third sample image, construct the input conditions of the commentator network for the current time step, and input the constructed input conditions into the commentator network to obtain the Q value corresponding to the input conditions of the current time step.

[0220] C6: If the current time step is not the last time step, then based on the fusion result of the current time step corresponding to the third sample image, calculate the intermediate result of the next time step of the current time step, and take the next time step as the new current time step, and return the step of constructing the input content of the current time step of the diffusion network based on the intermediate result of the current time step and the third sample image; or, if the current time step is the last time step, return the step of obtaining the third sample image.

[0221] C7: In response to the fulfillment of the loss calculation conditions, based on the Q-values ​​corresponding to the input conditions at each time step and the reward values ​​of the repaired images at each time step, calculate the loss of the performer network and the loss of the critic network. If the reinforcement learning network has not converged based on the calculated losses of the performer network and the critic network, adjust the network parameters of the reinforcement learning network.

[0222] The image fusion network in this application is a reinforcement learning network that follows the Deep Deterministic Policy Gradient (DDPG) paradigm. The reinforcement learning network can include an actor network and a critic network (two networks in the classic reinforcement learning paradigm AC, where the actor network outputs the policy and the critic network outputs the value assessment). When training the image fusion network, a third sample image is first acquired for training, and the first time step is used as the current time step. Based on the intermediate results of the current time step and the third sample image, the input content for the current time step of the trained diffusion network is constructed. The diffusion network then obtains the prediction result for the current time step corresponding to the third sample image. The input content for the current time step can include the input content for the first task and the input content for the second task of the diffusion network at the current time step (before construction, the third sample image can be de-degraded using the trained initial repair network, and the de-degradation result can be used to construct the input content). Correspondingly, the prediction result for the current time step corresponding to the third sample image can include the prediction results for the first and second tasks obtained through the trained diffusion model.

[0223] After that, as Figure 6As shown, the Performer Network can be invoked to generate a fusion strategy for the current time step based on the prediction result of the third sample image at the current time step. The input to the Performer Network can include three parts: the intermediate result of the current time step, the depth map of the third sample image and its de-degradation result (which can be stitched together to obtain the first result; the images can be stitched together at the channel level to obtain the first result), and the prediction result of the current time step (the prediction results of the first task and the second task, which can be stitched together to obtain the second result; specifically, the second result can be stitched together at the channel level). These three parts can be stitched together to obtain a stitched result (including the intermediate result, the first result, and the second result). The stitched result and the encoded result are input into the Performer Network to obtain the output result. The encoded result is obtained by encoding the identity features and facial description information extracted from the de-degradation result of the third sample image from the text. The output result is then split and reassembled (in a similar way to the aforementioned method, which will not be elaborated here) to obtain the fusion strategy for the current time step. Specifically, the output results include intermediate results, first output, second output, and third output corresponding to the first and second results, respectively. The first output can be obtained by splitting and recombining the output results, and the fusion strategy for the current time step can be extracted from the first output. The second and third outputs are not used.

[0224] In this process, identity features and facial description information can be used as input to the performer network as cue information. That is, identity features and facial description information serve as abstract input to the performer network, while the splicing result serves as concrete input to the performer network, and together they are used to generate the fusion strategy for the current time step.

[0225] Based on the generated fusion strategy, the prediction results of the first task and the prediction results of the second task of the third sample image are fused to obtain the fusion result of the current time step corresponding to the third sample image.

[0226] For example, the input content of the Performer Network It can include:

[0227] ;in, Let represent the state of the image fusion network at time step t, which is an instantaneous description of the environment. ; This is an intermediate result at the current time step; The prediction result for the first task is obtained by passing the third sample image through a diffusion network; I represents the prediction result of the second task obtained from the third sample image through a diffusion network; I represents the third sample image. The depth map of the de-degradation result of the initial training network after training the third sample image; The facial description information is the de-degradation result of the initial training network after training the third sample image; ID is the identity feature of the ground truth image of the third sample image; specifically, .

[0228] The process by which the performer network generates the fusion strategy for the current time step can be represented by the following formula:

[0229] ;in, The fusion strategy for the current time step corresponding to the third sample image can also be understood as the action of the image fusion network at time step t. For performers' network.

[0230] After obtaining the fusion result, the fusion result can be decoded to obtain the repaired image corresponding to the current time step of the third sample image, and the reward value of the repaired image can be calculated. Furthermore, based on the fusion strategy corresponding to the current time step of the third sample image, the input conditions of the critic network for the current time step are constructed and input into the critic network to obtain the Q value corresponding to the input conditions of the current time step.

[0231] like Figure 7 As shown, the input to the critic network can contain three parts: the intermediate results and fusion strategy at the current time step (which can be concatenated to obtain the third result), the depth map of the third sample image and its de-degradation result (which can be concatenated to obtain the fourth result), and the prediction results at the current time step (the prediction results of the first task and the prediction results of the second task, which can be concatenated to obtain the fifth result). These three parts can be concatenated to obtain a concatenated result (containing the third, fourth, and fifth results). The concatenated result and the encoded result are input into the critic network to obtain the output result. The encoded result is obtained by encoding the identity features and facial description information extracted from the de-degradation result of the third sample image from the text. The output result is split and recombined (the method can be similar to the above, and will not be elaborated here) to obtain the Q value. Specifically, the output result contains the fourth output, fifth output, and sixth output corresponding to the third, fourth, and fifth results, respectively. Among them, the output result can be split and recombined to obtain the fourth output, and the Q value can be extracted from the fourth output. The fifth and sixth outputs are not used.

[0232] In this process, identity features and facial description information can be used as input to the critic network as cue information. That is, identity features and facial description information serve as abstract input to the critic network, while the concatenation result serves as concrete input to the critic network, and together they are used to generate the Q value of the current time step.

[0233] It should be noted that the stitching involved in this application refers to the stitching between images, which is a channel-level stitching; while the stitching between tensors or feature vectors is a sequence stitching.

[0234] For example, the input conditions for the current time step of the critic network. It can include:

[0235] ;

[0236] That is, the input conditions of the critic network at the current time step include the actions and states of the image fusion network, specifically,

[0237] .

[0238] The process by which the critic network processes the input conditions at the current time step can be represented by the following formula: ;in, For the critics' network; The Q-value is the prediction of the commentator network at time step t.

[0239] If the current time step is not the last time step, iteration needs to continue. Based on the fusion result of the current time step corresponding to the third sample image, the intermediate result of the next time step can be calculated, and this next time step can be used as the new current time step for re-prediction via the diffusion network. If the current time step is the last time step, a new third sample image can be selected to train the image fusion network.

[0240] If the loss calculation conditions are met, the losses of the performer network and the critic network can be calculated based on the Q-values ​​corresponding to the input conditions at each time step and the reward values ​​of the repaired images at each time step. If the reinforcement learning network is determined to have not converged based on its losses, the network parameters of the reinforcement learning network are adjusted, and the reinforcement learning network is trained again using the third sample image.

[0241] The image fusion network in this application is a reinforcement learning network, comprising a performer network and a critic network. During training, the prediction result for the current time step corresponding to the third sample image is obtained through the trained diffusion network. The performer network generates a fusion strategy for the current time step. The prediction results for the first and second tasks at the current time step are fused using this fusion strategy to obtain the final fusion result. The resulting fusion result balances improving image quality with preserving identity information. For the fusion result at the current time step, the restored image is decoded, and its reward value is calculated. The critic network obtains the corresponding Q-value, which is then used to calculate the losses of the performer and critic networks, thereby evaluating whether the reinforcement learning network has converged. The reward value and Q-value comprehensively describe the restoration result, ensuring that the reinforcement learning network can balance improving image quality with preserving identity information during image restoration.

[0242] For example, in one implementation, the method of training the image fusion network also includes:

[0243] In response to obtaining the input conditions of two adjacent time steps, based on the input conditions of these two adjacent time steps, the reward value of the repaired image in the previous time step, and the fusion strategy of the previous time step, a set of state information corresponding to the target time step is constructed, and the constructed set of state information corresponding to the target time step is cached in a predetermined cache space. The target time step is the previous time step among the two adjacent time steps; accordingly,

[0244] In response to satisfying the loss calculation conditions, based on the calculated Q-values ​​corresponding to the input conditions at each time step and the reward values ​​of the repaired images at each time step, the losses of the performer network and the critic network are calculated, including:

[0245] In response to the fulfillment of the loss calculation conditions, a specified number of state information sets are selected from the predetermined cache space. The loss of the commentator network is calculated based on the Q-value corresponding to the input conditions of the target time step in each selected state information set, the reward value of the repaired image at the target time step, and the Q-value corresponding to the input conditions of the next time step in each selected state information set. The loss of the performer network is calculated based on the Q-value corresponding to the fusion strategy at the target time step.

[0246] When training an image fusion network, given the input conditions of two adjacent time steps, a set of state information corresponding to the target time step can be constructed based on the input conditions of these two adjacent time steps, the reward value of the repaired image in the previous time step, and the fusion strategy of the previous time step. This set of state information is then cached in a predetermined storage space for subsequent loss calculation. The adjacent time steps can be t and t-1, and the target time step is time step t.

[0247] For example, the set of state information corresponding to the target time step can be sample , The reward value for the target time step; This serves as the input condition for the next time step adjacent to the target time step. The state information set can be understood as including: the reward value of the current time step, the action of the current time step, the state of the current time step, and the state of the next time step.

[0248] Accordingly, the loss calculation conditions can be as follows: a target number (e.g., 100) sets of state information are stored in a predetermined cache space; if the loss calculation conditions are met, a specified number (e.g., 10) sets of state information can be selected from the predetermined storage space, and the input conditions of the target time step in each selected set of state information can be applied. The corresponding Q-value and reward value for the repaired image, as well as the input conditions for the next time step in each set of state information. The loss of the critic network is calculated based on the corresponding Q-value; and the loss of the performer network is calculated based on the Q-value corresponding to the fusion strategy at that time step.

[0249] For example, the performer network and the critic network perform iterative optimization at each time step; for each target time step in the selected set of state information, the loss of the critic network is calculated as follows:

[0250] ;in, The loss of the commentator network; N is the specified number; The input conditions of the critic network at the target time step t are: Q-value output at that time The truth value calculated based on Temporal Difference Learning (TD):

[0251] ;

[0252] For a and A network of critics with the same structure As the target network, its update is an update of the target network (momentum update). , , and All are in the current , and In the case of using This serves as the content of the next time step after the fusion strategy is implemented and the state is updated; These are the weighting coefficients. Specifically, when calculating the loss of the commentator network, the commentator network... The network parameters are the network parameters at the current time step t.

[0253] The loss of the performer network is calculated as follows:

[0254] ;

[0255] in, This is a loss for the performers' network.

[0256] When calculating the loss of the performer network, the critic network... The network parameters are fixed; that is, the network parameters of the critic network used to calculate the loss of the performer network at different time steps are the same.

[0257] For example, in one implementation, the third sample image is an image acquired that requires image restoration;

[0258] And / or,

[0259] Based on the fusion strategy corresponding to the current time step of the third sample image, the input conditions for the current time step of the commentator network are constructed, including:

[0260] Based on the fusion strategy corresponding to the third sample image at the current time step, the current intermediate results, the measurement results of the current time step regarding the first task, the prediction results regarding the second task, and the depth map of the third sample image and the de-degradation result of the third sample image, the input conditions of the commentator network at the current time step are constructed.

[0261] And / or,

[0262] Based on the fusion result at the current time step, the restored image at the current time step is decoded, including:

[0263] Based on the fusion result at the current time step, the initial diffusion intermediate result corresponding to the current third sample image is inferred, and based on the inferred initial diffusion intermediate result, the repaired image at the current time step is decoded.

[0264] The third sample image can be an image that has been acquired and requires image restoration; that is, the third sample image can be a real image acquired. Furthermore, the image fusion network can be trained based on the real third sample image, without needing to rely on the ground truth image of the third sample image. This reduces the difficulty of acquiring images required for training the image fusion network and improves training efficiency.

[0265] The input conditions for the current time step of the critic network can be obtained from the intermediate results of the current time step. The prediction results for the first task at the current time step. And predictions regarding the second task The network is constructed by combining the depth map of the third sample image and the de-degradation result of the third sample image with the identity features and facial description information extracted from the de-degradation result of the third sample image; furthermore, the input conditions of the current time step of the commentator network can also include the fusion strategy of the current time step. Therefore, the critic network can accurately assess the value of the performer network's output by considering the specific circumstances.

[0266] When decoding the fusion result, the initial diffusion intermediate result corresponding to the current third sample image can be inferred from the fusion result at the current time step, and the decoded image at the current time step can be decoded based on the inferred initial diffusion intermediate result (which can be achieved by calling the decoder in the diffusion network). The specific implementation process is similar to Flow Matching, and will not be elaborated here.

[0267] For example, in one implementation, calculating the reward value of the repaired image at the current time step includes:

[0268] Determine the facial features in the repair image at the current time step corresponding to the third sample image, and determine the first type of reward value based on the similarity between the determined facial features and the facial features in the first specified image; the first specified image is an image that contains the object in the third sample image and does not have a repair requirement;

[0269] The first major model is invoked to perform anomaly detection on the repaired image at the current time step corresponding to the third sample image, and the second type of reward value is determined based on the anomaly detection results output by the first major model.

[0270] The second major model is invoked to perform content consistency analysis on the repaired image at the current time step corresponding to the third sample image and the second specified image, and the third type of reward value is determined based on the analysis results of the second major model.

[0271] Based on the first type of reward value, the second type of reward value, and the third type of reward value, the reward value of the repaired image at the current time step is determined.

[0272] The reward value for image restoration can include: identity consistency reward and query reward. First, based on the facial features in the restored image, the similarity between the facial features and those in the first specified image can be calculated to determine the reward value for identity consistency (i.e., the first type of reward value). The first specified image can be an image containing objects from the third sample image that does not require restoration; that is, the identity of the person in the first specified image is consistent with the identity of the person in the third sample image, and the first specified image does not require image restoration. For example, the first specified image is an image of the real person corresponding to the third sample image, not a high-quality image corresponding to the third sample image (i.e., the first specified image is not identical to the person's pose, background, etc., in the third sample image, but its image quality is higher than the third sample image).

[0273] For example, the identity consistency reward value is calculated as follows:

[0274] ;

[0275] in This indicates calculating the cosine similarity between two normalized vectors. This represents a facial recognition model for extracting identity features. It is the first specified image.

[0276] The query reward is obtained through querying the large model; given a prompt question, the large model makes a judgment, and the query reward is obtained based on the output of the large model. Let the image to be queried be denoted as... Large model by Input image, Input text (the input text is the converted English text), output text; query rewards include: anomaly queries and content consistency queries.

[0277] The calculation method for the reward value (second type of reward value) for abnormal queries is as follows:

[0278] This refers to performing anomaly checks on the cheeks, eyes, mouth, ears, and nose.

[0279] ;

[0280] If there are no abnormalities in part X, the score is increased by 1; if there are abnormalities in part X, the score is decreased by 1.

[0281] The content consistency query process involves using a large model to query the de-degradation results of a third sample image. The content is then judged by the large model. Is the content consistent with Consistent.

[0282] The reward value (third type of reward value) for content consistency queries is calculated as follows:

[0283] ; that is, query and Are the facial expressions, eyes, nose, and mouth of the (second specified image) consistent?

[0284] ;

[0285] ;

[0286] ;

[0287] The reward value for image restoration is calculated as follows:

[0288] ;in, , , These are the weighting coefficients for the reward values ​​of identity consistency, abnormal queries, and content consistency queries, respectively; and > > .

[0289] In this application, a first type of reward value is determined by comparing the facial features of the repaired image with those of a first specified image; and anomaly detection and content consistency analysis are performed using a first large model and a second large model (which may be the same or different) to determine a second type of reward value and a third type of reward value. Thus, the reward value of the repaired image at the current time step is obtained by comprehensively evaluating the repaired image. By using this reward value for loss calculation, the accuracy of image repair can be ensured, while also improving image quality and preserving identity information during image repair.

[0290] The image fusion network of this application will now be described with reference to another embodiment.

[0291] The image fusion network is trained using real low-resolution images (i.e., the third sample image is an image that has been acquired and has the need for image restoration), and uses a large model (for calculating the second and third types of reward values) and an identity recognition model (for calculating the first type of reward value for identity consistency) to obtain rewards.

[0292] The training process for image fusion networks includes:

[0293] 1. Select a real low-resolution image (third sample image) and initialize a feature map from the noise. , t=Tmax;

[0294] 2. Using diffusion networks, predictions were obtained respectively. and ;

[0295] 3. Using performer networks to generate fusion strategies ;

[0296] 4. Perform fusion according to the fusion strategy to obtain the fusion result at the current time step:

[0297] ;

[0298] 5. Based on the Flow Matching paradigm, reverse-engineer the initial diffusion intermediate results. :

[0299] ;

[0300] 6. Use the stable diffusion3 decoder Decoding yields the repaired image at the current time step. : ;

[0301] 7. To Calculate the reward function And calculate the state at the next time step. sample this set of state information Recorded in the replay buffer (i.e., the experience replay pool, corresponding to the aforementioned pre-defined cache space).

[0302] 8. After feeding 100 samples into the replay buffer, sample 10 sets from the replay buffer. Calculate the loss for actor and critic using this sample, and update the gradient.

[0303] 9. Update according to the following formula :

[0304] ;

[0305] 10. Proceed to the next time step If at this time If the next third sample image is selected for training, otherwise the diffusion network is used to generate the next time step image. and .

[0306] Image inpainting process of image fusion network as follows Figure 8As shown, specifically:

[0307] 1. Initialize noise , t=Tmax;

[0308] 2. Based on the intermediate results at the current time step, construct the input content for the first task and the input content for the second task, and use a diffusion network to calculate the prediction results for the first task respectively. Predictions for the second task ;

[0309] 3. Using performer networks to generate fusion strategies (This is not shown in the diagram; instead, a fusion strategy is used directly for fusion.)

[0310] 4. Combine the prediction results of the first task and the prediction results of the second task according to the fusion strategy; ;

[0311] 5. Calculate the intermediate results for the next time step based on the Flow Matching generation paradigm. :

[0312] ;

[0313] 6. t = t - 1, determine if t = 0? If not, i.e., t > 0, return to step 2, i.e., use the intermediate results of the next time step to perform predictions for the first and second tasks respectively. If yes, use the stable diffusion3 decoder. The repaired image is calculated as follows: .

[0314] Image fusion networks primarily perform quality-preserving predictions for diffusion networks during the iterative inference process. Hobo identity prediction By employing a supervised reinforcement learning approach, the network learns a fusion strategy on real-world data that preserves the identity information of the output image while eliminating anomalies and maintaining image quality. This allows for image restoration by simultaneously improving image quality and retaining identity information. Furthermore, the image fusion network is trained using real low-resolution images (without requiring corresponding high-resolution images), breaking the constraint of pixel-level data pairs. This approach significantly improves model performance by accumulating low-cost data, enhancing the adaptability of this solution in real-world scenarios.

[0315] To better understand this solution, the image restoration method provided in this application will be described below with reference to a specific embodiment.

[0316] This application takes into account the characteristics of the diffusion model: 1. Latent space fusion capability: The latent space diffusion model method iterates in the feature space. Compared with models that directly output the image space result in one step, the diffusion model can still maintain the manifold after modifying the intermediate results in the latent space, making it suitable for fusion and content correction. 2. Multi-task compatibility: The diffusion model trained on big data contains rich prior knowledge and can generate high-quality images for different prompts. One model can integrate multiple tasks (such as the first and second tasks involved in this application).

[0317] Furthermore, the same diffusion model, through appropriate training, can output both high-quality images and images that preserve content (identity) (i.e., the prediction results for the first task of quality-preserving prediction and the prediction results for the second task of identity-preserving prediction). During inference, learning appropriate strategies to fuse images in the latent space can achieve both improved image quality and preservation of identity information.

[0318] Based on the above ideas, this application proposes an image inpainting method combining frequency domain constraints and reinforcement learning. It consists of three parts: an initial inpainting network, a diffusion network, and an image fusion network, as shown in the logical relationship diagram below. Figure 2 As shown, during the training process, the initial repair network 210, the diffusion network 220, and the image fusion network 230 are trained sequentially. During the training of the image fusion network, the iteration at each time step needs to generate the corresponding prediction result through the diffusion network.

[0319] The initial repair network is used to control the fineness of de-degradation, which can remove interference, complete the structure, and restrict the generation of truth values;

[0320] The diffusion network is built on a self-distillation diffusion network and can generate high-definition images (first task) and generate identity-preserving images (second task) according to different input content.

[0321] Image fusion networks are used to fuse the prediction results of the first task and the prediction results of the second task.

[0322] The initial repair network can include a discriminator and a generator; the diffusion network can include a backbone network, and its output can include backbone output data and facial output data, enabling self-distillation learning of the backbone network; the image fusion network can be a reinforcement learning network, including a performer network and a critic network.

[0323] The training process for the initial repair network, diffusion network, and image fusion network is as follows:

[0324] 1. Train the initial repair network. A discriminator is used in this process. (Frequency domain priors for perceiving facial regions) Helps the generator Learn frequency domain information specific to facial regions. After training, it can be applied based on a selected specified frequency domain threshold. Control the generation of images The level of detail;

[0325] 2. Training the Diffusion Network. The training process of the diffusion network uses a confidence-based facial region self-distillation algorithm. To reduce illusions, an initial repair network is used during training to constrain the training ground truth. After training, a diffusion model is obtained. Following the working principle of the diffusion network, each step predicts a denoising direction, and the prediction result is output after multiple iterations. Depending on the input conditions, the prediction result output by the diffusion network also differs, denoted as different denoising directions (i.e., the first task and the second task): If the input condition is a de-degraded image obtained through the initial repair network... After iterative denoising, a high-resolution image of the de-degraded image with detailed restoration is generated. The intermediate result of each step is recorded as the prediction result of the first task. If the input condition is an identity feature Depth map And facial descriptive information (such as facial attributes) This generates a high-resolution image that preserves the identity information; the intermediate result at this point is recorded as the prediction result of the second task. The diffusion network performs different prediction tasks based on different input content, but the network structure of the diffusion network is the same regardless of the input conditions, and the training paradigm follows the facial region self-distillation training algorithm; the two tasks differ only in the data settings (input content).

[0326] 3. Train the image fusion network. The image fusion network uses reinforcement learning to generate a fusion strategy for quality-assured predictions during the inference process of the diffusion network. Identity Prediction To integrate.

[0327] The training process for each network can be found in the above embodiments and will not be repeated here.

[0328] The image restoration process is as follows:

[0329] 1. Use the initial repair network. Based on the specified frequency domain threshold. The target image is passed through the generator of the initial repair network to obtain the de-degradation result image. Extract facial attributes from the resulting image. and depth map ;

[0330] 2. Enter the diffusion network, initialize the intermediate results of the diffusion model as pure noise, and iterate from t=Tmax to t=0.

[0331] a. Use a diffusion network to degenerate the resulting graph. and specified frequency domain threshold As a conditional input, obtain the prediction result of the first task for ensuring quality in the current step. ;

[0332] b. Using diffusion networks, based on identity features Depth map Facial attributes As input conditions, obtain the prediction result of the second task for the current step, which involves protecting identity information. ;

[0333] 3. Enter the image fusion network, and... , Depth map, identity features, facial attributes As a conditional input, the fusion strategy for the current time step is calculated. ;

[0334] 4. Use a fusion strategy Will and Perform fusion to obtain the fusion result at the current time step. ;

[0335] 5. Based on the flow matching calculation formula, calculate the intermediate result for the next time step by combining the intermediate result of the current time step with the prediction result of the current time step, and update the result. .

[0336] 6. Decode the intermediate result at t=0 using a decoder to obtain the output image. For the repaired image of the target image.

[0337] Furthermore, for diffusion networks and / or image fusion networks, the number of time steps in their training process can be greater than the number of time steps in the image inpainting process (i.e., the inference process). For example, the time steps in the image inpainting process are sampled from the time steps in the training process. This allows for more time steps to be used for iteration during training, ensuring the accuracy of the diffusion and image fusion network training; and it also allows for fewer time steps in the image inpainting process, enabling faster image inpainting and improving its efficiency.

[0338] This application proposes a facial image restoration method that combines reinforcement learning to improve image quality and preserve personal identity information. The trained diffusion network has the ability to generate images while preserving both identity and quality. The reinforcement learning network is trained based on real data and can fuse images that preserve both quality and identity to obtain the restored image, thus achieving both improved image quality and preservation of identity information in image restoration.

[0339] Based on the above method embodiments, this application also provides an image restoration apparatus, such as... Figure 12 As shown, the device includes:

[0340] The acquisition module 1210 is used to acquire the target image to be repaired;

[0341] Module 1220 is invoked to call the pre-trained initial insulation network to perform image de-degradation processing on the target image and obtain the de-degradation result of the target image;

[0342] The repair module 1230 is used to repair and optimize the de-degradation result according to a predetermined repair and optimization method to obtain the image repair result of the target image;

[0343] The planned repair and optimization methods include:

[0344] The de-degradation results are processed based on a pre-trained diffusion network used to perform the first and second tasks. The prediction results for the first task and the prediction results for the second task obtained by the diffusion network when processing the de-degradation results of the target image are used as the content to be fused, and fusion processing is performed by a pre-trained image fusion network.

[0345] The first task is to optimize the quality of the image input to the diffusion network to output an optimized prediction result. The second task is to output a prediction result that retains the identity information in the input image, based on the image input to the diffusion network.

[0346] Optionally, the repair module is specifically used for:

[0347] Taking the first time step as the current time step, based on the intermediate results of the current time step and the de-degradation results of the target image, the input content of the diffusion network for the current time step regarding the first task and the input content regarding the second task are constructed. The constructed input content is input into the diffusion network to obtain the prediction results of the first task and the second task for the current time step.

[0348] Based on the prediction results of the first task and the second task at the current time step, the image fusion network is invoked so that the image fusion network performs result fusion processing on the prediction results of the first task and the second task at the current time step to obtain the fusion result at the current time step.

[0349] If the current time step is the last time step, the restored image of the current time step is decoded based on the fusion result of the current time step, and the restored image of the current time step is used as the image restoration result of the target image; otherwise, based on the fusion result of the current time step, the intermediate result of the next time step is calculated, and the next time step is used as the new current time step. The prediction results of the first task and the prediction results of the second task based on the current time step are returned, and the steps of the image fusion network are invoked.

[0350] Optionally, the initial inpainting network is trained based on the first sample image and the ground truth image of the first sample image, where the ground truth image of any sample image is the image that represents the expected inpainting result of that sample image;

[0351] The first sample image is the image obtained after image degradation processing of a predetermined initial image. The ground truth image of the first sample image is the image obtained by low-pass filtering the image spectrum of the initial image according to the target frequency domain threshold set for the first sample image.

[0352] Correspondingly, the module is called, specifically for:

[0353] Based on a specified frequency domain threshold, the initial repair network is invoked to perform image de-degradation processing on the target image based on the specified frequency domain threshold, and the de-degradation result of the target image is obtained.

[0354] Optionally, the initial repair network can be trained in the following ways:

[0355] The vector features of the first sample image and the target frequency domain threshold are input into the initial repair network so that the initial repair network uses the vector features of the target frequency domain threshold as the first reference prompt to perform image de-degradation processing on the first sample image, and obtain the de-degradation result of the first sample image.

[0356] Based on the de-degradation result of the first sample image and the ground truth image of the first sample image, the loss of the initial instigation network is calculated;

[0357] In response to the loss characteristic of the initial repair network indicating that the initial repair network has not converged, the network parameters of the initial repair network are adjusted.

[0358] Accordingly, based on a specified frequency domain threshold, the initial repair network is invoked to perform image de-degradation processing on the target image based on the specified frequency domain threshold, obtaining the de-degradation result of the target image, including:

[0359] The target image and the vector features of a specified frequency domain threshold are input into the initial inpainting network, so that the initial inpainting network uses the vector features of the specified frequency domain threshold as the first reference prompt to perform image de-degradation processing on the target image, and obtain the de-degradation result of the target image.

[0360] Optionally, the initial repair network includes a generator and a discriminator;

[0361] The vector features of the first sample image and the target frequency domain threshold are input into the initial inpainting network, so that the initial inpainting network uses the vector features of the target frequency domain threshold as the first reference cue to perform image de-degradation processing on the first sample image, obtaining the de-degradation result of the first sample image, including:

[0362] The vectors of the first sample image and the target frequency domain threshold are input into the generator in the initial repair network, so that the generator in the initial repair network uses the vector features of the target frequency domain threshold as the first reference prompt to perform image de-degradation processing on the first sample image, and obtain the de-degradation result of the first sample image.

[0363] The initial training methods for the repair network also include:

[0364] Based on the de-degradation result of the first sample image and the ground truth image of the first sample image, the input content of the discriminator in the initial repair network is constructed;

[0365] The input content of the constructed discriminator is input into the discriminator so that the discriminator can perform discrimination processing on the received input content and obtain the discrimination result of the first sample image;

[0366] Based on the de-degradation result of the first sample image and the ground truth image of the first sample image, the loss of the initial inpainting network is calculated, including:

[0367] Based on the difference between the de-degradation result of the first sample image and the ground truth image of the first sample image, the loss is calculated to obtain the first type of loss;

[0368] Based on the discrimination results of the first sample image, the loss is calculated to obtain the second type of loss;

[0369] The loss of the initial repair network is determined based on the first and second type losses of the initial repair network.

[0370] Optionally, based on the de-degradation result of the first sample image and the ground truth image of the first sample image, the input content of the discriminator in the initial instigation network is constructed, including:

[0371] Based on the learnable parameters set for the discriminator, the target frequency domain threshold is processed to obtain the prior features corresponding to the target frequency domain threshold.

[0372] The prior features corresponding to the target frequency domain threshold and the facial region features of the de-degradation result of the first sample image are concatenated to obtain the first concatenated feature corresponding to the first sample image. The prior features corresponding to the target frequency domain threshold and the facial region features of the ground image of the first sample image are concatenated to obtain the second concatenated feature corresponding to the first sample image.

[0373] The input content of the constructed discriminator is input into the discriminator so that the discriminator performs discrimination processing on the received input content to obtain the discrimination result of the first sample image, including:

[0374] The first stitching feature corresponding to the first sample image is input into the discriminator so that the discriminator performs discrimination processing on the received first stitching feature to obtain the discrimination result of the first sample image with respect to the first stitching feature;

[0375] The second stitching feature of the first sample image is input into the discriminator so that the discriminator can perform discrimination processing on the received second stitching feature to obtain the discrimination result of the first sample image with respect to the second stitching feature;

[0376] Based on the discrimination results of the first sample image, loss is calculated to obtain the second type of loss, including:

[0377] Based on the discrimination results of the first sample image regarding the first stitching feature and the discrimination results of the first sample image regarding the second stitching feature, loss is calculated to obtain the second type of loss.

[0378] Optionally, based on the discrimination result of the first sample image regarding the first stitching feature and the discrimination result of the first sample image regarding the second stitching feature, a loss is calculated to obtain a second type of loss, including:

[0379] From the discrimination results of the first splicing features of the first sample image, extract the result content of the prior features corresponding to the target frequency domain threshold, and score the first discrimination score of the first sample image based on the extracted result content.

[0380] From the discrimination results of the first sample image regarding the second stitching features, extract the result content of the prior features corresponding to the target frequency domain threshold, and score based on the extracted result content to obtain the second discrimination score of the first sample image;

[0381] Based on the first discriminant score of the first sample image, determine the adversarial loss;

[0382] The discriminator loss is determined based on the first and second discriminant scores of the first sample image.

[0383] Alternatively, the training methods for the diffusion network include:

[0384] The second sample image is processed using the trained initial inpainting network to obtain the de-degradation result of the second sample image.

[0385] According to the predetermined processing method, the de-degradation result of the second sample image is subjected to prediction processing for the first task and prediction processing for the second task through a diffusion network to obtain the prediction results for the first task and the prediction results for the second task at each time step.

[0386] Based on the prediction results for the first task at each time step and the ground truth image of the second sample image, calculate the loss of the diffusion network for the first task; and based on the prediction results for the second task at each time step and the ground truth image of the second sample image, calculate the loss of the diffusion network for the second task.

[0387] Based on the loss of the diffusion network with respect to the first task and the loss with respect to the second task, determine whether the diffusion network has converged. If it has not converged, adjust the network parameters of the diffusion network.

[0388] The predetermined processing method includes: for each time step, based on the intermediate results of that time step, the learnable parameters set for the diffusion network, the de-degradation results of the second sample image, and the set second reference cue, constructing the input content required for the first task at that time step, and inputting the constructed input content into the diffusion network to perform the first task;

[0389] Based on the intermediate results of this time step, the learnable parameters set for the diffusion network, the depth map of the de-degradation result of the second sample image, and the identity features and facial description information of the objects in the de-degradation result of the second sample image, the input content required for the second task at this time step is constructed, and the constructed input content is input into the diffusion network to perform the second task.

[0390] Optionally, based on the prediction results for the first task at each time step and the ground truth image of the second sample image, the loss of the diffusion network for the first task is calculated, including:

[0391] For each time step, the loss for the first task is calculated based on the prediction result for the first task at that time step, the intermediate result for that time step, the time parameters for that time step, and the latent space features of the ground truth image of the second sample image.

[0392] Based on the loss of the first task at each time step, determine the loss of the diffusion network for the first task;

[0393] And / or,

[0394] Based on the prediction results for the second task at each time step and the ground truth image of the second sample image, the loss of the diffusion network for the second task is calculated, including:

[0395] For each time step, the loss for the second task is calculated based on the prediction result for the second task at that time step, the intermediate result at that time step, the time parameters at that time step, and the latent space features of the reference image corresponding to the second sample image.

[0396] Based on the loss of the second task at each time step, determine the loss of the diffusion network with respect to the second task;

[0397] The methods for obtaining the reference image corresponding to the second sample image include:

[0398] For each of the multiple frequency domain thresholds set, the vector features of the second sample image and the frequency domain threshold are input into the trained initial repair network to obtain the de-degradation result under that frequency domain threshold.

[0399] From the de-degradation results obtained at each frequency domain threshold, select an image that satisfies the similarity condition between the de-degradation result of the second sample image and the ground truth image of the second sample image, and obtain the reference image corresponding to the second sample image.

[0400] Optionally, the diffusion network includes a backbone network;

[0401] Training methods for diffusion networks also include:

[0402] For each time step, obtain the output data obtained by the backbone network processing at least one input content at that time step; wherein, at least one input content is at least one of the input content required for the first task and the input content required for the second task;

[0403] For the obtained output data, the data corresponding to the learnable parameters of the diffusion network is extracted to obtain the facial output data, and the data corresponding to the intermediate results at that time step is extracted to obtain the backbone output data; the obtained facial output data is linearly processed to obtain the mean and variance at that time step, and the obtained backbone output data is converted into the intermediate image at that time step; and, based on the obtained variance and mean, the loss for the facial output is calculated, and based on the obtained variance, mean, and intermediate image, the loss for the backbone network is calculated.

[0404] Based on the loss of the diffusion network for the first task and the loss for the second task, determine whether the diffusion network has converged, including:

[0405] Based on the loss of the diffusion network for the first task, the loss for the second task, and the calculated loss for the backbone network and the loss for the facial output, it is determined whether the diffusion network has converged.

[0406] Optionally, the image fusion network is a reinforcement learning network, which includes a performer network and a critic network;

[0407] Methods for training image fusion networks include:

[0408] Obtain the third sample image;

[0409] Using the first time step as the current time step, and based on the intermediate results of the current time step and the third sample image, the input content of the trained diffusion network for the current time step is constructed. The constructed input content is then input into the diffusion network to obtain the prediction result of the current time step corresponding to the third sample image.

[0410] Based on the prediction results of the current time step corresponding to the third sample image, the performer network is invoked to generate the fusion strategy of the current time step corresponding to the third sample image;

[0411] Based on the generated fusion strategy, the prediction results of the current time step corresponding to the third sample image are fused to obtain the fusion result of the current time step corresponding to the third sample image.

[0412] Based on the fusion result of the current time step corresponding to the third sample image, the repaired image of the current time step corresponding to the third sample image is decoded and the reward value of the repaired image of the current time step is calculated; and based on the fusion strategy of the current time step corresponding to the third sample image, the input conditions of the current time step of the critic network are constructed and the constructed input conditions are input into the critic network to obtain the Q value corresponding to the input conditions of the current time step.

[0413] If the current time step is not the last time step, then based on the fusion result of the current time step corresponding to the third sample image, calculate the intermediate result of the next time step of the current time step, and take the next time step as the new current time step, and return the step of constructing the input content of the current time step of the diffusion network based on the intermediate result of the current time step and the third sample image; or, if the current time step is the last time step, return the step of obtaining the third sample image.

[0414] In response to the fulfillment of the loss calculation conditions, based on the Q-values ​​corresponding to the input conditions at each time step and the reward values ​​of the repaired images at each time step, the losses of the performer network and the critic network are calculated. If the reinforcement learning network fails to converge based on the calculated losses of the performer network and the critic network, the network parameters of the reinforcement learning network are adjusted.

[0415] Alternatively, the image fusion network can be trained in other ways as follows:

[0416] In response to obtaining the input conditions of two adjacent time steps, based on the input conditions of these two adjacent time steps, the reward value of the repaired image in the previous time step, and the fusion strategy of the previous time step, a set of state information corresponding to the target time step is constructed, and the constructed set of state information corresponding to the target time step is cached in a predetermined cache space. The target time step is the previous time step among the two adjacent time steps; accordingly,

[0417] In response to satisfying the loss calculation conditions, based on the calculated Q-values ​​corresponding to the input conditions at each time step and the reward values ​​of the repaired images at each time step, the losses of the performer network and the critic network are calculated, including:

[0418] In response to the fulfillment of the loss calculation conditions, a specified number of state information sets are selected from the predetermined cache space. The loss of the commentator network is calculated based on the Q-value corresponding to the input conditions of the target time step in each selected state information set, the reward value of the repaired image at the target time step, and the Q-value corresponding to the input conditions of the next time step in each selected state information set. The loss of the performer network is calculated based on the Q-value corresponding to the fusion strategy at the target time step.

[0419] Optionally, the third sample image is an image acquired that requires image restoration;

[0420] And / or,

[0421] Based on the fusion strategy corresponding to the current time step of the third sample image, the input conditions for the current time step of the commentator network are constructed, including:

[0422] Based on the fusion strategy corresponding to the third sample image at the current time step, the current intermediate results, the measurement results of the current time step regarding the first task, the prediction results regarding the second task, and the depth map of the third sample image and the de-degradation result of the third sample image, the input conditions of the commentator network at the current time step are constructed.

[0423] And / or,

[0424] Based on the fusion result at the current time step, the restored image at the current time step is decoded, including:

[0425] Based on the fusion result at the current time step, the initial diffusion intermediate result corresponding to the current third sample image is inferred, and based on the inferred initial diffusion intermediate result, the repaired image at the current time step is decoded.

[0426] Optionally, the reward value of the repaired image at the current time step is calculated, including:

[0427] Determine the facial features in the repair image at the current time step corresponding to the third sample image, and determine the first type of reward value based on the similarity between the determined facial features and the facial features in the first specified image; the first specified image is an image that contains the object in the third sample image and does not have a repair requirement;

[0428] The first major model is invoked to perform anomaly detection on the repaired image at the current time step corresponding to the third sample image, and the second type of reward value is determined based on the anomaly detection results output by the first major model.

[0429] The second major model is invoked to perform content consistency analysis on the repaired image at the current time step corresponding to the third sample image and the second specified image, and the third type of reward value is determined based on the analysis results of the second major model.

[0430] Based on the first type of reward value, the second type of reward value, and the third type of reward value, the reward value of the repaired image at the current time step is determined.

[0431] This application also provides an electronic device, such as... Figure 13 As shown, it includes:

[0432] Memory 1301 is used to store computer programs;

[0433] The processor 1302 is used to implement any image restoration method when executing a program stored in the memory 1301.

[0434] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 1302, the communication interface, and the memory 1301 communicating with each other via the communication bus.

[0435] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0436] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0437] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0438] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0439] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described image restoration methods.

[0440] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the image restoration methods described above.

[0441] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state disk (SSD), etc.

[0442] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0443] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0444] The above are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. An image restoration method, characterized in that, The method includes: Obtain the target image to be repaired; A pre-trained initial insulation network is invoked to perform image de-degradation processing on the target image, thereby obtaining the de-degradation result of the target image; The de-degradation result is repaired and optimized according to the predetermined repair and optimization method to obtain the image restoration result of the target image; The predetermined repair and optimization methods include: The de-degradation result is processed based on a pre-trained diffusion network used to perform the first and second tasks, and the prediction results for the first task and the prediction results for the second task obtained by the diffusion network when processing the de-degradation result of the target image are used as the content to be fused, and fusion processing is performed by a pre-trained image fusion network. The first task is to perform quality optimization on the image input to the diffusion network to output an optimized prediction result; the second task is to output a prediction result that retains the identity information in the input image on the image input to the diffusion network. The training methods for the diffusion network include: The second sample image is processed using the trained initial inpainting network to obtain the de-degradation result of the second sample image. According to the predetermined processing method, the de-degradation result of the second sample image is subjected to prediction processing for the first task and prediction processing for the second task through the diffusion network to obtain the prediction results for the first task and the prediction results for the second task at each time step. Based on the prediction results for the first task at each time step and the ground truth image of the second sample image, the loss of the diffusion network for the first task is calculated; and based on the prediction results for the second task at each time step and the ground truth image of the second sample image, the loss of the diffusion network for the second task is calculated. Based on the loss of the diffusion network with respect to the first task and the loss with respect to the second task, it is determined whether the diffusion network has converged. If it has not converged, the network parameters of the diffusion network are adjusted. The predetermined processing method includes: for each time step, based on the intermediate results of that time step, the learnable parameters set for the diffusion network, the de-degradation result of the second sample image, and the set second reference cue, constructing the input content required for the first task at that time step, and inputting the constructed input content into the diffusion network to perform the first task; wherein, the second reference cue is used to enable the diffusion network to understand the level of detail of the initial repair network in de-degrading the second sample image; Based on the intermediate results of this time step, the learnable parameters set for the diffusion network, the depth map of the de-degradation result of the second sample image, and the identity features and facial description information of the object in the de-degradation result of the second sample image, the input content required for the second task at this time step is constructed, and the constructed input content is input into the diffusion network to perform the second task.

2. The method according to claim 1, characterized in that, The process of repairing and optimizing the de-degradation result according to a predetermined repair and optimization method to obtain the image restoration result of the target image includes: Using the first time step as the current time step, based on the intermediate results of the current time step and the de-degradation results of the target image, the input content of the diffusion network for the current time step regarding the first task and the input content regarding the second task are constructed. The constructed input content is then input into the diffusion network to obtain the prediction results of the first task and the second task for the current time step. Based on the prediction results of the first task and the second task at the current time step, the image fusion network is invoked so that the image fusion network performs result fusion processing on the prediction results of the first task and the second task at the current time step to obtain the fusion result at the current time step. In response to the current time step being the last time step, based on the obtained fusion result of the current time step, the repaired image of the current time step is decoded and used as the image repair result of the target image; otherwise, based on the obtained fusion result of the current time step, the intermediate result of the next time step of the current time step is calculated and the next time step is used as the new current time step. The prediction results of the first task and the prediction results of the second task based on the obtained current time step are returned, and the steps of the image fusion network are invoked.

3. The method according to claim 1 or 2, characterized in that, The initial repair network is trained based on the first sample image and the ground truth image of the first sample image. The ground truth image of any sample image is the image that represents the expected repair result of that sample image. The first sample image is: the image obtained after image degradation processing of a predetermined initial image; the true image of the first sample image is: the image obtained by low-pass filtering the image spectrum of the initial image according to the target frequency domain threshold set for the first sample image. Accordingly, the invocation of the pre-trained initial instigation network to perform image de-degradation processing on the target image to obtain the de-degradation result of the target image includes: Based on a specified frequency domain threshold, the initial repair network is invoked to perform image de-degradation processing on the target image based on the specified frequency domain threshold, thereby obtaining the de-degradation result of the target image.

4. The method according to claim 3, characterized in that, The training methods for the initial repair network include: The vector features of the first sample image and the target frequency domain threshold are input into the initial repair network, so that the initial repair network uses the vector features of the target frequency domain threshold as a first reference prompt to perform image de-degradation processing on the first sample image, and obtains the de-degradation result of the first sample image. Based on the de-degradation result of the first sample image and the ground truth image of the first sample image, the loss of the initial repair network is calculated; In response to the loss of the initial repair network indicating that the initial repair network has not converged, the network parameters of the initial repair network are adjusted. Accordingly, the step of invoking the initial repair network based on a specified frequency domain threshold to perform image de-degradation processing on the target image based on the specified frequency domain threshold, and obtaining the de-degradation result of the target image, includes: The target image and the vector features of the specified frequency domain threshold are input into the initial repair network, so that the initial repair network uses the vector features of the specified frequency domain threshold as a first reference prompt to perform image de-degradation processing on the target image, thereby obtaining the de-degradation result of the target image.

5. The method according to claim 4, characterized in that, The initial repair network includes a generator and a discriminator; The step of inputting the vector features of the first sample image and the target frequency domain threshold into the initial restoration network, so that the initial restoration network uses the vector features of the target frequency domain threshold as a first reference cue to perform image de-degradation processing on the first sample image, and obtaining the de-degradation result of the first sample image, includes: The vectors of the first sample image and the target frequency domain threshold are input into the generator in the initial repair network, so that the generator in the initial repair network uses the vector features of the target frequency domain threshold as a first reference prompt to perform image de-degradation processing on the first sample image to obtain the de-degradation result of the first sample image. The training method for the initial repair network also includes: Based on the de-degradation result of the first sample image and the ground truth image of the first sample image, the input content of the discriminator in the initial repair network is constructed; The input content of the constructed discriminator is input into the discriminator so that the discriminator performs discrimination processing on the received input content and obtains the discrimination result of the first sample image; The calculation of the loss of the initial instigation network based on the de-degradation result of the first sample image and the ground truth image of the first sample image includes: Based on the difference between the de-degradation result of the first sample image and the ground truth image of the first sample image, loss is calculated to obtain the first type of loss; Based on the discrimination result of the first sample image, loss is calculated to obtain the second type of loss; The loss of the initial repair network is determined based on the first type of loss and the second type of loss of the initial repair network.

6. The method according to claim 5, characterized in that, The input content for constructing the discriminator in the initial inpainting network based on the de-degradation result of the first sample image and the ground truth image of the first sample image includes: Based on the learnable parameters set for the discriminator, the target frequency domain threshold is processed to obtain the prior features corresponding to the target frequency domain threshold; The prior features corresponding to the target frequency domain threshold and the facial region features of the de-degradation result of the first sample image are concatenated to obtain the first concatenated feature corresponding to the first sample image. The prior features corresponding to the target frequency domain threshold and the facial region features of the ground truth image of the first sample image are concatenated to obtain the second concatenated feature corresponding to the first sample image. The step of inputting the constructed discriminator's input content into the discriminator, so that the discriminator performs discrimination processing on the received input content to obtain the discrimination result of the first sample image, includes: The first stitching feature corresponding to the first sample image is input into the discriminator, so that the discriminator performs discrimination processing on the received first stitching feature to obtain the discrimination result of the first sample image with respect to the first stitching feature; The second stitching feature of the first sample image is input into the discriminator, so that the discriminator performs discrimination processing on the received second stitching feature to obtain the discrimination result of the first sample image with respect to the second stitching feature; The loss calculation based on the discrimination result of the first sample image yields a second type of loss, including: Based on the discrimination results of the first sample image regarding the first stitching feature and the discrimination results of the first sample image regarding the second stitching feature, loss is calculated to obtain the second type of loss.

7. The method according to claim 6, characterized in that, The discrimination result based on the first sample image and the discrimination result based on the second stitching feature of the first sample image are used to calculate the loss, resulting in a second type of loss, including: From the discrimination results of the first sample image regarding the first stitching feature, extract the result content of the prior feature corresponding to the target frequency domain threshold, and score based on the extracted result content to obtain the first discrimination score of the first sample image; From the discrimination results of the first sample image regarding the second stitching features, extract the result content of the prior features corresponding to the target frequency domain threshold, and score based on the extracted result content to obtain the second discrimination score of the first sample image; Based on the first discriminant score of the first sample image, the adversarial loss is determined; The discriminator loss is determined based on the first and second discriminant scores of the first sample image.

8. The method according to claim 1, characterized in that, The calculation of the loss of the diffusion network with respect to the first task, based on the prediction results for the first task at each time step and the ground truth image of the second sample image, includes: For each time step, the loss for the first task is calculated based on the prediction result for the first task at that time step, the intermediate result for that time step, the time parameters for that time step, and the latent space features of the ground truth image of the second sample image. Based on the loss of the first task at each time step, the loss of the diffusion network with respect to the first task is determined; And / or, The calculation of the loss of the diffusion network with respect to the second task, based on the prediction results for the second task at each time step and the ground truth image of the second sample image, includes: For each time step, the loss for the second task is calculated based on the prediction result for the second task at that time step, the intermediate result at that time step, the time parameters at that time step, and the latent space features of the reference image corresponding to the second sample image. Based on the loss of the second task at each time step, the loss of the diffusion network with respect to the second task is determined; The method for obtaining the reference image corresponding to the second sample image includes: For each of the multiple frequency domain thresholds set, the vector features of the second sample image and the frequency domain threshold are input into the trained initial repair network to obtain the de-degradation result under that frequency domain threshold. From the de-degradation results obtained at each frequency domain threshold, select an image that satisfies the similarity condition between the de-degradation result of the second sample image and the ground truth image of the second sample image, and obtain the reference image corresponding to the second sample image.

9. The method according to claim 1, characterized in that, The diffusion network includes a backbone network; The training methods for the diffusion network also include: For each time step, the output data obtained by the backbone network processing at least one input content at that time step is acquired; wherein, the at least one input content is at least one of the input content required for the first task and the input content required for the second task; For the obtained output data, the data corresponding to the learnable parameters of the diffusion network is extracted to obtain facial output data, and the data corresponding to the intermediate results of the time step is extracted to obtain backbone output data; the obtained facial output data is linearly processed to obtain the mean and variance of the time step, and the obtained backbone output data is converted into the intermediate image of the time step; and, based on the obtained variance and mean, the loss with respect to the facial output is calculated, and based on the obtained variance, mean, and intermediate image, the loss with respect to the backbone network is calculated. The step of determining whether the diffusion network has converged based on the loss of the first task and the loss of the second task includes: Based on the loss of the diffusion network with respect to the first task, the loss with respect to the second task, and the calculated loss with respect to the backbone network and the loss with respect to the facial output, it is determined whether the diffusion network has converged.

10. The method according to claim 1 or 2, characterized in that, The image fusion network is a reinforcement learning network, which includes a performer network and a critic network; The methods for training the image fusion network include: Obtain the third sample image; Using the first time step as the current time step, and based on the intermediate results of the current time step and the third sample image, the input content of the trained diffusion network for the current time step is constructed. The constructed input content is then input into the diffusion network to obtain the prediction result of the current time step corresponding to the third sample image. Based on the prediction result of the current time step corresponding to the third sample image, the performer network is invoked to generate a fusion strategy for the current time step corresponding to the third sample image; Based on the generated fusion strategy, the prediction results of the current time step corresponding to the third sample image are fused to obtain the fusion result of the current time step corresponding to the third sample image. Based on the fusion result of the current time step corresponding to the third sample image, the repaired image of the current time step corresponding to the third sample image is decoded and the reward value of the repaired image of the current time step is calculated; and based on the fusion strategy of the current time step corresponding to the third sample image, the input conditions of the commentator network at the current time step are constructed and the constructed input conditions are input into the commentator network to obtain the Q value corresponding to the input conditions at the current time step. If the current time step is not the last time step, then based on the fusion result of the current time step corresponding to the third sample image, calculate the intermediate result of the next time step of the current time step, and take the next time step as the new current time step, and return the step of constructing the input content of the current time step of the diffusion network based on the intermediate result of the current time step and the third sample image; or, if the current time step is the last time step, return the step of obtaining the third sample image. In response to the fulfillment of the loss calculation conditions, the loss of the performer network and the loss of the critic network are calculated based on the Q-value corresponding to the input conditions at each time step and the reward value of the repaired image at each time step. If the reinforcement learning network fails to converge based on the calculated losses of the performer network and the critic network, the network parameters of the reinforcement learning network are adjusted.

11. The method according to claim 10, characterized in that, The method of training the image fusion network also includes: In response to obtaining the input conditions of two adjacent time steps, a state information set corresponding to the target time step is constructed based on the input conditions of the two adjacent time steps, the reward value of the repaired image of the previous time step, and the fusion strategy of the previous time step. This constructed state information set corresponding to the target time step is then cached in a predetermined cache space. The target time step is the previous time step among the two adjacent time steps. Accordingly, In response to satisfying the loss calculation conditions, based on the calculated Q-values ​​corresponding to the input conditions at each time step and the reward values ​​of the repaired images at each time step, the loss of the performer network and the loss of the critic network are calculated, including: In response to the fulfillment of the loss calculation conditions, a specified number of state information sets are selected from the predetermined cache space. The loss of the critic network is calculated based on the Q-value corresponding to the input conditions of the target time step in each selected state information set, the reward value of the repaired image at the target time step, and the Q-value corresponding to the input conditions of the next time step in each selected state information set. The loss of the performer network is also calculated based on the Q-value corresponding to the fusion strategy at the target time step.

12. The method according to claim 10, characterized in that, The third sample image is an image acquired that requires image restoration. And / or, The fusion strategy based on the current time step corresponding to the third sample image, which constructs the input conditions for the current time step of the commentator network, includes: Based on the fusion strategy of the current time step corresponding to the third sample image, the current intermediate result, the measurement result of the current time step regarding the first task, the prediction result regarding the second task, and the depth map of the third sample image and the de-degradation result of the third sample image, the input conditions of the commentator network for the current time step are constructed. And / or, The process of decoding the restored image at the current time step based on the fusion result includes: Based on the fusion result at the current time step, the initial diffusion intermediate result corresponding to the current third sample image is inferred, and based on the inferred initial diffusion intermediate result, the repaired image at the current time step is decoded.

13. The method according to claim 10, characterized in that, The reward value of the repaired image at the current time step obtained by calculation includes: The facial features in the repair image at the current time step corresponding to the third sample image are determined. Based on the similarity between the determined facial features and the facial features in the first specified image, a first type of reward value is determined. The first specified image is an image that contains the object of the third sample image and does not have a repair requirement. The first major model is invoked to perform anomaly detection on the repaired image at the current time step corresponding to the third sample image, and the second type of reward value is determined based on the anomaly detection result output by the first major model. The second major model is invoked to perform content consistency analysis on the repaired image at the current time step corresponding to the third sample image and the second specified image, and the third type of reward value is determined based on the analysis results of the second major model; wherein, the second specified image is the de-degradation result of the third sample image; Based on the first type of reward value, the second type of reward value, and the third type of reward value, the reward value of the repaired image at the current time step is determined.

14. An image restoration device, characterized in that, The device includes: The acquisition module is used to acquire the target image to be repaired; The calling module is used to call the pre-trained initial instigation network to perform image de-degradation processing on the target image and obtain the de-degradation result of the target image; The repair module is used to repair and optimize the de-degradation result according to a predetermined repair and optimization method to obtain the image repair result of the target image; The predetermined repair and optimization methods include: The de-degradation result is processed based on a pre-trained diffusion network used to perform the first and second tasks, and the prediction results for the first task and the prediction results for the second task obtained by the diffusion network when processing the de-degradation result of the target image are used as the content to be fused, and fusion processing is performed by a pre-trained image fusion network. The first task is to perform quality optimization on the image input to the diffusion network to output an optimized prediction result; the second task is to output a prediction result that retains the identity information in the input image on the image input to the diffusion network. The training methods for the diffusion network include: The second sample image is processed using the trained initial inpainting network to obtain the de-degradation result of the second sample image. According to the predetermined processing method, the de-degradation result of the second sample image is subjected to prediction processing for the first task and prediction processing for the second task through the diffusion network to obtain the prediction results for the first task and the prediction results for the second task at each time step. Based on the prediction results for the first task at each time step and the ground truth image of the second sample image, the loss of the diffusion network for the first task is calculated; and based on the prediction results for the second task at each time step and the ground truth image of the second sample image, the loss of the diffusion network for the second task is calculated. Based on the loss of the diffusion network with respect to the first task and the loss with respect to the second task, it is determined whether the diffusion network has converged. If it has not converged, the network parameters of the diffusion network are adjusted. The predetermined processing method includes: for each time step, based on the intermediate results of that time step, the learnable parameters set for the diffusion network, the de-degradation result of the second sample image, and the set second reference cue, constructing the input content required for the first task at that time step, and inputting the constructed input content into the diffusion network to perform the first task; wherein, the second reference cue is used to enable the diffusion network to understand the level of detail of the initial repair network in de-degrading the second sample image; Based on the intermediate results of this time step, the learnable parameters set for the diffusion network, the depth map of the de-degradation result of the second sample image, and the identity features and facial description information of the object in the de-degradation result of the second sample image, the input content required for the second task at this time step is constructed, and the constructed input content is input into the diffusion network to perform the second task.

15. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-13.

Images