Image enhancement method, target recognition method, device and medium

Abstract

The application relates to an image enhancement method, a target identification method, equipment and a medium, and belongs to the technical field of computers. The method comprises the following steps: inputting a target image to be enhanced into a pre-trained image enhancement network, encoding the target image through a self-encoder in the image enhancement network, inputting obtained encoding features into a first diffusion model in an image enhancement model for enhancement, inputting fused encoding features and the obtained enhanced encoding features into a self-decoder in the image enhancement model, and obtaining an enhanced image corresponding to the target image; and the output result of the self-encoder is corrected based on a preset lookup table during training of the image enhancement network, and the image enhancement network is trained based on the corrected encoding result. The method can solve the problem that the enhancement effect of a traditional neural network model used for image enhancement on an image with more missing features is poor, improve the adaptability of the network to an extreme scene, and improve the enhancement effect of the target image.

Description

Technical Field

[0001] This application relates to an image enhancement method, a target recognition method, an apparatus, and a medium, belonging to the field of computer technology. Background Technology

[0002] Image enhancement technology refers to the processing of features in an image to improve its visual effect and enhance its quality. Image enhancement technology can be applied to scenarios such as indoor surveillance and checkpoint monitoring, and can improve the brightness of low-light images. With increasing demands for pre-defined quality, image enhancement technology generally also needs to restore detailed information such as color and texture of the image.

[0003] To improve image enhancement results, traditional image enhancement algorithms are based on neural network models, which include, but are not limited to, the following:

[0004] The first type is Convolutional Neural Networks (CNNs). During training, CNNs use low-quality images (images requiring image augmentation) as input and high-quality images (images not requiring augmentation) as training targets. A loss function is used for iterative training of the network. When performing image augmentation, the target image is input into the trained CNN, and the augmented image is output.

[0005] The second type is Generative Adversarial Networks (GANs). GANs use low-quality images as input and high-quality images as training targets, undergoing iterative training in an adversarial process between the generator and the discriminator. When performing image augmentation, the target image to be augmented is input into the trained generator, and the augmented image is output.

[0006] However, traditional neural network models used for image enhancement are less effective at enhancing images with many missing features, such as those with low contrast, missing or occluded targets. Summary of the Invention

[0007] This application provides an image enhancement method, a target recognition method, an apparatus, and a medium, which can improve the enhancement effect on images with significant feature loss. This application provides the following technical solutions:

[0008] Firstly, an image enhancement method is provided, the method comprising:

[0009] Acquire the target image to be enhanced;

[0010] The target image is input into a pre-trained image augmentation network. The target image is then encoded by the autoencoder in the image augmentation network. The resulting encoded features are then input into the first diffusion model in the image augmentation model for enhancement. The enhanced encoded features and the original encoded features are then fused and input into the autodecoder in the image augmentation model to obtain the enhanced image corresponding to the target image.

[0011] During training, the output of the autoencoder in the image enhancement network is corrected based on a preset lookup table, and the image enhancement network is trained based on the corrected encoding result.

[0012] Optionally, the training process of the image augmentation network includes:

[0013] Acquire a first image, which meets preset quality requirements;

[0014] Based on the first image and the preset lookup table, the initial autoencoder and the initial autodecoder are iteratively trained to learn the first model parameters of the autoencoder and the autodecoder, so as to obtain the trained autoencoder and the trained autodecoder.

[0015] Acquire a second image and a third image, wherein the quality of the second image is higher than that of the third image, and the quality of the second image meets the preset quality requirements;

[0016] With the first model parameters fixed, the initial diffusion model is iteratively trained based on the second image, the third image, and the preset lookup table to learn the second model parameters of the first diffusion model, thereby obtaining the first diffusion model.

[0017] Optionally, the step of fixing the first model parameters and iteratively training the initial diffusion model based on the second image, the third image, and the preset lookup table to learn the second model parameters of the first diffusion model and obtain the first diffusion model includes:

[0018] The second image and the third image are respectively input into the trained autoencoder to obtain the encoded data corresponding to the second image and the encoded data corresponding to the third image;

[0019] The pre-defined lookup table is used to correct the encoded data corresponding to the second image, resulting in corrected encoded data.

[0020] The encoded data corresponding to the third image and the number of diffusion steps t are input into the initial diffusion model to obtain the model output result; where t is a positive integer;

[0021] After fusing the model output and the encoded data corresponding to the third image, the result is input into the trained self-decoder to obtain the network output corresponding to the third image.

[0022] The encoded data corresponding to the second image is input into the trained self-decoder to obtain the network output result corresponding to the second image.

[0023] Based on the model output, the corrected encoded data, the network output corresponding to the second image, and the network output corresponding to the third image, the loss information of the initial diffusion model is determined.

[0024] The initial diffusion model is iteratively trained based on the loss information to obtain the second model parameters.

[0025] Optionally, determining the loss information of the initial diffusion model based on the model output, the corrected encoded data, the network output corresponding to the second image, and the network output corresponding to the third image includes:

[0026] Obtain the actual noise added during t-step forward diffusion using the encoded data corresponding to the third image as input;

[0027] Based on the corrected encoded data, determine the diffusion result at step t obtained when the initial diffusion model performs diffusion at step t;

[0028] Determine the model output result obtained after performing t-step reverse diffusion on the diffusion result of step t;

[0029] Based on the model output and the diffusion result at step t, determine the prediction noise added during forward diffusion;

[0030] A first loss value is determined based on the actual noise and the predicted noise;

[0031] Based on the network output result corresponding to the second image, the network output result corresponding to the third image, the intermediate result of the self-decoder when determining the network output result corresponding to the second image, and the intermediate result of the self-decoder when determining the network output result corresponding to the third image, a second loss value is determined;

[0032] The loss information is determined based on the first loss value and the second loss value.

[0033] Optionally, each second image corresponds to at least two third images;

[0034] Accordingly, determining the loss information based on the first loss value and the second loss value includes:

[0035] Obtain the diffusion results at step t corresponding to different third images;

[0036] Based on the at least two third images and the diffusion result at step t corresponding to each third image, the third loss value is determined;

[0037] The loss information is determined based on the first loss value, the second loss value, and the third loss value.

[0038] Optionally, after iteratively training the initial diffusion model based on the second image, the third image, and the preset lookup table to learn the second model parameters of the first diffusion model and obtain the first diffusion model, the method further includes:

[0039] Acquire a fourth image and a fifth image, wherein the quality of the fourth image is higher than that of the fifth image, and the quality of the fourth image meets the preset quality requirements;

[0040] The model parameters of the autoencoder and the second model parameters are fixed, and the trained autodecoder is iteratively trained based on the fourth image and the fifth image to adjust the model parameters of the autodecoder. The autodecoder is then determined based on the adjusted model parameters.

[0041] Optionally, acquiring the second and third images includes:

[0042] Obtain the second image;

[0043] The second image is degraded using a pre-trained degradation model to obtain the third image. The degradation model is trained using a set of degraded images, which includes images corresponding to different scenes to be enhanced.

[0044] Optionally, the pre-trained degradation model includes a second diffusion model, wherein the degradation images used by the second diffusion model are obtained by image acquisition based on the different image enhancement scenarios to be performed.

[0045] Optionally, the pre-trained degradation model includes a recurrent consistent generative adversarial network, wherein the target in the high-quality image in the degradation image set used by the recurrent consistent generative adversarial network is the same as or different from the target in the degradation image.

[0046] Optionally, acquiring the second and third images includes:

[0047] Obtain the second image;

[0048] For each second image, a superimposed degradation operation is performed on the second image according to at least two preset degradation strategies and the superposition probability corresponding to each degradation strategy;

[0049] If the number of times the second image is overlaid and degraded reaches a preset number, the third image is output.

[0050] Optionally, the method further includes:

[0051] The teacher model is determined based on the first diffusion model;

[0052] The teacher model is used to guide a pre-created continuous-time student model to perform distillation training, resulting in a trained student network. The student network is used to sample noise during the calculation of the first diffusion model, and the sampled noise data is used for back diffusion to obtain the enhanced encoded features.

[0053] The continuous-time student model has learnable parameters, which are used to match the output of the teacher model at any time step.

[0054] Optionally, the first diffusion model is built on a U-Net neural network, which includes a shrinking path and an expanding path. The shrinking path and the expanding path include at least two residual modules, and each residual module includes a first convolutional module and a second convolutional module connected in sequence.

[0055] The diffusion step t currently input to the first diffusion model is converted into scalar parameters after the embedding transformation of the time feature matrix, and then fused with the output of the first convolution module. The fused result is then input to the second convolution module.

[0056] Optionally, after inputting the target image into a pre-trained image augmentation network to obtain the augmented image corresponding to the target image, the method further includes:

[0057] The enhanced image is input into a pre-trained target recognition network to obtain the target recognition result.

[0058] Optionally, before inputting the enhanced image into the pre-trained target recognition network, the method further includes:

[0059] Obtain the fifth image and the target label corresponding to the fifth image;

[0060] The fifth image is input into the image enhancement network to obtain the enhanced fifth image;

[0061] The enhanced fifth image and the target label are used to learn the model parameters of the target recognition network to obtain the target recognition network.

[0062] Secondly, a target recognition method is provided, the method comprising:

[0063] Acquire the image to be recognized;

[0064] The image to be identified is input into a pre-trained target recognition network to obtain the target recognition result corresponding to the image to be identified;

[0065] The target recognition network is trained using the enhanced fifth image and the target label corresponding to the fifth image after the image enhancement network enhances the fifth image. The image enhancement network includes an autoencoder, a first diffusion model, and a self-decoder connected in sequence. During training, the output of the autoencoder is corrected based on a preset lookup table, and the image enhancement network is trained based on the corrected encoding result.

[0066] Thirdly, an electronic device is provided, the device including a processor and a memory; the memory stores a program, the program being loaded and executed by the processor to implement the image enhancement method provided in the first aspect; or the target recognition method provided in the second aspect.

[0067] Fourthly, a computer-readable storage medium is provided, wherein a program is stored therein, which, when executed by a processor, is used to implement the image enhancement method provided in the first aspect; or the target recognition method provided in the second aspect.

[0068] The beneficial effects of this application include at least the following: by encoding the target image to obtain encoded features and then inputting them into a first diffusion model for enhancement, and then generating an enhanced image based on the enhanced encoded features, on the one hand, since the diffusion model has better enhancement performance than neural networks such as CNN or GAN, the introduction of the first diffusion model into the network enhancement model can improve the enhancement effect of the target image; on the other hand, since the input of the traditional diffusion model is a low-quality image, and the diffusion model does not have feature extraction capabilities, it may not be able to predict the corresponding residual image for denoising for target images with many missing features. However, in this embodiment, after encoding the target image using an autoencoder, the encoded features are input into the first diffusion model. Even for images with many missing features, the first diffusion model can obtain more image features after processing by the encoder, thereby improving the adaptability of the first diffusion model to extreme scenarios (i.e., scenarios where many images with many missing features are collected), and thus improving the adaptability of the entire network to extreme scenarios; it can solve the problem that the traditional neural network models used for image enhancement have poor enhancement effects on images with many missing features, and improve the enhancement effect of the target image.

[0069] In addition, by introducing a pre-set lookup table during network training, the encoding features of the encoder can be corrected. Since the pre-set lookup table is set based on the target features of the target to be identified, the autoencoder can learn model parameters that conform to the target features during network training. The first diffusion model and the autodecoder can be trained based on the encoded data that conforms to the target features, thereby improving the rationality of the entire network output.

[0070] Meanwhile, since the lookup process of the preset lookup table is relatively slow, and the trained autoencoder can output encoded features that conform to the target features, the image enhancement efficiency can be improved while ensuring the rationality of the network by not having the encoded features output by the encoder corrected by the preset lookup table during the image enhancement process.

[0071] In addition, when establishing the initial diffusion model and the first diffusion model, setting the calculation process to convert the time feature matrix vector into a scalar parameter and fuse it with the output of the first convolution model can reduce the amount of computation and improve the model calculation speed.

[0072] In addition, the computation speed of the model can be further improved by modifying and / or simplifying the model.

[0073] In addition, by using a pre-defined lookup table during the training of the autoencoder, the first diffusion model, and the self-decoder, the overall accuracy of the network in recovering target features can be improved, thus enhancing the accuracy of image enhancement.

[0074] In addition, when determining the loss information corresponding to the initial diffusion model, it is necessary to combine not only the model output results but also the output results of the entire network, which can improve the model performance of the diffusion model.

[0075] In addition, incorporating consistency loss during the training of the first diffusion model can prevent large differences in the enhancement results of different degraded images corresponding to the same image, which could lead to low accuracy in subsequent target recognition and improve the target recognition effect on the enhanced image.

[0076] In addition, by using a degradation model to acquire degraded images during the training of the first diffusion model, the efficiency of acquiring degraded images can be improved. On the other hand, since the degradation model can simulate various degradation factors in real-world scenes, the realism of degraded images can also be improved.

[0077] Furthermore, by building a degradation model based on a diffusion model, the realism of the degraded image can be improved because the diffusion model can learn the actual degradation factors. Alternatively, a degradation model built using a recurrent consistency generative adversarial network (GAN) does not require setting pairs of high-quality and low-quality images, thus reducing the complexity of obtaining training data and improving training efficiency. Alternatively, using a probability stacking method to degrade high-quality images does not require pre-training of the neural network model, thus improving degradation efficiency.

[0078] In addition, by fine-tuning the model parameters of the self-decoder after obtaining the trained self-decoder, the problem of a large gap between the high-quality image output by the self-decoder and the original low-quality image, which would lead to low accuracy in subsequent target recognition, can be prevented, thereby improving the accuracy of target recognition on the enhanced image.

[0079] Furthermore, by retraining the target recognition network using the enhanced images, the applicability of the recognition network to the enhanced images can be improved, thereby increasing recognition accuracy.

[0080] In addition, by using the trained student network to sample noise during the first diffusion model calculation, the sampling efficiency of the diffusion model can be improved, and the model calculation speed can be increased.

[0081] In addition, both the enhanced image to be recognized and the image to be recognized without image enhancement model processing can be input into the target recognition network. This can improve the versatility of the target recognition network while ensuring its target recognition accuracy.

[0082] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, the preferred embodiments of this application are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0083] Figure 1 This is a flowchart of an image enhancement method provided in one embodiment of this application;

[0084] Figure 2 This is a schematic diagram illustrating the calculation principle of a diffusion model provided in one embodiment of this application;

[0085] Figure 3 This is a schematic diagram of an image enhancement process provided in one embodiment of this application;

[0086] Figure 4 This is a flowchart of a training method for an image enhancement model provided in one embodiment of this application;

[0087] Figure 5 This is a flowchart of a target recognition method provided in one embodiment of this application;

[0088] Figure 6 This is a block diagram of an image enhancement apparatus provided in one embodiment of this application;

[0089] Figure 7 This is a block diagram of a target recognition device provided in one embodiment of this application;

[0090] Figure 8 This is a block diagram of a training apparatus for an image enhancement model provided in one embodiment of this application;

[0091] Figure 9 This is a block diagram of an electronic device provided in one embodiment of this application. Detailed Implementation

[0092] The specific embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate this application, but are not intended to limit the scope of this application.

[0093] Optionally, this application uses the image enhancement methods provided in various embodiments in electronic devices as examples for illustration. The electronic device is a device with computing capabilities, such as a terminal or server. The terminal can be a mobile phone, computer, tablet computer, scanner, electronic eye, surveillance camera, etc. This embodiment does not limit the type of electronic device.

[0094] Figure 1 This is a flowchart of an image enhancement method provided in one embodiment of this application, which includes at least the following steps:

[0095] Step 101: Obtain the target image to be enhanced.

[0096] The target image to be enhanced refers to the image that is to be input into the image enhancement model.

[0097] In one example, the target image may be captured by an electronic device (which has image capture capability) or sent by another device; the target image may be a frame in a video stream or a single image. This embodiment does not limit the implementation method of the target image.

[0098] In another example, the target image is obtained by extracting the target region from the original image acquired by the electronic device. Since the enhanced image is typically used for target recognition, and the target to be identified usually only occupies a portion of the original image, extracting the target region from the original image removes redundant parts, reduces the computational load of the image enhancement model, and improves its accuracy. In this case, the extracted target image includes the target to be identified.

[0099] Optionally, in this application, the target to be identified includes, but is not limited to, the face region, the head and shoulder region, the whole vehicle region, or the license plate region, etc. This embodiment does not limit the implementation method of the target to be identified.

[0100] For example, an electronic device extracts a 256*256 sRGB image from the original image, thus obtaining the target image. In actual implementation, the size of the target image can also be smaller or larger; this embodiment does not limit the size of the target image.

[0101] Methods for extracting target regions include, but are not limited to:

[0102] The first method involves determining the target region based on a preset location in the original image, and then cropping the target region to obtain the target image. In some scenarios, the location of the target in the original image is relatively fixed. For example, in a vehicle checkpoint, the target usually appears in the central area of ​​the original image. Therefore, determining the target region based on a preset location can ensure both the accuracy and efficiency of the extracted target image.

[0103] The preset position is determined based on the fixed position of the target to be identified in the original image. The preset position can be the center point of the image, the center of the upper edge of the image, or the lower edge of the image, etc. This embodiment does not limit the implementation method of the preset position.

[0104] The second method involves receiving a selection operation on a target region in the original image, cropping the selected region, and obtaining the target image. The selection operation includes, but is not limited to, selecting a target region or moving a cropping window of a fixed size. This embodiment does not limit the implementation of the selection operation. In this case, by having the user select the target region of the original image, the accuracy of target region extraction can be improved.

[0105] The third method involves performing target detection on the original image to obtain the target region, and then cropping the target region to obtain the target image.

[0106] This embodiment does not list all the methods for extracting the target region.

[0107] Optionally, the target image is an image that does not meet the preset quality requirements. If target recognition is required after the target image is enhanced, the preset quality requirements can be determined based on the recognition capability of the target recognition network. That is, images that meet the preset quality requirements can be recognized by the target recognition network, while images that do not meet the preset quality requirements cannot be recognized by the target recognition network.

[0108] In other embodiments, the preset quality requirements can also be determined based on user settings. This embodiment does not limit the setting method of the preset quality requirements.

[0109] The preset quality requirements include, but are not limited to, at least one of the following: image brightness is greater than a brightness threshold; image resolution is greater than a resolution threshold; image sharpness is greater than a sharpness threshold; image noise level is lower than a preset noise level. In other embodiments, the preset quality requirements may also be other requirements determined based on parameters that affect image quality. This embodiment does not limit the specific content of the preset quality requirements.

[0110] At this time, the electronic device can compare the current image with the preset quality requirements. If the preset quality requirements are met, the current image is processed as the target image; if the preset quality requirements are not met, the current image is used as the target image to trigger step 102.

[0111] In practice, electronic devices may also avoid comparing the current image with preset quality requirements and instead directly process the current image as the target image. In this case, there is no need to set preset quality requirements in the electronic device.

[0112] Step 102: Input the target image into a pre-trained image augmentation network. After the target image is encoded by the autoencoder in the image augmentation network, the obtained encoded features are input into the first diffusion model in the image augmentation model for enhancement. The enhanced encoded features and the fused encoded features are then input into the autodecoder in the image augmentation model to obtain the enhanced image corresponding to the target image.

[0113] In this process, the output of the autoencoder is corrected based on a preset lookup table during training, and the image enhancement network is trained based on the corrected encoding result.

[0114] The diffusion model (including the first diffusion model in this embodiment) is a process of gradually denoising data from purely noisy data through neural network learning. (Refer to...) Figure 2 A diffusion model typically includes two diffusion processes:

[0115] 1. Fixed forward diffusion process: In this step, Gaussian noise is gradually added to the input data x0 until a purely noisy image x is obtained. T ;

[0116] 2. Learnable reverse diffusion process: In this step, the image is gradually denoised from the pure noise until the image x0 output by the model is obtained.

[0117] Specifically, for a T-step diffusion model, the index of each step is t. During forward diffusion, starting from the data input to the diffusion model, some Gaussian noise is randomly generated at each step. This generated noise is then gradually added to the input data. When T is sufficiently large, the resulting noisy image approximates a Gaussian noise image, for example, in Denoising Diffusion Probabilistic Models (DDPM), where T = 1000. During backward diffusion, from the noisy image x... T Initially (training uses real images with added noise, sampling uses random noise), x is learned through a neural network. T-1 To x T The noise is added, and then the final image is obtained by gradually removing the noise.

[0118] In traditional diffusion models, the image to be denoised is typically input into the model. However, if the image was captured in extreme environments, its features may be severely lacking, resulting in poor denoising performance from the diffusion model.

[0119] In this embodiment, the target image is not directly input into the first diffusion model. Instead, the target image is first encoded, and the resulting encoded features are input into the first diffusion model. At this time, the first diffusion model fits the residual of the target image in the feature domain, which can improve the adaptability to extreme environments.

[0120] Specifically, refer to Figure 3 The computational process of the image enhancement model shown is as follows: The target image I is fed into an autoencoder to obtain the encoded features I. z ; Encode feature I z Random noise coding N t The current diffusion step number t (initialized to the total number of diffusion steps T) is fed into the first diffusion model to obtain the noise feature z added at each step of the diffusion process at the current diffusion step number. t Based on z t The residual image N obtained from the current inverse diffusion is calculated. t-1 Determine if t equals 1; if not, then set N. t Replace with N t-1 , t = t-1, encoding feature I z The data remains unchanged and is fed into the first diffusion model network for repeated diffusion to obtain the residual image N corresponding to each diffusion. t -1; if so, then z+N0 is fed into the self-decoder to obtain the enhanced image I'.

[0121] The residual image N predicted by each inverse diffusion t-1 The calculation method is expressed by the following formula:

[0122]

[0123]

[0124] Where, when t = T, N t The noise matrix is ​​Gaussian; the current t is less than T, N t This is the residual image of the previous inverse diffusion output; σ t For hyperparameters related to diffusion step t, This represents the variance parameter of the t-th diffusion step. When t is greater than 1, z is a random Gaussian matrix. When t = T, ..., 2, in, This represents a standard normal distribution; when t = 1, z = 0; β represents the diffusion coefficient at the t-th diffusion step. t z represents the Gaussian distribution parameters for the t-th diffusion step. t This represents the noise feature added at each step during the diffusion process, up to the current diffusion step number.

[0125] Optionally, if the target image is an image extracted from the original image, then after obtaining the enhanced image, the enhanced image can be pasted back into the target area corresponding to the original image.

[0126] In one example, when pasting the enhanced image back into the original image, the color difference and / or brightness difference between the enhanced image and the original image can be compared; if the color difference and / or brightness difference is greater than a preset threshold, the color and / or brightness of the enhanced image are corrected to reduce the color difference and / or brightness difference.

[0127] for example: Figure 3 In the process, the enhanced image output from the decoder is input to the color correction module to correct the color and / or brightness of the enhanced image, resulting in a corrected image.

[0128] Based on the above, the enhanced image output by the image augmentation network is generally used for target recognition. However, for target images with many missing features, the encoded features output by the encoder may deviate significantly from the target features to be recognized. This leads to poor reliability of the image augmentation network's output.

[0129] In this embodiment, the image enhancement network comprises three sequentially connected parts: an autoencoder, a first diffusion model, and a self-decoder. During network training, a preset lookup table is introduced to correct the encoder's encoding features. Since the preset lookup table is set based on the target features of the target to be identified, the autoencoder can learn model parameters that conform to the target features during network training. The first diffusion model and the self-decoder can be trained based on encoded data that conforms to the target features, thereby improving the rationality of the entire network's output.

[0130] Generally, the lookup process of a pre-defined lookup table is relatively slow, while a trained autoencoder can output encoded features that match the target features. Therefore, in this embodiment, during image enhancement, the encoded features output by the encoder are no longer corrected by the pre-defined lookup table, which can improve image enhancement efficiency while ensuring network rationality. In other words, in this embodiment, the pre-defined lookup table is only used during network training and does not exist during image enhancement.

[0131] Schematic illustration: The preset lookup table can be implemented as a codebook structure, pre-stored with feature data of the target features. The specific process of training an image augmentation network using the preset lookup table is described in the following embodiment, which will not be repeated here.

[0132] In summary, the image enhancement method provided in this embodiment encodes the target image to obtain coded features, inputs these features into a first diffusion model for enhancement, and then generates an enhanced image based on these enhanced coded features. On the one hand, since the diffusion model has better enhancement performance compared to neural networks such as CNN or GAN, introducing the first diffusion model into the network enhancement model can improve the enhancement effect of the target image. On the other hand, since the input of traditional diffusion models is a low-quality image, and the diffusion model does not have feature extraction capabilities, it may not be able to predict the corresponding residual image for denoising target images with many missing features. However, in this embodiment, after encoding the target image using an autoencoder, the coded features are input into the first diffusion model. Even for images with many missing features, the first diffusion model can acquire more image features after processing by the encoder, thereby improving the adaptability of the first diffusion model to extreme scenarios (i.e., scenarios where many images with many missing features are collected), and thus improving the adaptability of the entire network to extreme scenarios. This can solve the problem that traditional neural network models used for image enhancement have poor enhancement effects on images with many missing features, and improve the enhancement effect of the target image.

[0133] In addition, by introducing a pre-set lookup table during network training, the encoding features of the encoder can be corrected. Since the pre-set lookup table is set based on the target features of the target to be identified, the autoencoder can learn model parameters that conform to the target features during network training. The first diffusion model and the autodecoder can be trained based on the encoded data that conforms to the target features, thereby improving the rationality of the entire network output.

[0134] Meanwhile, since the lookup process of the preset lookup table is relatively slow, and the trained autoencoder can output encoded features that conform to the target features, the image enhancement efficiency can be improved while ensuring the rationality of the network by not having the encoded features output by the encoder corrected by the preset lookup table during the image enhancement process.

[0135] Based on the above embodiments, the training process of the image enhancement network will be described below. (Reference) Figure 4 The training process includes the following steps:

[0136] Step 401: Obtain the first image.

[0137] The first image refers to the image used to train the autoencoder and autodecoder, and there are usually many first images. The first image meets the preset quality requirements. The first image can be acquired by an electronic device, obtained from an existing image database, or downloaded from the network. This embodiment does not limit the acquisition method of the first image.

[0138] Step 402: Iteratively train the initial autoencoder and initial autodecoder based on the first image and a preset lookup table to learn the first model parameters of the autoencoder and autodecoder, and obtain the trained autoencoder and trained autodecoder.

[0139] The initial autoencoder (with the same model structure as the trained autoencoder) and the initial autodecoder (with the same model structure as the trained autodecoder) are built based on the U-Net neural network.

[0140] Each first image serves as both the input and the training target for training the autoencoder and autodecoder, thereby training their image recovery capabilities.

[0141] The process involves iteratively training an initial autoencoder and an initial autodecoder based on a first image and a preset lookup table to learn the first model parameters of the autoencoder and autodecoder, resulting in trained autoencoders and trained autodecoders. This includes: inputting the first image into the initial autoencoder to obtain encoded data; determining the template encoded data with the highest similarity to the encoded data in the preset lookup table to obtain corrected encoded data; inputting the corrected encoded data into the initial decoder to obtain a decoding result; and determining the loss information corresponding to the initial autoencoder and initial autodecoder based on the first image, the decoding result, and the corrected encoded data, and then iteratively training the initial decoder and initial autodecoder based on this loss information to learn the first model parameters, resulting in trained autoencoders and trained autodecoders.

[0142] In one example, the first model parameters are obtained through phased iterative training, with different loss functions used in each phase. Illustratively, the iterative training phases of the first model parameters include a first phase and a second phase. The first phase uses the first loss function, while the second phase uses both the first and second loss functions.

[0143] The first loss function is used to train the image reconstruction capabilities of the autoencoder and autodecoder, as well as the consistency between the encoding result of the trained autoencoder and the lookup result from the preset lookup table.

[0144] The first loss function includes a first term, a second term, and a third term. The first term is used to determine the image reconstruction loss of the autoencoder and the autodecoder. The second term is used to determine the loss of the lookup process in the preset lookup table. The third term is used to constrain the output of the autoencoder to be consistent with the lookup result.

[0145] Schematic, the first loss function L ac This can be expressed by the following formula:

[0146]

[0147] Where, x hq The first image is input to the initial autoencoder. For x hq The decoding result obtained after autoencoder and autodecoder; sg represents the gradient disconnection operation; codebook is the lookup (or mapping) process of the predefined lookup table; E(x hq ) represents the encoded data output by the initial autoencoder; codebook[E(x hq )] represents the corrected encoded data.

[0148] The second loss function is used to train the consistency between the image output by the autodecoder and the original image.

[0149] Schematic, the second loss function is adversarial loss, in which case the second loss function L gan This can be expressed by the following formula:

[0150]

[0151] Where D represents a single discriminator, which is used for adversarial training with the self-written decoder, D(x hq The ) represents the discriminator's judgment result on the first image. This represents the discriminator's judgment result on the decoding result after the first image has been processed sequentially by the autoencoder and the autodecoder.

[0152] During the first phase of iterative training, the first model parameters are learned based on the loss information output by the first loss function.

[0153] During the second phase of iterative training, the first model parameters are learned based on the loss information output by the first and second loss functions.

[0154] Optionally, learning the first model parameters based on the loss information output by the first loss function and the second loss function includes: obtaining the first weight of the first loss function and the second weight of the second loss function; determining the sum of the product of the loss information output by the first loss function and the first weight, and the product of the loss information output by the second loss function and the second weight, to obtain a weighted sum; and learning the first model parameters based on the weighted sum.

[0155] In this embodiment, the first weight is greater than the second weight, and the second weight can be 1 or 0.01. In actual implementation, the values ​​of the first weight and the second weight can also be other values. This embodiment does not limit the values ​​of the first weight and the second weight.

[0156] In other embodiments, the sum of the loss information output by the first loss function and the loss information output by the second loss function can also be directly calculated. This embodiment does not limit the way the first loss function and the second loss function are used.

[0157] Optionally, the threshold for the number of iterations in the first stage and the threshold for the number of iterations in the second stage are the same, for example, both are 25k times; or, the threshold for the number of iterations in the first stage and the threshold for the number of iterations in the second stage are different, for example, the threshold for the number of iterations in the first stage is less than the threshold for the number of iterations in the second stage. This embodiment does not limit the threshold for the number of iterations in each stage.

[0158] Step 403: Acquire the second image and the third image. The quality of the second image is higher than that of the third image, and the quality of the second image meets the preset quality requirements.

[0159] The second and third images refer to the images used to train the first diffusion model. The dataset to which the second image belongs may be the same as or different from the dataset to which the first image belongs.

[0160] In one example, the second image was acquired in an environment that meets preset quality requirements, while the third image was acquired in an environment that does not meet preset quality requirements. However, the number of second and third images acquired is usually enormous, which leads to low acquisition efficiency for the second and third images.

[0161] In another example, obtaining a second image and a third image includes: obtaining the second image; degrading the second image using a pre-trained degradation model to obtain the third image; wherein the degradation model is trained using a set of degraded images, which includes images corresponding to different image enhancement scenarios.

[0162] Since the degraded image set includes images corresponding to different image enhancement scenarios, it can cover various degraded factors. Therefore, the degraded model trained based on the degraded image set can also cover different degraded factors. Thus, the third image obtained by using the degraded model can also cover different degraded factors, thereby simulating different degraded situations in the real environment and ensuring the model performance of the first diffusion model obtained using the third image.

[0163] Alternatively, the degraded model can be implemented in ways including but not limited to the following:

[0164] The first type: The pre-trained degradation model includes a second diffusion model. The degradation images used by the second diffusion model are obtained by image acquisition based on different image enhancement scenarios.

[0165] The second diffusion model is used to degrade images that meet the preset quality requirements to images that do not meet the preset quality requirements. During training, the images input to the second diffusion model are images that meet the preset quality requirements (i.e., high-quality images in the following text), and the training target is the residual image between the images that do not meet the preset quality requirements (i.e., degraded images) and the images that meet the preset quality requirements.

[0166] In this approach, the degraded image set used to train the degraded model is based on real-world scene acquisition. This allows for the acquisition of degraded factors caused by the environment and those caused during image capture, covering a variety of degraded conditions and ensuring the richness and realism of the degraded image set, thereby improving the performance of the obtained first diffusion model.

[0167] The second method involves a pre-trained de-quality model that includes a recurrent consistent generative adversarial network (RCA). This RCA uses high-quality images from the de-quality image set as targets that may or may not be the same as the targets in the de-quality images. In other words, there is no need for a one-to-one correspondence between high-quality and de-quality images in the de-quality image set, which can improve the acquisition efficiency of the second and third images.

[0168] In another example, acquiring the second and third images includes: acquiring the second image; for each second image, performing a superimposed degradation operation on the second image according to at least two preset degradation strategies and the superimposed probability corresponding to each degradation strategy; and outputting the third image when the number of superimposed degradation operations on the second image reaches a preset number.

[0169] The process involves performing a superposition degradation operation on the second image according to at least two preset degradation strategies and the superposition probability corresponding to each degradation strategy. This includes: randomly generating the superposition probability corresponding to each degradation strategy; and performing a superposition operation on the second image for degradation strategies with a superposition probability greater than a probability threshold to obtain the second image after this degradation.

[0170] Optionally, the preset number of times can be 1 or at least 2. In actual shooting scenarios, the same degradation factor may be introduced multiple times. Therefore, by setting the preset number of times to at least 2, the simulation effect of degradation operation can be improved, and the realism of the third image can be enhanced.

[0171] Schematic, at least two degradation strategies cover various degradation factors that cause image degradation. The at least two degradation strategies include, but are not limited to, at least two of the following: brightness adjustment, Gaussian noise addition, Poisson noise addition, Gaussian blur, motion blur, and JPEG preservation degradation strategies.

[0172] By acquiring degraded images through the above methods, the efficiency of acquiring third images can be improved, thereby increasing the training samples for the first diffusion model and improving the performance of the first diffusion model after training.

[0173] In addition, different second images can be degraded using the same degrading method, such as using the second diffusion model for all of them. Alternatively, different degrading methods can be used, such as using the second diffusion model for some of them and using the probability superposition method for others. This embodiment does not limit the degrading method of the second image.

[0174] Step 404: Fix the first model parameters, and iteratively train the initial diffusion model based on the second image, the third image and the preset lookup table to learn the second model parameters of the first diffusion model and obtain the first diffusion model.

[0175] Traditional diffusion models are based on the U-Net neural network. The network structure typically includes: a first convolutional module, four consecutive contraction paths, a central module, four consecutive expansion paths, and a second convolutional module connected sequentially. The first and second convolutional modules have identical structures, each consisting of a 2D convolutional layer and a Mish layer. Each contraction path consists of two consecutive residual modules and a downsampling layer connected sequentially. The central module consists of two consecutive residual modules. Each expansion path mainly consists of two consecutive residual modules and an upsampling layer connected sequentially. The residual modules in the contraction, central, and expansion paths have identical structures; each residual module consists of two consecutive convolutional modules. The output of the first convolutional module in each residual module is combined with the temporal feature matrix t at the current t-th diffusion step. e The inputs are summed and then fed into the second convolutional module. The input to the residual module, after being processed by two consecutive convolutional modules, is then summed with itself to become the output of the residual module. The convolutional modules in the residual modules of the contraction path, center module, and expansion path have the same structure as the first and second convolutional modules.

[0176] The output of the first convolutional module is added to the low-resolution image information Ie output by the low-resolution encoder and then input into the first contraction path. The output of the first contraction path is input into the second contraction path and simultaneously added to the output of the third expansion path before being input into the fourth expansion path. The output of the second contraction path is input into the third contraction path and simultaneously added to the output of the second expansion path before being input into the third expansion path. The output of the third contraction path is input into the fourth contraction path and simultaneously added to the output of the first expansion path before being input into the second expansion path. The output of the fourth contraction path is input into the center module and simultaneously added to the output of the center module before being input into the first expansion path.

[0177] However, the computational cost of the aforementioned traditional diffusion models is typically high, leading to low training and computational efficiency. Furthermore, when directly applying this diffusion model to the image enhancement scenario described in this application, there is a possibility that the electronic device may lack the necessary operating environment for the model, resulting in incompatibility with the device.

[0178] Based on the above-mentioned technical problems, this embodiment improves the traditional diffusion model, and the improvement is described below.

[0179] Firstly, both the first diffusion model and the initial diffusion model are based on the U-Net neural network. The U-Net neural network includes a shrinking path and an expanding path. The shrinking path and the expanding path include at least two residual modules. Each residual module includes a first convolutional module and a second convolutional module connected in sequence. The diffusion step t currently input to the first diffusion model is converted into scalar parameters after the time feature matrix is ​​embedded and then fused with the output of the first convolutional module. The fused result is then input to the second convolutional module.

[0180] The other parts of the first diffusion model and the initial diffusion model, as well as the relevant explanations of the contraction and expansion paths, are described in the traditional diffusion model above, and will not be repeated here in this embodiment. Unlike the traditional diffusion model, this embodiment does not use the time feature matrix t... e Instead of directly adding it to the output of the first convolutional module, the time feature matrix t is used. e The parameters are converted into scalar parameters and used for calculation with the output of the first convolution module. This reduces the amount of computation and improves computational efficiency without affecting the fusion effect.

[0181] In one example, the way to convert a temporal feature matrix into scalar parameters is to input the temporal feature matrix into a series of convolutional and fully connected layers to obtain the scalar parameters shift and scale.

[0182] Accordingly, the scalar parameters are fused with the output of the first convolutional module by multiplying the scale with the output of the first convolutional module and adding shift to obtain the fused result.

[0183] Secondly, this embodiment can also modify the model of the U-Net neural network.

[0184] Generally, the network structure of the first and second convolutional modules includes sequentially connected convolutional layers, normalization layers, and activation layers. The activation layers are implemented using the Silu (Sigmoid Linear Unit) activation function. In this embodiment, Silu is modified to a Rectified Linear Unit (ReLU), which can further reduce the computational cost of the model.

[0185] In some embodiments, an attention module is connected after the first and second convolutional modules. This attention module can be a QKV (Query-Key-Value) attention mechanism. In this embodiment, the QKV attention mechanism is modified to a channel attention mechanism, which can further reduce the computational cost of the model.

[0186] Thirdly, this embodiment can also simplify the U-Net neural network model.

[0187] To illustrate, the number of channels in the first two contraction paths of the four contraction paths and the last two expansion paths of the four expansion paths is reduced to half of the original number, thereby reducing the complexity of the model and the amount of computation required.

[0188] All of the above improvements can reduce the computational load of the image enhancement model without affecting its performance.

[0189] When training the initial diffusion model, the first model parameters are fixed, and the initial diffusion model is iteratively trained based on the second image, the third image, and a preset lookup table to learn the second model parameters of the first diffusion model, thus obtaining the first diffusion model. This includes: inputting the second image and the third image into the trained autoencoder to obtain the encoded data z corresponding to the second image. hq The encoded data z corresponding to the third image lq Use a preset lookup table to encode the data z corresponding to the second image. hq After correction, the corrected encoded data codebook[z] is obtained. hq ]; Encode the data z corresponding to the third image lq Input the diffusion step number t into the initial diffusion model to obtain the model output; then input the model output and the encoded data z corresponding to the third image. lq After fusion, the input is given to the trained self-decoder to obtain the network output corresponding to the third image; the encoded data z corresponding to the second image is then processed. hq Input the trained self-decoder to obtain the network output result corresponding to the second image; based on the model output result, the corrected encoded data, the network output result corresponding to the second image and the network output result corresponding to the third image, determine the loss information of the initial diffusion model; based on the loss information, iteratively train the initial diffusion model to obtain the parameters of the second model.

[0190] Specifically, based on the model output, the corrected encoded data, the network output corresponding to the second image, and the network output corresponding to the third image, the loss information of the initial diffusion model is determined, including:

[0191] Obtain the encoded data z corresponding to the third image. lq The actual noise added during t-step forward spread of the input ∈; based on the corrected encoded data codebook[z hq Determine the diffusion result at step t of the initial diffusion model; determine the model output after performing t-step reverse diffusion on the diffusion result at step t; based on the model output and the diffusion result at step t, determine the prediction noise added during forward diffusion. σThe first loss value is determined based on the actual noise and the predicted noise; the second loss value is determined based on the network output result corresponding to the second image, the network output result corresponding to the third image, the intermediate result of the self-decoder when determining the network output result corresponding to the second image, and the intermediate result of the self-decoder when determining the network output result corresponding to the third image; and the loss information is determined based on the first loss value and the second loss value.

[0192] The calculation process for the first loss value is expressed by the following formula:

[0193]

[0194] Among them, codebook[z hq ] represents the corrected encoded data, z t This represents the diffusion result at step t, where x0 represents the encoded data corresponding to the third image, and z0 is the model output. This represents the diffusion coefficient at the t-th diffusion step.

[0195] The intermediate result of the self-decoder refers to the result of the output of the specified hidden layer in the decoder, or it can be the result of the output of all hidden layers. This embodiment does not limit the source of the intermediate result.

[0196] The second loss value is determined based on the perceptual loss function of the self-decoder. Schematic, the second loss value L... p Through the following

[0197] The formula yields:

[0198]

[0199] in, Represents the network layers of the self-decoder; j represents the hidden layer of the output and the output layer of the self-decoder, C j H j W j Z represents the number of channels, length, and width of the j-th layer of the decoder, respectively, where j is a positive integer. γ is the weight of the output of the j-th layer (which can be an intermediate result or the network output). The weights for different network layers may be the same or different. Schematic, the γ of the output layer is greater than the γ of the hidden layer. hq This represents the encoded data corresponding to the second image, z. lq This represents the encoded data corresponding to the third image.

[0200] In one example, determining loss information based on a first loss value and a second loss value includes: obtaining the weights corresponding to the first loss value and the second loss value; and determining the weighted sum of the first loss value and the second loss value based on the weights to obtain the loss information.

[0201] For example, the weight of the first loss value is 1, and the weight of the second loss value is 0.01. In other embodiments, the weights corresponding to the first loss value and the second loss value can also be other values. This embodiment does not limit the way the weights are set.

[0202] In other embodiments, the loss information may also be the sum of the first loss value and the second loss value. This embodiment does not limit the method of determining the loss information.

[0203] In some scenarios, the enhanced image still needs to undergo target recognition. Common target recognition methods include inputting the enhanced image into a pre-trained target recognition network to obtain the target recognition result.

[0204] Image enhancement using image augmentation models can fill in missing information about the target object, but the supplemented information is random, which can lead to a decrease in target recognition capability to some extent. Therefore, to improve the recognition capability of the target recognition network, in one example of this embodiment, when training the first diffusion model, at least two third images corresponding to each second image can be used for training to train the consistency of different third images during the denoising process. That is, each second image corresponds to at least two third images. Accordingly, loss information is determined based on the first loss value and the second loss value, including: obtaining the t-th step diffusion result corresponding to different third images; determining the third loss value based on at least two third images and the t-th step diffusion result corresponding to each third image; and determining the loss information based on the first loss value, the second loss value, and the third loss value.

[0205] Schematic, the calculation process of the third loss value Ls is represented by the following formula:

[0206] Ls=||∈ φ (y t x W-tuurb ,t)-∈ δ (y t x W-turb ,t)+cond φ -cond δ || 2

[0207] Where ∈ represents the diffusion result of the third image obtained after different degrees of degradation of the same second image at time t, and cond represents the third image corresponding to different degrees of degradation.

[0208] The loss information can be determined based on a weighted sum or a summation, and this embodiment does not limit the method of determining the loss information.

[0209] In order to improve the recognition ability of the target recognition network, in another example of this embodiment, the model parameters of the self-decoder can be fine-tuned after the first diffusion model is trained, so as to improve the adaptability of the output of the self-decoder to the target recognition network.

[0210] Specifically, after step 404, the process further includes: acquiring the fourth image and the fifth image; fixing the model parameters of the autoencoder and the second model parameters; iteratively training the trained autodecoder based on the fourth image and the fifth image to adjust the model parameters of the autodecoder; and determining the autodecoder based on the adjusted model parameters.

[0211] The fourth image is of higher quality than the fifth image, and its quality meets the preset quality requirements. The dataset to which the fourth image belongs may be the same as or different from the training set to which the second image belongs, and the dataset to which the fifth image belongs may be the same as or different from the training set to which the third image belongs.

[0212] Schematic, the object recognition network includes an encoder, and correspondingly, the trained self-decoder is iteratively trained based on the fourth and fifth images to adjust the model parameters of the self-decoder, including:

[0213] The fourth image I hq Input the encoder in the target recognition network to obtain the encoded data corresponding to the fourth image; input the fifth image I lq The encoder in the target recognition network is input to obtain the encoded data corresponding to the fifth image; the encoded data corresponding to the fourth image and the fifth image are input into the first diffusion model and the trained self-decoder respectively to obtain the current decoding result of the self-decoder; the model parameters of the self-decoder are fine-tuned; the encoded data corresponding to the fourth image and the fifth image are input into the first diffusion model and the fine-tuned self-decoder respectively to obtain the adjusted decoding result of the self-decoder; the loss information is determined based on the current decoding result and the adjusted decoding result, and the model parameters are iteratively fine-tuned based on the loss information to obtain the adjusted model parameters.

[0214] In one example, the loss information is determined based on the encoded data corresponding to the fourth image, the intermediate result of the encoder output when determining the encoded data corresponding to the fourth image, the encoded data corresponding to the fifth image, the intermediate result of the encoder output when determining the encoded data corresponding to the fifth image, and the current decoding result and the adjusted decoding result. For example: loss information L id The calculation process is represented by the following formula:

[0215]

[0216] in, I represents the network layers of the encoder in the target recognition network.lq For the fifth image, I hq The fourth image, x is the current decoding result, x id The adjusted decoding result is shown; i represents the hidden layer and output layer of the encoder output in the target recognition network, and C represents the output of the target recognition network. i H i W i γ' represents the number of channels, length, and width of the encoder's i-th layer, respectively, where i is a positive integer. γ' is the weight of the output result of the i-th layer (which can be an intermediate result or the network output result). The weights corresponding to different network layers may be the same or different. Schematic, the γ' of the output layer is greater than the γ' of the hidden layer.

[0217] To improve the recognition capability of the target recognition network, in another example of this embodiment, before inputting the enhanced image into the pre-trained target recognition network, the method further includes: obtaining a fifth image and the target label corresponding to the fifth image; inputting the fifth image into the image enhancement network to obtain the enhanced fifth image; and using the enhanced fifth image and the target label to learn the model parameters of the target recognition network to obtain the target recognition network.

[0218] In this way, since the target recognition network is retrained using the enhanced images, its adaptability to the enhanced images can be improved, thus increasing the accuracy of target recognition.

[0219] Optionally, the fifth image is an image that does not meet the preset quality requirements.

[0220] In addition, the initial parameters of the target recognition network in this embodiment can enable target recognition on images that meet preset quality requirements. That is, the target recognition network is trained using images that meet preset quality requirements and the target labels corresponding to those images. The enhanced fifth image is used to adjust the initial parameters of the target recognition network to improve the network's adaptability to the enhanced images.

[0221] In summary, in this embodiment, when establishing the initial diffusion model and the first diffusion model, the time feature matrix is ​​converted into a scalar parameter and fused with the output of the first convolution model during the calculation process, which can reduce the amount of computation and improve the model calculation speed.

[0222] In addition, the computation speed of the model can be further improved by modifying and / or simplifying the model.

[0223] In addition, by using a pre-defined lookup table during the training of the autoencoder, the first diffusion model, and the self-decoder, the overall accuracy of the network in recovering target features can be improved, thus enhancing the accuracy of image enhancement.

[0224] In addition, when determining the loss information corresponding to the initial diffusion model, it is necessary to combine not only the model output results but also the output results of the entire network, which can improve the model performance of the diffusion model.

[0225] In addition, incorporating consistency loss during the training of the first diffusion model can prevent large differences in the enhancement results of different degraded images corresponding to the same image, which could lead to low accuracy in subsequent target recognition and improve the target recognition effect on the enhanced image.

[0226] In addition, by using a degradation model to acquire degraded images during the training of the first diffusion model, the efficiency of acquiring degraded images can be improved. On the other hand, since the degradation model can simulate various degradation factors in real-world scenes, the realism of degraded images can also be improved.

[0227] Furthermore, by building a degradation model based on a diffusion model, the realism of the degraded image can be improved because the diffusion model can learn the actual degradation factors. Alternatively, a degradation model built using a recurrent consistency generative adversarial network (GAN) does not require setting pairs of high-quality and low-quality images, thus reducing the complexity of obtaining training data and improving training efficiency. Alternatively, using a probability stacking method to degrade high-quality images does not require pre-training of the neural network model, thus improving degradation efficiency.

[0228] In addition, by fine-tuning the model parameters of the self-decoder after obtaining the trained self-decoder, the problem of a large gap between the high-quality image output by the self-decoder and the original low-quality image, which would lead to low accuracy in subsequent target recognition, can be prevented, thereby improving the accuracy of target recognition on the enhanced image.

[0229] Furthermore, by retraining the target recognition network using the enhanced images, the applicability of the recognition network to the enhanced images can be improved, thereby increasing recognition accuracy.

[0230] Optionally, based on the above embodiments, in step 102, the first diffusion model needs to use a sampler to sample noise during the calculation process. Traditional sampling methods include using a discrete-time denoising diffusion implicit models (DDIM) sampler to sample from the model. However, the diffusion steps of the first diffusion model are generally one thousand steps or more, which leads to low computational efficiency. Therefore, before step 102, the method further includes: determining a teacher model based on the first diffusion model; using the teacher model to guide a pre-created continuous-time student model for distillation training to obtain a trained student network. This student network is used to sample noise during the calculation of the first diffusion model, so as to use the sampled noise data for back-diffusion to obtain enhanced encoded features; wherein, the continuous-time student model has learnable parameters, which are used to match the output of the teacher model at any time step.

[0231] The distillation training method is a two-step distillation to improve sampling efficiency.

[0232] In this embodiment, by using the trained student network to sample noise during the first diffusion model calculation process, the sampling efficiency of the diffusion model can be improved, and the model calculation speed can be increased.

[0233] Based on the above embodiments, this application also provides a target recognition method. Figure 5 This is a flowchart of a target recognition method provided in one embodiment of this application. The method includes at least the following steps:

[0234] Step 501: Obtain the image to be recognized.

[0235] The image to be identified can be the enhanced image output in step 102, or it can be an image that has not been processed by the image enhancement model. This embodiment does not limit the type of image to be identified.

[0236] Step 502: Input the image to be recognized into a pre-trained target recognition network to obtain the target recognition result corresponding to the image to be recognized.

[0237] In one example, the target recognition network is trained using the enhanced fifth image and the target label corresponding to the fifth image after the image enhancement network enhances the fifth image. The image enhancement network includes an autoencoder, a first diffusion model, and a self-decoder connected in sequence. During training, the output of the autoencoder is corrected based on a preset lookup table, and the image enhancement network is trained based on the corrected encoding result.

[0238] For detailed instructions regarding this example, please refer to [link / reference]. Figure 4 The embodiments described above will not be repeated here.

[0239] In summary, in this embodiment, either the enhanced image to be recognized or the image to be recognized without image enhancement model processing can be input into the target recognition network. This not only ensures the target recognition accuracy of the target recognition network but also improves its versatility.

[0240] Figure 6 This is a block diagram of an image enhancement apparatus provided in one embodiment of this application. The apparatus includes at least the following modules: an image acquisition module 610 and an image enhancement module 620.

[0241] Image acquisition module 610 is used to acquire the target image to be enhanced;

[0242] The image enhancement module 620 is used to input the target image into a pre-trained image enhancement network, encode the target image through the autoencoder in the image enhancement network, input the obtained encoded features into the first diffusion model in the image enhancement model for enhancement, fuse the obtained enhanced encoded features with the encoded features and input them into the autodecoder in the image enhancement model to obtain the enhanced image corresponding to the target image;

[0243] During training, the output of the autoencoder in the image enhancement network is corrected based on a preset lookup table, and the image enhancement network is trained based on the corrected encoding result.

[0244] For relevant details, please refer to the above method implementation examples.

[0245] It should be noted that the image enhancement device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the image enhancement device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image enhancement device and the image enhancement method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0246] Figure 7 This is a block diagram of a target recognition device according to an embodiment of this application. The device includes at least the following modules: an image acquisition module 710 and a target recognition module 720.

[0247] Image acquisition module 710 is used to acquire the image to be recognized;

[0248] The target recognition module 720 is used to input the image to be recognized into a pre-trained target recognition network to obtain the target recognition result corresponding to the image to be recognized;

[0249] The target recognition network is trained using the enhanced fifth image and the target label corresponding to the fifth image after the image enhancement network enhances the fifth image. The image enhancement network includes an autoencoder, a first diffusion model, and a self-decoder connected in sequence. During training, the output of the autoencoder is corrected based on a preset lookup table, and the target recognition network is trained based on the corrected encoding result.

[0250] For relevant details, please refer to the above method implementation examples.

[0251] It should be noted that the target recognition device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the target recognition device can be divided into different functional modules to complete all or part of the functions described above. In addition, the target recognition device and the target recognition method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0252] Figure 8 This is a block diagram of a training apparatus for an image enhancement network according to an embodiment of this application. The image enhancement network includes an autoencoder, a first diffusion model, and a self-decoder connected in sequence. The autoencoder encodes an input target image, the resulting encoded features are input into the first diffusion model for enhancement, and the enhanced encoded features are fused with the original encoded features and input into the self-decoder to obtain the enhanced image corresponding to the target image. The apparatus includes at least the following modules: a first acquisition module 810, a first training module 820, a second acquisition module 830, and a second training module 840.

[0253] The first acquisition module 810 is used to acquire a first image, wherein the first image meets a preset quality requirement;

[0254] The first training module 820 is used to iteratively train the initial autoencoder and the initial autodecoder based on the first image and the preset lookup table, so as to learn the first model parameters of the autoencoder and the autodecoder and obtain the trained autoencoder and the trained autodecoder.

[0255] The second acquisition module 830 is used to acquire a second image and a third image, wherein the quality of the second image is higher than the quality of the third image, and the quality of the second image meets the preset quality requirements;

[0256] The second training module 840 is used to fix the parameters of the first model and iteratively train the initial diffusion model based on the second image, the third image and the preset lookup table to learn the second model parameters of the first diffusion model and obtain the first diffusion model.

[0257] For relevant details, please refer to the above method implementation examples.

[0258] It should be noted that the image enhancement network training device provided in the above embodiments is only illustrated by the division of the above functional modules when training the image enhancement network. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the image enhancement network training device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image enhancement network training device and the image enhancement network training method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0259] Figure 9 This is a block diagram of an electronic device provided in one embodiment of this application. The device includes at least a processor 901 and a memory 902.

[0260] Processor 901 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 901 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 901 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 901 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 901 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0261] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 902 are used to store at least one instruction, which is executed by the processor 901 to implement the image enhancement or target recognition method provided in the method embodiments of this application.

[0262] In some embodiments, the electronic device may also optionally include: a peripheral device interface and at least one peripheral device. The processor 901, memory 902, and peripheral device interface can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface via a bus, signal line, or circuit board. Indicatively, peripheral devices include, but are not limited to: radio frequency circuitry, a touch display screen, audio circuitry, and a power supply.

[0263] Of course, electronic devices may also include fewer or more components, and this embodiment does not limit this.

[0264] Optionally, this application also provides a computer-readable storage medium storing a program that is loaded and executed by a processor to implement the image enhancement or target recognition method of the above method embodiments.

[0265] Optionally, this application also provides a computer product including a computer-readable storage medium storing a program that is loaded and executed by a processor to implement the image enhancement or target recognition method of the above method embodiments.

[0266] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0267] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An image enhancement method, characterized in that, The method includes: Acquire the target image to be enhanced; The target image is input into a pre-trained image augmentation network. The target image is encoded by the autoencoder in the image augmentation network. The encoded features are then input into the first diffusion model in the image augmentation network for enhancement. The enhanced encoded features and the encoded features are then fused and input into the autodecoder in the image augmentation network to obtain the enhanced image corresponding to the target image. During training, the output of the autoencoder in the image enhancement network is corrected based on a preset lookup table, and the image enhancement network is trained based on the corrected encoding result. The training process of the image augmentation network includes: acquiring a first image that meets a preset quality requirement; iteratively training an initial autoencoder and an initial autodecoder based on the first image and the preset lookup table to learn the first model parameters of the autoencoder and the autodecoder, thereby obtaining the trained autoencoder and the trained autodecoder; acquiring a second image and a third image, wherein the quality of the second image is higher than that of the third image and the quality of the second image meets the preset quality requirement; fixing the first model parameters; iteratively training an initial diffusion model based on the second image, the third image, and the preset lookup table to learn the second model parameters of the first diffusion model, thereby obtaining the first diffusion model; The step of iteratively training the initial autoencoder and initial autodecoder based on the first image and the preset lookup table to learn the first model parameters of the autoencoder and the autodecoder includes: inputting the first image into the initial autoencoder to obtain encoded data; determining the template encoded data with the highest similarity to the encoded data in the preset lookup table to obtain corrected encoded data; inputting the corrected encoded data into the initial decoder to obtain a decoding result; and determining the loss information corresponding to the initial autoencoder and initial autodecoder based on the first image, the decoding result, and the corrected encoded data. The step of fixing the first model parameters and iteratively training an initial diffusion model based on the second image, the third image, and the preset lookup table to learn the second model parameters of the first diffusion model and obtain the first diffusion model includes: inputting the second image and the third image into the trained autoencoder to obtain encoded data corresponding to the second image and encoded data corresponding to the third image; using the preset lookup table to correct the encoded data corresponding to the second image to obtain corrected encoded data; inputting the encoded data corresponding to the third image and the diffusion step number t into the initial diffusion model to obtain the model output result; where t is a positive integer; fusing the model output result and the encoded data corresponding to the third image and inputting them into the trained autodecoder to obtain the network output result corresponding to the third image; inputting the encoded data corresponding to the second image into the trained autodecoder to obtain the network output result corresponding to the second image; determining the loss information of the initial diffusion model based on the model output result, the corrected encoded data, the network output result corresponding to the second image, and the network output result corresponding to the third image; and iteratively training the initial diffusion model based on the loss information to obtain the second model parameters.

2. The method according to claim 1, characterized in that, The step of determining the loss information of the initial diffusion model based on the model output, the corrected encoded data, the network output corresponding to the second image, and the network output corresponding to the third image includes: Obtain the actual noise added during t-step forward diffusion using the encoded data corresponding to the third image as input; Based on the corrected encoded data, determine the diffusion result at step t obtained when the initial diffusion model performs diffusion at step t; Determine the model output result obtained after performing t-step reverse diffusion on the diffusion result of step t; Based on the model output and the diffusion result at step t, determine the prediction noise added during forward diffusion; A first loss value is determined based on the actual noise and the predicted noise; Based on the network output result corresponding to the second image, the network output result corresponding to the third image, the intermediate result of the self-decoder when determining the network output result corresponding to the second image, and the intermediate result of the self-decoder when determining the network output result corresponding to the third image, a second loss value is determined; The loss information is determined based on the first loss value and the second loss value.

3. The method according to claim 2, characterized in that, Each second image corresponds to at least two third images; Accordingly, determining the loss information based on the first loss value and the second loss value includes: Obtain the diffusion results at step t corresponding to different third images; Based on the at least two third images and the diffusion result at step t corresponding to each third image, the third loss value is determined; The loss information is determined based on the first loss value, the second loss value, and the third loss value.

4. The method according to claim 1, characterized in that, After iteratively training the initial diffusion model based on the second image, the third image, and the preset lookup table to learn the second model parameters of the first diffusion model and obtain the first diffusion model, the method further includes: Acquire a fourth image and a fifth image, wherein the quality of the fourth image is higher than that of the fifth image, and the quality of the fourth image meets the preset quality requirements; The model parameters of the autoencoder and the second model parameters are fixed, and the trained autodecoder is iteratively trained based on the fourth image and the fifth image to adjust the model parameters of the autodecoder. The autodecoder is then determined based on the adjusted model parameters.

5. The method according to claim 1, characterized in that, The acquisition of the second and third images includes: Obtain the second image; The second image is degraded using a pre-trained degradation model to obtain the third image. The degradation model is trained using a set of degraded images, which includes images corresponding to different scenes to be enhanced.

6. The method according to claim 5, characterized in that, The pre-trained degradation model includes a second diffusion model, wherein the degraded images used by the second diffusion model are obtained based on image acquisition from the different image enhancement scenarios to be performed; or... The pre-trained degradation model includes a recurrent consistent generative adversarial network, wherein the target in the high-quality image in the degradation image set used by the recurrent consistent generative adversarial network is the same as or different from the target in the degradation image.

7. The method according to any one of claims 1 to 6, characterized in that, The first diffusion model is based on a U-Net neural network, which includes a shrinking path and an expanding path. The shrinking path and the expanding path each include at least two residual modules, and each residual module includes a first convolutional module and a second convolutional module connected in sequence. The diffusion step t currently input to the first diffusion model is converted into a scalar parameter after the embedding transformation of the time feature matrix, and then fused with the output of the first convolution module. The fused result is then input to the second convolution module.

8. The method according to any one of claims 1 to 6, characterized in that, After inputting the target image into a pre-trained image augmentation network to obtain the augmented image corresponding to the target image, the method further includes: Obtain the fifth image and the target label corresponding to the fifth image; The fifth image is input into the image enhancement network to obtain the enhanced fifth image; The enhanced fifth image and the target label are used to learn the model parameters of the target recognition network, thus obtaining the target recognition network.

9. A target recognition method, characterized in that, The method includes: Acquire the image to be recognized; The image to be identified is input into a pre-trained target recognition network to obtain the target recognition result corresponding to the image to be identified; The target recognition network is obtained by enhancing the fifth image using the pre-trained image enhancement network of claim 1, and then training the enhanced fifth image and the target label corresponding to the fifth image. The image enhancement network includes an autoencoder, a first diffusion model, and a self-decoder connected in sequence. During training, the output result of the autoencoder is corrected based on a preset lookup table, and the image enhancement network is trained based on the corrected encoding result.

10. An electronic device, characterized in that, The device includes a processor and a memory; the memory stores a program that is loaded and executed by the processor to implement the image enhancement method as described in any one of claims 1 to 8; or to implement the target recognition method as described in claim 9.

11. A computer-readable storage medium, characterized in that, The storage medium stores a program that, when executed by a processor, is used to implement the image enhancement method as described in any one of claims 1 to 8; or to implement the target recognition method as described in claim 9.

Images