An image inpainting method, device, computer equipment and storage medium

Abstract

The present application relates to the field of image restoration, and discloses a kind of image restoration method, device, computer equipment and storage medium to improve the fidelity of restored image.The method comprises: local occlusion is carried out to training image, and mask image is obtained;First feature vector of mask image is extracted;According to first feature vector, initial restored image is reconstructed;Multiple loss values of training image and initial restored image are calculated;Based on bat algorithm, the weight of each loss value is adjusted, the minimum total loss value is obtained, and the optimal weight combination corresponding to the minimum total loss value;According to optimal weight combination, the model parameters of initial image restoration model are adjusted, and the image restoration model is obtained;Second feature vector of incomplete image is extracted;Second feature vector is input into image restoration model, and target restored image is reconstructed.

Description

Technical Field

[0001] This invention relates to the field of image restoration, and more particularly to an image restoration method, apparatus, computer device, and storage medium. Background Technology

[0002] With the development of artificial intelligence, AI-based image restoration technology has been widely used, and various image restoration models have emerged. These include technologies such as Convolutional Neural Network (CNN) and Transformer (a deep learning model that uses a self-attention mechanism) to restore damaged or incomplete images. Image restoration technology can be widely used in many scenarios such as disease assistance diagnosis, health management, and remote consultation.

[0003] In existing technologies, most image restoration is based on the Transformer framework to improve the image restoration model, thereby enhancing the effect of image restoration. However, under the current technological background, the information loss is relatively large when using Transformer for image restoration, which is not conducive to improving the fidelity of the restored image. Summary of the Invention

[0004] This invention provides an image restoration method, apparatus, computer device, and storage medium to improve the fidelity of restored images.

[0005] An image restoration method, comprising:

[0006] The training image is partially occluded to obtain a mask image;

[0007] Extract the first feature vector of the mask image;

[0008] Based on the first feature vector, the initial restored image is reconstructed;

[0009] Calculate multiple loss values ​​for the training image and the initial restored image;

[0010] Based on the bat algorithm, the weights of each loss value are adjusted to obtain the minimum total loss value and the optimal weight combination corresponding to the minimum total loss value;

[0011] Based on the optimal weight combination, the model parameters of the initial image restoration model are adjusted to obtain the image restoration model;

[0012] Extract the second feature vector from the incomplete image;

[0013] The second feature vector is input into the image restoration model to reconstruct the target restored image.

[0014] In one possible design, the plurality of loss values ​​include: regression loss, gradient loss, adversarial loss, perceptual loss, and style loss of the training image and the initial restored image.

[0015] In one possible design, extracting the first feature vector of the mask image includes:

[0016] The mask image is divided into multiple mask image blocks;

[0017] Extract the feature vector corresponding to each of the masked image blocks, and use it as the first feature vector of the masked image.

[0018] In one possible design, the step of adjusting the weights of each loss value based on the bat algorithm to obtain the minimum total loss value and the optimal weight combination corresponding to the minimum total loss value includes:

[0019] Initialize the weights of all the aforementioned loss values ​​to obtain a first total loss value and a first weight combination;

[0020] The algorithm parameters in the bat algorithm are adjusted iteratively to obtain the second total loss value and the second weight combination for each iteration, and the number of iterations is recorded.

[0021] When the number of iterations reaches the expected number, the minimum value of the second total loss value is selected as the minimum total loss value.

[0022] The second weight combination corresponding to the minimum value of the second total loss is taken as the optimal weight combination.

[0023] In one possible design, the iterative adjustment of the algorithm parameters in the bat algorithm to obtain a second total loss value and a second weight combination for each iteration includes:

[0024] Determine whether the algorithm parameters are greater than the expected value;

[0025] If it is greater than the current optimal solution of the bat algorithm, a second weight combination is obtained by global search.

[0026] Based on the second weight combination, the second total loss value is obtained, and the algorithm parameters are updated.

[0027] In one possible design, after determining whether the algorithm parameters are greater than the expected value, the method further includes:

[0028] If it is less than or equal to the current optimal solution, then a second weight combination is obtained by local search.

[0029] Based on the second weight combination, the second total loss value is obtained, and the algorithm parameters are updated.

[0030] In one possible design, prior to extracting the first feature vector of the mask image, the method further includes:

[0031] Determine whether the mask image has missing pixels;

[0032] If present, the feature probabilities of the masked image are sampled from the pre-encoded codebook to obtain the first feature vector.

[0033] An image restoration apparatus, comprising:

[0034] The occlusion module is used to partially occlude the training image to obtain a mask image;

[0035] The first quantization module is used to extract the first feature vector of the mask image;

[0036] The reconstruction module is used to reconstruct the initial restored image based on the first feature vector;

[0037] The calculation module is used to calculate multiple loss values ​​between the training image and the initial restored image;

[0038] The loss module is used to calculate the optimal weight for each loss value based on the bat algorithm, and to obtain the minimum loss value;

[0039] The first output module is used to adjust the model parameters of the initial image restoration model according to the optimal weights, so as to obtain the image restoration model.

[0040] The second quantization module is used to extract the second feature vector of the incomplete image;

[0041] The second output module is used to input the second feature vector into the image restoration model to reconstruct the target restored image.

[0042] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the image restoration method described above.

[0043] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the image restoration method described above.

[0044] The aforementioned image restoration method, apparatus, computer equipment, and storage medium perform masking on the training image to obtain a mask image. Then, feature vectors are extracted from the mask image. Next, based on the feature vectors, an initial restored image is reconstructed. Multiple loss values ​​between the training image and the initial restored image are calculated. Finally, based on the bat algorithm, the optimal weight for each loss value is calculated, resulting in the minimum loss value. Based on the optimal weights, the model parameters of the initial image restoration model are adjusted to obtain the image restoration model. Because this image restoration method comprehensively considers different loss values ​​and derives the optimal weight for each loss value, the resulting image restoration model effectively improves the problem of excessive information loss after image restoration, thereby increasing the fidelity of the target restored image. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of an application environment for an image restoration method according to an embodiment of the present invention;

[0047] Figure 2 This is a schematic flowchart of an image restoration method according to an embodiment of the present invention;

[0048] Figure 3 This is a schematic flowchart of an image restoration method according to an embodiment of the present invention;

[0049] Figure 4 This is a flowchart illustrating an image restoration method according to an embodiment of the present invention.

[0050] Figure 5 This is a flowchart illustrating an image restoration method according to an embodiment of the present invention.

[0051] Figure 6 This is a flowchart illustrating an image restoration method according to an embodiment of the present invention.

[0052] Figure 7 This is a schematic diagram of an image restoration device according to an embodiment of the present invention;

[0053] Figure 8 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] The image restoration method provided in this embodiment of the invention can be applied to, for example... Figure 1 In this application environment, the terminal device communicates with the server via a network. The terminal device transmits training images to the server to train the image restoration model. Additionally, the terminal device also transmits the actual incomplete images that need to be restored to the server to reconstruct the target restored image. The terminal device can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be a standalone server or a server cluster consisting of multiple servers.

[0056] In medical applications, incomplete images and target reconstruction images constitute medical images. The final target reconstruction image will include objects classified as lesions, i.e., the parts of the body where diseases have occurred. Medical images refer to images of internal tissues obtained non-invasively for medical treatment or research, such as images of the stomach, abdomen, heart, knees, and brain. These include images generated by medical instruments such as CT (Computed Tomography), MRI (Magnetic Resonance Imaging), ultrasound (Ultrasonic), X-ray images, electroencephalograms (EEGs), and optical imaging.

[0057] In one embodiment, such as Figure 2 As shown, an image restoration method is provided, which is applied to... Figure 1 Taking a server as an example, the explanation includes training and application methods. The training method specifically includes the following steps:

[0058] S10: Partially occlude the training image to obtain a mask image.

[0059] The server acquires complete images as training images, and then performs occlusion processing on the training images to obtain partially occluded mask images to simulate real incomplete images. Methods for acquiring complete images include, but are not limited to, acquisition via terminal devices, automatic image acquisition scripts, and manual acquisition; methods for partially occluding the training images include, but are not limited to, normalizing image pixel values ​​and performing image data augmentation.

[0060] S20: Extract the first feature vector of the mask image.

[0061] After obtaining the mask image, the feature vector of the mask image, also known as the first feature vector, is extracted. Methods for extracting the first feature vector include, but are not limited to, the SIFT (Scale Invariant Feature Transform) algorithm and the HOG (Histogram of Oriented Grids) feature detection algorithm.

[0062] S30: Based on the first feature vector, reconstruct the initial restored image.

[0063] After obtaining the first feature vector, the first feature vector is decoded to reconstruct the initial restored image. The initial restored image refers to the image initially repaired based on the Transformer framework. Methods for reconstructing the initial restored image include, but are not limited to, introducing discriminator adversarial training to improve reconstruction performance.

[0064] S40: Calculate multiple loss values ​​for the training image and the initial restored image.

[0065] Different loss functions are used to calculate different loss values ​​between the training image and the initial restored image. These different loss functions include, but are not limited to, the least squares loss function, the cross-entropy loss function, and the regression loss function; the loss value refers to the loss value between the training image and the initial restored image calculated by the corresponding loss function.

[0066] S50: Based on the bat algorithm, adjust the weight of each loss value to obtain the minimum total loss value and the optimal weight combination corresponding to the minimum total loss value.

[0067] After obtaining multiple loss values, the Bat Algorithm (BA) is applied to select the minimum total loss value and the optimal weight combination for the masked image. The Bat Algorithm is a heuristic intelligent algorithm that simulates the echolocation principle used by bats for hunting. The total loss value refers to the sum of multiple loss values; however, in this embodiment, since a corresponding weight is introduced into each loss value, the total loss value is actually the weighted sum of multiple loss values.

[0068] Specifically, the optimal weight combination is α∈[α g ,α a ,α p ,α s A single-objective unconstrained optimization problem can be viewed as an equation:

[0069] minf(α),[α=(αg ,α a ,α p ,α s )]

[0070] S60: Based on the optimal weight combination, adjust the model parameters of the initial image restoration model to obtain the image restoration model.

[0071] After determining the optimal weight combination, the model parameters of the initial image restoration model are adjusted based on this optimal weight combination to improve the accuracy of the final image restoration model. The initial image restoration model can be an image restoration model based on the Transformer framework, a CNN-based image restoration model, etc.

[0072] In this embodiment, the initial image inpainting model preferably adopts the PUT (P-VQVAE UQ-Transformer, short for Patch-based Vector Quantized Variational Auto-Encoder Un-QuantizedTransformer) model. Therefore, when adjusting the model parameters of the PUT model in step S60, it is essentially adjusting the weights of the first feature vector according to the optimal weight combination, thereby reconstructing the mask image and deriving the image inpainting model.

[0073] The application method of this image restoration model includes the following specific steps:

[0074] S70: Extract the second feature vector of the incomplete image.

[0075] After the trained image restoration model is deployed on the server, the server receives the incomplete image and extracts the second feature vector from the incomplete image. The method for extracting the second feature vector here is the same as the method for extracting the first feature vector in step S20.

[0076] S80: Input the second feature vector into the image restoration model to reconstruct the target restored image.

[0077] By inputting the second feature vector into the image restoration model, the target restored image can be obtained based on the second feature vector.

[0078] It should be noted that because this image restoration method comprehensively considers different loss values ​​and derives the optimal weight for each loss value, the resulting image restoration model effectively improves the problem of excessive loss information after image restoration, thereby increasing the fidelity and restoration efficiency of the target restored image. Furthermore, since the image restoration method employs the bat algorithm to find the optimal solution for the loss value, the trained image restoration model has a simple structure, few parameters, strong robustness, and is easy to implement.

[0079] In one embodiment, step S40, namely the multiple loss values, specifically includes: regression loss of the training image and the initial restored image, gradient loss, adversarial loss, perceptual loss, and style loss. Among them, the regression loss includes, but is not limited to, mean squared error (also known as MSE or L2 loss), mean absolute error (also known as MAE or L1 loss), Huber loss (smoothed mean absolute error), etc.

[0080] Specifically, the total loss value L of the initial restored image rec As the target value for algorithm optimization, L rec The calculation formula is:

[0081] L rec =L pixel +α g L grad +α a L adv +α p L perc +α s L style

[0082] Among them, L pixel L1 loss represents the difference between pixel values ​​in the training image and the initial restored image. grad For the gradient loss of the two images, L adv For the adversarial loss between two images, L perc L is the perceptual loss between two images. style This represents the style loss between the two images. Additionally, α in the formula represents the weight, and the optimal weight combination in subsequent step S50 is α∈[α...]. g ,α a ,α p ,α s Therefore, in the subsequent steps of this embodiment, the calculated minimum total loss value will be obtained by adding the L1 loss of the pixel values ​​in the two images to the weighted sum of the gradient loss, adversarial loss, perceptual loss, and style loss of the training image and the initial restored image.

[0083] It should be noted that different loss functions are used to calculate different loss values. Multiple loss values ​​of the training image and the initial restored image are considered comprehensively. By calculating their optimal weight combination, the minimum total loss value is obtained, thereby ensuring the fidelity of the final generated target restored image.

[0084] In one embodiment, such as Figure 3 As shown, step S20, which involves extracting the first feature vector of the mask image, specifically includes the following steps:

[0085] S21: Divide the mask image into multiple mask image blocks.

[0086] S22: Extract the feature vector corresponding to each of the masked image blocks as the first feature vector of the masked image.

[0087] In steps S21-S22, the mask image is segmented into multiple mask image blocks, and the vector features corresponding to each mask image are extracted as the first feature vector of the mask image. This step essentially extracts the spatial relationship features of the mask image and establishes an index. In step S21, in addition to simply dividing the image uniformly into several regular sub-blocks, the image can also be automatically segmented according to the objects or color regions contained in the image. Then, in step S22, the image feature vector of each region is extracted based on these regions and used as the first feature vector of the mask image.

[0088] Specifically, an Auto-Encoder algorithm is used for feature extraction of the mask image. Its decoder consists entirely of fully connected layers, and each mask image block is encoded independently. In this embodiment, due to the use of the PUT model, the Auto-Encoder employs P-VQVAE (Patch-based Vector Quantized Variational Auto-Encoder).

[0089] It should be noted that dividing the mask image into multiple mask image blocks and encoding each mask image block independently avoids interference between them, which helps to extract more discriminative features, making the final restored target image more accurate and faithful.

[0090] In one embodiment, such as Figure 4 As shown, step S50, which involves adjusting the weights of each loss value based on the bat algorithm to obtain the minimum total loss value and the optimal weight combination corresponding to the minimum total loss value, specifically includes the following steps:

[0091] S51: Initialize the weights of all the loss values ​​to obtain the first total loss value and the first weight combination.

[0092] S52: Iteratively adjust the algorithm parameters in the bat algorithm, obtain the second total loss value and the second weight combination for each iteration, and record the number of iterations.

[0093] S53: When the number of iterations reaches the expected number, the minimum value of the second total loss value is selected as the minimum total loss value.

[0094] S54: Take the second weight combination corresponding to the minimum value of the second total loss value as the optimal weight combination.

[0095] Steps S51 - S54 are essentially a process of calculating the optimal weight combination and the minimum total loss through the bat algorithm. In the bat algorithm, to reflect the decisions of the bats, the bat positions change randomly. Among them, the algorithm parameters include but are not limited to position parameters, speed parameters, frequency parameters, loudness parameters, emission rate parameters, etc. The position parameter in the algorithm parameters is the weight parameter, and the best bat position finally obtained by the bat algorithm is the best weight. In addition, the weights of each generation are based on the total loss value L of the initial restored image rec Calculate the total loss value of each generation according to the calculation formula of the optimized target value.

[0096] Specifically, step S51 is the first iteration. In this iteration, first, initialize several virtual bats, that is, initialize n virtual bats. Then, mark a certain bat among the n bats as the i-th bat (i is a positive integer, and its range is 0 < i ≤ n). Finally, mark the i-th bat as <α i (t), v i (t), f ri (t), A i (t), r i (t)>, and initialize these algorithm parameters. Among them, α i (t) refers to the position parameter of the i-th bat in the t-th generation, v i (t) is the speed parameter of the i-th bat in the t-th generation, f ri (t) is the frequency parameter, A i (t) is the loudness parameter, r i (t) is the emission rate parameter. In addition, the position parameter And the speed parameter Therefore, in this step, the weight α of the loss value is initialized i (t), so α i (t) is the first weight combination. According to the first weight combination, calculate the first total loss value through the calculation formula of the total loss value L rec Optimized target value.

[0097] In step S52, enter the next iteration. First, record the previous iteration number as t, so this iteration is the (t + 1)-th. Second, it is necessary to adjust each algorithm parameter. That is, on the one hand, the speed parameter will be updated to Among them, the best position found so far is p(t) = (p1(t), p2(t), …, p k (t), …, p D (t)), the above The equation can be viewed as velocity The combination of p(t) and its influence. On the other hand, the frequency parameter f ri The update method for (t) is as follows:

[0098]

[0099] in, and These are the maximum and minimum frequency values, respectively, while rand1 is a uniformly distributed random number within [0,1]. Due to the position parameter... Therefore, the position parameter α in the (t+1)th iteration i (t+1), that is, the second weight combination can be updated to:

[0100]

[0101] Here, rand3 is a random number generated uniformly within the range [0,1]. Then, according to α... i (t+1), update L rec The value is the second total loss value.

[0102] In step S53, the expected number of iterations is preset. When the expected number of iterations t+1 is reached, the iteration stops, and the minimum value of the second total loss value in each previous iteration is selected as the minimum total loss value for the training image and the restored image.

[0103] In step S54, based on the selected minimum total loss value, the corresponding second weight combination is obtained, which serves as the optimal weight combination for the minimum total loss value.

[0104] It should be noted that the bat algorithm is a heuristic search algorithm that measures the quality of a bat's position by solving a fitness function value. It compares the process of individual survival of the fittest to an optimization and search process, continuously replacing poor solutions with better ones until the optimal solution is found. This effectively balances and adjusts the relevant parameters of the algorithm, making the image restoration model structure simple and improving the efficiency and performance of actual image restoration.

[0105] In one embodiment, such as Figure 5 As shown, step S52, which iteratively adjusts the algorithm parameters in the bat algorithm to obtain the second total loss value and the second weight combination for each iteration, specifically includes the following steps:

[0106] S521: Determine whether the algorithm parameters are greater than the expected value.

[0107] S522: If it is greater than, then the second weight combination is obtained by globally searching the current optimal solution of the bat algorithm.

[0108] S523: Based on the second weight combination, derive the second total loss value and update the algorithm parameters.

[0109] When adjusting the algorithm parameters, the frequency parameter f was updated. ri After (t), in order to reflect the bat's decision, the bat's position will change randomly. Assume the expected value of the change is rand2, where rand2 is a uniformly distributed random number within [0,1]. In step S521, it is necessary to determine whether the emissivity parameter r1(t) in the algorithm parameters is greater than the expected value rand2.

[0110] In steps S522-S523, if rand2 <r i If (t), meaning the algorithm parameters are greater than the expected value, then the i-th bat will execute the following global search pattern to find the optimal solution for the bat's next position:

[0111]

[0112] Based on the above formula, the second weight combination is obtained. After updating the position of the i-th bat, the corresponding loudness parameter A is... i (t+1) and the emissivity parameter r1(t+1) are then updated as follows:

[0113] A i (t+1)=βA i (t)

[0114] r1(t+1)=r(0)×(1-e -γt )

[0115] Here, β>0 and γ>0 are two predefined parameters, and A(0) and r(0) are the loudness parameter and emissivity parameter initialized during the first iteration, respectively.

[0116] It should be noted that steps S521-S523, which initialize a set of random weight combinations and then iteratively search for the minimum total loss value, demonstrate that this algorithm is simple in structure, has few parameters, is robust, and is easy to understand and implement. Due to these characteristics, its application to image inpainting models effectively improves the efficiency and performance of image inpainting.

[0117] In one embodiment, such as Figure 5 As shown, after step S521, i.e., after determining whether the algorithm parameters are greater than the expected value, the image restoration method further includes the following steps:

[0118] S91: If it is less than or equal to the current optimal solution, then a second weight combination is obtained by local search.

[0119] S92: Based on the second weight combination, the second total loss value is obtained, and the algorithm parameters are updated.

[0120] After determining whether the emissivity parameter r1(t) in the algorithm parameters is greater than the expected value rand2, if rand2≥r i If (t), meaning the algorithm parameter is less than or equal to the expected value, then the i-th bat will execute the following local search mode, also known as the local search mode, to find the optimal solution for the next position:

[0121]

[0122] in It is a random number generated from a uniform distribution within the range [-1, 1]. The average loudness of all bats is calculated by the following formula:

[0123]

[0124] Based on the above formula, the second weight combination is obtained. After updating the position of the i-th bat, the corresponding loudness parameter A is updated using the same method as in step S523 above. i (t+1) and emissivity parameter r1(t+1).

[0125] It's important to note that, in addition to searching for the global minimum total loss (the optimal solution) through algorithms, this method also generates new local minimum total losses around the minimum total loss value by observing the random flight positions of the bats. This effectively considers the interactions and influences between individual bats, making the final optimal weight combination more accurate and effective, thus improving the accuracy of the image restoration model and better ensuring the fidelity of the restored image. Furthermore, due to the search for local optima, compared to other methods, it doesn't require adjusting too many parameters, resulting in better accuracy and effectiveness.

[0126] In one embodiment, in step S52, each time the weight α of the (t+1)th generation bat is calculated... i (t+1), L was updated. rec After setting the value, the current historical best weight value needs to be updated once, until the number of iterations reaches the expected number.

[0127] In one embodiment, such as Figure 6 As shown, before step S20, that is, before extracting the first feature vector of the mask image, the image restoration method further includes the following steps:

[0128] S101: Determine whether the mask image has missing pixels.

[0129] S102: If it exists, the feature probability of the mask image is sampled from the pre-encoded codebook to obtain the first feature vector.

[0130] For all masked images, the UQ Transformer in the PUT model outputs a probability, which is the probability of the features in the codebook corresponding to the masked image or masked image block, also known as the feature probability.

[0131] When any pixel is missing in the mask image, the corresponding probability is sampled, and the feature vector obtained from the codebook is used as the feature of the mask image block and sent to the decoder for image reconstruction.

[0132] It should be noted that determining whether there are missing pixels in the mask image effectively prevents unnecessary computational waste when there are no missing pixels, thus ensuring the performance of the image restoration model and the efficiency of image restoration.

[0133] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0134] In one embodiment, an image restoration apparatus is provided, which corresponds one-to-one with the image restoration methods described in the above embodiments. For example... Figure 7 As shown, the image restoration device includes an occlusion module 10, a first quantization module 20, a reconstruction module 30, a calculation module 40, a loss module 50, a first output module 60, a second quantization module 70, and a second output module 80. Detailed descriptions of each functional module are as follows:

[0135] The occlusion module 10 is used to partially occlude the training image to obtain a mask image;

[0136] The first quantization module 20 is used to extract the first feature vector of the mask image;

[0137] The reconstruction module 30 is used to reconstruct the initial restored image based on the first feature vector;

[0138] The calculation module 40 is used to calculate multiple loss values ​​between the training image and the initial restored image;

[0139] Loss module 50 is used to calculate the optimal weight of each loss value based on the bat algorithm, and to obtain the minimum loss value;

[0140] The first output module 60 is used to adjust the model parameters of the initial image restoration model according to the optimal weights to obtain the image restoration model.

[0141] The second quantization module 70 is used to extract the second feature vector of the incomplete image;

[0142] The second output module 80 is used to input the second feature vector into the image restoration model to reconstruct the target restored image.

[0143] In one embodiment, the plurality of loss values ​​of the calculation module 40 include regression loss, gradient loss, adversarial loss, perceptual loss, and style loss of the training image and the initial restored image.

[0144] In one embodiment, the first quantization module 20 is specifically used for:

[0145] The mask image is divided into multiple mask image blocks;

[0146] Extract the feature vector corresponding to each of the masked image blocks, and use it as the first feature vector of the masked image.

[0147] In one embodiment, the loss module 50 is specifically used for:

[0148] Initialize the weights of all the aforementioned loss values ​​to obtain a first total loss value and a first weight combination;

[0149] The algorithm parameters in the bat algorithm are adjusted iteratively to obtain the second total loss value and the second weight combination for each iteration, and the number of iterations is recorded.

[0150] When the number of iterations reaches the expected number, the minimum value of the second total loss value is selected as the minimum total loss value.

[0151] The second weight combination corresponding to the minimum value of the second total loss is taken as the optimal weight combination.

[0152] In one embodiment, the loss module 50 is further configured to:

[0153] Determine whether the algorithm parameters are greater than the expected value;

[0154] If it is greater than the current optimal solution of the bat algorithm, a second weight combination is obtained by global search.

[0155] If it is less than or equal to the current optimal solution, then a second weight combination is obtained by local search.

[0156] Based on the second weight combination, the second total loss value is obtained, and the algorithm parameters are updated.

[0157] In one embodiment, prior to the first quantization module 20, the image restoration apparatus further includes:

[0158] The judgment module 90 is used to determine whether there are missing pixels in the mask image;

[0159] The sampling module 100 is used to sample the feature probabilities of the mask image from the pre-encoded codebook if the mask image has missing pixels, so as to obtain the first feature vector.

[0160] For specific limitations regarding the image restoration device, please refer to the limitations of the image restoration method described above, which will not be repeated here. Each module in the aforementioned image restoration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0161] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores acquired image data, as well as various images and data generated during image processing. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements an image restoration method.

[0162] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0163] The training image is partially occluded to obtain a mask image;

[0164] Extract the first feature vector of the mask image;

[0165] Based on the first feature vector, the initial restored image is reconstructed;

[0166] Calculate multiple loss values ​​for the training image and the initial restored image;

[0167] Based on the bat algorithm, the weights of each loss value are adjusted to obtain the minimum total loss value and the optimal weight combination corresponding to the minimum total loss value;

[0168] Based on the optimal weight combination, the model parameters of the initial image restoration model are adjusted to obtain the image restoration model;

[0169] Extract the second feature vector from the incomplete image;

[0170] The second feature vector is input into the image restoration model to reconstruct the target restored image.

[0171] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0172] The training image is partially occluded to obtain a mask image;

[0173] Extract the first feature vector of the mask image;

[0174] Based on the first feature vector, the initial restored image is reconstructed;

[0175] Calculate multiple loss values ​​for the training image and the initial restored image;

[0176] Based on the bat algorithm, the weights of each loss value are adjusted to obtain the minimum total loss value and the optimal weight combination corresponding to the minimum total loss value;

[0177] Based on the optimal weight combination, the model parameters of the initial image restoration model are adjusted to obtain the image restoration model;

[0178] Extract the second feature vector from the incomplete image;

[0179] The second feature vector is input into the image restoration model to reconstruct the target restored image.

[0180] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0181] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0182] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for image restoration, characterized in that, include: The training image is partially occluded to obtain a mask image; Extract the first feature vector of the mask image; Based on the first feature vector, the initial restored image is reconstructed; Calculate multiple loss values ​​for the training image and the initial restored image; Based on the bat algorithm, the weights of each loss value are adjusted to obtain the minimum total loss value and the optimal weight combination corresponding to the minimum total loss value; Based on the optimal weight combination, the model parameters of the initial image restoration model are adjusted to obtain the image restoration model; Extract the second feature vector from the incomplete image; The second feature vector is input into the image restoration model to reconstruct the target restored image; The step of adjusting the weights of each loss value based on the Bat Algorithm to obtain the minimum total loss value and the optimal weight combination corresponding to the minimum total loss value includes: initializing the weights of all loss values ​​to obtain a first total loss value and a first weight combination; iteratively adjusting the algorithm parameters in the Bat Algorithm to obtain a second total loss value and a second weight combination for each iteration, and recording the number of iterations; when the number of iterations reaches the expected number, selecting the minimum value of the second total loss value as the minimum total loss value; and using the second weight combination corresponding to the minimum value of the second total loss value as the optimal weight combination. The algorithm parameters include an emissivity parameter and a loudness parameter. The iterative adjustment of the algorithm parameters in the bat algorithm to obtain a second total loss value and a second weight combination for each iteration includes: determining whether the emissivity parameter in the algorithm parameters is greater than the expected value; if it is greater, then obtaining the second weight combination by globally searching the current optimal solution of the bat algorithm; if it is less than or equal to, then obtaining the second weight combination by locally searching the current optimal solution and combining it with the average loudness of all bats; and obtaining the second total loss value based on the second weight combination and updating the algorithm parameters.

2. The repair method as described in claim 1, characterized in that, The multiple loss values ​​include: regression loss, gradient loss, adversarial loss, perceptual loss, and style loss of the training image and the initial restored image.

3. The repair method as described in claim 1, characterized in that, The extraction of the first feature vector of the mask image includes: The mask image is divided into multiple mask image blocks; Extract the feature vector corresponding to each of the masked image blocks, and use it as the first feature vector of the masked image.

4. The repair method as described in claim 1, characterized in that, Before extracting the first feature vector of the masked image, the method further includes: Determine whether the mask image has missing pixels; If it exists, the feature probabilities of the masked image are sampled from the pre-encoded codebook to obtain the first feature vector.

5. An image restoration apparatus, characterized in that, include: The occlusion module is used to partially occlude the training image to obtain a mask image; The first quantization module is used to extract the first feature vector of the mask image; The reconstruction module is used to reconstruct the initial restored image based on the first feature vector; The calculation module is used to calculate multiple loss values ​​between the training image and the initial restored image; The loss module is used to adjust the weight of each loss value based on the bat algorithm to obtain the minimum total loss value and the optimal weight combination corresponding to the minimum total loss value. The first output module is used to adjust the model parameters of the initial image restoration model according to the optimal weight combination, so as to obtain the image restoration model. The second quantization module is used to extract the second feature vector of the incomplete image; The second output module is used to input the second feature vector into the image restoration model to reconstruct the target restored image; The step of adjusting the weights of each loss value based on the Bat Algorithm to obtain the minimum total loss value and the optimal weight combination corresponding to the minimum total loss value includes: initializing the weights of all loss values ​​to obtain a first total loss value and a first weight combination; iteratively adjusting the algorithm parameters in the Bat Algorithm to obtain a second total loss value and a second weight combination for each iteration, and recording the number of iterations; when the number of iterations reaches the expected number, selecting the minimum value of the second total loss value as the minimum total loss value; and using the second weight combination corresponding to the minimum value of the second total loss value as the optimal weight combination. The algorithm parameters include an emissivity parameter and a loudness parameter. The iterative adjustment of the algorithm parameters in the bat algorithm to obtain a second total loss value and a second weight combination for each iteration includes: determining whether the emissivity parameter in the algorithm parameters is greater than the expected value; if it is greater, then obtaining the second weight combination by globally searching the current optimal solution of the bat algorithm; if it is less than or equal to, then obtaining the second weight combination by locally searching the current optimal solution and combining it with the average loudness of all bats; and obtaining the second total loss value based on the second weight combination and updating the algorithm parameters.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the repair method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the repair method as described in any one of claims 1 to 4.

Images