Image restoration method and system

Abstract

The invention relates to an image restoration method and system, and specifically, the method comprises the steps: inputting a to-be-restored image into an encoder to obtain hidden variables, obtaining the shortest distance between each feature of the hidden variables and a codebook, and sorting the features in the hidden variables according to the shortest distances; taking the sorted first feature as an initial feature, and determining the embedding of the initial feature mapped into a codebook; acquiring a neighborhood of the initial feature in the hidden variable, if the neighborhood is not mapped into the codebook, taking the distance between the initial feature and the neighborhood as a target distance, and determining the embedding of the neighborhood mapped into the codebook according to the target distance and / or the neighbor of the neighborhood in the codebook; when all neighborhoods of the initial features are mapped to the embedding of the codebook, deleting the initial features from the sequence; the discretization result of the hidden variables is obtained until all the characteristics of the hidden variables are mapped to the embedding of the codebook; and inputting a discretization result into a decoder to obtain a repaired image.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image restoration method and system. Background Art

[0002] Image inpainting primarily fills in missing areas in an image, ensuring that the restored image maintains consistency in structure, texture, and color with the original image. This technology has applications in a variety of fields, including cultural relic restoration, medical image completion, photo restoration, video editing, and image de-occlusion. Traditional image inpainting methods primarily include diffusion methods based on partial differential equations (PDEs) and texture synthesis methods. The former uses a diffusion model to propagate pixel information to fill missing areas and is suitable for image inpainting with small missing areas; the latter utilizes similar textures for filling and is more effective for inpainting images with repetitive textures. However, these traditional methods often struggle to generate content that matches the distribution of the real image when inpainting images with large missing areas or complex structures.

[0003] Compared to traditional VAEs, VQ-VAE can better capture the local structure and global semantics of data, making it more applicable to image restoration tasks. Combining VQ-VAE with autoregressive models (such as PixelCNN) or Transformer can further improve the realism and coherence of the restored area. However, because VQ-VAE introduces a codebook, it discretizes the latent variables output by the encoder using the nearest neighbor method. Since each feature in the latent variable is replaced with the feature in the codebook that is most similar to the latent variable, the difference between the two features in the latent variable increases after discretization, which can lead to discontinuity, distortion, or even unnaturalness in the restored image. Improving the continuity and naturalness of the restored image is crucial for image restoration. Summary of the Invention

[0004] In the current image restoration, due to the discretization process of the latent variables of the VQ-VAE model, the nearest neighbor is used to map the latent variables to the codebook, which may increase the latent variable error and thus make the image restoration effect unsatisfactory.

[0005] In a first aspect of the present invention, there is provided an image restoration method, the method comprising the following steps: S1: Input the image to be repaired into the encoder to obtain a latent variable, obtain the shortest distance from each feature of the latent variable to the embedding in the codebook, and sort the features in the latent variable according to the shortest distance; S2: Take the first feature after sorting as the initial feature and determine the embedding of the initial feature mapped to the codebook; obtain the neighborhood of the initial feature in the latent variable. If the neighborhood is not mapped to the embedding in the codebook, the distance between the initial feature and the neighborhood is used as the target distance. The embedding of the neighborhood mapped to the codebook is determined based on the target distance and / or the neighborhood of the neighborhood in the codebook; when all neighbors of the initial feature are mapped to the embedding of the codebook, the initial feature is deleted from the sorting; S3, repeat S2 continuously until all features of the latent variable are mapped to the embedding of the codebook to obtain the discretization result of the latent variable; S4, inputs the discretization result into the decoder to obtain the restored image.

[0006] Preferably, the determining of embedding the initial feature map into the codebook is specifically as follows: If the initial feature has been mapped to an embedding in the codebook, the embedding is used as the embedding mapped to the codebook by the initial feature; If the initial feature is not mapped to an embedding in the codebook, the embedding in the codebook closest to the initial feature is used as the embedding mapped to the codebook.

[0007] Preferably, the embedding of the neighborhood mapping to the codebook is determined according to the target distance and / or the neighbors of the neighborhood in the codebook, specifically: Calculate the k nearest neighbors of the neighborhood in the codebook, calculate the distance between the embedding of the initial feature map to the codebook and each of the k nearest neighbors, and use the nearest neighbor with the closest distance to the target as the embedding of the neighborhood map to the codebook.

[0008] Preferably, the embedding of the neighborhood mapping to the codebook is determined according to the target distance and / or the neighbors of the neighborhood in the codebook, specifically: Obtain the neighbors of the neighborhood in the latent variable, and if the neighbors in the latent variable have been mapped to the embedding of the codebook, add the neighbors to the neighbor set; calculate the distance from the neighborhood to each neighbor in the neighbor set; Get the k nearest neighbors of the neighborhood in the codebook. For each nearest neighbor, calculate the average deviation between the distance from the neighborhood to each neighbor in the neighbor set and the distance from the nearest neighbor to each neighbor in the neighbor set mapped to the codebook embedding. The nearest neighbor with the minimum average deviation is used as the embedding of the neighborhood mapped to the codebook.

[0009] Preferably, the average deviation of the distance from the computing neighborhood to each element in the neighbor set and the distance from each nearest neighbor to each element in the neighbor set mapped to the codebook embedding is specifically: Calculating the distance from the neighborhood to each neighbor in the neighbor set to obtain a distance set on the latent variable; Calculating the distance from the nearest neighbor to the embedding of each neighbor in the neighbor set on the codebook to obtain a distance set on the codebook; The distance set on the latent variable and the distance set on the codebook are subtracted bit by bit, and then the absolute value is taken. Then, the average of all absolute values ​​is calculated, and the average of the absolute values ​​is used as the average deviation.

[0010] In a second aspect of the present invention, an image restoration system is provided, the system comprising the following modules: An encoding module is used to input the image to be repaired into an encoder to obtain a latent variable, obtain the shortest distance from each feature of the latent variable to the embedding in the codebook, and sort the features in the latent variable according to the shortest distance; A mapping module is configured to use the first feature after sorting as the initial feature and determine the embedding of the initial feature mapped to the codebook; obtain the neighborhood of the initial feature in the latent variable, and if the neighborhood is not mapped to the embedding in the codebook, use the distance between the initial feature and the neighborhood as the target distance, and determine the embedding of the neighborhood mapped to the codebook based on the target distance and / or the neighborhood of the neighborhood in the codebook; when all neighbors of the initial feature are mapped to the embedding of the codebook, the initial feature is deleted from the sorting; An iterative module, configured to continuously repeat the mapping module until all features of the latent variable are mapped to the embedding of the codebook to obtain a discretization result of the latent variable; The restoration module is used to input the discretization result into the decoder to obtain the restored image.

[0011] Preferably, the determining of embedding the initial feature map into the codebook is specifically as follows: If the initial feature has been mapped to an embedding in the codebook, the embedding is used as the embedding mapped to the codebook by the initial feature; If the initial feature is not mapped to an embedding in the codebook, the embedding in the codebook closest to the initial feature is used as the embedding mapped to the codebook.

[0012] Preferably, the embedding of the neighborhood mapping to the codebook is determined according to the target distance and / or the neighbors of the neighborhood in the codebook, specifically: Calculate the k nearest neighbors of the neighborhood in the codebook, calculate the distance between the embedding of the initial feature map to the codebook and each of the k nearest neighbors, and use the nearest neighbor with the closest distance to the target as the embedding of the neighborhood map to the codebook.

[0013] Preferably, the embedding of the neighborhood mapping to the codebook is determined according to the target distance and / or the neighbors of the neighborhood in the codebook, specifically: Obtain the neighbors of the neighborhood in the latent variable, and if the neighbors in the latent variable have been mapped to the embedding of the codebook, add the neighbors to the neighbor set; calculate the distance from the neighborhood to each neighbor in the neighbor set; Get the k nearest neighbors of the neighborhood in the codebook. For each nearest neighbor, calculate the average deviation between the distance from the neighborhood to each neighbor in the neighbor set and the distance from the nearest neighbor to each neighbor in the neighbor set mapped to the codebook embedding. The nearest neighbor with the minimum average deviation is used as the embedding of the neighborhood mapped to the codebook.

[0014] Preferably, the average deviation of the distance from the computing neighborhood to each element in the neighbor set and the distance from each nearest neighbor to each element in the neighbor set mapped to the codebook embedding is specifically: Calculating the distance from the neighborhood to each neighbor in the neighbor set to obtain a distance set on the latent variable; Calculating the distance from the nearest neighbor to the embedding of each neighbor in the neighbor set on the codebook to obtain a distance set on the codebook; The distance set on the latent variable and the distance set on the codebook are subtracted bit by bit, and then the absolute value is taken. Then, the average of all absolute values ​​is calculated, and the average of the absolute values ​​is used as the average deviation.

[0015] The VQ-VAE model of the present invention considers the neighborhood relationship between latent variable features during the discretization process. By calculating the distance between the initial feature and the neighborhood as the target distance and determining the neighborhood mapping based on the target distance and / or the neighborhood's nearest neighbors in the codebook, it can effectively maintain the continuity of the image and avoid unnatural breaks or mutations. This improves the restoration quality and makes the restored image more natural and realistic. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of Example 1; Figure 2 Schematic diagram of replacing features in latent variables with embeddings in the codebook; Figure 3 Schematic diagram of finding the nearest neighbor for latent variable embedding from the codebook in the original VQ-VAE model; Figure 4 Schematic diagram of the relationship between initial features and neighborhood; Figure 5 A schematic diagram of the relationship between neighbors and their neighbors. DETAILED DESCRIPTION

[0017] In this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0019] Example 1 like Figure 1 As shown, Figure 1 An image restoration method is provided, comprising the following steps: S1: Input the image to be repaired into the encoder to obtain a latent variable, obtain the shortest distance from each feature of the latent variable to the embedding in the codebook, and sort the features in the latent variable according to the shortest distance; The image to be restored is input into an encoder, which converts the image into latent variables. In one embodiment, the encoder uses the encoder in the VQ-VAE model. The latent variables output by the encoder are continuous values. More specifically, the encoder includes at least one activation layer or Reinforced Luminance (ReLU) layer. In the original VQ-VAE model, the latent variables have multiple features, each of which is a vector. Figure 2 The figure shows the process of discretizing latent variables through the codebook in the original VQ-VAE model. Since the original VQ-VAE model uses the most similar embedding method to determine the replacement or mapping of features in the latent variables, it is possible that feature 1 and feature 2 are close in the latent variables. However, since feature 1 is mapped to the most similar embedding and feature 2 is mapped to the most similar embedding, after mapping, the distance between feature 1 and feature 2 in the discrete results becomes larger, as shown in the figure. Figure 3 As shown, Figure 3In the latent variable, the distance between features 1 and 2 is 2, but after mapping to the discrete result, the distance becomes 1, which may lead to the inpainting problem mentioned above. In the present invention, for each feature in the latent variable, a most similar feature is found in the codebook and the distance between them is calculated. After obtaining the most similar embedding (code) of the latent variable feature from the codebook, the distance between the latent variable feature and the most similar embedding is calculated, and the features in the latent variable are sorted in ascending order of these distances. For example, the image to be inpainted is encoded to generate a latent variable containing 6 features, and the codebook contains 20 embeddings. The distances between these 6 features and the 20 embeddings are calculated to obtain a 6x20 distance matrix. For each feature, the embedding with the closest distance is found and the shortest distance is recorded. The features are then sorted according to these 6 shortest distances, preferably in ascending order. Assume that after sorting, the order of the latent variable features is S3, S1, S4, S2, S6, and S5. In one embodiment, the latent variable is an h×w×c matrix, where h is the height, w is the width, and c is the number of channels. The latent variable contains h×w features, each of which is a 1×c vector. The codebook is a k×c matrix, which means that the codebook has k embeddings (also called codes), each of which is a 1×c vector.

[0020] S2: Take the first feature after sorting as the initial feature and determine the embedding of the initial feature mapped to the codebook; obtain the neighborhood of the initial feature in the latent variable. If the neighborhood is not mapped to the embedding in the codebook, the distance between the initial feature and the neighborhood is used as the target distance. The embedding of the neighborhood mapped to the codebook is determined based on the target distance and / or the neighborhood of the neighborhood in the codebook; when all neighbors of the initial feature are mapped to the embedding of the codebook, the initial feature is deleted from the sorting; S3, repeat S2 continuously until all features of the latent variable are mapped to the embedding of the codebook to obtain the discretization result of the latent variable; In step S1, the features in the latent variable are sorted by their distance from the codebook embedding. The first feature after sorting is selected as the initial feature. For this initial feature, the most appropriate embedding is found in the codebook to represent it. In one embodiment, the distance between the initial feature and all embeddings in the codebook is calculated, and the embedding with the closest distance is selected as the mapping of the initial feature. The mapping refers to establishing an association between the two. After all features of the latent variable are mapped, the features in the latent variable are replaced with the embeddings in the codebook to obtain the discretized result.

[0021] In yet another embodiment, in subsequent iterations, it is possible that the initial feature has been mapped to an embedding in the codebook in the previous step, and determining the embedding mapped to the codebook by the initial feature is specifically as follows: if the initial feature has been mapped to an embedding in the codebook, the embedding is used as the embedding mapped to the codebook by the initial feature; if the initial feature is not mapped to an embedding in the codebook, the embedding in the codebook that is closest to the initial feature is used as the embedding mapped to the codebook by the initial feature.

[0022] In the latent variable, each feature has its adjacent features, which form the neighborhood of the feature. The features in the neighborhood are close to the initial features in space. Figure 4 A neighborhood of the initial feature is shown, preferably, the neighborhood is a 4-neighborhood or an 8-neighborhood. If a neighborhood has not yet been mapped to the codebook, the distance between the initial feature and the neighborhood feature is calculated, and this distance is used as the target distance. In order to maintain the relationship between the neighborhood feature and the initial feature, an embedding is found in the codebook so that the distance between the embedding and the neighborhood feature is close to the target distance. This ensures that the discretized features can maintain the original relative relationship. When all the neighborhoods of the initial feature have been mapped to the codebook, it means that the processing of the initial feature has been completed and it can be deleted from the sorting in order to process the next feature.

[0023] Repeat step S2 until all features in the latent variable are mapped into the codebook. When all features are mapped into the codebook, the discretized representation of the latent variable is obtained.

[0024] In one embodiment, the embedding of the neighborhood mapping to the codebook is determined according to the target distance and / or the neighbors of the neighborhood in the codebook, specifically: Calculate the k nearest neighbors of the neighborhood in the codebook, calculate the distance between the embedding of the initial feature map to the codebook and each of the k nearest neighbors, and use the nearest neighbor with the closest distance to the target as the embedding of the neighborhood map to the codebook.

[0025] For a neighborhood, the k most similar embeddings are found in the codebook. Similarity refers to the distance between feature vectors; closer distances indicate more similar features. The mapping of the initial feature in the codebook has been determined, meaning an embedding has been found to represent the initial feature. The distances between this initial feature embedding and the k nearest neighbor embeddings are then calculated. After calculating the distances between the initial feature embedding and the k nearest neighbor embeddings, the most suitable embedding is selected to represent the neighborhood feature. The optimality criterion is that the distance between this embedding and the initial feature embedding should be closest to the previously calculated target distance (i.e., the distance between the initial feature and the neighborhood feature). This approach maintains the relative relationship between the neighborhood feature and the initial feature as much as possible, allowing the discretized features to better preserve the original information. Suppose the initial feature has a neighborhood, and the distance between them in the latent variable, also known as the target distance, is 5. Three nearest neighbor embeddings are found in the codebook, with distances of 3, 6, and 8 to the initial feature embedding, respectively. Therefore, the embedding with a distance of 6 is selected as the mapping for the neighborhood feature because it is closest to the target distance of 5.

[0026] In yet another optional embodiment, the embedding of the neighborhood mapping to the codebook is determined according to the target distance and / or the neighbors of the neighborhood in the codebook, specifically: Obtain the neighbors of the neighborhood in the latent variable, and if the neighbors in the latent variable have been mapped to the embedding of the codebook, add the neighbors to the neighbor set; calculate the distance from the neighborhood to each neighbor in the neighbor set; Get the k nearest neighbors of the neighborhood in the codebook. For each nearest neighbor, calculate the average deviation between the distance from the neighborhood to each neighbor in the neighbor set and the distance from the nearest neighbor to each neighbor in the neighbor set mapped to the codebook embedding. The nearest neighbor with the minimum average deviation is used as the embedding of the neighborhood mapped to the codebook.

[0027] For a neighborhood of the initial feature, find its neighbors in the latent variable; the neighbors refer to the features adjacent to the feature in the latent variable space. Figure 5The neighbors of the neighborhood are shown. After finding these neighbor features, determine whether they have been mapped to the embedding in the codebook. If a neighbor has been mapped to the codebook embedding, it can be used to help determine the mapping of the current neighborhood feature; these mapped neighbor features are added to a neighbor set. Calculate the distance between the current neighborhood feature and each neighbor feature in the neighbor set. Similar to the previous embodiment, find the k embeddings most similar to the current neighborhood feature in the codebook as candidate mappings. For each of the k nearest neighbors, calculate the distance from it to the codebook embedding mapped to each neighbor in the neighbor set, calculate the distance from the current neighborhood feature to each neighbor in the neighbor set, and compare it with the distance from the nearest neighbor to the neighbor set calculated previously, calculate their absolute errors, average these absolute errors to obtain an average deviation, which represents the consistency of the nearest neighbor embedding and the neighborhood feature in terms of the relationship with the neighbor set. Select the one with the smallest average deviation among the k nearest neighbors as the mapping of the current neighborhood feature.

[0028] Assume that the first feature after sorting is F1. As the initial feature, the embedding closest to F1 in the codebook is E1. Map F1 to E1. F1's neighborhood includes features F2 and F3. F2 has not yet been mapped, while F3 has been mapped to embedding E3. Since F2 has not yet been mapped, find F2's neighbors in the latent variable, assuming they are F3-F5. F3 has been mapped to E3, while F4-F5 have not. Since F3 has been mapped, F3 is added to the neighbor set. Find F2's three nearest neighbor embeddings in the codebook: E2a, E2b, and E2c. Calculate the distance from E2a to E3, denoted as D2a, and the distance from F2 to F3, denoted as D2. Calculate |D2a - D2| and obtain the absolute error. Since there is only one element in the neighbor set, F3, this absolute error is the average deviation of E2a. Repeat the above steps for E2b and E2c, calculating the average deviation of E2b and E2c. Assuming that E2a has the smallest average deviation, F2 is mapped to E2a.

[0029] The average deviation of the distance from the calculated neighborhood to each element in the neighbor set and the distance from each nearest neighbor to each element in the neighbor set mapped to the codebook embedding is specifically: Calculating the distance from the neighborhood to each neighbor in the neighbor set to obtain a distance set on the latent variable; Calculating the distance from the nearest neighbor to the embedding of each neighbor in the neighbor set on the codebook to obtain a distance set on the codebook; The distance set on the latent variable and the distance set on the codebook are subtracted bit by bit, and then the absolute value is taken. Then, the average of all absolute values ​​is calculated, and the average of the absolute values ​​is used as the average deviation.

[0030] Compute the distance from the neighborhood feature to each neighbor in the neighbor set. The neighbor set is the set of neighbor features mapped to the codebook. This distance calculation is performed in the latent variable space, hence the term "distance on latent variables." Assuming there are n neighbors in the neighbor set, we obtain a set of n distance values, called the distance set on latent variables.

[0031] Compute the distance from the nearest neighbor (i.e., one of the k nearest neighbors) to the codebook embedding of each neighbor in the neighbor set. This distance calculation is performed in codebook space and is called the on-codebook distance. Similarly, if there are n neighbors in the neighbor set, the resulting set of n distance values ​​is called the on-codebook distance set.

[0032] The distance values ​​in the distance set on the latent variable and the distance set on the codebook are subtracted bit by bit, and then the absolute value is taken. That is, for each neighbor in the neighbor set, the following absolute error is calculated: |(distance from the nearest neighbor to the neighbor's codebook embedding) - (distance from the neighbor to the neighbor)|. The average of these n absolute errors is the average deviation.

[0033] Suppose the neighborhood feature is F2, whose neighbor set includes F1 and F3, and the nearest neighbor embedding of F2 is E2a. Calculate the distance D2_F1 from F2 to F1 and the distance D2_F3 from F2 to F3. The distance set on the latent variables is {D2_F1, D2_F3}. Calculate the distance D2a_E1 from E2a to the codebook embedding E1 of F1 and the distance D2a_E3 from E2a to the codebook embedding E3 of F3. The distance set on the codebook is {D2a_E1, D2a_E3}. Calculate the absolute errors |D2a_E1 - D2_F1| and |D2a_E3 - D2_F3|. Calculate the average of these two absolute errors as the mean deviation of E2a.

[0034] S4, inputs the discretization result into the decoder to obtain the restored image.

[0035] The discretization result is the quantized and indexed result of the latent variables extracted by the encoder. It is a vector in which each element represents a feature of the image in the latent space. Unlike the original latent variables, each element in the discretization result is an integer corresponding to an embedding in the codebook. The decoder takes the discretization result as input and, through a series of convolution and upsampling operations, ultimately generates the restored image.

[0036] Example 2 like Figure 4 As shown, an image restoration system is provided, the system comprising the following modules: An encoding module is used to input the image to be repaired into an encoder to obtain a latent variable, obtain the shortest distance from each feature of the latent variable to the embedding in the codebook, and sort the features in the latent variable according to the shortest distance; A mapping module is configured to use the first feature after sorting as the initial feature and determine the embedding of the initial feature mapped to the codebook; obtain the neighborhood of the initial feature in the latent variable, and if the neighborhood is not mapped to the embedding in the codebook, use the distance between the initial feature and the neighborhood as the target distance, and determine the embedding of the neighborhood mapped to the codebook based on the target distance and / or the neighborhood of the neighborhood in the codebook; when all neighbors of the initial feature are mapped to the embedding of the codebook, the initial feature is deleted from the sorting; An iterative module, configured to continuously repeat the mapping module until all features of the latent variable are mapped to the embedding of the codebook to obtain a discretization result of the latent variable; The restoration module is used to input the discretization result into the decoder to obtain the restored image.

[0037] Preferably, the determining of embedding the initial feature map into the codebook is specifically as follows: If the initial feature has been mapped to an embedding in the codebook, the embedding is used as the embedding mapped to the codebook by the initial feature; If the initial feature is not mapped to an embedding in the codebook, the embedding in the codebook closest to the initial feature is used as the embedding mapped to the codebook.

[0038] Preferably, the embedding of the neighborhood mapping to the codebook is determined according to the target distance and / or the neighbors of the neighborhood in the codebook, specifically: Calculate the k nearest neighbors of the neighborhood in the codebook, calculate the distance between the embedding of the initial feature map to the codebook and each of the k nearest neighbors, and use the nearest neighbor with the closest distance to the target as the embedding of the neighborhood map to the codebook.

[0039] Preferably, the embedding of the neighborhood mapping to the codebook is determined according to the target distance and / or the neighbors of the neighborhood in the codebook, specifically: Obtain the neighbors of the neighborhood in the latent variable, and if the neighbors in the latent variable have been mapped to the embedding of the codebook, add the neighbors to the neighbor set; calculate the distance from the neighborhood to each neighbor in the neighbor set; Get the k nearest neighbors of the neighborhood in the codebook. For each nearest neighbor, calculate the average deviation between the distance from the neighborhood to each neighbor in the neighbor set and the distance from the nearest neighbor to each neighbor in the neighbor set mapped to the codebook embedding. The nearest neighbor with the minimum average deviation is used as the embedding of the neighborhood mapped to the codebook.

[0040] Preferably, the average deviation of the distance from the computing neighborhood to each element in the neighbor set and the distance from each nearest neighbor to each element in the neighbor set mapped to the codebook embedding is specifically: Calculating the distance from the neighborhood to each neighbor in the neighbor set to obtain a distance set on the latent variable; Calculating the distance from the nearest neighbor to the embedding of each neighbor in the neighbor set on the codebook to obtain a distance set on the codebook; The distance set on the latent variable and the distance set on the codebook are subtracted bit by bit, and then the absolute value is taken. Then, the average of all absolute values ​​is calculated, and the average of the absolute values ​​is used as the average deviation.

[0041] Finally, the present invention further provides a computer program, which, when executed, implements the method described in Example 1. In addition, the present invention further provides a computer-readable storage medium, which stores the computer program.

[0042] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using a general-purpose hardware platform, or alternatively, through a combination of hardware and software. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it, and other embodiments may also be used. Although the present invention has been described in detail with reference to the aforementioned embodiments, ordinary technicians in this field should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An image restoration method, characterized in that: The method comprises the following steps: S1, input the image to be repaired into the encoder to obtain a latent variable, obtain the shortest distance from each feature of the latent variable to the embedding in the codebook, and sort the features in the latent variable according to the shortest distance; S2, taking the first feature after sorting as the initial feature, determining the embedding of the initial feature mapped to the codebook; obtaining the neighborhood of the initial feature in the latent variable, if the neighborhood is not mapped to the embedding in the codebook, taking the distance between the initial feature and the neighborhood as the target distance, and determining the embedding of the neighborhood mapped to the codebook according to the target distance and / or the nearest neighbors of the neighborhood in the codebook; when all the neighbors of the initial feature are mapped to the embedding of the codebook, the initial feature is deleted from the sorting; S3, repeat S2 continuously until all the features of the latent variable are mapped to the embedding of the codebook to obtain the discretization result of the latent variable; S4, input the discretization result into the decoder to obtain the restored image.

2. The method according to claim 1, characterized in that The determination of embedding the initial feature map into the codebook is specifically as follows: If the initial feature has been mapped to an embedding in the codebook, mapping the embedding to the codebook as the initial feature; If the initial feature is not mapped to an embedding in the codebook, the embedding in the codebook that is closest to the initial feature is used as the embedding mapped to the codebook.

3. The method according to claim 1, characterized in that The embedding of the neighborhood mapping to the codebook is determined according to the target distance and / or the neighbors of the neighborhood in the codebook, specifically: The k nearest neighbors of the neighborhood in the codebook are calculated, the embedding of the initial feature mapping to the codebook and the distance between each of the k nearest neighbors are calculated, and the nearest neighbor with the closest distance to the target is used as the embedding of the neighborhood mapping to the codebook.

4. The method according to claim 1, characterized in that The embedding of the neighborhood mapping to the codebook is determined according to the target distance and / or the neighbors of the neighborhood in the codebook, specifically: Obtaining a neighbor of the neighborhood in the latent variable, and if the neighbor in the latent variable has been mapped to the embedding of the codebook, adding the neighbor to the neighbor set; Calculating the distance from the neighborhood to each neighbor in the neighbor set; Get k nearest neighbors of the neighborhood in the codebook, and for each nearest neighbor, calculate the average deviation of the distance from the neighborhood to each neighbor in the neighbor set and the distance from the nearest neighbor to each neighbor in the neighbor set mapped to the codebook embedding, and use the nearest neighbor with the minimum average deviation as the embedding of the neighborhood mapped to the codebook.

5. The method according to claim 4, characterized in that The average deviation of the distance from the calculated neighborhood to each element in the neighbor set and the distance from the nearest neighbor to each neighbor in the neighbor set mapped to the codebook embedding is specifically: Calculate the distance from the neighborhood to each neighbor in the neighbor set to obtain a distance set on the latent variable; Calculate the distance from the nearest neighbor to the embedding of each neighbor in the neighbor set on the codebook to obtain a distance set on the codebook; The distance set on the latent variable and the distance set on the codebook are subtracted bit by bit, and then the absolute value is taken. Then the average value of all absolute values ​​is calculated, and the average value of the absolute values ​​is taken as the average deviation.

6. An image restoration system, characterized in that: The system includes the following modules: The encoding module is used to input the image to be repaired into the encoder to obtain a latent variable, obtain the shortest distance from each feature of the latent variable to the embedding in the codebook, and sort the features in the latent variable according to the shortest distance; A mapping module is used to take the first feature after sorting as the initial feature, determine the embedding of the initial feature mapped to the codebook; obtain the neighborhood of the initial feature in the latent variable, if the neighborhood is not mapped to the embedding in the codebook, then take the distance between the initial feature and the neighborhood as the target distance, and determine the embedding of the neighborhood mapped to the codebook according to the target distance and / or the nearest neighbors of the neighborhood in the codebook; when all neighbors of the initial feature are mapped to the embedding of the codebook, the initial feature is deleted from the sorting; An iterative module, used for continuously repeating the mapping module until all features of the latent variable are mapped to the embedding of the codebook to obtain a discretization result of the latent variable; The restoration module is used to input the discretization result into the decoder to obtain the restored image.

7. The system according to claim 6, characterized in that The determination of embedding the initial feature map into the codebook is specifically as follows: If the initial feature has been mapped to an embedding in the codebook, mapping the embedding to the codebook as the initial feature; If the initial feature is not mapped to an embedding in the codebook, the embedding in the codebook that is closest to the initial feature is used as the embedding mapped to the codebook.

8. The system according to claim 6, characterized in that The embedding of the neighborhood mapping to the codebook is determined according to the target distance and / or the neighbors of the neighborhood in the codebook, specifically: The k nearest neighbors of the neighborhood in the codebook are calculated, the embedding of the initial feature mapping to the codebook and the distance between each of the k nearest neighbors are calculated, and the nearest neighbor with the closest distance to the target is used as the embedding of the neighborhood mapping to the codebook.

9. The system according to claim 6, characterized in that The embedding of the neighborhood mapping to the codebook is determined according to the target distance and / or the neighbors of the neighborhood in the codebook, specifically: Obtaining a neighbor of the neighborhood in the latent variable, and if the neighbor in the latent variable has been mapped to the embedding of the codebook, adding the neighbor to the neighbor set; Calculating the distance from the neighborhood to each neighbor in the neighbor set; Get k nearest neighbors of the neighborhood in the codebook, and for each nearest neighbor, calculate the average deviation of the distance from the neighborhood to each neighbor in the neighbor set and the distance from the nearest neighbor to each neighbor in the neighbor set mapped to the codebook embedding, and use the nearest neighbor with the minimum average deviation as the embedding of the neighborhood mapped to the codebook.

10. The system according to claim 9, characterized in that The average deviation of the distance from the calculated neighborhood to each element in the neighbor set and the distance from each nearest neighbor to each element in the neighbor set mapped to the codebook embedding is specifically: Calculate the distance from the neighborhood to each neighbor in the neighbor set to obtain a distance set on the latent variable; Calculate the distance from the nearest neighbor to the embedding of each neighbor in the neighbor set on the codebook to obtain a distance set on the codebook; The distance set on the latent variable and the distance set on the codebook are subtracted bit by bit, and then the absolute value is taken. Then the average value of all absolute values ​​is calculated, and the average value of the absolute values ​​is taken as the average deviation.

Images