Method and system for restoring missing parts of an image

Abstract

The present invention discloses a method and system for restoring a missing part of an image, comprising: preprocessing the image to be restored to obtain Y; decomposing Y and performing singular value decomposition to obtain U, ∑, V T , multiplying the U,∑ matrices to obtain the weight matrix is ​​input into the pre-trained multi-layer fully connected autoencoder layer, and further training is performed to convert it into a feature vector to capture the global information of the image; the first pre-trained CAE-MC network is input, and further training is performed to extract more accurate and rich local features and potential information of each channel dimension, learn the complex relationships within the image, and use the low-rank characteristics of the image to complete the image, obtaining a preliminary single-channel restored image; after synthesizing the three-channel image, the second pre-trained CAE-MC network is input, and further training is performed to further extract features and learn the complex relationships within the image, obtaining a more accurate and comprehensive restored image. The present invention can restore the missing part of the image.

Description

Technical Field

[0001] The present invention relates to the technical field of picture restoration, and in particular to a method and system for restoring a missing portion of a picture. Background Art

[0002] Many images are often partially missing due to various reasons such as collection cost limitations, equipment loss, and environmental factors. Therefore, how to restore the missing parts of the images has become an urgent problem to be solved. Summary of the Invention

[0003] The present invention provides a method and system for restoring missing parts of an image to solve the problems existing in the above-mentioned prior art. The technical solution is as follows:

[0004] In one aspect, a method for restoring a missing portion of an image is provided, comprising:

[0005] S1. Preprocess the image to be restored to obtain a three-channel image matrix Y;

[0006] S2. Decompose the three-channel image matrix Y into three single-channel matrices, and then perform singular value decomposition on the matrix of each channel to obtain U, ∑, V T Three matrices, multiply the first two matrices U and ∑ matrices to obtain the weight matrix;

[0007] S3, inputting the weight matrix into a pre-trained multi-layer fully connected autoencoder layer of the overall restoration model, wherein the overall restoration model is composed of the multi-layer fully connected autoencoder layer, a first convolutional autoencoder-based matrix completion subnetwork CAE-MC network, and a second CAE-MC network, wherein the multi-layer fully connected autoencoder layer continues to train and converts the weight matrix into a feature vector to capture the global information of the image;

[0008] S4. Input the feature vector into the first pre-trained CAE-MC network. The first CAE-MC network continues to train and extract more accurate and rich local features and potential information in each channel dimension, learns the complex relationships within the image, and uses the low-rank characteristics of the image to complete the image and obtain a preliminary single-channel restored image.

[0009] S5. After synthesizing the preliminary single-channel restored images of the three channels into a three-channel image, the images are input into the pre-trained second CAE-MC network. The second CAE-MC network continues to train and further extracts features and learns the complex relationships within the image to obtain a more accurate and comprehensive restored image.

[0010] Optionally, the preprocessing in S1 specifically includes:

[0011] Automatically identify the 0 or 255 pixels in the image, the 0 or 255 pixels are the missing parts in the image, and obtain the three-channel image matrix Where m is the height of the image and n is the width of the image;

[0012] Alternatively, if the user specifies that the pixels of a certain area in the image are a missing part, the missing part is filled with 0 pixels to obtain the three-channel image matrix Y.

[0013] Optionally, the pre-training of the overall recovery model specifically includes:

[0014] Preprocess a picture containing the missing part to obtain a three-channel image matrix Y';

[0015] Decompose the three-channel image matrix Y' into three single-channel matrices, and then perform singular value decomposition on the matrix of each channel to obtain U', ∑', V' T Three matrices, multiply the first two matrices U' and ∑' to get the weight matrix, expressed as

[0016] The weight matrix Input a multi-layer fully connected autoencoder layer, which is composed of multiple layers of fully connected layers. The number of neurons in the first layer is r, and the weight matrix As a random initialization of the first layer latent variables of the multi-layer fully connected autoencoder layer instead of input, pre-training the multi-layer fully connected autoencoder layer;

[0017] After the pre-training of the multi-layer fully connected autoencoder layer is completed, pre-training the first CAE-MC network;

[0018] When the multi-layer fully connected autoencoder layer and the first CAE-MC network of the three channels are pre-trained, the second CAE-MC network of the three channels is pre-trained until the second CAE-MC network is pre-trained, at which time the overall recovery model is also pre-trained.

[0019] Optionally, because only one picture is used in pre-training, the method will continue to train the model based on the pre-training during the process of restoring the picture to be restored, and migrate the network weights of the overall restoration model that has been pre-trained. Other network weights use the network weights obtained by pre-training. However, for the first-layer hidden variables of the multi-layer fully connected autoencoder layer, the network weights obtained by pre-training are not used. Instead, a weight matrix obtained based on the picture to be restored is used to fully utilize the advantages of the pre-training weights, and combined with the characteristics of the current picture to be restored, the method will continue to perform training for a smaller number of times, update all network weights, and restore the picture to be restored.

[0020] Optionally, the CAE-MC network is composed of multiple encoders and multiple decoders. The multiple encoders are sequentially cascaded to compress high-dimensional data into low-dimensional feature representations and extract features. Each encoder is composed of a convolution layer, a ReLU activation function, and a batch normalization BN. The multiple decoders are also sequentially cascaded to restore the original image based on the features extracted by the encoder compression. Each decoder is composed of a transposed convolution, a ReLU activation function, and a batch normalization BN.

[0021] The input and output channels of the first CAE-MC network are 1 respectively, and the input and output channels of the second CAE-MC network are 3 respectively.

[0022] In another aspect, a system for restoring a missing portion of an image is provided, the system comprising:

[0023] The preprocessing module is used to preprocess the image to be restored to obtain a three-channel image matrix Y;

[0024] Decomposition module, used to decompose the three-channel image matrix Y into three single-channel matrices, and then perform singular value decomposition on the matrix of each channel to obtain U, ∑, V T Three matrices, multiply the first two matrices U and ∑ matrices to obtain the weight matrix;

[0025] a conversion module, configured to input the weight matrix into a pre-trained multi-layer fully connected autoencoder layer of a holistic restoration model, wherein the holistic restoration model is composed of the multi-layer fully connected autoencoder layer, a first convolutional autoencoder-based matrix completion subnetwork CAE-MC network, and a second CAE-MC network, wherein the multi-layer fully connected autoencoder layer continues to train and converts the weight matrix into a feature vector to capture global information of the image;

[0026] A preliminary restoration module is used to input the feature vector into the first pre-trained CAE-MC network. The first CAE-MC network continues to train and extract more accurate and rich local features and potential information for each channel dimension, learns the complex relationships within the image, and uses the low-rank characteristics of the image to complete the image and obtain a preliminary single-channel restored image;

[0027] The restoration module is used to synthesize the preliminary single-channel restored images of the three channels into a three-channel image, and then input it into a second pre-trained CAE-MC network. The second CAE-MC network continues training and further extracts features and learns the complex relationships within the image to obtain a more accurate and comprehensive restored image.

[0028] Optionally, the preprocessing module is specifically configured to:

[0029] Automatically identify the 0 or 255 pixels in the image, the 0 or 255 pixels are the missing parts in the image, and obtain the three-channel image matrix Where m is the height of the image and n is the width of the image;

[0030] Alternatively, if the user specifies that the pixels of a certain area in the image are a missing part, the missing part is filled with 0 pixels to obtain the three-channel image matrix Y.

[0031] Optionally, the pre-training of the overall recovery model specifically includes:

[0032] Preprocess a picture containing the missing part to obtain a three-channel image matrix Y';

[0033] Decompose the three-channel image matrix Y' into three single-channel matrices, and then perform singular value decomposition on the matrix of each channel to obtain U', ∑', V' T Three matrices, multiply the first two matrices U' and ∑' to get the weight matrix, expressed as

[0034] The weight matrix Input a multi-layer fully connected autoencoder layer, which is composed of multiple layers of fully connected layers. The number of neurons in the first layer is r, and the weight matrix As a random initialization of the first layer latent variables of the multi-layer fully connected autoencoder layer instead of input, pre-training the multi-layer fully connected autoencoder layer;

[0035] After the pre-training of the multi-layer fully connected autoencoder layer is completed, pre-training the first CAE-MC network;

[0036] When the multi-layer fully connected autoencoder layer and the first CAE-MC network of the three channels are pre-trained, the second CAE-MC network of the three channels is pre-trained until the second CAE-MC network is pre-trained, at which time the overall recovery model is also pre-trained.

[0037] Optionally, because only one picture is used in pre-training, the system will continue to train the model based on the pre-training during the process of restoring the picture to be restored, and migrate the network weights of the overall restoration model completed by pre-training. Other network weights use the network weights obtained by pre-training. However, for the first-layer hidden variables of the multi-layer fully connected autoencoder layer, the network weights obtained by pre-training are not used. Instead, the weight matrix obtained based on the picture to be restored is used to fully utilize the advantages of the pre-training weights, and combined with the characteristics of the current picture to be restored, the training is continued for a smaller number of times, all network weights are updated, and the picture to be restored is restored.

[0038] Optionally, the CAE-MC network is composed of multiple encoders and multiple decoders. The multiple encoders are sequentially cascaded to compress high-dimensional data into low-dimensional feature representations and extract features. Each encoder is composed of a convolution layer, a ReLU activation function, and a batch normalization BN. The multiple decoders are also sequentially cascaded to restore the original image based on the features extracted by the encoder compression. Each decoder is composed of a transposed convolution, a ReLU activation function, and a batch normalization BN.

[0039] The input and output channels of the first CAE-MC network are 1 respectively, and the input and output channels of the second CAE-MC network are 3 respectively.

[0040] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above-mentioned method for restoring a missing portion of an image.

[0041] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned method for restoring a missing portion of an image.

[0042] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0043] This method can effectively restore missing parts of images without requiring long-term pre-training with a large amount of image data. Compared with other methods, it has a faster iteration speed and better restoration effect. Specifically:

[0044] 1) Using a picture containing the missing part and undergoing multiple training iterations, the low-rank property of the image is utilized to obtain the restored (completed) picture.

[0045] 2) Using singular value decomposition to replace the random initialization process of hidden variables in traditional neural networks has been proven to accelerate network convergence, reduce the number of iterations, and improve the accuracy of image restoration (matrix completion).

[0046] 3) A matrix completion subnetwork (CAE-MC) based on convolutional autoencoders is incorporated to enhance the processing capability of multi-channel data such as images. After the first stage multi-layer fully connected autoencoder layer captures the global information of the image, the first CAE-MC network is introduced to increase further processing of the data, deeply explore the local features and potential information of the image data, learn the complex relationship within the image, and use the low-rank characteristics of the image to achieve image completion and perform preliminary single-channel recovery. Then, for three-channel color image data, a second CAE-MC network is introduced to further extract features and learn the complex relationship within the image, thereby obtaining the final recovery result. The traditional method requires converting the three-channel color data into a two-dimensional matrix form. This flattening process inevitably loses the spatial structure and details of the original data, affecting the integrity of the data. Therefore, the method of the present invention can enhance the processing capability of image details and obtain higher matrix completion accuracy compared to the traditional method in which each channel is processed separately. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 This is a flow chart of a method for restoring a missing portion of an image provided by an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of the overall network model structure provided by an embodiment of the present invention;

[0050] Figure 3 1. It is a schematic diagram comparing the traditional method initialization and the singular value initialization provided by the embodiment of the present invention;

[0051] Figure 4 Schematic diagram of the overall network model training process provided by an embodiment of the present invention;

[0052] Figure 5 This is a first CAE-MC network diagram provided by an embodiment of the present invention;

[0053] Figure 6 is a second CAE-MC network diagram provided by an embodiment of the present invention;

[0054] Figure 7 This is a block diagram of a system for restoring missing parts of an image provided by an embodiment of the present invention;

[0055] Figure 8It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0057] The embodiment of the present invention provides a method for restoring a missing portion of an image, which can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The flowchart of a method for restoring a missing portion of an image is shown. The processing flow of the method may include the following steps:

[0058] S1. Preprocess the image to be restored to obtain a three-channel image matrix Y;

[0059] There are usually two sources for the missing parts of the image to be restored: one is that the 0 or 255 pixels in the image are missing, and the other is that the user specifies a certain area of ​​pixels in the image as the missing part. Therefore, there are also two corresponding preprocessing methods.

[0060] Optionally, the preprocessing in S1 specifically includes:

[0061] Automatically identify the 0 or 255 pixels in the image, where the 0 or 255 pixels are missing parts of the image, and obtain a three-channel image matrix Where m is the height of the image and n is the width of the image;

[0062] Alternatively, if the user specifies that the pixels of a certain area in the image are missing, the missing part is filled with 0 pixels to obtain a three-channel image matrix Y.

[0063] S2. Decompose the three-channel image matrix Y into three single-channel matrices (red channel, green channel, blue channel), and then perform singular value decomposition on the matrix of each channel to obtain U, Σ, V T Three matrices, multiply the first two matrices U and ∑ matrices to obtain the weight matrix;

[0064] S3. Inputting the weight matrix into a pre-trained multi-layer fully connected autoencoder layer of the overall restoration model, wherein the overall restoration model is composed of the multi-layer fully connected autoencoder layer, a first convolutional autoencoder-based matrix completion subnetwork (CAE-MC) network, and a second CAE-MC network, wherein the multi-layer fully connected autoencoder layer continues to train and converts the weight matrix into a feature vector to capture the global information of the image;

[0065] S4. Input the feature vector into the first pre-trained CAE-MC network. The first CAE-MC network continues to train and extract more accurate and rich local features and potential information in each channel dimension, learns the complex relationships within the image, and uses the low-rank characteristics of the image to complete the image and obtain a preliminary single-channel restored image.

[0066] S5. After synthesizing the preliminary single-channel restored images of the three channels into a three-channel image, the images are input into the pre-trained second CAE-MC network. The second CAE-MC network continues to train and further extracts features and learns the complex relationships within the image to obtain a more accurate and comprehensive restored image.

[0067] Optionally, the pre-training of the overall recovery model specifically includes:

[0068] Preprocess a picture containing the missing part to obtain a three-channel image matrix Y';

[0069] Decompose the three-channel image matrix Y' into three single-channel matrices, and then perform singular value decomposition on the matrix of each channel to obtain U', ∑', V' T Three matrices, multiply the first two matrices U' and ∑' to get the weight matrix, expressed as

[0070] The weight matrix Input a multi-layer fully connected autoencoder layer, which is composed of multiple layers of fully connected layers, and the number of neurons in the first layer is r, such as Figure 2 As shown, the weight matrix As the random initialization of the first layer hidden variable of the multi-layer fully connected autoencoder layer instead of the input (if there is no singular value decomposition process, then use a picture as input, after passing through the first layer of the multi-layer fully connected autoencoder layer, then the output weight matrix can be naturally obtained, whose size is m×r, and the size of this matrix is ​​consistent with the weight matrix after singular value decomposition mentioned above, as shown Figure 3 As shown, the weight matrix after singular value decomposition can be used to replace the input matrix, that is, it can be used as the initialization parameter replacement of the neural network. Because of random initialization, the value is uncertain, and then it needs to be continuously updated through iteration. The embodiment of the present invention uses the weight matrix as input, which is equivalent to giving a very accurate value at the beginning, and the number of iterations will be much less), pre-training the multi-layer fully connected autoencoder layer;

[0071] After the pre-training of the multi-layer fully connected autoencoder layer is completed, pre-training the first CAE-MC network;

[0072] When the multi-layer fully connected autoencoder layer and the first CAE-MC network of the three channels are pre-trained, the second CAE-MC network of the three channels is pre-trained until the second CAE-MC network is pre-trained, at which time the overall recovery model is also pre-trained.

[0073] The embodiment of the present invention uses a picture for training, that is, a picture containing a missing part is used to obtain a restored (completed) picture after multiple training iterations. The reason why it can be restored is that the low-rank characteristics of the image are utilized. For example, there is a correlation between rows and rows or between columns in the picture, so the correlation (low-rank characteristics) within the picture can be used to restore the missing part. The deep learning model can extract high-dimensional features from the image and map these features to the required matrix space, and the trained model can mine the internal features of the image. The model of the embodiment of the present invention mainly includes a fully connected layer (the main structure of the multi-layer fully connected autoencoder layer) and a convolutional layer (the main structure of the CAE-MC network). The fully connected layer is responsible for converting the input image into a feature vector to capture the global information of the image. Then, the convolutional layer further processes these features to extract more refined local features and patterns. In order to achieve better extraction effect, the embodiment of the present invention sets multiple convolutional layers. Through this layer-by-layer extraction and conversion, the model can learn the complex relationship within the image, thereby realizing the restoration of the image.

[0074] However, because the embodiment of the present invention only uses one picture for training, unlike other deep learning models that use a large number of pictures for training (the embodiment of the present invention only uses one picture for training, which has the advantage of not only not needing to collect a large number of pictures, but also making the model training process simpler and faster), in order to obtain more accurate and robust recovery results, the embodiment of the present invention uses the above-mentioned process of training with only one picture as pre-training. In the process of restoring other pictures to be restored, the model will continue to be trained on the basis of pre-training, and the network weights of the overall recovery model completed by pre-training will be transferred over. Other network weights use the network weights obtained by pre-training, but for The first-layer hidden variables of the multi-layer fully connected autoencoder layer do not use the network weights obtained by pre-training, but use the weight matrix obtained based on the image to be restored to fully utilize the advantages of the pre-training weights, and combine with the characteristics of the current image to be restored, and then continue to train for a smaller number of times (until the maximum iteration round is reached or the preset error threshold is reached, at which time the network training will stop. In this way, it can ensure that the network has higher performance and generalization ability when migrating to other image data completion (missing part restoration) tasks), update all network weights, and restore the image to be restored (the recovery speed will be very fast, and the recovery effect is comparable). The entire training process is as follows Figure 4 shown.

[0075] Alternatively, as Figure 5 (the first CAE-MC network) and 6 (the second CAE-MC network), the CAE-MC network consists of multiple encoders d i and multiple decoders u j The multiple encoders are sequentially cascaded (using multiple encoders has better results), compressing high-dimensional data into low-dimensional feature representations and extracting features. Each encoder is composed of a convolution layer, a ReLU activation function, and a batch normalization BN. The multiple decoders are also sequentially cascaded (using multiple decoders has better results), and the original image is restored based on the features extracted by the encoder compression. Each decoder is composed of a transposed convolution, a ReLU activation function, and a batch normalization BN.

[0076] The input and output channels of the first CAE-MC network are 1 respectively, and the input and output channels of the second CAE-MC network are 3 respectively.

[0077] like Figure 7 As shown, an embodiment of the present invention further provides a system for restoring missing parts of an image, the system comprising:

[0078] A preprocessing module 710 is used to preprocess the image to be restored to obtain a three-channel image matrix Y;

[0079] Decomposition module 720 is used to decompose the three-channel image matrix Y into three single-channel matrices, and then perform singular value decomposition on the matrix of each channel to obtain U, ∑, V T Three matrices, multiply the first two matrices U and ∑ matrices to obtain the weight matrix;

[0080] a conversion module 730 configured to input the weight matrix into a pre-trained multi-layer fully connected autoencoder layer of a holistic restoration model, wherein the holistic restoration model is composed of the multi-layer fully connected autoencoder layer, a first convolutional autoencoder-based matrix completion subnetwork CAE-MC network, and a second CAE-MC network, wherein the multi-layer fully connected autoencoder layer continues to train and converts the weight matrix into a feature vector to capture global information of the image;

[0081] A preliminary restoration module 740 is configured to input the feature vector into a pre-trained first CAE-MC network. The first CAE-MC network continues training and extracts more accurate and rich local features and potential information for each channel dimension, learns the complex relationships within the image, and utilizes the low-rank characteristics of the image to complete the image and obtain a preliminary single-channel restored image.

[0082] Restoration module 750 is used to synthesize the preliminary single-channel restored images of the three channels into a three-channel image, and then input the pre-trained second CAE-MC network. The second CAE-MC network continues to train and further extract features and learn the complex relationships within the image to obtain a more accurate and comprehensive restored image.

[0083] Optionally, the preprocessing module is specifically configured to:

[0084] Automatically identify the 0 or 255 pixels in the image, the 0 or 255 pixels are the missing parts in the image, and obtain the three-channel image matrix Where m is the height of the image and n is the width of the image;

[0085] Alternatively, if the user specifies that the pixels of a certain area in the image are a missing part, the missing part is filled with 0 pixels to obtain the three-channel image matrix Y.

[0086] Optionally, the pre-training of the overall recovery model specifically includes:

[0087] Preprocess a picture containing the missing part to obtain a three-channel image matrix Y';

[0088] Decompose the three-channel image matrix Y' into three single-channel matrices, and then perform singular value decomposition on the matrix of each channel to obtain U', ∑', V' T Three matrices, multiply the first two matrices U' and ∑' to get the weight matrix, expressed as

[0089] The weight matrix Input a multi-layer fully connected autoencoder layer, which is composed of multiple layers of fully connected layers. The number of neurons in the first layer is r, and the weight matrix As a random initialization of the first layer latent variables of the multi-layer fully connected autoencoder layer instead of input, pre-training the multi-layer fully connected autoencoder layer;

[0090] After the pre-training of the multi-layer fully connected autoencoder layer is completed, pre-training the first CAE-MC network;

[0091] When the multi-layer fully connected autoencoder layer and the first CAE-MC network of the three channels are pre-trained, the second CAE-MC network of the three channels is pre-trained until the second CAE-MC network is pre-trained, at which time the overall recovery model is also pre-trained.

[0092] Optionally, because only one picture is used in pre-training, the system will continue to train the model based on the pre-training during the process of restoring the picture to be restored, and migrate the network weights of the overall restoration model completed by pre-training. Other network weights use the network weights obtained by pre-training. However, for the first-layer hidden variables of the multi-layer fully connected autoencoder layer, the network weights obtained by pre-training are not used. Instead, the weight matrix obtained based on the picture to be restored is used to fully utilize the advantages of the pre-training weights, and combined with the characteristics of the current picture to be restored, the training is continued for a smaller number of times, all network weights are updated, and the picture to be restored is restored.

[0093] Optionally, the CAE-MC network is composed of multiple encoders and multiple decoders. The multiple encoders are sequentially cascaded to compress high-dimensional data into low-dimensional feature representations and extract features. Each encoder is composed of a convolution layer, a ReLU activation function, and a batch normalization BN. The multiple decoders are also sequentially cascaded to restore the original image based on the features extracted by the encoder compression. Each decoder is composed of a transposed convolution, a ReLU activation function, and a batch normalization BN.

[0094] The input and output channels of the first CAE-MC network are 1 respectively, and the input and output channels of the second CAE-MC network are 3 respectively.

[0095] The functional structure of a system for restoring a missing portion of an image provided by an embodiment of the present invention corresponds to that of a method for restoring a missing portion of an image provided by an embodiment of the present invention, and will not be described in detail here.

[0096] Figure 8 1 is a structural diagram of an electronic device 800 provided in an embodiment of the present invention. The electronic device 800 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 801 and one or more memories 802, wherein the memory 802 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 801 to implement the steps of the above-mentioned method for recovering missing parts of an image.

[0097] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions. The instructions are executable by a processor in a terminal to perform the above-described method for restoring a missing portion of an image. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0098] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for restoring a missing portion of an image, characterized in that: The method comprises: S1. Preprocess the image to be restored to obtain a three-channel image matrix Y; S2. Decompose the three-channel image matrix Y into three single-channel matrices, and then perform singular value decomposition on the matrix of each channel to obtain U, Σ, V T Three matrices, multiply the first two matrices U and Σ to get the weight matrix; S3, inputting the weight matrix into a pre-trained multi-layer fully connected autoencoder layer of the overall restoration model, wherein the overall restoration model is composed of the multi-layer fully connected autoencoder layer, a first convolutional autoencoder-based matrix completion subnetwork CAE-MC network, and a second CAE-MC network, wherein the multi-layer fully connected autoencoder layer continues to train and converts the weight matrix into a feature vector to capture the global information of the image; S4. Input the feature vector into the first pre-trained CAE-MC network. The first CAE-MC network continues to train and extract more accurate and rich local features and potential information in each channel dimension, learns the complex relationships within the image, and uses the low-rank characteristics of the image to complete the image and obtain a preliminary single-channel restored image. S5. After synthesizing the preliminary single-channel restored images of the three channels into a three-channel image, the images are input into the pre-trained second CAE-MC network. The second CAE-MC network continues to train and further extracts features and learns the complex relationships within the image to obtain a more accurate and comprehensive restored image.

2. The method according to claim 1, characterized in that The preprocessing in S1 specifically includes: Automatically identify the 0 or 255 pixels in the image, the 0 or 255 pixels are the missing parts in the image, and obtain the three-channel image matrix Where m is the height of the image and n is the width of the image; Alternatively, if the user specifies that the pixels of a certain area in the image are a missing part, the missing part is filled with 0 pixels to obtain the three-channel image matrix Y.

3. The method according to claim 1, characterized in that The pre-training of the overall recovery model specifically includes: Preprocess a picture containing the missing part to obtain a three-channel image matrix Y'; Decompose the three-channel image matrix Y' into three single-channel matrices, and then perform singular value decomposition on the matrix of each channel to obtain U', ∑', V' T Three matrices, multiply the first two matrices U' and ∑' to get the weight matrix, expressed as The weight matrix Input a multi-layer fully connected autoencoder layer, which is composed of multiple layers of fully connected layers. The number of neurons in the first layer is r, and the weight matrix As a random initialization of the first layer latent variables of the multi-layer fully connected autoencoder layer instead of input, pre-training the multi-layer fully connected autoencoder layer; After the pre-training of the multi-layer fully connected autoencoder layer is completed, pre-training the first CAE-MC network; When the multi-layer fully connected autoencoder layer and the first CAE-MC network of the three channels are pre-trained, the second CAE-MC network of the three channels is pre-trained until the second CAE-MC network is pre-trained, at which time the overall recovery model is also pre-trained.

4. The method according to claim 3, characterized in that Because only one picture is used in pre-training, the method will continue to train the model based on the pre-training during the process of restoring the picture to be restored, and transfer the network weights of the overall restoration model completed by pre-training. Other network weights use the network weights obtained by pre-training. However, for the first-layer hidden variables of the multi-layer fully connected autoencoder layer, the network weights obtained by pre-training are not used. Instead, a weight matrix obtained based on the picture to be restored is used to fully utilize the advantages of the pre-training weights and combine the characteristics of the current picture to be restored. After that, a smaller number of trainings are continued to update all network weights and restore the picture to be restored.

5. The method according to claim 1, wherein The CAE-MC network consists of multiple encoders and multiple decoders. The multiple encoders are cascaded sequentially to compress high-dimensional data into low-dimensional feature representations and extract features. Each encoder consists of a convolutional layer, a ReLU activation function, and a batch normalization (BN). The multiple decoders are also cascaded sequentially to restore the original image based on the features extracted by the encoder compression. Each decoder consists of a transposed convolution, a ReLU activation function, and a batch normalization (BN). The input and output channels of the first CAE-MC network are 1 respectively, and the input and output channels of the second CAE-MC network are 3 respectively.

6. A system for restoring missing parts of an image, characterized in that: The system comprises: The preprocessing module is used to preprocess the image to be restored to obtain a three-channel image matrix Y; Decomposition module, used to decompose the three-channel image matrix Y into three single-channel matrices, and then perform singular value decomposition on the matrix of each channel to obtain U, Σ, V T Three matrices, multiply the first two matrices U and Σ to get the weight matrix; a conversion module, configured to input the weight matrix into a pre-trained multi-layer fully connected autoencoder layer of a holistic restoration model, wherein the holistic restoration model is composed of the multi-layer fully connected autoencoder layer, a first convolutional autoencoder-based matrix completion subnetwork CAE-MC network, and a second CAE-MC network, wherein the multi-layer fully connected autoencoder layer continues to train and converts the weight matrix into a feature vector to capture global information of the image; A preliminary restoration module is used to input the feature vector into the first pre-trained CAE-MC network. The first CAE-MC network continues to train and extract more accurate and rich local features and potential information for each channel dimension, learns the complex relationships within the image, and uses the low-rank characteristics of the image to complete the image and obtain a preliminary single-channel restored image; The restoration module is used to synthesize the preliminary single-channel restored images of the three channels into a three-channel image, and then input it into a second pre-trained CAE-MC network. The second CAE-MC network continues training and further extracts features and learns the complex relationships within the image to obtain a more accurate and comprehensive restored image.

7. The system according to claim 6, characterized in that The preprocessing module is specifically used to: Automatically identify the 0 or 255 pixels in the image, the 0 or 255 pixels are the missing parts in the image, and obtain the three-channel image matrix Where m is the height of the image and n is the width of the image; Alternatively, if the user specifies that the pixels of a certain area in the image are a missing part, the missing part is filled with 0 pixels to obtain the three-channel image matrix Y.

8. The system according to claim 6, wherein: The pre-training of the overall recovery model specifically includes: Preprocess a picture containing the missing part to obtain a three-channel image matrix Y'; Decompose the three-channel image matrix Y' into three single-channel matrices, and then perform singular value decomposition on the matrix of each channel to obtain U', ∑', V' T Three matrices, multiply the first two matrices U' and ∑' to get the weight matrix, expressed as The weight matrix Input a multi-layer fully connected autoencoder layer, which is composed of multiple layers of fully connected layers. The number of neurons in the first layer is r, and the weight matrix As a random initialization of the first layer latent variables of the multi-layer fully connected autoencoder layer instead of input, pre-training the multi-layer fully connected autoencoder layer; After the pre-training of the multi-layer fully connected autoencoder layer is completed, pre-training the first CAE-MC network; When the multi-layer fully connected autoencoder layer and the first CAE-MC network of the three channels are pre-trained, the second CAE-MC network of the three channels is pre-trained until the second CAE-MC network is pre-trained, at which time the overall recovery model is also pre-trained.

9. The system according to claim 8, characterized in that Because only one picture is used in pre-training, the system will continue to train the model based on pre-training during the process of restoring the picture to be restored, and transfer the network weights of the overall restoration model completed by pre-training. Other network weights use the network weights obtained by pre-training. However, for the first-layer hidden variables of the multi-layer fully connected autoencoder layer, the pre-trained network weights are not used. Instead, the weight matrix obtained based on the picture to be restored is used to make full use of the advantages of the pre-training weights and combine with the characteristics of the current picture to be restored. After that, a smaller number of trainings are continued to update all network weights and restore the picture to be restored.

10. The system according to claim 6, wherein: The CAE-MC network consists of multiple encoders and multiple decoders. The multiple encoders are cascaded sequentially to compress high-dimensional data into low-dimensional feature representations and extract features. Each encoder consists of a convolutional layer, a ReLU activation function, and a batch normalization (BN). The multiple decoders are also cascaded sequentially to restore the original image based on the features extracted by the encoder compression. Each decoder consists of a transposed convolution, a ReLU activation function, and a batch normalization (BN). The input and output channels of the first CAE-MC network are 1 respectively, and the input and output channels of the second CAE-MC network are 3 respectively.

Images