Method and system for recovering missing part of picture

Abstract

The invention discloses a method and a system for recovering a missing part of a picture. The method comprises the following steps: preprocessing a to-be-recovered picture to obtain Y; decomposing the Y and performing singular value decomposition to obtain U, sigma and VT, multiplying the U and sigma matrixes to obtain a weight matrix, inputting the weight matrix into the pre-trained multi-layer full-connection auto-encoder layer, continuing training and converting the weight matrix into a feature vector to capture global information of the image; inputting a pre-trained first CAE-MC network, continuing to train and extract more accurate and rich local features and potential information of each channel dimension, learning a complex relationship in the image, and complementing the image by using a low-rank characteristic of the image to obtain a preliminary single-channel recovery image; and after a three-channel picture is synthesized, inputting the three-channel picture into a second CAE-MC network which completes pre-training, continuing training, and further extracting features and learning a complex relation in the image to obtain a more accurate and comprehensive recovered picture. According to the invention, the missing part of the picture can be recovered.

Description

Technical Field

[0001] The present invention relates to the technical field of picture restoration, and in particular to a method and system for restoring a missing part of a picture. Background Art

[0002] Many pictures are often partially missing due to various reasons such as collection cost limitations, equipment loss, and environmental factors. Therefore, how to restore the missing parts of the pictures has become an urgent problem to be solved. Summary of the invention

[0003] The present invention provides a method and system for restoring missing parts of an image, which is used to solve the problems existing in the above-mentioned prior art. The technical solution is as follows:

[0004] On the one hand, a method for restoring a missing portion of an image is provided, comprising:

[0005] S1. Preprocess the image to be restored to obtain a three-channel image matrix Y;

[0006] S2, decompose the three-channel image matrix Y into three single-channel matrices, and then perform singular value decomposition on the matrix of each channel to obtain U, ∑, V T Three matrices, multiply the first two matrices U and ∑ matrices to get the weight matrix;

[0007] S3, inputting the weight matrix into a multi-layer fully connected autoencoder layer of a pre-trained overall restoration model, wherein the overall restoration model is composed of the multi-layer fully connected autoencoder layer, a first matrix completion subnetwork CAE-MC network based on a convolutional autoencoder, and a second CAE-MC network, wherein the multi-layer fully connected autoencoder layer continues to train and converts the weight matrix into a feature vector to capture the global information of the image;

[0008] S4, inputting the feature vector into the first pre-trained CAE-MC network, the first CAE-MC network continues to train and extract more accurate and rich local features and potential information of each channel dimension, learns the complex relationship within the image, and uses the low-rank characteristics of the image to complete the image, and obtain a preliminary single-channel restored image;

[0009] S5. After synthesizing the preliminary single-channel restored images of the three channels into a three-channel image, the images are input into the pre-trained second CAE-MC network. The second CAE-MC network continues to train and further extracts features and learns the complex relationships within the image to obtain a more accurate and comprehensive restored image.

[0010] Optionally, the preprocessing in S1 specifically includes:

[0011] Automatically identify 0 or 255 pixels in the image, where the 0 or 255 pixels are missing parts of the image, and obtain the three-channel image matrix Where m is the height of the image and n is the width of the image;

[0012] Or if the user specifies that the pixels of a certain area in the image are the missing part, the missing part is filled with 0 pixels to obtain the three-channel image matrix Y.

[0013] Optionally, the pre-training of the overall recovery model specifically includes:

[0014] Preprocess a picture containing the missing part to obtain a three-channel image matrix Y';

[0015] Decompose the three-channel image matrix Y' into three single-channel matrices, and then perform singular value decomposition on the matrix of each channel to obtain U', ∑', V' T Three matrices, multiply the first two matrices U' and ∑' to get the weight matrix, expressed as

[0016] The weight matrix Input a multi-layer fully connected autoencoder layer, the multi-layer fully connected autoencoder layer is composed of multiple layers of fully connected layers, the number of neurons in the first layer is r, the weight matrix As a random initialization substitute input for the first layer hidden variables of the multi-layer fully connected autoencoder layer, pre-training the multi-layer fully connected autoencoder layer;

[0017] After the pre-training of the multi-layer fully connected autoencoder layer is completed, pre-training the first CAE-MC network;

[0018] When the multi-layer fully connected autoencoder layer and the first CAE-MC network of the three channels are pre-trained, the second CAE-MC network of the three channels is pre-trained until the second CAE-MC network is pre-trained, at which time the overall recovery model is also pre-trained.

[0019] Optionally, because only one picture is used in the pre-training, the method will continue to train the model based on the pre-training during the process of restoring the picture to be restored, and the network weights of the overall restoration model completed by pre-training will be transferred over. Other network weights all use the network weights obtained by pre-training, but the first layer hidden variables of the multi-layer fully connected autoencoder layer do not use the network weights obtained by pre-training. Instead, the weight matrix obtained based on the picture to be restored is used to fully utilize the advantages of the pre-training weights, and combined with the characteristics of the current picture to be restored, continue to perform fewer training times, update all network weights, and restore the picture to be restored.

[0020] Optionally, the CAE-MC network is composed of multiple encoders and multiple decoders, the multiple encoders are cascaded sequentially to compress high-dimensional data into low-dimensional feature representations and extract features, and each encoder is composed of a convolution layer, a ReLU activation function, and a batch normalization BN; the multiple decoders are also cascaded sequentially to restore the original image according to the features extracted by the encoder compression, and each decoder is composed of a transposed convolution, a ReLU activation function, and a batch normalization BN;

[0021] The input and output channels of the first CAE-MC network are 1 respectively, and the input and output channels of the second CAE-MC network are 3 respectively.

[0022] On the other hand, a system for restoring a missing portion of an image is provided, the system comprising:

[0023] A preprocessing module is used to preprocess the image to be restored to obtain a three-channel image matrix Y;

[0024] The decomposition module is used to decompose the three-channel image matrix Y into three single-channel matrices, and then perform singular value decomposition on the matrix of each channel to obtain U, ∑, V T Three matrices, multiply the first two matrices U and ∑ matrices to get the weight matrix;

[0025] A conversion module, used for inputting the weight matrix into a multi-layer fully connected autoencoder layer of a pre-trained overall restoration model, wherein the overall restoration model is composed of the multi-layer fully connected autoencoder layer, a first convolutional autoencoder-based matrix completion subnetwork CAE-MC network and a second CAE-MC network, wherein the multi-layer fully connected autoencoder layer continues to train and converts the weight matrix into a feature vector to capture the global information of the image;

[0026] A preliminary restoration module is used to input the feature vector into the first pre-trained CAE-MC network, and the first CAE-MC network continues to train and extract more accurate and rich local features and potential information of each channel dimension, learns the complex relationship within the image, and uses the low-rank characteristics of the image to complete the image and obtain a preliminary single-channel restored image;

[0027] The restoration module is used to synthesize the preliminary single-channel restored images of the three channels into a three-channel image, and then input the pre-trained second CAE-MC network. The second CAE-MC network continues to train and further extracts features and learns the complex relationships within the image to obtain a more accurate and comprehensive restored image.

[0028] Optionally, the preprocessing module is specifically used to:

[0029] Automatically identify 0 or 255 pixels in the image, where the 0 or 255 pixels are missing parts of the image, and obtain the three-channel image matrix Where m is the height of the image and n is the width of the image;

[0030] Or if the user specifies that the pixels of a certain area in the image are the missing part, the missing part is filled with 0 pixels to obtain the three-channel image matrix Y.

[0031] Optionally, the pre-training of the overall recovery model specifically includes:

[0032] Preprocess a picture containing the missing part to obtain a three-channel image matrix Y';

[0033] Decompose the three-channel image matrix Y' into three single-channel matrices, and then perform singular value decomposition on the matrix of each channel to obtain U', ∑', V' T Three matrices, multiply the first two matrices U' and ∑' to get the weight matrix, expressed as

[0034] The weight matrix Input a multi-layer fully connected autoencoder layer, the multi-layer fully connected autoencoder layer is composed of multiple layers of fully connected layers, the number of neurons in the first layer is r, the weight matrix As a random initialization substitute input for the first layer hidden variables of the multi-layer fully connected autoencoder layer, pre-training the multi-layer fully connected autoencoder layer;

[0035] After the pre-training of the multi-layer fully connected autoencoder layer is completed, pre-training the first CAE-MC network;

[0036] When the multi-layer fully connected autoencoder layer and the first CAE-MC network of the three channels are pre-trained, the second CAE-MC network of the three channels is pre-trained until the second CAE-MC network is pre-trained, at which time the overall recovery model is also pre-trained.

[0037] Optionally, because only one picture is used in the pre-training, the system will continue to train the model based on the pre-training during the process of restoring the picture to be restored, and migrate the network weights of the overall restoration model completed by pre-training. Other network weights use the network weights obtained by pre-training, but the first layer of hidden variables of the multi-layer fully connected autoencoder layer does not use the pre-trained network weights. Instead, a weight matrix obtained based on the picture to be restored is used to fully utilize the advantages of the pre-training weights and combine the characteristics of the current picture to be restored. After that, a smaller number of trainings are continued to update all network weights and restore the picture to be restored.

[0038] Optionally, the CAE-MC network is composed of multiple encoders and multiple decoders, the multiple encoders are cascaded sequentially to compress high-dimensional data into low-dimensional feature representations and extract features, and each encoder is composed of a convolution layer, a ReLU activation function, and a batch normalization BN; the multiple decoders are also cascaded sequentially to restore the original image according to the features extracted by the encoder compression, and each decoder is composed of a transposed convolution, a ReLU activation function, and a batch normalization BN;

[0039] The input and output channels of the first CAE-MC network are 1 respectively, and the input and output channels of the second CAE-MC network are 3 respectively.

[0040] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above-mentioned method for restoring a missing portion of an image.

[0041] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned method for restoring a missing portion of an image.

[0042] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0043] The present invention can effectively restore the missing part of the image without long-term pre-training with a large amount of image data, and has a faster iteration speed and better restoration effect than other methods. Specifically:

[0044] 1) Using a picture containing the missing part and undergoing multiple training iterations, the low-rank characteristics of the image are utilized to obtain the restored (completed) picture.

[0045] 2) Using singular value decomposition to replace the random initialization process of hidden variables in traditional neural networks has been proven to accelerate network convergence, reduce the number of iterations, and improve the accuracy of image restoration (matrix completion).

[0046] 3) A matrix completion subnetwork (CAE-MC) based on a convolutional autoencoder is incorporated to enhance the processing capability of multi-channel data such as images. After the first stage multi-layer fully connected autoencoder layer captures the global information of the image, the first CAE-MC network is introduced to increase the further processing of the data, deeply explore the local features and potential information of the image data, learn the complex relationship inside the image, and use the low-rank characteristics of the image to achieve the completion of the image, perform preliminary single-channel recovery, and then, for the three-channel color image data, introduce the second CAE-MC network to further extract features and learn the complex relationship inside the image, so as to obtain the final recovery result. The traditional method needs to convert the three-channel color data into a two-dimensional matrix form. This flattening process will inevitably lose the spatial structure and details of the original data, affecting the integrity of the data. Therefore, the method of the present invention can enhance the processing capability of image details and obtain higher matrix completion accuracy compared to the way in which each channel of the traditional method is processed separately. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 This is a flow chart of a method for restoring a missing portion of an image provided by an embodiment of the present invention;

[0049] Figure 2 It is a schematic diagram of the overall network model structure provided by an embodiment of the present invention;

[0050] Figure 3 1 is a schematic diagram comparing the traditional method initialization and the singular value initialization provided by the embodiment of the present invention;

[0051] Figure 4 It is a schematic diagram of the overall network model training process provided by an embodiment of the present invention;

[0052] Figure 5 This is a first CAE-MC network schematic diagram provided by an embodiment of the present invention;

[0053] Figure 6 is a second CAE-MC network schematic diagram provided by an embodiment of the present invention;

[0054] Figure 7 It is a block diagram of a system for restoring missing parts of an image provided by an embodiment of the present invention;

[0055] Figure 8It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0057] The embodiment of the present invention provides a method for restoring a missing portion of an image, which can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The flowchart of a method for restoring a missing portion of an image is shown. The processing flow of the method may include the following steps:

[0058] S1. Preprocess the image to be restored to obtain a three-channel image matrix Y;

[0059] There are usually two sources for the missing parts of the image to be restored: one is that the 0 or 255 pixels in the image are the missing parts, and the other is that the user specifies that the pixels in a certain area in the image are the missing parts. Therefore, there are also two corresponding preprocessing methods.

[0060] Optionally, the preprocessing in S1 specifically includes:

[0061] Automatically identify 0 or 255 pixels in the image, where the 0 or 255 pixels are missing parts of the image, and obtain a three-channel image matrix Where m is the height of the image and n is the width of the image;

[0062] Or if the user specifies that the pixels of a certain area in the image are the missing part, the missing part is filled with 0 pixels to obtain a three-channel image matrix Y.

[0063] S2, decompose the three-channel image matrix Y into three single-channel matrices (red channel, green channel, blue channel), and then perform singular value decomposition on the matrix of each channel to obtain U, Σ, V T Three matrices, multiply the first two matrices U and ∑ matrices to get the weight matrix;

[0064] S3, inputting the weight matrix into a multi-layer fully connected autoencoder layer of a pre-trained overall restoration model, wherein the overall restoration model is composed of the multi-layer fully connected autoencoder layer, a first convolutional autoencoder-based matrix completion subnetwork (CAE-MC) network and a second CAE-MC network, wherein the multi-layer fully connected autoencoder layer continues to train and converts the weight matrix into a feature vector to capture the global information of the image;

[0065] S4, inputting the feature vector into the first pre-trained CAE-MC network, the first CAE-MC network continues to train and extract more accurate and rich local features and potential information of each channel dimension, learns the complex relationship within the image, and uses the low-rank characteristics of the image to complete the image, and obtain a preliminary single-channel restored image;

[0066] S5. After synthesizing the preliminary single-channel restored images of the three channels into a three-channel image, the images are input into the pre-trained second CAE-MC network. The second CAE-MC network continues to train and further extracts features and learns the complex relationships within the image to obtain a more accurate and comprehensive restored image.

[0067] Optionally, the pre-training of the overall recovery model specifically includes:

[0068] Preprocess a picture containing the missing part to obtain a three-channel image matrix Y';

[0069] Decompose the three-channel image matrix Y' into three single-channel matrices, and then perform singular value decomposition on the matrix of each channel to obtain U', ∑', V' T Three matrices, multiply the first two matrices U' and ∑' to get the weight matrix, expressed as

[0070] The weight matrix Input a multi-layer fully connected autoencoder layer, wherein the multi-layer fully connected autoencoder layer is composed of multiple layers of fully connected layers, and the number of neurons in the first layer is r, such as Figure 2 As shown, the weight matrix As the random initialization of the first layer hidden variables of the multi-layer fully connected autoencoder layer instead of the input (if there is no singular value decomposition process, then use a picture as the input, after passing through the first layer of the multi-layer fully connected autoencoder layer, then the output weight matrix can be naturally obtained, and its size is m×r. The size of this matrix is ​​consistent with the size of the weight matrix after the singular value decomposition mentioned above, such as Figure 3 As shown, the weight matrix after singular value decomposition can be used to replace the input matrix, that is, it can be used as the initialization parameter replacement of the neural network. Because of random initialization, the value is uncertain, and then it needs to be continuously updated through iteration. The embodiment of the present invention uses the weight matrix as input, which is equivalent to giving a very accurate value at the beginning, and the number of iterations will be much less), pre-training the multi-layer fully connected autoencoder layer;

[0071] After the pre-training of the multi-layer fully connected autoencoder layer is completed, pre-training the first CAE-MC network;

[0072] When the multi-layer fully connected autoencoder layer and the first CAE-MC network of the three channels are pre-trained, the second CAE-MC network of the three channels is pre-trained until the second CAE-MC network is pre-trained, at which time the overall recovery model is also pre-trained.

[0073] The embodiment of the present invention uses a picture for training, that is, a picture containing a missing part can be restored (completed) after multiple training iterations. The reason why it can be restored is that the low-rank characteristics of the image are utilized. For example, there is a correlation between rows and rows or between columns in the picture, so the correlation (low-rank characteristics) between the internal parts of the picture can be used to restore the missing part. The deep learning model can extract high-dimensional features from the image and map these features to the required matrix space, and the trained model can mine the internal features of the image. The model of the embodiment of the present invention mainly includes a fully connected layer (the main structure of a multi-layer fully connected autoencoder layer) and a convolutional layer (the main structure of the CAE-MC network). The fully connected layer is responsible for converting the input image into a feature vector to capture the global information of the image. Then, the convolutional layer further processes these features to extract more refined local features and patterns. In order to achieve better extraction effect, the embodiment of the present invention sets multiple convolutional layers. Through this layer-by-layer extraction and conversion, the model can learn the complex relationship inside the image, thereby realizing the restoration of the image.

[0074] However, because the embodiment of the present invention only uses one picture for training, unlike other deep learning models that use a large number of pictures for training (the embodiment of the present invention only uses one picture for training, which has the advantage of not only not needing to collect a large number of pictures, but also making the model training process simpler and faster), in order to obtain a more accurate and robust recovery result, the embodiment of the present invention uses the above-mentioned process of training with only one picture as pre-training, and in the process of restoring other pictures to be restored, the model will continue to be trained on the basis of pre-training, and the network weights of the overall recovery model completed by pre-training will be transferred over, and other network weights will use the network weights obtained by pre-training, but for The first-layer hidden variables of the multi-layer fully connected autoencoder layer do not use the network weights obtained by pre-training, but use the weight matrix obtained based on the image to be restored, so as to make full use of the advantages of the pre-training weights, and combine with the characteristics of the current image to be restored, and then continue to perform a small number of training times (until the maximum iteration round is reached or the preset error threshold is reached, at which time the network training will stop. In this way, it can ensure that the network has higher performance and generalization ability when migrating to other image data completion (missing part restoration) tasks), update all network weights, and restore the image to be restored (the recovery speed will be very fast, and the recovery effect is equivalent). The entire training process is as follows: Figure 4 shown.

[0075] Alternatively, if Figure 5 (the first CAE-MC network) and 6 (the second CAE-MC network), the CAE-MC network consists of multiple encoders d i and multiple decoders u j The multiple encoders are sequentially cascaded (using multiple encoders has better effect), compressing high-dimensional data into low-dimensional feature representation and extracting features. Each encoder is composed of a convolution layer, a ReLU activation function and a batch normalization BN. The multiple decoders are also sequentially cascaded (using multiple decoders has better effect), and the original picture is restored according to the features extracted by the encoder compression. Each decoder is composed of a transposed convolution, a ReLU activation function and a batch normalization BN.

[0076] The input and output channels of the first CAE-MC network are 1 respectively, and the input and output channels of the second CAE-MC network are 3 respectively.

[0077] like Figure 7 As shown, an embodiment of the present invention further provides a system for recovering a missing portion of an image, the system comprising:

[0078] A preprocessing module 710 is used to preprocess the image to be restored to obtain a three-channel image matrix Y;

[0079] Decomposition module 720 is used to decompose the three-channel image matrix Y into three single-channel matrices, and then perform singular value decomposition on the matrix of each channel to obtain U, ∑, V T Three matrices, multiply the first two matrices U and ∑ matrices to get the weight matrix;

[0080] A conversion module 730 is used to input the weight matrix into a multi-layer fully connected autoencoder layer of a pre-trained overall restoration model, wherein the overall restoration model is composed of the multi-layer fully connected autoencoder layer, a first convolutional autoencoder-based matrix completion subnetwork CAE-MC network, and a second CAE-MC network, wherein the multi-layer fully connected autoencoder layer continues to train and converts the weight matrix into a feature vector to capture the global information of the image;

[0081] A preliminary restoration module 740 is used to input the feature vector into the first pre-trained CAE-MC network, and the first CAE-MC network continues to train and extract more accurate and rich local features and potential information of each channel dimension, learns the complex relationship within the image, and uses the low-rank characteristics of the image to complete the image and obtain a preliminary single-channel restored image;

[0082] The restoration module 750 is used to synthesize the preliminary single-channel restored images of the three channels into a three-channel image, and then input the pre-trained second CAE-MC network. The second CAE-MC network continues to train and further extracts features and learns the complex relationships within the image to obtain a more accurate and comprehensive restored image.

[0083] Optionally, the preprocessing module is specifically used to:

[0084] Automatically identify 0 or 255 pixels in the image, where the 0 or 255 pixels are missing parts of the image, and obtain the three-channel image matrix Where m is the height of the image and n is the width of the image;

[0085] Or if the user specifies that the pixels of a certain area in the image are the missing part, the missing part is filled with 0 pixels to obtain the three-channel image matrix Y.

[0086] Optionally, the pre-training of the overall recovery model specifically includes:

[0087] Preprocess a picture containing the missing part to obtain a three-channel image matrix Y';

[0088] Decompose the three-channel image matrix Y' into three single-channel matrices, and then perform singular value decomposition on the matrix of each channel to obtain U', ∑', V' T Three matrices, multiply the first two matrices U' and ∑' to get the weight matrix, expressed as

[0089] The weight matrix Input a multi-layer fully connected autoencoder layer, the multi-layer fully connected autoencoder layer is composed of multiple layers of fully connected layers, the number of neurons in the first layer is r, the weight matrix As a random initialization substitute input for the first layer hidden variables of the multi-layer fully connected autoencoder layer, pre-training the multi-layer fully connected autoencoder layer;

[0090] After the pre-training of the multi-layer fully connected autoencoder layer is completed, pre-training the first CAE-MC network;

[0091] When the multi-layer fully connected autoencoder layer and the first CAE-MC network of the three channels are pre-trained, the second CAE-MC network of the three channels is pre-trained until the second CAE-MC network is pre-trained, at which time the overall recovery model is also pre-trained.

[0092] Optionally, because only one picture is used in the pre-training, the system will continue to train the model based on the pre-training during the process of restoring the picture to be restored, and migrate the network weights of the overall restoration model completed by pre-training. Other network weights use the network weights obtained by pre-training, but the first layer of hidden variables of the multi-layer fully connected autoencoder layer does not use the pre-trained network weights. Instead, a weight matrix obtained based on the picture to be restored is used to fully utilize the advantages of the pre-training weights and combine the characteristics of the current picture to be restored. After that, a smaller number of trainings are continued to update all network weights and restore the picture to be restored.

[0093] Optionally, the CAE-MC network is composed of multiple encoders and multiple decoders, the multiple encoders are cascaded sequentially to compress high-dimensional data into low-dimensional feature representations and extract features, and each encoder is composed of a convolution layer, a ReLU activation function, and a batch normalization BN; the multiple decoders are also cascaded sequentially to restore the original image according to the features extracted by the encoder compression, and each decoder is composed of a transposed convolution, a ReLU activation function, and a batch normalization BN;

[0094] The input and output channels of the first CAE-MC network are 1 respectively, and the input and output channels of the second CAE-MC network are 3 respectively.

[0095] The functional structure of a system for restoring a missing portion of an image provided by an embodiment of the present invention corresponds to a method for restoring a missing portion of an image provided by an embodiment of the present invention, and will not be described in detail herein.

[0096] Figure 8 It is a structural diagram of an electronic device 800 provided in an embodiment of the present invention. The electronic device 800 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 801 and one or more memories 802, wherein at least one instruction is stored in the memory 802, and the at least one instruction is loaded and executed by the processor 801 to implement the steps of the above-mentioned method for restoring the missing part of the image.

[0097] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, and the instructions can be executed by a processor in a terminal to complete the above-mentioned method for recovering a missing portion of an image. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0098] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for restoring a missing portion of an image, characterized in that: The method comprises: S1. Preprocess the image to be restored to obtain a three-channel image matrix Y; S2, decompose the three-channel image matrix Y into three single-channel matrices, and then perform singular value decomposition on the matrix of each channel to obtain U, Σ, V T Three matrices, multiply the first two matrices U and Σ to get the weight matrix; S3, inputting the weight matrix into a multi-layer fully connected autoencoder layer of a pre-trained overall restoration model, wherein the overall restoration model is composed of the multi-layer fully connected autoencoder layer, a first matrix completion subnetwork CAE-MC network based on a convolutional autoencoder, and a second CAE-MC network, wherein the multi-layer fully connected autoencoder layer continues to train and converts the weight matrix into a feature vector to capture the global information of the image; S4, inputting the feature vector into the first pre-trained CAE-MC network, the first CAE-MC network continues to train and extract more accurate and rich local features and potential information of each channel dimension, learns the complex relationship within the image, and uses the low-rank characteristics of the image to complete the image, and obtain a preliminary single-channel restored image; S5. After synthesizing the preliminary single-channel restored images of the three channels into a three-channel image, the images are input into the pre-trained second CAE-MC network. The second CAE-MC network continues to train and further extracts features and learns the complex relationships within the image to obtain a more accurate and comprehensive restored image.

2. The method according to claim 1, characterized in that The preprocessing in S1 specifically includes: Automatically identify 0 or 255 pixels in the image, where the 0 or 255 pixels are missing parts of the image, and obtain the three-channel image matrix Where m is the height of the image and n is the width of the image; Or if the user specifies that the pixels of a certain area in the image are the missing part, the missing part is filled with 0 pixels to obtain the three-channel image matrix Y.

3. The method according to claim 1, characterized in that The pre-training of the overall recovery model specifically includes: Preprocess a picture containing the missing part to obtain a three-channel image matrix Y'; Decompose the three-channel image matrix Y' into three single-channel matrices, and then perform singular value decomposition on the matrix of each channel to obtain U', ∑', V' T Three matrices, multiply the first two matrices U' and ∑' to get the weight matrix, expressed as The weight matrix Input a multi-layer fully connected autoencoder layer, the multi-layer fully connected autoencoder layer is composed of multiple layers of fully connected layers, the number of neurons in the first layer is r, the weight matrix As a random initialization substitute input for the first layer hidden variables of the multi-layer fully connected autoencoder layer, pre-training the multi-layer fully connected autoencoder layer; After the pre-training of the multi-layer fully connected autoencoder layer is completed, pre-training the first CAE-MC network; When the multi-layer fully connected autoencoder layer and the first CAE-MC network of the three channels are pre-trained, the second CAE-MC network of the three channels is pre-trained until the second CAE-MC network is pre-trained, at which time the overall recovery model is also pre-trained.

4. The method according to claim 3, characterized in that Because only one picture is used in pre-training, the method will continue to train the model based on the pre-training during the process of restoring the picture to be restored, and transfer the network weights of the overall restoration model completed by pre-training. Other network weights use the network weights obtained by pre-training, but the first layer hidden variables of the multi-layer fully connected autoencoder layer do not use the network weights obtained by pre-training. Instead, a weight matrix obtained based on the picture to be restored is used to fully utilize the advantages of the pre-training weights, and combined with the characteristics of the current picture to be restored, continue to perform a smaller number of training times, update all network weights, and restore the picture to be restored.

5. The method according to claim 1, characterized in that The CAE-MC network is composed of multiple encoders and multiple decoders. The multiple encoders are cascaded in sequence to compress high-dimensional data into low-dimensional feature representation and extract features. Each encoder is composed of a convolution layer, a ReLU activation function and a batch normalization BN. The multiple decoders are also cascaded in sequence to restore the original image according to the features extracted by the encoder compression. Each decoder is composed of a transposed convolution, a ReLU activation function and a batch normalization BN. The input and output channels of the first CAE-MC network are 1 respectively, and the input and output channels of the second CAE-MC network are 3 respectively.

6. A system for restoring missing parts of an image, characterized in that: The system comprises: A preprocessing module is used to preprocess the image to be restored to obtain a three-channel image matrix Y; The decomposition module is used to decompose the three-channel image matrix Y into three single-channel matrices, and then perform singular value decomposition on the matrix of each channel to obtain U, Σ, V T Three matrices, multiply the first two matrices U and Σ to get the weight matrix; A conversion module, used for inputting the weight matrix into a multi-layer fully connected autoencoder layer of a pre-trained overall restoration model, wherein the overall restoration model is composed of the multi-layer fully connected autoencoder layer, a first convolutional autoencoder-based matrix completion subnetwork CAE-MC network and a second CAE-MC network, wherein the multi-layer fully connected autoencoder layer continues to train and converts the weight matrix into a feature vector to capture the global information of the image; A preliminary restoration module is used to input the feature vector into the first pre-trained CAE-MC network, and the first CAE-MC network continues to train and extract more accurate and rich local features and potential information of each channel dimension, learns the complex relationship within the image, and uses the low-rank characteristics of the image to complete the image and obtain a preliminary single-channel restored image; The restoration module is used to synthesize the preliminary single-channel restored images of the three channels into a three-channel image, and then input the pre-trained second CAE-MC network. The second CAE-MC network continues to train and further extracts features and learns the complex relationships within the image to obtain a more accurate and comprehensive restored image.

7. The system according to claim 6, characterized in that The preprocessing module is specifically used for: Automatically identify 0 or 255 pixels in the image, where the 0 or 255 pixels are missing parts of the image, and obtain the three-channel image matrix Where m is the height of the image and n is the width of the image; Or if the user specifies that the pixels of a certain area in the image are the missing part, the missing part is filled with 0 pixels to obtain the three-channel image matrix Y.

8. The system according to claim 6, characterized in that The pre-training of the overall recovery model specifically includes: Preprocess a picture containing the missing part to obtain a three-channel image matrix Y'; Decompose the three-channel image matrix Y' into three single-channel matrices, and then perform singular value decomposition on the matrix of each channel to obtain U', ∑', V' T Three matrices, multiply the first two matrices U' and ∑' to get the weight matrix, expressed as The weight matrix Input a multi-layer fully connected autoencoder layer, the multi-layer fully connected autoencoder layer is composed of multiple layers of fully connected layers, the number of neurons in the first layer is r, the weight matrix As a random initialization substitute input for the first layer hidden variables of the multi-layer fully connected autoencoder layer, pre-training the multi-layer fully connected autoencoder layer; After the pre-training of the multi-layer fully connected autoencoder layer is completed, pre-training the first CAE-MC network; When the multi-layer fully connected autoencoder layer and the first CAE-MC network of the three channels are pre-trained, the second CAE-MC network of the three channels is pre-trained until the second CAE-MC network is pre-trained, at which time the overall recovery model is also pre-trained.

9. The system according to claim 8, characterized in that Because only one picture is used in pre-training, the system will continue to train the model based on the pre-training during the process of restoring the picture to be restored, and transfer the network weights of the overall restoration model completed by pre-training. Other network weights use the network weights obtained by pre-training. However, for the first layer of hidden variables of the multi-layer fully connected autoencoder layer, the pre-trained network weights are not used. Instead, the weight matrix obtained based on the picture to be restored is used to make full use of the advantages of the pre-training weights and combine the characteristics of the current picture to be restored. After that, a smaller number of trainings are continued to update all network weights and restore the picture to be restored.

10. The system according to claim 6, characterized in that The CAE-MC network is composed of multiple encoders and multiple decoders. The multiple encoders are cascaded in sequence to compress high-dimensional data into low-dimensional feature representation and extract features. Each encoder is composed of a convolution layer, a ReLU activation function and a batch normalization BN. The multiple decoders are also cascaded in sequence to restore the original image according to the features extracted by the encoder compression. Each decoder is composed of a transposed convolution, a ReLU activation function and a batch normalization BN. The input and output channels of the first CAE-MC network are 1 respectively, and the input and output channels of the second CAE-MC network are 3 respectively.

Images