Image restoration method, device, equipment, medium and product

Abstract

The present disclosure provides an image restoration method, device, equipment, medium and product, the method comprising: obtaining an image to be restored, then determining multiple similar images of the image to be restored from an image library, including a first similar image and a second similar image, fusing the image to be restored and the first similar image and the second similar image, and inputting the fused image into an image restoration model, serializing and encoding the feature map extracted from the fused image through the first branch of the image restoration model to obtain a first encoding result, respectively encoding a sub-feature map set divided according to different scales of the feature map through the second branch of the image restoration model, and fully connecting the encoding results of the sub-feature maps in the sub-feature map set to obtain a second encoding result, and obtaining a restored image according to the first encoding result and the second encoding result. In this way, the authenticity of the image restoration is improved, and the user experience is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image restoration method, apparatus, device, computer-readable storage medium, and computer program product. Background Art

[0002] As image processing technology continues to mature, users' demand for image restoration through image processing technology is gradually increasing. Image restoration refers to the restoration of missing parts in an image. Specifically, image restoration refers to restoring unknown information in an image based on known information in the image.

[0003] A typical image restoration scheme is to determine the restoration area in the image to be restored, determine a reference area for the restoration area, and then predict the pixel value of the restoration area through a neural network model based on the pixel value of the reference area of ​​the image, thereby achieving image restoration. However, this image restoration technology may cause distortion such as ripples and distortions in the restoration area, which does not meet the user's requirements for the authenticity of image restoration.

[0004] How to improve the authenticity of image restoration has become an urgent problem to be solved. Summary of the invention

[0005] The purpose of the present disclosure is to provide an image restoration method, apparatus, device, computer-readable storage medium and computer program product, which can perform high-fidelity restoration on the image to be restored and improve the user experience.

[0006] In a first aspect, the present disclosure provides an image restoration method, the method comprising:

[0007] Obtain the image to be repaired;

[0008] Determining a plurality of similar images of the image to be restored from an image library, wherein the plurality of similar images at least include a first similar image and a second similar image;

[0009] The image to be repaired, the first similar image and the second similar image are fused, and the fused image is input into an image repair model. The feature map extracted from the fused image is serialized and encoded through the first branch of the image repair model to obtain a first encoding result. The sub-feature map sets divided by the feature map at different scales are respectively encoded through the second branch of the image repair model, and the encoding results of the sub-feature maps in the sub-feature map set are fully connected to obtain a second encoding result. The repaired image is obtained according to the first encoding result and the second encoding result.

[0010] In a second aspect, the present disclosure provides an image restoration device, characterized in that the device comprises:

[0011] An acquisition module, used for acquiring an image to be repaired;

[0012] A determination module, configured to determine a plurality of similar images of the image to be restored from an image library, wherein the plurality of similar images at least include a first similar image and a second similar image;

[0013] A fusion module is used to fuse the image to be repaired with the first similar image and the second similar image, and input the fused image into an image repair model, serialize and encode the feature map extracted from the fused image through the first branch of the image repair model to obtain a first encoding result, respectively encode a set of sub-feature maps divided by the feature map at different scales through the second branch of the image repair model, and fully connect the encoding results of the sub-feature maps in the sub-feature map set to obtain a second encoding result, and obtain a repaired image according to the first encoding result and the second encoding result.

[0014] In a third aspect, the present disclosure provides an electronic device, comprising: a storage device on which a computer program is stored; and a processing device for executing the computer program in the storage device to implement the steps of the method described in the first aspect of the present disclosure.

[0015] In a fourth aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect of the present disclosure.

[0016] In a fifth aspect, the present disclosure provides a computer program product comprising instructions, which, when executed on a device, enables the device to execute the steps of the method described in the first aspect.

[0017] It can be seen from the above technical solutions that the present disclosure has at least the following advantages:

[0018] In the above technical solution, the electronic device obtains the image to be repaired, and then determines multiple similar images of the image to be repaired from the image library, including the first similar image and the second similar image, fuses the image to be repaired and the first similar image and the second similar image, and inputs the fused image into the image repair model, serializes and encodes the feature map extracted from the fused image through the first branch of the image repair model to obtain a first encoding result, and encodes the sub-feature map set of the feature map divided according to different scales through the second branch of the image repair model, and fully connects the encoding results of the sub-feature map in the sub-feature map set to obtain a second encoding result, and obtains the repaired image according to the first encoding result and the second encoding result. On the one hand, for the repair of the image, not only can the repair area be predicted based on the known area (reference area) of the image to be repaired, but also the similar image can be used as a known area to predict the repair area of ​​the image to be repaired, which increases the reliable data for predicting the image of the repair area, thereby effectively improving the effect of image repair. On the other hand, the two branches of the image repair model can repair the image to be repaired based on the feature maps of different scales, making the image repair more accurate, improving the authenticity of the image repair, and improving the user experience.

[0019] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical method of the embodiments of the present disclosure, the drawings required for use in the embodiments are briefly introduced below.

[0021] Figure 1 A schematic diagram of a process of an image restoration method provided by an embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram of an image restoration provided by an embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram of a process of obtaining similar images through a neural network model provided in an embodiment of the present disclosure;

[0024] Figure 4 A schematic diagram of performing image restoration using an image restoration model provided by an embodiment of the present disclosure;

[0025] Figure 5 A schematic diagram of a flow chart of another image restoration method provided by an embodiment of the present disclosure;

[0026] Figure 6 A schematic diagram of the structure of an image restoration device provided by an embodiment of the present disclosure;

[0027] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] The terms "first" and "second" in the embodiments of the present disclosure are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0029] First, some technical terms involved in the embodiments of the present disclosure are introduced.

[0030] Image processing technology generally refers to the processing of digital images, specifically the technology of analyzing and processing digital images through computers. Image processing technology can perform various types of processing on images, such as repairing images with missing parts, which is image restoration technology.

[0031] Image restoration technology refers to determining the restoration area and reference area in the image to be restored, and restoring the restoration area based on the reference area. The image to be restored can be an image with some patterns missing, or an image whose clarity does not meet the user's requirements.

[0032] The image restoration technology usually used is to predict the pixel values ​​of the restoration area through a neural network model based on the pixel values ​​of the reference area in the image to be restored, thereby achieving image restoration. This image restoration method only restores the image from the perspective of pixels, which may cause distortions such as ripples and distortions in the restored area. In addition, when a large part of the pattern is missing in the image to be restored, this method cannot accurately determine the missing content and does not meet the user's requirements for the authenticity of image restoration.

[0033] In view of this, the present disclosure provides an image restoration method, which is applied to electronic devices. An electronic device refers to a device with data processing capabilities, such as a server or a terminal. Among them, the terminal includes but is not limited to a smart phone, a tablet computer, a laptop computer, a personal digital assistant (PDA) or a smart wearable device. The server can be a cloud server, such as a central server in a central cloud computing cluster, or an edge server in an edge cloud computing cluster. Of course, the server can also be a server in a local data center. A local data center refers to a data center directly controlled by a user.

[0034] Specifically, the electronic device obtains an image to be repaired, then determines multiple similar images of the image to be repaired from an image library, fuses the image to be repaired with a first similar image and a second similar image, inputs the fused image into an image repair model, serializes and encodes a feature map extracted from the fused image through a first branch of the image repair model to obtain a first encoding result, and then encodes a set of sub-feature maps divided into different scales according to the feature map through a second branch of the image repair model, and fully connects the encoding results of the sub-feature maps in the sub-feature map set to obtain a second encoding result, and obtains the repaired image according to the first encoding result and the second encoding result.

[0035] On the one hand, image restoration can not only predict the area to be restored based on the known area (reference area) of the image to be restored, but also predict the area to be restored of the image to be restored using similar images as known areas, which increases the reliable data for predicting the image of the area to be restored, thereby effectively improving the effect of image restoration. On the other hand, the two branches of the image restoration model can restore the image to be restored based on feature maps of different scales, making the image restoration more accurate, improving the authenticity of the image restoration, and improving the user experience.

[0036] In order to make the technical solution of the present disclosure clearer and easier to understand, the following takes the electronic device as an example. Figure 1 As shown, the image restoration method provided by the embodiment of the present disclosure is introduced, and the method comprises the following steps:

[0037] S102: The terminal obtains the image to be restored.

[0038] The image to be repaired may be an image with a partial loss, or an image whose clarity does not meet the user's requirements. In this embodiment, the image to be repaired is an image with a partial loss as an example. Figure 2 As shown in A, there is a blank (missing) in the upper left part of the figure.

[0039] The terminal can obtain the image to be restored through various ways, for example, the terminal can capture the image to be restored through a camera, or the terminal can obtain the image to be restored through the user's input operation on the corresponding page. In some possible implementations, the user has a paper photo with a part missing, and the user can take a photo of the paper photo through a camera to obtain the image to be restored in a digital format. In other possible implementations, the user has a digital image to be restored saved in other terminals, so the image to be restored can be transmitted to the terminal through any transmission method, so that the terminal obtains the image to be restored.

[0040] S104: The terminal determines a plurality of similar images to the image to be restored from the image library.

[0041] The image library stores a large number of images, and the terminal can search the image library for multiple similar images similar to the image to be restored, including a first similar image and a second similar image.

[0042] In some possible implementations, the multiple similar images and the image to be restored are images of the same subject at different times or angles. For example, if the image to be restored is a photographed image of a certain building, the terminal can obtain other similar images with the image subject being the building in the image library. Among them, the similar images and the image to be restored may only differ in shooting angle, shooting distance, light, etc., and thus can be used as reference data for restoring the image to be restored. In other possible implementations, the multiple similar images and the image to be restored are images taken at the same location.

[0043] like Figure 2 As shown, the image A to be restored is an image of a building, and the multiple similar images B and C determined can be images of the building at different times and light intensities.

[0044] The terminal can determine multiple similar images of the image to be restored from the image library in a variety of ways. In some possible implementations, the image parameters include the shooting location of the image, and the terminal can obtain images shot at the same location from the image library based on the shooting location, and then determine whether the images shot at the same location are similar images by comparing the similarity between the images shot at the same location and the image to be restored. For example, the image to be restored is an image of a famous building taken in front of the famous building, so other images shot at the same location can be obtained from the image library, and then compared to see if they are similar to the image to be restored. If they are similar, they can be determined to be similar images to the image to be restored.

[0045] In some other possible implementations, the image parameters include the shooting time of the image, and the terminal can obtain images shot at the same time from the image library according to the shooting time, and then determine whether the images shot at the same time are similar images by comparing the similarity between the images shot at the same time and the image to be repaired. For example, the image to be repaired is an image of the sky taken during a meteor shower, so other images of the sky taken at the same time can be obtained from the image library, and then compared to see if they are similar to the image to be repaired. If they are similar, they can be determined to be similar images to the image to be repaired.

[0046] The terminal can also determine multiple similar images of the image to be repaired from the image library through the neural network model, such as Figure 3 As shown, the method for determining multiple similar images of an image to be repaired from an image library by using a neural network model comprises the following steps:

[0047] S302: The terminal inputs the image to be repaired into a feature comparison model to obtain a search feature of the image to be repaired.

[0048] The feature comparison model is used to determine the search features for searching in the image library from the image to be repaired. The search features are the features of the image to be repaired required for finding similar images. The feature comparison model is a trained neural network model. By searching in the image feature library using the search features output by the feature comparison model, similar features corresponding to similar images similar to the image to be repaired can be obtained, thereby determining similar images.

[0049] Specifically, the feature comparison model can use convolutional neural networks (CNN) to downsample the image to be repaired by 4 times to obtain a feature map with a size of 1 / 16 of the original image. After flattening the feature map into a one-dimensional sequence, it is sent to an N-layer encoder for encoding to obtain an encoded one-dimensional sequence, and then the encoded one-dimensional sequence is converted into a two-dimensional feature to obtain the search feature of the image to be repaired.

[0050] S304: The terminal obtains a plurality of similar features similar to the search feature from the image feature library according to the search feature.

[0051] The image feature library is a feature library that has a one-to-one correspondence with the image library. The images in the image library correspond to the features in the image feature library. The feature comparison model is a feature comparison model obtained by training a neural network model with a large number of similar images. The search feature output by the feature comparison model can obtain multiple similar features similar to the search feature in the image feature library.

[0052] S306: The terminal obtains multiple similar images of the image to be restored from the image library according to the multiple similar features.

[0053] There is a one-to-one correspondence between the image library and the image feature library, so the terminal can determine multiple similar images of the image to be repaired from the image library according to similar features.

[0054] In this way, the terminal can determine multiple similar images of the image to be repaired from the image library through the neural network model, thereby obtaining more reference data for image repair.

[0055] Normally, image restoration only uses the reference area in the image as known data to determine the unknown data of the restoration area. However, in the present invention, similar images of the image to be restored are determined, and the similar images are also used as known data to determine the unknown data of the restoration area, which effectively increases the amount of known data and thus can achieve a better restoration effect.

[0056] S106: The terminal fuses the image to be repaired, the first similar image and the second similar image, and inputs the fused image into the image repair model to obtain the repaired image through the image repair model.

[0057] The terminal fuses (concat) the image to be repaired and the multiple similar images to obtain a fused image. The multiple similar images may include a first similar image and a second similar image. In this embodiment, the multiple similar images are taken as an example to introduce the first similar image and the second similar image. Figure 4 shown.

[0058] The terminal uses the image restoration model to first downsample the fused image by 8 times through a 4-layer convolutional neural network (CNN), and then downsamples it by 2 times through a downsampling layer to obtain a feature map with a size of 1 / 256 of the original image.

[0059] The image restoration model consists of two branches. The first branch flattens the feature map to obtain a one-dimensional sequence, and then inputs the one-dimensional sequence into an N-layer encoder for encoding to obtain a first encoding result. The second branch segments the feature map at different scales, such as using feature Figure 1 The feature map is segmented into windows of / 4, 1 / 16, and 1 / 64 to generate a set of sub-feature maps. The feature maps in the sub-feature sets are then flattened to obtain the corresponding one-dimensional sequences. These one-dimensional sequences are input into the N-layer encoder for encoding. Then, the sequences of different scales output by the encoder are mapped back to the same length through the fully connected (FC) layer to obtain the second encoding result. Finally, the first encoding result and the second encoding result are fused through the FC layer to obtain the final output, thereby realizing the restoration of the image to be repaired.

[0060] Since the image restoration model has two branches and the feature map is segmented at different scales in the second branch, the image to be restored can be restored based on feature maps of multiple different scales, thereby making the image restoration more accurate, improving the authenticity of the image restoration, and improving the user experience.

[0061] The image restoration model can be obtained by training with training images including multiple similar images, wherein the multiple similar images include a mask image, and the mask image is an image obtained by masking the multiple similar images. The training process of the image restoration model specifically includes:

[0062] The terminal obtains a training feature map from a training image including multiple similar images. Specifically, the terminal fuses multiple similar images, and then downsamples the fused image by 8 times through a 4-layer convolutional neural network, and then downsamples it by 2 times through a downsampling layer to obtain a training feature map with a size of 1 / 256 of the original image.

[0063] The first branch of the image restoration model flattens the training feature map to obtain a one-dimensional sequence, and then inputs the one-dimensional sequence into the N-layer encoder for encoding to obtain the first training encoding result. The second branch of the image restoration model uses the training feature map to Figure 1 The training feature graph is segmented at different scales with windows of / 4, 1 / 16, and 1 / 64 to generate a set of training sub-feature graphs. The training feature graphs in the training sub-feature sets are then flattened to obtain the corresponding one-dimensional sequences. These one-dimensional sequences are then input into the encoder of the N layers for encoding. The sequences of different scales output by the encoder are mapped back to the same length through the FC layer to obtain the second training encoding result. Finally, the first training encoding result and the second training encoding result are fused through the FC layer to obtain the final output, thereby realizing the restoration of the mask image. The terminal can compare the repaired mask image with the training image before the mask and update the parameters of the image restoration model.

[0064] The terminal executing the image restoration method in this embodiment and the terminal performing model training may be the same terminal or different terminals. In some possible implementations, the terminal may transmit its trained image restoration model to multiple other terminals, so that the multiple other terminals may directly use the image restoration model to implement the image restoration method in this disclosure.

[0065] Based on the above description, the present disclosure provides an image restoration method. The terminal obtains an image to be restored, then determines multiple similar images of the image to be restored from an image library, fuses the image to be restored and the multiple similar images, and inputs the fused image into an image restoration model, serializes and encodes the feature map extracted from the fused image through the first branch of the image restoration model to obtain a first encoding result, and respectively encodes a set of sub-feature maps divided according to different scales of the feature map through the second branch of the image restoration model, and fully connects the encoding results of the sub-feature maps in the sub-feature map set to obtain a second encoding result, and obtains the restored image according to the first encoding result and the second encoding result.

[0066] In this way, on the one hand, the terminal can not only predict the area to be repaired based on the known area (reference area) of the image to be repaired, but also use similar images as known areas to predict the area to be repaired of the image to be repaired, which increases the reliable data for predicting the image of the area to be repaired, thereby effectively improving the effect of image repair. On the other hand, the two branches of the image repair model can repair the image to be repaired based on feature maps of different scales, making the image repair more accurate, improving the authenticity of the image repair, and improving the user experience.

[0067] In some possible implementations, the subject of the image to be restored is a famous building, such as Figure 2 As shown in the figure, the upper left part is missing. Figure 5 As shown, the repair process of the image to be repaired includes the following steps:

[0068] S502: The terminal obtains the image to be restored.

[0069] The image to be repaired in this embodiment is as follows Figure 2 As shown, the terminal can convert the paper image to be restored into a digital image to be restored by photographing, or directly obtain the digital image to be restored.

[0070] S504: The terminal inputs the image to be repaired into a feature comparison model to obtain a search feature of the image to be repaired.

[0071] The feature comparison model can use a convolutional neural network to downsample the image to be repaired by 4 times to obtain a feature map with a size of 1 / 16 of the original image. The feature map is then flattened and converted into a one-dimensional sequence, which is then sent to an N-layer encoder for encoding to obtain an encoded one-dimensional sequence, which is then converted into a two-dimensional feature to obtain the search feature of the image to be repaired.

[0072] S506: The terminal obtains similar features similar to the search feature from the image feature library according to the search feature.

[0073] The terminal obtains, according to the search feature, a plurality of similar features similar to the search feature from an image feature library having a one-to-one correspondence with the image library.

[0074] S508: The terminal obtains multiple similar images of the image to be restored from the image library according to the multiple similar features.

[0075] Since the features in the image feature library correspond one-to-one to the images in the image library, the terminal can determine similar images corresponding to the multiple similar features in the image library based on the multiple similar features, so that multiple similar images of the image to be repaired can be obtained.

[0076] S510: The terminal fuses the image to be repaired with multiple similar images, inputs the fused image into an image repair model, and obtains a repaired image through the image repair model.

[0077] Specifically, the terminal fuses the image to be repaired with multiple similar images to obtain a fused image. Then, through the image repair model, the fused image is first downsampled 8 times through a 4-layer convolutional neural network, and then downsampled 2 times through a downsampling layer to obtain a feature map with a size of 1 / 256 of the original image.

[0078] Furthermore, the terminal flattens the feature map through the first branch of the image restoration model to obtain a one-dimensional sequence, and then inputs the one-dimensional sequence into an N-layer encoder for encoding to obtain a first encoding result.

[0079] Furthermore, the terminal uses the feature map through the second branch of the image restoration model. Figure 1 The feature graph is segmented at different scales using windows of / 4, 1 / 16, and 1 / 64 to generate a sub-feature graph set, and then the feature graphs in the sub-feature set are flattened to obtain the corresponding one-dimensional sequences, and then these one-dimensional sequences are input into the N-layer encoder for encoding, and the sequences of different scales output by the encoder are mapped back to the same length through the fully connected layer to obtain the second encoding result. The processing of the feature graph by the first branch and the second branch of the model can be performed simultaneously.

[0080] Finally, the terminal fuses the first encoding result and the second encoding result through the FC layer to obtain the final output, thereby realizing the restoration of the image to be restored.

[0081] In some possible implementations, the image to be repaired can be Figure 2 As shown in A in , the similar images of the image to be repaired can be Figure 2 As shown in B and C in the figure, B is the first similar image and C is the second similar image. Figure 2 D is the restored image. Since the reference data in the restoration process includes not only the known part in A, but also similar images B and C, there is more reference data, so it can be restored more accurately.

[0082] Figure 6 is a schematic diagram of an image restoration device according to an exemplary disclosed embodiment. Figure 6 As shown, the image restoration device 600 includes:

[0083] An acquisition module 602 is used to acquire an image to be restored;

[0084] A determination module 604 is used to determine a plurality of similar images of the image to be restored from an image library, wherein the plurality of similar images at least include a first similar image and a second similar image;

[0085] A fusion module 606 is used to fuse the image to be repaired with the first similar image and the second similar image, and input the fused image into an image repair model, serialize and encode the feature map extracted from the fused image through the first branch of the image repair model to obtain a first encoding result, encode a set of sub-feature maps divided by the feature map at different scales through the second branch of the image repair model, and fully connect the encoding results of the sub-feature maps in the sub-feature map set to obtain a second encoding result, and obtain a repaired image according to the first encoding result and the second encoding result.

[0086] Optionally, the image restoration model is obtained by training with training images, the training images include a plurality of similar images, the plurality of similar images include a mask image, and the mask image is obtained by masking the similar images.

[0087] Optionally, the image restoration model is trained in the following manner:

[0088] Extracting a training feature map from the training image;

[0089] Serializing and encoding the training feature map through the first branch of the image restoration model to obtain a first training encoding result;

[0090] The training sub-feature graph sets obtained by dividing the training feature graph according to different scales are respectively encoded by the second branch of the image restoration model, and the encoding results of the training sub-feature graphs in the training sub-feature graph sets are fully connected to obtain a second training encoding result;

[0091] Restoring the mask image according to the first training encoding result and the second training encoding result;

[0092] The image restoration model parameters are updated according to the restored mask image and the training image before masking.

[0093] Optionally, the determining module 604 may be used to:

[0094] Inputting the image to be repaired into a feature comparison model to obtain a search feature of the image to be repaired;

[0095] Acquire multiple similar features similar to the search feature from an image feature library according to the search feature;

[0096] According to the multiple similar features, multiple similar images of the image to be restored are obtained from an image library, and the images in the image library correspond one to one with the features in the image feature library.

[0097] Optionally, the determining module 604 is specifically configured to:

[0098] The image to be repaired is input into a feature comparison model, the image to be repaired is downsampled by the feature comparison model to obtain a feature map of the image to be repaired, and the feature map is encoded to obtain a search feature of the image to be repaired.

[0099] Optionally, the multiple similar images and the image to be restored are images of the same subject at different times or at different angles.

[0100] Optionally, the multiple similar images and the image to be restored are images shot at the same location.

[0101] The functions of the above modules have been described in detail in the method steps in the previous embodiment and will not be repeated here.

[0102] Reference below Figure 7 , which shows a schematic diagram of the structure of an electronic device 700 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0103] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0104] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0105] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0106] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0107] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0108] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0109] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: performs text detection on the image to obtain a text area in the image, wherein the text area includes multiple text lines; constructs a graph network model according to the text area, wherein each text line in the text area is a node of the graph network model; classifies the nodes in the graph network model by a node classification model, and classifies the edges between the nodes in the graph network model by an edge classification model; obtains at least one key-value pair in the image according to the classification results of the nodes and the classification results of the edges. The computer program code for performing the operations of the present disclosure can be written in one or more programming languages ​​or a combination thereof, wherein the programming languages ​​include but are not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0110] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0111] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a module does not, in some cases, limit the module itself.

[0112] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0113] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0114] According to one or more embodiments of the present disclosure, Example 1 provides an image restoration method, the method comprising: obtaining an image to be restored; determining multiple similar images of the image to be restored from an image library, the multiple similar images including at least a first similar image and a second similar image; fusing the image to be restored with the first similar image and the second similar image, and inputting the fused image into an image restoration model, serializing and encoding a feature map extracted from the fused image through a first branch of the image restoration model to obtain a first encoding result, respectively encoding a set of sub-feature maps divided by the feature map at different scales through a second branch of the image restoration model, and fully connecting the encoding results of the sub-feature maps in the set of sub-feature maps to obtain a second encoding result, and obtaining a restored image according to the first encoding result and the second encoding result.

[0115] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, wherein the image restoration model is obtained by training with training images, the training images include multiple similar images, the multiple similar images include mask images, and the mask images are obtained by masking the similar images.

[0116] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 2, and the image restoration model is trained in the following manner: extracting a training feature map from the training image; serializing and encoding the training feature map through the first branch of the image restoration model to obtain a first training encoding result; respectively encoding a set of training sub-feature maps divided according to different scales of the training feature map through the second branch of the image restoration model, and fully connecting the encoding results of the training sub-feature maps in the training sub-feature map set to obtain a second training encoding result; repairing the mask image according to the first training encoding result and the second training encoding result; updating the image restoration model parameters according to the repaired mask image and the training image before masking.

[0117] According to one or more embodiments of the present disclosure, Example 4 provides the method of Example 1, wherein determining multiple similar images of the image to be repaired from an image library comprises: inputting the image to be repaired into a feature comparison model to obtain a search feature of the image to be repaired; obtaining multiple similar features similar to the search feature from an image feature library based on the search feature; obtaining multiple similar images of the image to be repaired from the image library based on the multiple similar features, wherein the images in the image library correspond one-to-one to the features in the image feature library.

[0118] According to one or more embodiments of the present disclosure, Example 5 provides the method of Example 4, wherein the inputting the image to be repaired into a feature comparison model to obtain the search features of the image to be repaired comprises: inputting the image to be repaired into a feature comparison model, downsampling the image to be repaired through the feature comparison model to obtain a feature map of the image to be repaired, and encoding the feature map to obtain the search features of the image to be repaired.

[0119] According to one or more embodiments of the present disclosure, Example 6 provides a method described in any one of Examples 1 to 5, wherein the multiple similar images and the image to be repaired are images of the same subject at different times or at different angles.

[0120] According to one or more embodiments of the present disclosure, Example 7 provides the method described in any one of Examples 1 to 5, wherein the multiple similar images and the image to be restored are images taken at the same location.

[0121] According to one or more embodiments of the present disclosure, Example 8 provides an image restoration device, which includes: an acquisition module, used to acquire an image to be restored; a determination module, used to determine multiple similar images of the image to be restored from an image library, and the multiple similar images include at least a first similar image and a second similar image; a fusion module, used to fuse the image to be restored with the first similar image and the second similar image, and input the fused image into an image restoration model, serialize and encode the feature map extracted from the fused image through the first branch of the image restoration model to obtain a first encoding result, and respectively encode a set of sub-feature maps divided by the feature map at different scales through the second branch of the image restoration model, and fully connect the encoding results of the sub-feature maps in the sub-feature map set to obtain a second encoding result, and obtain the restored image according to the first encoding result and the second encoding result.

[0122] According to one or more embodiments of the present disclosure, Example 9 provides the device of Example 8, wherein the image restoration model is trained by training images, the training images include multiple similar images, the multiple similar images include mask images, and the mask images are obtained by masking the similar images.

[0123] According to one or more embodiments of the present disclosure, Example 10 provides the device of Example 9, wherein the image restoration model is trained in the following manner: extracting a training feature map from the training image; serializing and encoding the training feature map through the first branch of the image restoration model to obtain a first training encoding result; respectively encoding a set of training sub-feature maps divided according to different scales of the training feature map through the second branch of the image restoration model, and fully connecting the encoding results of the training sub-feature maps in the training sub-feature map set to obtain a second training encoding result; repairing the mask image according to the first training encoding result and the second training encoding result; and updating the image restoration model parameters according to the repaired mask image and the training image before masking.

[0124] According to one or more embodiments of the present disclosure, Example 11 provides the device of Example 8, wherein the determination module is specifically used to: input the image to be repaired into a feature comparison model to obtain a search feature of the image to be repaired; obtain multiple similar features similar to the search feature from an image feature library based on the search feature; obtain multiple similar images of the image to be repaired from an image library based on the multiple similar features, and the images in the image library correspond one-to-one to the features in the image feature library.

[0125] According to one or more embodiments of the present disclosure, Example 12 provides the device of Example 11, wherein the determination module is specifically used to: input the image to be repaired into a feature comparison model, downsample the image to be repaired through the feature comparison model to obtain a feature map of the image to be repaired, and encode the feature map to obtain a search feature of the image to be repaired.

[0126] According to one or more embodiments of the present disclosure, Example 13 provides the device described in any one of Examples 8 to 12, wherein the multiple similar images and the image to be repaired are images of the same subject at different times or at different angles.

[0127] According to one or more embodiments of the present disclosure, Example 14 provides the device described in any one of Examples 8 to 12, wherein the multiple similar images and the image to be restored are images taken at the same location.

[0128] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0129] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0130] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. An image restoration method, characterized in that: The method comprises: Obtain the image to be repaired; Determining a plurality of similar images of the image to be restored from an image library, wherein the plurality of similar images at least include a first similar image and a second similar image; The image to be repaired is fused with the first similar image and the second similar image, and the fused image is input into an image repair model, a feature map extracted from the fused image is serialized and encoded through a first branch of the image repair model to obtain a first encoding result, a sub-feature map set obtained by dividing the feature map according to different scales is respectively encoded through a second branch of the image repair model, and the encoding results of the sub-feature maps in the sub-feature map set are fully connected to obtain a second encoding result, and a repaired image is obtained according to the first encoding result and the second encoding result; The image restoration model is obtained by training with training images, the training images include multiple similar images, the multiple similar images include mask images, the mask images are obtained by masking the similar images, and the image restoration model is obtained by training with the following method: Extracting a training feature map from the training image; Serializing and encoding the training feature map through the first branch of the image restoration model to obtain a first training encoding result; The training sub-feature graph sets obtained by dividing the training feature graph according to different scales are respectively encoded by the second branch of the image restoration model, and the encoding results of the training sub-feature graphs in the training sub-feature graph sets are fully connected to obtain a second training encoding result; Restoring the mask image according to the first training encoding result and the second training encoding result; The image restoration model parameters are updated according to the restored mask image and the training image before masking.

2. The method according to claim 1, characterized in that The step of determining a plurality of similar images to the image to be repaired from an image library comprises: Inputting the image to be repaired into a feature comparison model to obtain a search feature of the image to be repaired; Acquire multiple similar features similar to the search feature from an image feature library according to the search feature; According to the multiple similar features, multiple similar images of the image to be restored are obtained from an image library, and the images in the image library correspond one to one with the features in the image feature library.

3. The method according to claim 2, characterized in that The step of inputting the image to be repaired into a feature comparison model to obtain a search feature of the image to be repaired includes: The image to be repaired is input into a feature comparison model, the image to be repaired is downsampled by the feature comparison model to obtain a feature map of the image to be repaired, and the feature map is encoded to obtain a search feature of the image to be repaired.

4. The method according to any one of claims 1 to 3, characterized in that: The multiple similar images and the image to be restored are all images of the same subject at different times or at different angles.

5. The method according to any one of claims 1 to 3, characterized in that: The multiple similar images and the image to be restored are images taken at the same location.

6. An image restoration device, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be repaired; A determination module, configured to determine a plurality of similar images of the image to be restored from an image library, wherein the plurality of similar images at least include a first similar image and a second similar image; A fusion module is used to fuse the image to be repaired with the first similar image and the second similar image, and input the fused image into an image repair model, serialize and encode the feature map extracted from the fused image through the first branch of the image repair model to obtain a first encoding result, respectively encode the sub-feature map sets divided by the feature map at different scales through the second branch of the image repair model, and fully connect the encoding results of the sub-feature maps in the sub-feature map set to obtain a second encoding result, and obtain the repaired image according to the first encoding result and the second encoding result; wherein the image repair model is obtained by training training images, the training images include multiple similar images, the multiple similar images include mask images, the mask images are obtained by masking the similar images, and the image repair model is trained in the following manner: Extracting a training feature map from the training image; Serializing and encoding the training feature map through the first branch of the image restoration model to obtain a first training encoding result; The training sub-feature graph sets obtained by dividing the training feature graph according to different scales are respectively encoded by the second branch of the image restoration model, and the encoding results of the training sub-feature graphs in the training sub-feature graph sets are fully connected to obtain a second training encoding result; Restoring the mask image according to the first training encoding result and the second training encoding result; The image restoration model parameters are updated according to the restored mask image and the training image before masking.

7. The device according to claim 6, characterized in that The determination module is specifically used for: Inputting the image to be repaired into a feature comparison model to obtain a search feature of the image to be repaired; Acquire multiple similar features similar to the search feature from an image feature library according to the search feature; According to the multiple similar features, multiple similar images of the image to be restored are obtained from an image library, and the images in the image library correspond one to one with the features in the image feature library.

8. The device according to claim 7, characterized in that The determination module is specifically used for: The image to be repaired is input into a feature comparison model, the image to be repaired is downsampled by the feature comparison model to obtain a feature map of the image to be repaired, and the feature map is encoded to obtain a search feature of the image to be repaired.

9. The device according to any one of claims 6 to 8, characterized in that The multiple similar images and the image to be restored are all images of the same subject at different times or at different angles.

10. The device according to any one of claims 6 to 8, characterized in that The multiple similar images and the image to be restored are images taken at the same location.

11. An electronic device, characterized in that: The electronic device comprises a processor and a memory; The processor is configured to execute instructions stored in the memory, so that the electronic device executes the method according to any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that: The method comprises instructions for instructing a device to execute the method according to any one of claims 1 to 5.

13. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 5.

Images