Image restoration methods, devices, equipment, media and products

Abstract

The present application provides an image restoration method, apparatus, device, medium, and product. The method comprises: obtaining an image to be restored, then inputting the image to be restored into a structural restoration model, downsampling the image to be restored through multiple branches of the structural restoration model to obtain a first feature sequence and a second feature sequence, converting the first feature sequence into a third feature sequence of the same length as the second feature sequence, fusing the third feature sequence with the second feature sequence, and performing structural restoration on the image to be restored based on the fused feature sequence to obtain an image with the structure of the image to be restored. In this way, a restored image with high restoration accuracy and good restoration effect can be obtained.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image restoration method, apparatus, device, computer-readable storage medium, and computer program product. Background Art

[0002] As image processing technology continues to mature, users have placed higher demands on the quality of image restoration through image processing technology. Image restoration refers to restoring unknown information in an image based on known information in the image, thereby repairing the missing parts of the image.

[0003] Typically, image restoration techniques identify a reference region and the area to be restored within the image to be restored. Using a neural network model, the pixel values of the area to be restored are determined based on the pixel values of the reference region. However, this type of image restoration technique can result in distortions such as ripples and distortions in the restored area, which does not meet user expectations for image restoration results.

[0004] How to improve the effect of image restoration has become an urgent problem that needs to be solved. Summary of the Invention

[0005] The purpose of the present disclosure is to provide an image restoration method, apparatus, device, computer-readable storage medium, and computer program product, which can restore an image from the perspective of the entire image to obtain a more realistic restoration effect.

[0006] In a first aspect, the present disclosure provides an image restoration method, the method comprising:

[0007] Obtain the image to be repaired;

[0008] The image to be repaired is input into a structural repair model, and the image to be repaired is downsampled through multiple branches of the structural repair model to obtain a first feature sequence and a second feature sequence. The first feature sequence is converted into a third feature sequence with the same length as the second feature sequence, and the third feature sequence is fused with the second feature sequence. The structure of the image to be repaired is repaired according to the fused feature sequence to obtain a first repaired image, which is an image in which the structure of the image to be repaired is repaired.

[0009] In a second aspect, the present disclosure provides an image restoration device, comprising:

[0010] An acquisition module, used to acquire the image to be repaired;

[0011] A structural repair module is used to input the image to be repaired into a structural repair model, downsample the image to be repaired through multiple branches of the structural repair model to obtain a first feature sequence and a second feature sequence, convert the first feature sequence into a third feature sequence with the same length as the second feature sequence, fuse the third feature sequence with the second feature sequence, and perform structural repair on the image to be repaired according to the fused feature sequence to obtain a first repaired image, which is an image that repairs the structure of the image to be repaired.

[0012] In a third aspect, the present disclosure provides an electronic device, comprising: a storage device on which a computer program is stored; and a processing device for executing the computer program in the storage device to implement the steps of the method described in the first aspect of the present disclosure.

[0013] In a fourth aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect of the present disclosure.

[0014] In a fifth aspect, the present disclosure provides a computer program product comprising instructions, which, when executed on a device, enables the device to execute the steps of the method described in the first aspect above.

[0015] It can be seen from the above technical solutions that the present disclosure has at least the following advantages:

[0016] In the above technical solution, an electronic device obtains an image to be repaired, then inputs the image to be repaired into a structural repair model. Multiple branches of the structural repair model downsample the image to be repaired to obtain a first feature sequence and a second feature sequence. The first feature sequence is converted into a third feature sequence of the same length as the second feature sequence, and the third feature sequence is fused with the second feature sequence. The structure of the image to be repaired is repaired based on the fused feature sequence, thereby obtaining an image with the structure of the image to be repaired repaired. Because the multiple branches of the structural repair model can downsample the image to be repaired at different scales, extract features of the image to be repaired at different scales, and repair the image to be repaired based on the fused results, a repaired image with high repair accuracy and good results can be obtained.

[0017] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical methods of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments.

[0019] Figure 1A schematic diagram of a process for an image restoration method provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of a structural repair model provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of another structural repair model provided in an embodiment of the present application;

[0022] Figure 4 A schematic diagram of another image restoration method provided in an embodiment of the present application;

[0023] Figure 5 A schematic diagram of a texture / color restoration model provided in an embodiment of the present disclosure;

[0024] Figure 6 A schematic structural diagram of an image restoration device provided in an embodiment of the present application;

[0025] Figure 7 A schematic structural diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] The terms "first" and "second" in the embodiments of this application are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of the features.

[0027] First, some technical terms involved in the embodiments of this application are introduced.

[0028] Image processing technology generally refers to the processing of digital images, specifically the analysis and processing of digital images through computers. Image processing technology can perform various types of image processing, such as repairing missing parts of an image, which is also known as image restoration technology.

[0029] With the continuous development of image processing technology, users have placed higher demands on the effectiveness of image restoration technology. Image restoration technology is aimed at the image to be restored, and determines the restoration area and reference area in the image to be restored. The restoration area can be an area where part of the pattern is missing or the clarity does not meet the user's requirements.

[0030] Typically, image restoration technology can directly predict the pixel values of the restoration area based on the pixel values of the reference area in the image to be restored through a neural network model, thereby restoring the restoration area of the image to be restored.

[0031] However, this restoration method, which directly predicts the pixel values of the repaired area through a model, repairs the image only from the perspective of pixel values, which may cause distortions such as ripples and distortions in the repaired area, and does not meet the user's requirements for the authenticity of image restoration.

[0032] In view of this, the present application provides an image restoration method, which is applied to electronic devices. An electronic device refers to a device with data processing capabilities, such as a server or a terminal. Among them, the terminal includes but is not limited to a smartphone, a tablet computer, a laptop computer, a personal digital assistant (PDA) or a smart wearable device. The server can be a cloud server, such as a central server in a central cloud computing cluster, or an edge server in an edge cloud computing cluster. Of course, the server can also be a server in a local data center. A local data center refers to a data center directly controlled by the user.

[0033] Specifically, the electronic device obtains an image to be repaired, and then inputs the image to be repaired into a structural repair model, downsamples the image to be repaired through multiple branches of the structural repair model to obtain a first feature sequence and a second feature sequence, and then converts the first feature sequence into a third feature sequence with the same length as the second feature sequence, and fuses the third feature sequence with the second feature sequence. According to the fused feature sequence, the structure of the image to be repaired is repaired, thereby obtaining a first repaired image that repairs the structure of the image to be repaired.

[0034] In this way, the structural repair model can repair the image to be repaired according to image features at different scales, and integrate features at different scales to perform structural repair on the model to be repaired, thereby achieving structural repair of the image to be repaired, improving the authenticity of image repair, and meeting users' needs for image repair.

[0035] Furthermore, the electronic device can also input the model after structural repair into the texture repair model and / or color repair model to perform texture repair and / or color repair, thereby achieving texture and / or color repair of the image to be repaired, and obtaining a repaired image that better meets user needs.

[0036] In order to make the technical solution of the present disclosure clearer and easier to understand, the following takes the electronic device as an example. Figure 1 As shown, the image restoration method provided by the embodiment of the present disclosure is introduced, and the method includes the following steps:

[0037] S102: The terminal obtains the image to be repaired.

[0038] The image to be restored may be partially missing or may not meet the user's definition. The terminal can obtain the image to be restored in a variety of ways. For example, in response to a user's confirmation request, the terminal may identify an image stored in the terminal as the image to be restored, and then access a storage unit in the terminal to retrieve the image to be restored. Alternatively, in response to a user's confirmation request, the terminal may identify an image stored in another device as the image to be restored, and then access the image to be restored from the other device. The terminal may also access the image to be restored by accessing other components. For example, the terminal may use a camera to capture a paper photograph to obtain a digital version of the image to be restored.

[0039] S104: The terminal inputs the image to be repaired into the structural repair model, downsamples the image to be repaired through multiple branches of the structural repair model to obtain a first feature sequence and a second feature sequence, converts the first feature sequence into a third feature sequence with the same length as the second feature sequence, fuses the third feature sequence with the second feature sequence, and performs structural repair on the image to be repaired according to the fused feature sequence to obtain a first repaired image.

[0040] Among them, the first repaired image is an image that repairs the structure of the image to be repaired. The conversion of the first feature sequence into a third feature sequence with the same length as the second feature sequence can be performed by upsampling the first feature sequence to obtain the third feature sequence. The conversion of the first feature sequence into a third feature sequence with the same length as the second feature sequence can also be performed by adding the first feature sequence and the fifth feature sequence and upsampling to obtain the third feature sequence, wherein the fifth feature sequence has the same length as the first feature sequence, the fifth feature sequence can be obtained by upsampling the fourth feature sequence, and the fourth feature sequence can be obtained by downsampling the image to be repaired by another branch of the structural repair model. The fusion of the third feature sequence and the second feature sequence can be performed by adding the third feature sequence and the second feature sequence and encoding and decoding them.

[0041] In some possible implementations, the length of the first feature sequence is four times the length of the fourth feature sequence, the length of the fifth feature sequence is the same as the length of the first feature sequence, the length of the second feature sequence is four times the length of the first feature sequence, and the length of the third feature sequence is the same as the length of the second feature sequence.

[0042] For example, Figure 2 As shown, the terminal inputs the image to be repaired into the structural repair model. Multiple branches of the structural repair model downsample the image to be repaired through convolution to obtain corresponding feature images, which are then downsampled again to obtain smaller feature maps. The feature maps are then flattened to obtain first, second, and fourth feature sequences of different lengths.

[0043] The structural restoration model encodes and decodes the fourth feature sequence to convert it into a fifth feature sequence of the same length as the first feature sequence. The fifth feature sequence is then added to the first feature sequence and encoded and decoded. The decoded result is then upsampled to obtain a third feature sequence. The structural restoration model adds the third feature sequence to the second feature sequence and encodes and decodes them to obtain a fused feature sequence. This fused feature sequence can then be used to perform structural restoration on the image to be restored, yielding the first restored image.

[0044] In some possible implementations, the electronic device may further downsample the image to be repaired through multiple branches of the structural repair model to obtain a fourth feature sequence, wherein the length of the fourth feature sequence is different from that of the first feature sequence and the second feature sequence.

[0045] like Figure 3 As shown in the figure, in the structural repair model, the image to be repaired is downsampled by three convolutional networks of different scales to obtain three feature maps of different scales. For example, if the size of the image to be repaired is 256*256, it is downsampled by 4 times by convolutional network 1 (convolution1, conv1) to obtain a feature map of size 64*64, downsampled by 8 times by convolutional network 2 (conv2) to obtain a feature map of size 32*32, and downsampled by 16 times by convolutional network 3 (conv3) to obtain a feature map of size 16*16.

[0046] The structural repair model then downsamples the feature maps, reducing their length to half of their original size, to obtain feature maps of sizes 32*32, 16*16, and 8*8, respectively. These feature maps are then flattened, converting the two-dimensional feature maps into one-dimensional feature sequences of lengths 1024, 256, and 64, respectively. The 64-length sequence is encoded by N encoders and decoded by N decoders. The resulting sequence is then upsampled to obtain a sequence of length 256. The upsampled 256-length sequence is then added to the flattened 256-length sequence. The resulting sum is then encoded and decoded. The resulting sequence is then further upsampled to obtain a sequence of length 1024. The upsampled 1024-length sequence is then added to the flattened 1024-length sequence. The resulting sum is then encoded and decoded to obtain the resulting feature sequence. Then, the structure of the image to be repaired is repaired according to the resulting feature sequence to obtain a first repaired image.

[0047] Because the structural restoration model has multiple branches, each branch can acquire structural features of the image to be restored from different scales. This allows the image to be restored based on these features at different scales, making the structural restoration of the image more accurate, improving the accuracy of image structural restoration and enhancing the user experience. Furthermore, the structural restoration model can learn from the spatial layout of the image, taking into account the uniform distribution of objects in the image, and thus repairing the rough outline of the image.

[0048] The structural repair model can be obtained by training with training images. For example, the terminal can mask the training image to obtain a masked image, where the size of the training image can be 256*256. Then, the three branches of the structural repair model are used to obtain training feature maps of different scales. For example, after conv1 is downsampled by 4 times, a training feature map of size 64*64 is obtained; after conv2 is downsampled by 8 times, a training feature map of size 32*32 is obtained; and after conv3 is downsampled by 16 times, a training feature map of size 16*16 is obtained.

[0049] The structural repair model then downsamples the training feature maps, reducing their length to half of their original size, to obtain training feature maps of sizes 32*32, 16*16, and 8*8, respectively. The training feature maps are then flattened to convert the two-dimensional training feature maps into one-dimensional training feature sequences of lengths 1024, 256, and 64, respectively. The 64-length sequence is encoded by N encoders and decoded by N decoders. The resulting training sequence is then used to repair the masked image, obtaining the first sub-repaired image. The first mean squared loss (MSE) is then calculated using the first sub-repaired image and the unmasked training image.

[0050] At the same time, the structural repair model upsamples the training result sequence to obtain a sequence of length 256, adds the upsampled sequence of length 256 to the flattened sequence of length 256, encodes and decodes the added result, and then uses the obtained training result sequence to repair the mask image to obtain a second sub-repaired image, and then calculates the second sub-repaired image and the unmasked training image to obtain the second mean square function.

[0051] The structural restoration model then upsamples the resulting training sequence to obtain a sequence of length 1024. This upsampled sequence of length 1024 is then added to the flattened sequence of length 1024. The resultant sum is encoded and decoded to obtain a resulting feature sequence. The structure of the image to be restored is then restored based on the resulting feature sequence, obtaining a first restored image. The first restored image is then compared to the unmasked training image to obtain a third mean square function.

[0052] In this way, the terminal can update the parameters of the structural model according to the first mean square function, the second mean square function, and the third mean square function to optimize the structural repair model. Specifically, the terminal can optimize the branch where conv1 is located in the structural repair model by the first mean square function, optimize the branch where conv1 and conv2 are located in the structural repair model by the second mean square function, and optimize the branch where conv1, conv2, and conv3 are located in the structural repair model by the third mean square function.

[0053] The terminal executing the image restoration method of this embodiment and the terminal performing structural model training can be the same terminal or different terminals. In some possible implementations, the terminal can transmit its trained structural restoration model to multiple other terminals, so that the multiple other terminals can directly use the structural restoration model to implement the image restoration method of this disclosure.

[0054] Based on the above description, the present disclosure provides an image restoration method. The terminal obtains an image to be restored, and then inputs the image to be restored into a structural restoration model, downsamples the image to be restored through multiple branches of the structural restoration model, obtains a first feature sequence and a second feature sequence, converts the first feature sequence into a third feature sequence of the same length as the second feature sequence, fuses the third feature sequence with the second feature sequence, and performs structural restoration on the image to be restored according to the fused feature sequence to obtain an image with the structure of the image to be restored restored. Since the multiple branches in the structural restoration model can downsample the image to be restored at different scales, and then extract the features of the image to be restored at different scales, and repair the image to be restored according to the fused results, a restoration image with high restoration accuracy and good effect can be obtained.

[0055] In some possible implementations, such as Figure 4 As shown, the image restoration method further includes the following steps:

[0056] S406: The terminal inputs the first repaired image into a texture repair model to obtain a second repaired image.

[0057] The second inpainted image is the image obtained by performing texture inpainting on the first inpainted image. Specifically, the terminal inputs the first inpainted image into the texture inpainting model, which downsamples the first inpainted image and then flattens it to obtain a sequence. The sequence is then sent to the encoder for encoding. The sequence is then converted into a two-dimensional feature map, which is sent to the deconvolution layer for deconvolution, followed by upsampling to obtain a feature map of the same size as the original image. The final result is obtained according to the fully connected (FC) layer, achieving texture inpainting of the first inpainted image.

[0058] For example, Figure 5 As shown in the figure, the first repaired image (image to be repaired) with a size of 256*256 is downsampled 8 times through the convolution layer to obtain a feature map with a size of 32*32, and then the feature map with a size of 32*32 is flattened to obtain a sequence with a length of 1024. The sequence with a length of 1024 is sent to N encoders for encoding, and the output result of the encoder is converted into a two-dimensional feature map with a size of 32*32. The two-dimensional feature map is then deconvolved to obtain a feature map with a size of 64*64. The feature map with a size of 64*64 is further upsampled to obtain a feature map with a size of 256*256. The final result is then obtained through the FC layer to realize texture repair of the first repaired image. The repaired image is the second repaired image.

[0059] The texture restoration model can be trained using texture training images. Specifically, the texture training image can be masked, and the masked texture training image is downsampled through a convolutional layer to obtain a feature map of the image. The sequence obtained by flattening the feature map is then fed into an encoder for encoding. The encoder output is converted into a feature map and deconvolved, followed by further upsampling. Finally, the texture of the masked texture training image is predicted through an FC layer, and the predicted result is compared with the texture training image to optimize the parameters of the texture restoration model.

[0060] S408: The terminal inputs the second restored image into the color restoration model to obtain a third restored image.

[0061] The third restored image is the image obtained by color restoration of the second restored image. Specifically, the terminal inputs the second restored image into the color restoration model, uses the model to downsample the second restored image, and then flattens it to obtain a sequence. The sequence is sent to the encoder for encoding, and then the sequence is converted into a two-dimensional feature map. The two-dimensional feature map is sent to the deconvolution layer for deconvolution, followed by upsampling to obtain a feature map of the same size as the original image, thereby obtaining the final result according to the fully connected layer, achieving color restoration of the second restored image.

[0062] For example, Figure 5 As shown in the figure, the second repaired image (image to be repaired) with a size of 256*256 is downsampled 8 times by the convolution layer to obtain a feature map with a size of 32*32, and then the feature map with a size of 32*32 is flattened to obtain a sequence with a length of 1024, and the sequence with a length of 1024 is sent to N encoders for encoding, and the output result of the encoder is converted into a two-dimensional feature map with a size of 32*32, and then the two-dimensional feature map is deconvolved to obtain a feature map with a size of 64*64, and the feature map with a size of 64*64 is further upsampled to obtain a feature map with a size of 256*256, and then the final result is obtained through the FC layer to realize color restoration of the second repaired image, and the repaired image is the third repaired image.

[0063] The color restoration model can be trained using color training images. Specifically, the color training image can be masked, and the masked color training image can be downsampled through a convolutional layer to obtain a feature map of the image. The sequence obtained by flattening the feature map is then fed into an encoder for encoding. The encoder output is converted into a feature map and deconvolved, followed by further upsampling. Finally, the color of the masked color training image is predicted through an FC layer, and the predicted result is compared with the color training image to optimize the parameters of the color restoration model.

[0064] The above S406 and S408 are optional steps. The terminal can perform texture repair on the first repaired image after structural repair through S406, and the terminal can also perform color repair on the first repaired image after structural repair through S408. The terminal can also perform texture repair on the first repaired image after structural repair through S406, and perform color repair on the second repaired image after texture repair through S408. The terminal that executes the image repair method in this embodiment and the terminal that performs texture model training and color model training can be the same terminal or different terminals. In some possible implementations, the terminal can transmit its trained texture repair model and / or color repair model to multiple other terminals, so that multiple other terminals can directly use the texture repair model and / or color repair model to implement the image repair method in the present disclosure.

[0065] When the method includes steps S406 and S408, the image restoration method can gradually achieve accurate restoration of the image to be restored from the three aspects of structure, texture, and color of the image to be restored, from the overall to the local. The structural restoration model, texture restoration model, and color restoration model are each obtained from corresponding training images, allowing the three models to learn the structural, texture, and color laws of the image, respectively. Each model accurately restores the function corresponding to the model, thereby improving the accuracy of the model restoration.

[0066] Figure 6 is a schematic diagram of an image restoration device according to an exemplary disclosed embodiment. Figure 6 As shown, the image restoration device 600 includes:

[0067] An acquisition module 602 is used to acquire an image to be restored;

[0068] The structural repair module 604 is used to input the image to be repaired into a structural repair model, downsample the image to be repaired through multiple branches of the structural repair model to obtain a first feature sequence and a second feature sequence, convert the first feature sequence into a third feature sequence with the same length as the second feature sequence, fuse the third feature sequence with the second feature sequence, and perform structural repair on the image to be repaired according to the fused feature sequence to obtain a first repaired image, which is an image that repairs the structure of the image to be repaired.

[0069] Optionally, the device further comprises:

[0070] The texture restoration module and / or color restoration module are used to input the first restoration image into a texture restoration model and / or a color restoration model to perform texture restoration and / or color restoration to obtain a second restoration image, where the second restoration image is an image obtained by performing texture restoration and / or color restoration on the first restoration image.

[0071] Optionally, the structure repair module 604 is further configured to:

[0072] Downsampling the image to be repaired through multiple branches of the structure repair model to obtain a fourth feature sequence;

[0073] The structural repair module is specifically used for:

[0074] The fourth feature sequence is upsampled and fused with the first feature sequence to obtain a third feature sequence with the same length as the second feature sequence.

[0075] Optionally, the device further comprises:

[0076] a texture restoration module, configured to input the first restoration image into a texture restoration model to perform texture restoration to obtain a second restoration image, wherein the second restoration image is an image obtained by performing texture restoration on the first restoration image;

[0077] The color restoration module is used to input the second restored image into a color restoration model to perform color restoration to obtain a third restored image, where the third restored image is an image obtained by performing color restoration on the second restored image.

[0078] Optionally, the length of the second feature sequence is four times the length of the first feature sequence.

[0079] Optionally, the length of the first characteristic sequence is four times the length of the fourth characteristic sequence.

[0080] Optionally, the structure repair module 604 is specifically configured to:

[0081] The third feature sequence is added to the second feature sequence, and encoding and decoding are performed to obtain a fused feature sequence.

[0082] Optionally, the structural repair model is trained in the following manner:

[0083] Acquire a training image, wherein the training image includes a mask image;

[0084] Downsampling the mask image through multiple branches of the structure repair model to obtain a first training feature sequence and a second training feature sequence, converting the first training feature sequence into a third training feature sequence of the same length as the second training feature sequence, fusing the third training feature sequence with the second training feature sequence, and performing structural repair on the mask image according to the fused training feature sequence to obtain a first training repaired image;

[0085] The parameters of the structure restoration model are updated according to the first training restoration image and the training image before masking.

[0086] Optionally, the structure repair module 604 is specifically configured to:

[0087] The first repaired image is input into a texture repair model and / or a color repair model, the first repaired image is downsampled and encoded through the texture repair model and / or the color repair model to obtain a fifth feature sequence, the fifth feature sequence is deconvolved to obtain a feature map, and texture repair and / or color repair are performed on the first repaired image according to the feature map to obtain a second repaired image.

[0088] The functions of the above modules have been described in detail in the method steps of the previous embodiment and will not be repeated here.

[0089] Reference below Figure 7 , which shows a schematic structural diagram of an electronic device 700 suitable for implementing the embodiments of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0090] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the electronic device 700 are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0091] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0092] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0093] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0094] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0095] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0096] The computer-readable medium carries one or more programs. When executed by the electronic device, the electronic device: performs text detection on an image to obtain a text region in the image, wherein the text region includes multiple text lines; constructs a graph network model based on the text region, wherein each text line in the text region is a node in the graph network model; classifies the nodes in the graph network model using a node classification model, and classifies the edges between the nodes in the graph network model using an edge classification model; and obtains at least one key-value pair in the image based on the classification results of the nodes and the classification results of the edges. The computer program code for performing the operations of the present disclosure can be written in one or more programming languages or a combination thereof, including but not limited to object-oriented programming languages such as Java, Smalltalk, and C++, as well as conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. Where a remote computer is involved, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0097] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0098] The modules described in the embodiments of the present disclosure may be implemented in software or hardware, wherein the name of a module does not necessarily limit the module itself.

[0099] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0100] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0101] According to one or more embodiments of the present disclosure, Example 1 provides an image restoration method, the method comprising: acquiring an image to be restored; inputting the image to be restored into a structural restoration model, downsampling the image to be restored through multiple branches of the structural restoration model to obtain a first feature sequence and a second feature sequence, converting the first feature sequence into a third feature sequence with the same length as the second feature sequence, fusing the third feature sequence with the second feature sequence, and performing structural restoration on the image to be restored according to the fused feature sequence to obtain a first restored image, wherein the first restored image is an image in which the structure of the image to be restored is restored.

[0102] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, which further includes: inputting the first repaired image into a texture repair model and / or a color repair model, performing texture repair and / or color repair, and obtaining a second repaired image, wherein the second repaired image is an image obtained by performing texture repair and / or color repair on the first repaired image.

[0103] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 1, which further includes: downsampling the image to be repaired through multiple branches of the structural repair model to obtain a fourth feature sequence; converting the first feature sequence into a third feature sequence with the same length as the second feature sequence includes: upsampling the fourth feature sequence and fusing it with the first feature sequence to obtain a third feature sequence with the same length as the second feature sequence.

[0104] According to one or more embodiments of the present disclosure, Example 4 provides the method of Example 1, which further includes: inputting the first repaired image into a texture repair model to perform texture repair, and obtaining a second repaired image, where the second repaired image is an image in which texture repair is performed on the first repaired image; inputting the second repaired image into a color repair model to perform color repair, and obtaining a third repaired image, where the third repaired image is an image in which color repair is performed on the second repaired image.

[0105] According to one or more embodiments of the present disclosure, Example 5 provides the method of Example 1, wherein the length of the second feature sequence is four times the length of the first feature sequence.

[0106] According to one or more embodiments of the present disclosure, Example 6 provides the method of Example 3, wherein the length of the first feature sequence is four times the length of the fourth feature sequence.

[0107] According to one or more embodiments of the present disclosure, Example 7 provides the method of Example 1, wherein the fusing of the third feature sequence and the second feature sequence includes: adding the third feature sequence and the second feature sequence, and encoding and decoding them to obtain a fused feature sequence.

[0108] According to one or more embodiments of the present disclosure, Example 8 provides the method of Example 1, and the structure repair model is trained in the following manner: obtaining a training image, wherein the training image includes a mask image; downsampling the mask image through multiple branches of the structure repair model to obtain a first training feature sequence and a second training feature sequence, converting the first training feature sequence into a third training feature sequence having the same length as the second training feature sequence, fusing the third training feature sequence with the second training feature sequence, and performing structural repair on the mask image according to the fused training feature sequence to obtain a first training repaired image; and updating the parameters of the structure repair model according to the first training repaired image and the training image before masking.

[0109] According to one or more embodiments of the present disclosure, Example 9 provides the method of Example 2, wherein the first repaired image is input into a texture repair model and / or a color repair model to perform texture repair and / or color repair to obtain a second repaired image, including: inputting the first repaired image into a texture repair model and / or a color repair model, downsampling and encoding the first repaired image through the texture repair model and / or the color repair model to obtain a fifth feature sequence, deconvolving the fifth feature sequence to obtain a feature map, and performing texture repair and / or color repair on the first repaired image according to the feature map to obtain a second repaired image.

[0110] According to one or more embodiments of the present disclosure, Example 10 provides an image restoration device, comprising: an acquisition module for acquiring an image to be restored; a structure restoration module for inputting the image to be restored into a structure restoration model, down-sampling the image to be restored through multiple branches of the structure restoration model to obtain a first feature sequence and a second feature sequence, converting the first feature sequence into a third feature sequence with the same length as the second feature sequence, fusing the third feature sequence with the second feature sequence, and performing structural restoration on the image to be restored according to the fused feature sequence to obtain a first restored image, wherein the first restored image is an image in which the structure of the image to be restored is restored.

[0111] According to one or more embodiments of the present disclosure, Example 11 provides the device of Example 10, which also includes: a texture restoration module and / or a color restoration module, used to input the first restoration image into a texture restoration model and / or a color restoration model, perform texture restoration and / or color restoration, and obtain a second restoration image, wherein the second restoration image is an image obtained by performing texture restoration and / or color restoration on the first restoration image.

[0112] According to one or more embodiments of the present disclosure, Example 12 provides the apparatus of Example 10, wherein the structure repair module is further used to: downsample the image to be repaired through multiple branches of the structure repair model to obtain a fourth feature sequence; the structure repair module is specifically used to: upsample the fourth feature sequence and fuse it with the first feature sequence to obtain a third feature sequence with the same length as the second feature sequence.

[0113] According to one or more embodiments of the present disclosure, Example 13 provides the device of Example 10, which further includes: a texture restoration module, used to input the first restoration image into a texture restoration model, perform texture restoration, and obtain a second restoration image, where the second restoration image is an image in which texture restoration is performed on the first restoration image; and a color restoration module, used to input the second restoration image into a color restoration model, perform color restoration, and obtain a third restoration image, where the third restoration image is an image in which color restoration is performed on the second restoration image.

[0114] According to one or more embodiments of the present disclosure, Example 14 provides the apparatus of Example 10, wherein the length of the second feature sequence is four times the length of the first feature sequence.

[0115] According to one or more embodiments of the present disclosure, Example 15 provides the apparatus of Example 12, wherein the length of the first feature sequence is four times the length of the fourth feature sequence.

[0116] According to one or more embodiments of the present disclosure, Example 16 provides the apparatus of Example 10, wherein the structure repair module is specifically configured to add the third feature sequence to the second feature sequence, and perform encoding and decoding to obtain a fused feature sequence.

[0117] According to one or more embodiments of the present disclosure, Example 17 provides the device of Example 10, wherein the structure repair model is trained in the following manner: obtaining a training image, wherein the training image includes a mask image; downsampling the mask image through multiple branches of the structure repair model to obtain a first training feature sequence and a second training feature sequence, converting the first training feature sequence into a third training feature sequence having the same length as the second training feature sequence, fusing the third training feature sequence with the second training feature sequence, performing structural repair on the mask image according to the fused training feature sequence to obtain a first training repaired image; and updating the parameters of the structure repair model according to the first training repaired image and the training image before masking.

[0118] According to one or more embodiments of the present disclosure, Example 18 provides the device of Example 11, and the structure repair module is specifically used to: input the first repaired image into a texture repair model and / or a color repair model, downsample and encode the first repaired image through the texture repair model and / or the color repair model to obtain a fifth feature sequence, deconvolute the fifth feature sequence to obtain a feature map, and perform texture repair and / or color repair on the first repaired image according to the feature map to obtain a second repaired image.

[0119] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0120] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0121] Although the subject matter has been described using language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims. Regarding the apparatus in the above-described embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method and will not be elaborated upon here.

Claims

1. An image restoration method, characterized in that: The method comprises: Obtain the image to be repaired; Inputting the image to be repaired into a structural repair model, downsampling the image to be repaired through multiple branches of the structural repair model to obtain a first feature sequence, a second feature sequence, and a fourth feature sequence of different lengths, converting the first feature sequence into a third feature sequence of the same length as the second feature sequence, fusing the third feature sequence with the second feature sequence, and performing structural repair on the image to be repaired based on the fused feature sequence to obtain a first repaired image, wherein the first repaired image is an image in which the structure of the image to be repaired is repaired. The structural repair model is further used to repair the contour of the image to be repaired by learning spatial layout information of the image to be repaired; The converting the first feature sequence into a third feature sequence having the same length as the second feature sequence includes: encoding and decoding the fourth feature sequence to convert it into a fifth feature sequence having the same length as the first feature sequence; adding the fifth feature sequence and the first feature sequence, encoding and decoding them, and then upsampling the decoded result to obtain the third feature sequence; The fusing of the third feature sequence and the second feature sequence includes: adding the third feature sequence and the second feature sequence and performing encoding and decoding to obtain the fused feature sequence.

2. The method according to claim 1, characterized in that The method further comprises: The first repaired image is input into a texture repair model and / or a color repair model to perform texture repair and / or color repair to obtain a second repaired image, where the second repaired image is an image obtained by performing texture repair and / or color repair on the first repaired image.

3. The method according to claim 1, characterized in that The method further comprises: Inputting the first repaired image into a texture repair model to perform texture repair to obtain a second repaired image, where the second repaired image is an image obtained by performing texture repair on the first repaired image; The second restored image is input into a color restoration model to perform color restoration to obtain a third restored image, where the third restored image is an image obtained by performing color restoration on the second restored image.

4. The method according to claim 1, wherein The length of the second signature sequence is four times the length of the first signature sequence.

5. The method according to claim 4, characterized in that The length of the first characteristic sequence is four times the length of the fourth characteristic sequence.

6. The method according to claim 1, characterized in that The fusing of the third characteristic sequence with the second characteristic sequence comprises: The third feature sequence is added to the second feature sequence, and encoding and decoding are performed to obtain a fused feature sequence.

7. The method according to claim 1, characterized in that The structural repair model is trained in the following way: Acquire a training image, wherein the training image includes a mask image; Downsampling the mask image through multiple branches of the structure repair model to obtain a first training feature sequence and a second training feature sequence, converting the first training feature sequence into a third training feature sequence of the same length as the second training feature sequence, fusing the third training feature sequence with the second training feature sequence, and performing structural repair on the mask image according to the fused training feature sequence to obtain a first training repaired image; The parameters of the structure restoration model are updated according to the first training restoration image and the training image before masking.

8. The method according to claim 2, characterized in that Inputting the first restored image into a texture restoration model and / or a color restoration model to perform texture restoration and / or color restoration to obtain a second restored image includes: The first repaired image is input into a texture repair model and / or a color repair model, the first repaired image is downsampled and encoded through the texture repair model and / or the color repair model to obtain a fifth feature sequence, the fifth feature sequence is deconvolved to obtain a feature map, and texture repair and / or color repair are performed on the first repaired image according to the feature map to obtain a second repaired image.

9. An image restoration device, characterized in that: The device comprises: An acquisition module, used to acquire the image to be repaired; a structural restoration module, configured to input the image to be restored into a structural restoration model, downsample the image to be restored through multiple branches of the structural restoration model to obtain a first feature sequence, a second feature sequence, and a fourth feature sequence of different lengths, convert the first feature sequence into a third feature sequence of the same length as the second feature sequence, fuse the third feature sequence with the second feature sequence, and perform structural restoration on the image to be restored based on the fused feature sequence to obtain a first restored image, wherein the first restored image is an image in which the structure of the image to be restored is restored. The structural restoration model is further configured to restore the contour of the image to be restored by learning spatial layout information of the image to be restored; Among them, the structure repair module is specifically used to: encode and decode the fourth feature sequence and convert it into a fifth feature sequence with the same length as the first feature sequence; add the fifth feature sequence and the first feature sequence and encode and decode them, and then upsample the decoding result to obtain the third feature sequence; add the third feature sequence and the second feature sequence and encode and decode them to obtain the fused feature sequence.

10. A device, characterized in that The device includes a processor and a memory; The processor is configured to execute instructions stored in the memory, so that the device performs the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that The method comprises instructions for instructing a device to execute the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to perform the method according to any one of claims 1 to 8.

Images