Image restoration method, device, electronic device, and computer storage medium

Abstract

This application provides an image restoration method, apparatus, electronic device, and computer storage medium. The image restoration method includes: acquiring an image to be restored and a corresponding quality score; extracting image features from the image to be restored; performing residual recovery processing on the image to be restored based on the acquired quality score to obtain a residual recovery processing result; and obtaining a restored image based on the residual recovery processing result. This application improves the image restoration effect.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image restoration method, apparatus, electronic device, and computer storage medium. Background Technology

[0002] During the acquisition, processing, transmission, and recording of images, image distortion or quality degradation is inevitable due to various possible factors such as imaging systems, camera methods, transmission media, and recording equipment, or other factors such as object movement and noise pollution. Therefore, in specific application scenarios, image inpainting is generally used to repair these images (referred to as images to be repaired).

[0003] However, current image restoration methods use a uniform restoration approach regardless of the degree of image distortion or quality degradation, failing to provide targeted restoration based on different levels of distortion or quality degradation, resulting in poor restoration outcomes. Summary of the Invention

[0004] This application provides an image restoration scheme to at least partially solve the above-mentioned problems.

[0005] According to a first aspect of the embodiments of this application, an image restoration method is provided, comprising: acquiring an image to be restored and a quality score corresponding to the image to be restored; extracting image features of the image to be restored; performing residual recovery processing on the image to be restored based on the acquired quality score to obtain a residual recovery processing result; and obtaining a restored image based on the residual recovery processing result.

[0006] According to a second aspect of the embodiments of this application, an image restoration apparatus is provided, comprising: an image acquisition unit for acquiring an image to be restored and a quality score corresponding to the image to be restored; a feature extraction unit for extracting image features of the image to be restored; a residual recovery unit for performing residual recovery processing on the image to be restored based on the acquired quality score to obtain a residual recovery processing result; and an image generation unit for obtaining a restored image based on the residual recovery processing result.

[0007] According to a third aspect of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the image restoration method described in the first aspect.

[0008] According to a third aspect of the embodiments of this application, a computer storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the image restoration method as described in the first aspect.

[0009] According to a fourth aspect of the embodiments of this application, an image restoration service is provided, wherein, in response to an image restoration command, the image restoration method as described in the first aspect is performed on the image pointed to by the image restoration command.

[0010] In the technical solution provided in this application embodiment, the image to be repaired corresponds to a quality score. When repairing the image to be repaired, residual recovery processing is performed on the image features used to characterize the image based on the quality score, so as to obtain the repaired image based on the processing result. On the one hand, the residual recovery processing method is more accurate in image repair than other methods; on the other hand, during the repair process, the intensity of image repair can be controlled by the quality score. Because images to be repaired with different degrees of distortion or quality degradation will correspond to different quality scores, controlling the repair intensity by the quality score avoids the phenomenon of over-repairing or under-repairing different images, thus improving the repair accuracy and the repair effect. Attached Figure Description

[0011] The following sections will describe some specific embodiments of the present application in a detailed manner by way of example and not limitation, with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar parts or components. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0012] Figure 1A This is a schematic diagram of the image restoration method in Embodiment 1 of this application;

[0013] Figure 1B This is a schematic diagram illustrating the application of the image restoration method according to an embodiment of this application in one application scenario;

[0014] Figure 2A This is a schematic diagram illustrating the structure of an image restoration model used in an embodiment of this application to restore the image to be restored.

[0015] Figure 2B This is an exemplary structural diagram of the residual block in an embodiment of this application;

[0016] Figure 2C This is a schematic diagram of the image restoration method in Embodiment 2 of this application;

[0017] Figure 2D This is a schematic diagram illustrating the structure of an embodiment of this application that uses n residual blocks to perform n residual recovery processes to repair the image to be repaired;

[0018] Figure 3 This is a schematic diagram of an embodiment of an image restoration device according to this application;

[0019] Figure 4 This is a schematic diagram of an electronic device according to this application. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0021] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0022] refer to Figure 1A In Embodiment 1 of this application, the image restoration method includes:

[0023] S101. Obtain the image to be repaired and the quality score corresponding to the image to be repaired.

[0024] In this embodiment, for example, the image to be repaired can be any suitable image, including but not limited to grayscale images, color images, etc.

[0025] In this embodiment, for example, the image to be repaired includes at least one of the following: video frame images, video cover images, search result images, recommended images, and information stream images from a video stream.

[0026] The quality score of the image to be repaired reflects its quality. In one feasible approach, a higher quality score indicates better quality, and a lower quality score indicates lower quality. Optionally, the quality score of the image to be repaired can range from 0 to 1. Factors affecting the quality of the image to be repaired include, but are not limited to, various possible factors such as the imaging system, imaging method, transmission medium, and recording equipment, or other possible factors such as object movement and noise pollution.

[0027] In this embodiment, the quality score of the image to be repaired can be performed manually, or the image quality can be evaluated using a set image quality evaluation model, or an appropriate scoring algorithm can be used to obtain the quality score of the image to be repaired.

[0028] In one alternative approach, the image to be repaired is evaluated using a predefined image quality assessment model to obtain a quality score. Obtaining the quality score using an image quality assessment model yields a more accurate score compared to other methods. The image quality assessment model can be implemented as any suitable network model, such as the Vgg13bn network model.

[0029] Further optionally, before performing image quality assessment on the image to be repaired using the established image quality assessment model, the method further includes: acquiring sample images for training the image quality assessment model, and quality score labels corresponding to the sample images, wherein the quality score labels are determined based on the signal-to-noise ratio (SNR) of the sample images and the similarity between the sample images and reference images; and training the image quality assessment model using the sample images and the quality score labels, thereby making the quality score of the image to be repaired obtained by the image quality assessment model more accurate when performing quality assessment on the image to be repaired.

[0030] Among them, the peak signal-to-noise ratio (PSNR) of a sample image is the ratio of the maximum possible power of a sample image to the power of destructive noise, usually expressed in logarithmic decibels. The structural similarity (SSIM) between a sample image and a reference image is a metric that measures the similarity between the two images.

[0031] In one alternative approach, the quality score label, determined based on the signal-to-noise ratio (SNR) of the sample image and the similarity between the sample image and the reference image, can be implemented as follows: the quality score label is determined by multiplying the SNR by the similarity. This product-based determination method further improves the accuracy of the quality score.

[0032] The specific implementation of training the image quality assessment model using the aforementioned sample images and their corresponding quality score labels can be achieved by referring to relevant model training methods; this application embodiment does not impose any limitations on this. The trained image quality assessment model can provide more accurate quality assessments for various images.

[0033] S102. Extract the image features of the image to be repaired.

[0034] Image features can be used to describe images; in this embodiment, they are used to describe the image to be repaired. For example, in step S102, the image features of the image to be repaired can be extracted using a convolutional layer in the image repair model structure. Alternatively, when extracting the image features of the image to be repaired, the convolutional layer can also convert the original image channels of the image to be repaired into multiple image channels that conform to the channel standard that can be processed by the residual recovery processing in step S103.

[0035] S103. Based on the obtained quality score, perform residual recovery processing on the image to be repaired based on the image features to obtain the residual recovery processing result.

[0036] In this embodiment, for example, the residual recovery processing result can specifically be the image features obtained by residual recovery processing.

[0037] In this embodiment, exemplarily, in step S103, the image to be repaired can be subjected to residual recovery processing one or more times based on the image features of the image to be repaired and the identity mapping function determined by the quality score for residual recovery processing, to obtain the residual recovery processing result. For a single residual recovery processing, the identity mapping function reflects the relationship between the image before and after processing based on the residuals. By using the identity mapping function, the accuracy of residual recovery processing can be improved.

[0038] Further, in one optional embodiment, the identity mapping function may include a first calculation factor and a second calculation factor, wherein the first calculation factor is an image residual value obtained based on the image feature estimate and a preset residual function, and the second calculation factor is the result of processing the image feature estimate with the quality score as a weight. In other embodiments, the identity mapping function may also take other forms.

[0039] S104. Based on the residual recovery processing result, the repaired image is obtained.

[0040] In this embodiment, for example, the repaired image can be obtained by using a convolutional layer in the image restoration model structure based on the residual recovery processing result.

[0041] like Figure 1BAs shown, taking the restoration of a video file's cover image in an application scenario as an example, a video file typically includes multiple frames. Assuming one frame is damaged, such as the video frame serving as the cover image, it can be used as the image to be restored. According to this embodiment, firstly, the image to be restored and its corresponding quality score are obtained; then, image features of the image to be restored are extracted using an appropriate method, such as feature extraction via convolution; next, residual recovery processing is performed based on these image features, wherein the intensity of the residual recovery processing is controlled according to the quality score to obtain the residual recovery processing result; finally, based on the residual recovery processing result, the restored image is obtained, and this restored image is used as the cover image of the video file.

[0042] The above example uses the cover image of a video file, but it is not limited to this. The embodiments of the present invention can be applied to other scenarios, such as video frame images in a video stream, search result images, recommendation images, and information stream images.

[0043] In a video stream, if one or more frames are damaged, the above-mentioned image restoration method can be used to repair them. After restoration, the damaged images can be replaced in the video stream. This avoids the impact of image quality on subsequent processing or playback of the video stream.

[0044] In the context of search results images, if a user performs an image search and one or more of the search results images are damaged, the image restoration method described above can be used to repair the damaged images before showing them to the user, ensuring that the user receives high-quality images.

[0045] In the recommended image scenario, similar to the search result image scenario, if there is a damaged image among the images to be recommended to the user, the above image restoration method can be used to restore the damaged image before recommending it to the user, so that the user can obtain a higher quality image.

[0046] In image-based information feeds (such as feeds), images guide users to view or browse the feed. If the image is damaged, this function is greatly diminished. Therefore, the image restoration methods described above can be used to repair the damaged image corresponding to the feed before displaying it.

[0047] In this embodiment, the image to be repaired corresponds to a quality score. When repairing the image, residual recovery processing is performed on the image features representing the image based on the quality score, so as to obtain the repaired image based on the processing result. On the one hand, residual recovery processing is more accurate than other methods for image repair; on the other hand, during the repair process, the intensity of image repair can be controlled by the quality score. Images with different degrees of distortion or quality degradation will correspond to different quality scores. Controlling the repair intensity by the quality score avoids the phenomenon of over- or under-repairing different images, thus improving the repair accuracy and the repair effect.

[0048] refer to Figure 2A In another embodiment of this application, the restoration of the image to be restored is illustrated by using an image restoration model.

[0049] The image inpainting model includes a pre-convolutional layer, n residual blocks (n≥8), and a post-convolutional layer, wherein the n residual blocks are connected to the pre-convolutional layer and the n residual blocks are connected to the post-convolutional layer. In an optional embodiment, n is 12 to achieve a balance between inpainting effect and inpainting efficiency.

[0050] In this process, the image features of the image to be repaired are first extracted using a pre-convolutional layer. Furthermore, the pre-convolutional layer can also convert the original image channels of the image to be repaired into multiple image channels that conform to the channel standard processed by the n residual blocks, facilitating subsequent processing by the n residual blocks. The n residual blocks are used to perform n residual restoration processes on the image to be repaired to obtain the residual restoration result. The n residual blocks are sequentially labeled as Residual Block 1, Residual Block 2, ..., Residual Block n according to the order of the residual restoration processes. In this embodiment, all residual blocks use the same identity mapping function. The post-convolutional layer is used to obtain the repaired image based on the residual restoration result. Additionally, the post-convolutional layer can also convert the image channels of the residual restoration result back into the original image channels, making the repaired image more closely resemble the original image.

[0051] refer to Figure 2B This is a schematic diagram of an exemplary structure of a residual block. (Example:) Figure 2BAs shown, each residual block includes two convolutional layers (referred to as convolutional layer 1 and convolutional layer 2) and a nonlinear activation layer (ReLU). The image feature estimate is obtained by taking the feature value of the image feature of the current residual block as input. The image feature estimate is then processed by convolutional layer 1, nonlinear activation layer (ReLU), and convolutional layer 2 in sequence, and then by convolutional processing, nonlinear processing, and convolutional processing respectively, to obtain the convolutional processing result of convolutional layer 2. Then, residual restoration processing is performed based on the convolutional processing result of convolutional layer 2, the image feature estimate, and the quality score of the image to be restored by the identity mapping function, and the processed image features are output according to the residual restoration processing result.

[0052] refer to Figure 2C In Embodiment 2 of this application, based on Figure 2A , 2B The provided image restoration model implements image restoration methods, including:

[0053] S201. Obtain the image to be repaired and the quality score corresponding to the image to be repaired.

[0054] The implementation of this step can refer to the implementation of step S101 in the above embodiment one, and will not be repeated here.

[0055] S202. Extract the image features of the image to be repaired through the pre-convolutional layer of the image restoration model.

[0056] S203. The image features of the image to be repaired are input into n consecutive residual blocks through the image restoration model. The image to be repaired is then subjected to n residual restoration processes through the n residual blocks according to the identity mapping function to obtain the residual restoration processing result.

[0057] In this embodiment, for example, step S203, which involves performing multiple residual recovery processes on the image to be repaired using the plurality of residual blocks according to the identity mapping function, includes:

[0058] For each residual block, the image feature estimate is obtained by taking the feature value of the image feature of the current residual block as input. The residual recovery process of the current residual block is performed according to the identity mapping function, and the processed image feature is output according to the residual recovery process result.

[0059] See Figure 2D Taking an application scenario that utilizes n residual blocks for n residual recovery processes as an example, in Figure 2DIn this diagram, the first convolutional layer is denoted as CONV_FIRST, and the n residual blocks are denoted as RESBLOCK1...RESBLOCKn. The last convolutional layer is denoted as CONV_LAST. The residual recovery result obtained from RESBLOCK1 is denoted as Residual Recovery Result 1, the residual recovery result obtained from RESBLOCK2 is denoted as Residual Recovery Result 2, and so on, until the residual recovery result obtained from RESBLOCKn is denoted as Residual Recovery Result n. As shown in the diagram, Residual Recovery Result 1 serves as the input to RESBLOCK2, Residual Recovery Result 2 serves as the input to RESBLOCK3, and so on, with the residual recovery result n-1 corresponding to RESBLOCKn-1 serving as the input to RESBLOCKn. After obtaining the residual recovery result n, the repaired image is obtained after processing with CONV_LAST.

[0060] Figure 2D In the illustrated application scenario diagram, the identity mapping function is: R0 = RI * score + RC2. Where R0 represents the output of the current residual block, R1 represents the input of the current residual block, score represents the quality score, and RC2 represents the processing result of convolutional layer 2 in the current residual block (refer to...). Figure 2B (as shown in the image).

[0061] S204. The restored image is obtained by the post-convolutional layer of the image restoration model based on the residual recovery processing result.

[0062] In this embodiment, for example, the repaired image is obtained based on the residual recovery processing result n of the residual block n.

[0063] For example, the above Figure 2D In this process, the kernel size used in the pre-convolutional layer, residual block, and post-convolutional layer can be 3*3. The stride used during convolution processing is 1, and the padding parameter is 1. The pre-convolutional layer, residual block convolutional layer, and post-convolutional layer can contain up to 64 kernels. Furthermore, if the image to be repaired is an RGB image, the pre-convolutional layer has 3 input image channels. Since it has 64 kernels, it outputs 64 image channels. Each residual block has 64 input and 64 output image channels. The post-convolutional layer has 64 input image channels. To ensure the repaired image is also an RGB image, it outputs 3 image channels.

[0064] Assume the image feature estimate input to each residual block is denoted as R1, the convolution result after processing by convolutional layer 2 is denoted as RC2, and the processed image feature output by each residual block is R0. The identity mapping function is defined as above: R0 = RI * score + RC2, or alternatively, R0 = RI * score + (1 - score) * RC2. For the current residual block (excluding the first residual block), its image feature estimate R1 is the image feature output of the previous residual block (i.e., R0 output by the previous residual block), and the image feature R0 output by the current residual block is the image feature estimate input to the next residual block (i.e., R1 input to the next residual block). Furthermore, the image feature input to the first residual block is the image feature extracted from the image to be repaired by the previous convolutional layer.

[0065] For R0 = RI * score + RC2, RC2, the result of processing the image feature estimate through convolutional layer 1, nonlinear activation layer (ReLU), and convolutional layer 2, is used as the first calculation factor (the result of processing the image feature estimate through residual blocks, or the image residual value obtained based on the image feature estimate and a preset residual function). RI * score, the product of the quality score and the image feature estimate, is used as the second calculation factor (the result of processing the image feature estimate with the quality score as the weight).

[0066] For R0 = RI*score + (1-score)*RC2, where RI*score is used as the second calculation factor and (1-score)*RC2 is used as the first calculation factor (the image residual value obtained from the image feature estimate and the preset residual function, and processed according to the residual weights). It can be seen that the sum of the weights in the second calculation factor (score) and the weights in the first calculation factor (1-score) is 1. Compared to R0 = RI*score + RC2, R0 = RI*score + (1-score)*RC2 converges faster.

[0067] According to this embodiment, the image to be repaired corresponds to a quality score. When repairing the image, residual recovery processing is performed on the image features used to characterize the image based on the quality score, so as to obtain the repaired image based on the processing result. On the one hand, the residual recovery processing method is more accurate in image repair compared to other methods; on the other hand, during the repair process, the intensity of image repair can be controlled by the quality score. Images with different degrees of distortion or quality degradation will correspond to different quality scores. Controlling the repair intensity by the quality score avoids the phenomenon of over-repairing or under-repairing different images, thus improving the repair accuracy and the repair effect.

[0068] Figure 3 This is a schematic diagram of an embodiment of an image restoration device according to this application, as shown below. Figure 3 As shown, it includes:

[0069] Image acquisition unit 301 is used to acquire the image to be repaired and the quality score corresponding to the image to be repaired;

[0070] Feature extraction unit 302 is used to extract image features of the image to be repaired;

[0071] The residual recovery unit 303 is used to perform residual recovery processing on the image to be repaired based on the image features according to the acquired quality score, and obtain the residual recovery processing result;

[0072] The image generation unit 304 is used to obtain the repaired image based on the residual recovery processing result.

[0073] Optionally, the residual recovery unit 303 is used to perform residual recovery processing on the image to be repaired more than once based on the image features of the image to be repaired and the identity mapping function determined by the quality score for residual recovery processing, so as to obtain the residual recovery processing result.

[0074] Optionally, the identity mapping function includes a first calculation factor and a second calculation factor, wherein the first calculation factor is the image residual value obtained based on the image feature estimate and a preset residual function, and the second calculation factor is the result of processing the image feature estimate with the quality score as a weight.

[0075] Optionally, the first calculation factor is the image residual value obtained based on the image feature estimate and the preset residual function, and the image residual value is processed according to the residual weight, wherein the sum of the residual weight and the quality score is 1.

[0076] Optionally, the residual recovery unit 303 is used to input the image features of the image to be repaired into a plurality of residual blocks that are sequentially connected, and to perform multiple residual recovery processes on the image to be repaired through the plurality of residual blocks according to the identity mapping function to obtain the residual recovery processing result.

[0077] Optionally, when the residual recovery unit 303 performs multiple residual recovery processes on the image to be repaired through the plurality of residual blocks according to the identity mapping function: for each residual block, the feature value of the image feature of the current residual block is used as the image feature estimate value, the residual recovery process of the current residual block is performed according to the identity mapping function, and the processed image feature is output according to the residual recovery process result.

[0078] Optionally, a pre-convolutional layer is connected before the plurality of residual blocks, and a post-convolutional layer is connected after the plurality of residual blocks; the pre-convolutional layer is used to convert the original image channels of the image to be repaired into a plurality of image channels that conform to the channel standard that the plurality of residual blocks can process; the post-convolutional layer is used to convert the image channels of the residual recovery processing result back into the original image channels.

[0079] Optionally, before obtaining the quality score of the image to be repaired, the method further includes: performing an image quality assessment on the image to be repaired using a set image quality evaluation model to obtain the quality score of the image to be repaired.

[0080] Optionally, the image restoration apparatus of this embodiment further includes: a training module, configured to acquire sample images for training the image quality evaluation model, and quality score labels corresponding to the sample images, before performing image quality evaluation on the image to be restored using the set image quality evaluation model, wherein the quality score labels are determined based on the signal-to-noise ratio (SNR) of the sample images and the similarity between the sample images and reference images; and to train the image quality evaluation model using the sample images and the quality score labels.

[0081] Optionally, the quality score is labeled as the product of the signal-to-noise ratio and the similarity.

[0082] Optionally, the image to be repaired includes at least one of the following: video frame images, video cover images, search result images, recommended images, and information stream images from a video stream.

[0083] The image restoration device of this embodiment is used to implement the corresponding image restoration methods in the foregoing multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0084] This application also provides an electronic device, which includes: a processor 401, a memory 402, a communication interface 403, and a communication bus 404, wherein the processor 401, the memory 402, and the communication interface 403 communicate with each other through the communication bus 404.

[0085] The memory 402 is used to store at least one executable instruction, which causes the processor 401 to perform an operation corresponding to the image restoration method described in any embodiment of this application.

[0086] This application also provides an image restoration service, which can be used to perform an image restoration method as described in any of the foregoing method embodiments on an image pointed to by an image restoration command in response to an image restoration command.

[0087] For example, the image restoration service can be deployed on the same platform as the client, or the image restoration service can be deployed on the server side and called via an API (Application Programming Interface).

[0088] For example, after receiving the image to be repaired, the client calls the image repair service deployed locally on the client. The image repair service repairs the image to be repaired in the manner described in the aforementioned multiple image repair method embodiments and then returns the repaired image.

[0089] For example, after receiving the image to be repaired, the client calls the image repair service through the external interface provided by the image repair service deployed on the server. This image repair service repairs the image as described in the aforementioned embodiments of image repair methods, and then returns the repaired image to the client through the external interface or other return interfaces. Alternatively, the client can send the image to be repaired to the server, which then repairs the image using its image repair service and returns it to the client.

[0090] This application also provides a computer storage medium storing a computer program that, when executed by a processor, implements the image restoration method as described in any embodiment.

[0091] Memory, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the method embodiments in this application. The processor executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory, thereby implementing the above-described method embodiments.

[0092] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created according to specific use. Furthermore, the memory may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state memory. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to a video server via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0093] The aforementioned electronic device and computer storage medium can execute the methods provided in the embodiments of this application, and have the corresponding functional modules and beneficial effects for executing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0094] The electronic devices in this application embodiments exist in various forms, including but not limited to:

[0095] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.

[0096] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0097] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players (such as iPods), handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0098] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0099] (5) Other electronic devices with data interaction functions.

[0100] Specific embodiments of the subject matter have now been described. Other embodiments are within the scope of the appended claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing can be advantageous.

[0101] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0102] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0103] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0104] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0105] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0106] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0107] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0108] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0109] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0110] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0111] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0112] This application can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific transactions or implement specific abstract data types. This application can also be practiced in distributed computing environments where transactions are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0113] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0114] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. An image restoration method, comprising: Obtain the image to be repaired and its corresponding quality score; Extract the image features of the image to be repaired; Based on the image features of the image to be repaired, and based on the identity mapping function determined by the quality score for residual recovery processing, the image to be repaired is subjected to residual recovery processing one or more times to obtain the residual recovery processing result. The identity mapping function includes a first calculation factor and a second calculation factor. The first calculation factor is the image residual value obtained based on the image feature estimate and the preset residual function. The second calculation factor is the result of processing the image feature estimate with the quality score as the weight. Based on the residual recovery processing results, the repaired image is obtained.

2. The image restoration method according to claim 1, wherein, The first calculation factor is the image residual value obtained based on the image feature estimate and the preset residual function, and the result of processing the image residual value according to the residual weight. The sum of the residual weight and the quality score is 1.

3. The image restoration method according to claim 1 or 2, wherein, The process involves performing residual recovery processing on the image to be repaired at least once, based on the image features of the image to be repaired and the identity mapping function determined by the quality score for residual recovery processing, to obtain the residual recovery processing result, including: The image features of the image to be repaired are input into multiple residual blocks that are sequentially connected. The residual blocks are then used to perform multiple residual recovery processes on the image to be repaired according to the identity mapping function, so as to obtain the residual recovery processing result.

4. The image restoration method according to claim 3, wherein, The step of performing multiple residual recovery processes on the image to be repaired using the multiple residual blocks according to the identity mapping function includes: For each residual block, the image feature estimate is obtained by taking the feature value of the image feature of the current residual block as input. The residual recovery process of the current residual block is performed according to the identity mapping function, and the processed image feature is output according to the residual recovery process result.

5. The image restoration method according to claim 3, wherein, The plurality of residual blocks are connected to a pre-convolutional layer and are connected to a post-convolutional layer. The preconvolutional layer is used to convert the original image channels of the image to be repaired into multiple image channels that conform to the channel standard that the multiple residual blocks can process; The post-convolutional layer is used to convert the image channels of the residual recovery processing result into the original image channels.

6. The image restoration method according to claim 1 or 2, wherein, Before obtaining the quality score of the image to be repaired, the method further includes: performing an image quality assessment on the image to be repaired using a set image quality evaluation model to obtain the quality score of the image to be repaired.

7. The image restoration method according to claim 6, wherein, Before performing image quality assessment on the image to be repaired using the established image quality assessment model, the method further includes: Obtain sample images for training the image quality assessment model, and quality score labels corresponding to the sample images, wherein the quality score labels are determined based on the signal-to-noise ratio (SNR) of the sample images and the similarity between the sample images and reference images; use the sample images and the quality score labels to train the image quality assessment model.

8. The image restoration method according to claim 7, wherein, The quality score is labeled as the product of the signal-to-noise ratio and the similarity.

9. An image restoration apparatus, comprising: An image acquisition unit is used to acquire the image to be repaired and the quality score corresponding to the image to be repaired. The feature extraction unit is used to extract image features of the image to be repaired; The residual recovery unit is used to perform residual recovery processing on the image to be repaired more than once based on the image features of the image to be repaired and the identity mapping function determined by the quality score for residual recovery processing, to obtain the residual recovery processing result. The identity mapping function includes a first calculation factor and a second calculation factor. The first calculation factor is the image residual value obtained based on the image feature estimate and the preset residual function. The second calculation factor is the result of processing the image feature estimate with the quality score as the weight. An image generation unit is used to obtain the repaired image based on the residual recovery processing result.

10. An electronic device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the image restoration method as described in any one of claims 1-8.

11. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the image restoration method as described in any one of claims 1-8.

12. A server-side application that deploys an image restoration service, wherein, In response to an image restoration command, the image restoration method of any one of claims 1-8 is performed on the image pointed to by the image restoration command.

Images