Image restoration method and system based on auxiliary information, terminal and storage medium

Abstract

The invention discloses an image restoration method and system based on auxiliary information, a terminal and a storage medium, and the method comprises the steps: obtaining a to-be-processed blurred image, and carrying out the information analysis of the to-be-processed blurred image, and obtaining target image information; performing optimization processing on a blurring kernel estimation scheme of the to-be-processed blurred image according to the target image information to obtain a target blurring kernel estimation scheme, and performing optimization processing on a restoration strategy of the to-be-processed blurred image according to the target image information to obtain a target restoration strategy; and repairing the to-be-processed blurred image according to the target blurring kernel estimation scheme and the target repairing strategy to obtain a deblurred image, and optimizing the quality of the deblurred image to obtain a target image. According to the invention, based on the auxiliary information related to the image, the deblurring effect of the image obtained after processing is better, and the corresponding imaging quality is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image restoration method, system, terminal and computer-readable storage medium based on auxiliary information. Background Art

[0002] A camera is a device that converts image signals into electrical signals and is widely used in mobile phones, computers, surveillance, medical and other fields. The imaging quality of a camera depends not only on its internal sensors, circuits and algorithms, but also on external factors such as lenses and glass covers. Among them, the lens is one of the most important components of a camera, which determines the camera's viewing angle, aperture, focal length and other parameters. It is also easily affected by the external environment, thus affecting the camera's imaging effect, which is mainly manifested in overall or partial image blur.

[0003] At present, the existing blurred image processing methods are not based on relevant information sources as auxiliary information, resulting in the overall effect of the processed blurred image is not very good, resulting in low image quality and affecting the user experience. Therefore, how to repair the blurred image based on auxiliary information to obtain a high-quality image is a difficult problem that needs to be solved in the field of image processing.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide an image restoration method, system, terminal and storage medium based on auxiliary information, aiming to solve the problem in the prior art that there is no relevant information source as auxiliary information, resulting in poor overall effect of the processed blurred image and low imaging quality.

[0006] To achieve the above object, the present invention provides an image restoration method based on auxiliary information, and the image restoration method based on auxiliary information comprises the following steps:

[0007] Acquire a blurred image to be processed, and perform information analysis on the blurred image to be processed to obtain target image information;

[0008] Optimizing the blur kernel estimation scheme of the blurred image to be processed according to the target image information to obtain a target blur kernel estimation scheme, and optimizing the restoration strategy of the blurred image to be processed according to the target image information to obtain a target restoration strategy;

[0009] The blurred image to be processed is repaired according to the target blur kernel estimation scheme and the target repair strategy to obtain a deblurred image, and the quality of the deblurred image is optimized to obtain a target image.

[0010] Optionally, in the image restoration method based on auxiliary information, the target image information includes time information, image scene information and image depth information;

[0011] The step of obtaining a blurred image to be processed and performing information analysis on the blurred image to be processed to obtain target image information specifically includes:

[0012] Acquire a blurred image to be processed, read metadata of the blurred image to be processed, acquire a corresponding image capture timestamp according to the metadata, and obtain the time information according to the image capture timestamp;

[0013] Performing scene classification processing on the blurred image to be processed to obtain a classification result, performing feature extraction on the blurred image to be processed according to the classification result to obtain scene features, and obtaining the image scene information according to the scene features;

[0014] Depth estimation is performed on the blurred image to be processed to obtain scene depth information, and smoothing is performed on the scene depth information to obtain image depth information.

[0015] Optionally, the image restoration method based on auxiliary information, wherein the step of performing depth estimation on the blurred image to be processed to obtain scene depth information, specifically includes:

[0016] Acquire the image viewing angle of the blurred image to be processed, wherein the image viewing angle includes a single viewing angle and a multi-viewing angle;

[0017] If the image perspective is a single perspective, feature extraction is performed on the blurred image to be processed through a deep convolutional network to obtain target features, and regional depth inference is performed based on the target features to obtain scene depth information, wherein the target features include texture features, illumination features, and object size features;

[0018] If the image viewing angle is multi-view, the disparity calculation is performed on the blurred image to be processed by a stereoscopic vision algorithm to obtain a position difference result, and the depth calculation is performed according to the position difference result to obtain scene depth information.

[0019] Optionally, the image restoration method based on auxiliary information, wherein the step of optimizing the blur kernel estimation scheme of the blurred image to be processed according to the target image information to obtain the target blur kernel estimation scheme, specifically comprises:

[0020] Obtaining a blur kernel estimation scheme for the blurred image to be processed, and inferring the illumination condition of the blurred image to be processed according to the time information to obtain the image illumination condition;

[0021] According to the image scene information, the blurred image to be processed is respectively identified and processed to obtain object information and background information, and fuzzy inference is performed according to the object information and the background information to obtain the target blur source;

[0022] Performing regional distance judgment on the blurred image to be processed according to the image depth information to obtain distance information of multiple different regions, and obtaining blur conditions of corresponding regions according to the distance information of all regions;

[0023] The blur kernel estimation scheme is optimized according to the image illumination condition, the target blur source and the blur conditions of all regions to obtain a target blur kernel estimation scheme.

[0024] Optionally, the image restoration method based on auxiliary information, wherein the restoration strategy of the blurred image to be processed is optimized according to the target image information to obtain the target restoration strategy, specifically comprising:

[0025] Acquire a restoration strategy for the blurred image to be processed, infer the exposure time of the blurred image to be processed according to the time information to obtain the image exposure time, and obtain a filter adjustment parameter of the blurred image to be processed according to the image scene information;

[0026] The restoration strategy is optimized according to the image exposure time, the filter adjustment parameters and the blurring conditions of all regions to obtain a target restoration strategy.

[0027] Optionally, the image restoration method based on auxiliary information, wherein the restoration process is performed on the blurred image to be processed according to the target blur kernel estimation scheme and the target restoration strategy to obtain a deblurred image, specifically comprising:

[0028] Deblurring the blurred image to be processed according to the target blur kernel estimation scheme and the target restoration strategy to obtain an initial clear image;

[0029] Performing error correction processing on the initial clear image to obtain a deblurred image, and performing quality evaluation on the deblurred image according to a preset image quality to obtain an evaluation result;

[0030] If the evaluation result is that the quality of the deblurred image meets the preset image quality, the deblurred image is output.

[0031] Optionally, the image restoration method based on auxiliary information, wherein the step of optimizing the quality of the deblurred image to obtain a target image, specifically comprises:

[0032] Performing a first optimization process on the quality of the deblurred image to obtain an optimized image, wherein the first optimization process includes contrast enhancement process, sharpening process and color correction process;

[0033] The optimized image is subjected to a second optimization process to obtain a target image, wherein the second optimization process includes noise suppression, artifact suppression and image edge refinement.

[0034] Optionally, in the auxiliary information-based image restoration method, the auxiliary information-based image restoration system comprises:

[0035] An image analysis module is used to obtain a fuzzy image to be processed, and perform information analysis on the fuzzy image to be processed to obtain target image information;

[0036] A scheme optimization module is used to optimize the blur kernel estimation scheme of the blurred image to be processed according to the target image information to obtain a target blur kernel estimation scheme, and optimize the restoration strategy of the blurred image to be processed according to the target image information to obtain a target restoration strategy;

[0037] The image restoration module is used to restore the blurred image to be processed according to the target blur kernel estimation scheme and the target restoration strategy to obtain a deblurred image, and optimize the quality of the deblurred image to obtain a target image.

[0038] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and an image repair program based on auxiliary information stored in the memory and executable on the processor, wherein the image repair program based on auxiliary information, when executed by the processor, implements the steps of the image repair method based on auxiliary information as described above.

[0039] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an image repair program based on auxiliary information, and when the image repair program based on auxiliary information is executed by a processor, the steps of the image repair method based on auxiliary information as described above are implemented.

[0040] In the present invention, a blurred image to be processed is obtained, and information analysis is performed on the blurred image to be processed to obtain target image information; the blur kernel estimation scheme of the blurred image to be processed is optimized according to the target image information to obtain a target blur kernel estimation scheme; the repair strategy of the blurred image to be processed is optimized according to the target image information to obtain a target repair strategy; the blurred image to be processed is repaired according to the target blur kernel estimation scheme and the target repair strategy to obtain a deblurred image, and the quality of the deblurred image is optimized to obtain a target image. The present invention can make the deblurring effect of the processed image better based on the auxiliary information related to the image, and improve the corresponding imaging quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of a preferred embodiment of the image restoration method based on auxiliary information in the present invention;

[0042] Figure 2 It is a schematic diagram of the principle of a preferred embodiment of the image restoration system based on auxiliary information in the present invention;

[0043] Figure 3 It is a schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0045] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0046] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0047] The image restoration method based on auxiliary information described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the image restoration method based on auxiliary information includes the following steps:

[0048] Step S10: Obtain a blurred image to be processed, and perform information analysis on the blurred image to be processed to obtain target image information.

[0049] Specifically, in an embodiment of the present invention, when a blurred image to be processed, i.e., a to-be-processed blurred image, is obtained, before the to-be-processed blurred image is processed, it is necessary to firstly perform information analysis on the to-be-processed blurred image to obtain corresponding target image information, wherein the target image information includes time information, image scene information, and image depth information. Specifically, first, it is necessary to obtain the time information of the to-be-processed blurred image, and obtain the corresponding image capture timestamp according to the metadata by reading the metadata of the to-be-processed blurred image, such as EXIF ​​(Exchangeable Image File) information, because the metadata usually records the specific time when the image is taken, and the time information is obtained according to the image capture timestamp; in addition, the lighting conditions (e.g., daytime, dusk, nighttime, etc.) when the to-be-processed blurred image is taken can also be inferred according to the image capture timestamp. If the to-be-processed blurred image is a multi-frame image, the light change trend can be analyzed by comparing images at different time points, and the dynamic changes of the scene (e.g., moving objects and light changes, etc.) can be inferred, which can be used for subsequent dynamic blur repair.

[0050] Secondly, it is necessary to obtain the image scene information of the blurred image to be processed. Specifically, a trained deep learning model (for example, a ResNet model, etc.) is used to perform scene classification on the blurred image to be processed so as to identify the scene type contained in the blurred image to be processed, such as indoor, outdoor, urban and natural, etc. Different areas in the blurred image to be processed can also be labeled and classified through a semantic segmentation model, such as sky areas, building areas, vegetation areas and road areas, etc. Then, feature extraction is performed on the blurred image to be processed according to the classification result to obtain scene features, extract key points in the blurred image to be processed, and analyze the distribution characteristics of these key points, so as to judge scene features such as depth changes and object shapes in the scene, and obtain the image scene information according to the scene features.

[0051] Finally, it is necessary to obtain the image depth information of the blurred image to be processed, wherein the image depth information refers to the distance information between each pixel point or area in the image and the camera. In simple terms, the image depth information describes the front-to-back position relationship of the object in the image relative to the camera, that is, the "far" relationship. Specifically, the image perspective of the blurred image to be processed is obtained, wherein the image perspective includes a single perspective and multiple perspectives; if the image perspective is a single perspective, a deep convolutional network is used to extract features of the blurred image to be processed to obtain target features, wherein the target features include texture features, illumination features and object size features, and regional depth inference is performed based on the target features to obtain scene depth information, which can help identify the degree of blur at different depth levels in the image; if the image perspective is multiple perspectives, a stereoscopic vision algorithm is used to calculate the parallax of the blurred image to be processed to obtain a position difference result, and depth calculation is performed based on the position difference result to obtain scene depth information; and the scene The depth information is smoothed to obtain image depth information to reduce the influence of noise and ensure the accuracy and stability of the image depth information. In addition, the image depth information can be combined with the brightness information of the blurred image to be processed for analysis to further optimize the depth estimation result, especially under the conditions of occlusion or complex lighting. More reliable image depth information can be provided through combined analysis. The role of the image depth information is that it can be used for scene understanding, that is, the image depth information can help understand the spatial relationship between objects in the image. For example, in street scene images, the image depth information can help distinguish between pedestrians in the foreground and buildings in the background, so that the blur of different depth areas can be processed in a targeted manner during image processing; it can also assist in deblurring, that is, the image depth information can be used to identify the degree of blur at different depths; for example, distant objects may be more blurred than nearby objects due to inaccurate focusing, and in the deblurring process, the image depth information can help optimize the estimation of the blur kernel, so that blur at different depth levels can be processed more effectively.

[0052] Step S20, optimizing the blur kernel estimation scheme of the blurred image to be processed according to the target image information to obtain a target blur kernel estimation scheme, optimizing the restoration strategy of the blurred image to be processed according to the target image information to obtain a target restoration strategy.

[0053] The step S20 comprises:

[0054] Step S21, obtaining a blur kernel estimation scheme of the blurred image to be processed, and inferring the illumination condition of the blurred image to be processed according to the time information to obtain the image illumination condition;

[0055] Step S22, performing recognition processing on the blurred image to be processed according to the image scene information to obtain object information and background information, and performing fuzzy inference according to the object information and the background information to obtain the target blur source;

[0056] Step S23, performing regional distance judgment on the blurred image to be processed according to the image depth information, obtaining distance information of multiple different regions, and obtaining blur conditions of corresponding regions according to the distance information of all regions;

[0057] Step S24, optimizing the blur kernel estimation scheme according to the image illumination condition, the target blur source and the blur conditions of all regions to obtain a target blur kernel estimation scheme;

[0058] Step S25, obtaining a restoration strategy for the blurred image to be processed, inferring the exposure time of the blurred image to be processed according to the time information to obtain the image exposure time, and obtaining a filter adjustment parameter of the blurred image to be processed according to the image scene information;

[0059] Step S26: Optimize the restoration strategy according to the image exposure time, the filter adjustment parameters and the blurring conditions of all regions to obtain a target restoration strategy.

[0060] Specifically, in an embodiment of the present invention, after obtaining the target image information of the blurred image to be processed, it is necessary to optimize the blur kernel estimation scheme and the restoration strategy of the blurred image to be processed. The optimization process of the blur kernel estimation scheme is specifically to obtain the blur kernel estimation scheme of the blurred image to be processed, and infer the lighting condition of the blurred image to be processed according to the time information to obtain the image lighting condition, because different lighting conditions may lead to different blur types. For example, night shooting may cause motion blur due to the long exposure time under low light conditions. The corresponding optimization measure is to adjust the type and size of the blur kernel according to this lighting condition. For example, A longer blur kernel may be required at night, so a longer exposure time is simulated; if the blurred image to be processed is a multi-frame image, the time information can help analyze the motion trajectory of the object or camera in the image, and the corresponding optimization measure is to determine the motion direction and speed through time series analysis, thereby optimizing the shape of the blur kernel (for example, the direction and length of the linear blur kernel); then, the blurred image to be processed is identified and processed according to the image scene information to obtain object information and background information, and blur inference is performed based on the object information and the background information to obtain the target blur source. The corresponding optimization measure is to adjust the local characteristics of the blur kernel according to the target blur source, for example, in the construction In the building scene, a directional blur kernel is used to simulate the blur of the building edge. The characteristics of different regions (for example, the sky, the ground, and the building, etc.) can also be determined by semantic segmentation of the blurred image to be processed, so as to help understand the distribution of blur. The corresponding optimization measure is to apply different blur kernel estimation strategies to different regions according to the results of semantic segmentation. For example, a larger blur kernel may be used for the sky region, while a finer blur kernel may be used for the building edge. Subsequently, the regional distance judgment is performed on the blurred image to be processed according to the image depth information to obtain the distance information of multiple different regions, and the distance information of the corresponding region is obtained according to the distance information of all regions. The corresponding optimization measure is to use a larger blur kernel for distant areas and a smaller blur kernel for nearby areas, so as to more accurately reflect the characteristics of depth of field blur; if the image perspective of the blurred image to be processed is multi-perspective, the image depth information can be used to adjust the stereoscopic characteristics of the blur kernel to ensure the consistency of the blur kernel under different perspectives, and the corresponding optimization measure is to dynamically adjust the three-dimensional shape of the blur kernel according to the depth information to make it better adapt to complex three-dimensional scenes; finally, the blur kernel estimation scheme is optimized according to the image lighting conditions, the target blur source and the blur conditions of all areas to obtain the target blur kernel estimation scheme.

[0061] The optimization process of the repair strategy is specifically as follows: obtaining the repair strategy of the blurred image to be processed, inferring the exposure time of the blurred image to be processed according to the time information, and obtaining the image exposure time. The weight parameters in the deblurring algorithm can be adjusted to make the repair strategy more in line with the actual shooting conditions. The corresponding optimization measure is to appropriately enhance the image smoothness under long exposure conditions to reduce the influence of noise on the deblurring effect. If there are light changes in multiple frames of images, the illumination balance processing can be adjusted according to the time information to reduce the interference of uneven illumination on the repair effect. The corresponding optimization measure is to perform illumination correction while deblurring to ensure the consistency of image brightness and color. Afterwards, the filter adjustment parameters of the blurred image to be processed are obtained according to the image scene information. For example, in natural scenes such as trees or grass, a smoother filter is used, and in buildings or road scenes, a sharper filter is used. Finally, the repair strategy is optimized according to the image exposure time, the filter adjustment parameters and the blurring conditions of all areas to obtain the target repair strategy.

[0062] Step S30: performing restoration processing on the blurred image to be processed according to the target blur kernel estimation scheme and the target restoration strategy to obtain a deblurred image, and optimizing the quality of the deblurred image to obtain a target image.

[0063] The step S30 comprises:

[0064] Step S31, deblurring the blurred image to be processed according to the target blur kernel estimation scheme and the target restoration strategy to obtain an initial clear image;

[0065] Step S32, performing error correction processing on the initial clear image to obtain a deblurred image, and performing quality evaluation on the deblurred image according to a preset image quality to obtain an evaluation result;

[0066] Step S33: if the evaluation result is that the quality of the deblurred image meets the preset image quality, outputting the deblurred image;

[0067] Step S34, performing a first optimization process on the quality of the deblurred image to obtain an optimized image, wherein the first optimization process includes contrast enhancement process, sharpening process and color correction process;

[0068] Step S35: performing a second optimization process on the optimized image to obtain a target image, wherein the second optimization process includes noise suppression, artifact suppression and image edge refinement.

[0069] Specifically, in an embodiment of the present invention, after obtaining the corresponding target blur kernel estimation scheme and the target repair strategy, it is necessary to perform repair processing on the blurred image to be processed according to the target blur kernel estimation scheme and the target repair strategy. The specific repair process is to deblur the blurred image to be processed according to the target blur kernel estimation scheme and the target repair strategy to obtain an initial clear image; the image after the deblurring process usually contains some residual blur effects or noise. In order to further refine the image quality, the initial clear image can be denoised; after the denoising process is completed, there may still be a certain residual error. In this case, the initial clear image can be error corrected by training the residual learning network to learn the residual and perform correction, so as to efficiently reduce the residual blur and artifacts, and obtain a deblurred image after the processing is completed. Afterwards, a variety of objective indicators, such as peak signal-to-noise ratio, structural similarity index and mean square error, are used to set corresponding image quality standards, and the preset image quality can be used to quantify the quality of the generated image, that is, the quality of the deblurred image is evaluated according to the preset image quality to obtain an evaluation result; if the evaluation result is that the quality of the deblurred image does not meet the preset image quality, it is necessary to further repair the blurred image to be processed; if the evaluation result is that the quality of the deblurred image meets the preset image quality, the deblurred image is output.

[0070] Subsequently, in order to further improve the quality of the deblurred image, especially in detail recovery and visual perception, in an embodiment of the present invention, the quality of the deblurred image is first optimized to obtain an optimized image, wherein the content of the first optimization includes contrast enhancement, sharpening and color correction; the contrast enhancement includes histogram equalization and adaptive histogram equalization, through which the global contrast of the deblurred image can be adjusted to make the brightness distribution of the deblurred image more uniform, thereby enhancing the detail performance of the deblurred image; on the basis of histogram equalization, adaptive histogram equalization performs equalization processing on the local area of ​​the deblurred image, which is particularly suitable for improving the local contrast when the image brightness is uneven. The sharpening process is to enhance the edge of the deblurred image by using the Laplace operator, which can highlight the details of the deblurred image, or to superimpose the high-frequency part of the deblurred image with the original image by nonlinear sharpening, so as to enhance the edge and detail area of ​​the image, thereby improving the overall clarity of the deblurred image. The color correction processing includes white balance adjustment and color saturation adjustment. Because the color of the image may be affected to a certain extent during the deblurring process, the white balance adjustment corrects the color temperature of the deblurred image to make the color of the deblurred image more natural; and improves the color saturation of the deblurred image, enhances the color expression of the deblurred image, and makes the deblurred image more vivid.

[0071] Afterwards, the optimized image is subjected to a second optimization process to obtain a target image, wherein the second optimization process includes noise suppression, artifact suppression and image edge refinement, wherein the noise suppression includes median filtering and bilateral filtering, wherein median filtering can eliminate noise that may be introduced during the deblurring process, especially salt noise and pepper noise, thereby ensuring the purity of the image, and bilateral filtering can retain the edge information of the image while removing noise. The artifact suppression includes residual learning and de-artifact filtering, wherein the residual learning network is used to eliminate artifacts and unnatural textures during the deblurring process, and a specially designed de-artifact filter is used to process block artifacts, stripe artifacts, etc. that may be generated during the deblurring process, thereby ensuring the improvement of the image visual quality. The image edge refinement is to identify and refine the edge portion of the optimized image through an edge detection algorithm, so that the optimized image is clearer and sharper in the edge area, or to enhance the edge of the image at different scales to ensure the expressiveness of the optimized image at various detail levels; in an embodiment of the present invention, the gamma value of the target image is adjusted to ensure that the brightness and contrast of the target image are in line with the visual habits of the human eye, thereby improving the user's viewing experience; the brightness range of the target image can also be stretched to ensure that the bright parts of the target image are brighter and the dark parts are darker, thereby enhancing the layering and three-dimensional sense of the target image. The present invention can not only effectively improve the visual effect of the image, but also make it reach higher standards in terms of clarity, contrast, color, details and noise control.

[0072] Furthermore, if Figure 2 As shown, based on the above-mentioned image restoration method based on auxiliary information, the present invention also provides an image restoration system based on auxiliary information, and the image restoration system based on auxiliary information includes:

[0073] The image analysis module 51 is used to obtain the blurred image to be processed, and perform information analysis on the blurred image to be processed to obtain target image information;

[0074] A scheme optimization module 52 is used to optimize the blur kernel estimation scheme of the blurred image to be processed according to the target image information to obtain a target blur kernel estimation scheme, and optimize the restoration strategy of the blurred image to be processed according to the target image information to obtain a target restoration strategy;

[0075] The image restoration module 53 is used to restore the blurred image to be processed according to the target blur kernel estimation scheme and the target restoration strategy to obtain a deblurred image, and optimize the quality of the deblurred image to obtain a target image.

[0076] Furthermore, if Figure 3As shown, based on the above-mentioned image restoration method based on auxiliary information, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 3 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0077] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed in the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, an image repair program 40 based on auxiliary information is stored on the memory 20, and the image repair program 40 based on auxiliary information can be executed by the processor 10, thereby realizing the image repair method based on auxiliary information in the present application.

[0078] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the image restoration method based on auxiliary information.

[0079] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0080] In one embodiment, when the processor 10 executes the image restoration program 40 based on the auxiliary information in the memory 20, the following steps are implemented:

[0081] Acquire a blurred image to be processed, and perform information analysis on the blurred image to be processed to obtain target image information;

[0082] Optimizing the blur kernel estimation scheme of the blurred image to be processed according to the target image information to obtain a target blur kernel estimation scheme, and optimizing the restoration strategy of the blurred image to be processed according to the target image information to obtain a target restoration strategy;

[0083] The blurred image to be processed is repaired according to the target blur kernel estimation scheme and the target repair strategy to obtain a deblurred image, and the quality of the deblurred image is optimized to obtain a target image.

[0084] Wherein, the target image information includes time information, image scene information and image depth information;

[0085] The step of obtaining a blurred image to be processed and performing information analysis on the blurred image to be processed to obtain target image information specifically includes:

[0086] Acquire a blurred image to be processed, read metadata of the blurred image to be processed, acquire a corresponding image capture timestamp according to the metadata, and obtain the time information according to the image capture timestamp;

[0087] Performing scene classification processing on the blurred image to be processed to obtain a classification result, performing feature extraction on the blurred image to be processed according to the classification result to obtain scene features, and obtaining the image scene information according to the scene features;

[0088] Depth estimation is performed on the blurred image to be processed to obtain scene depth information, and smoothing is performed on the scene depth information to obtain image depth information.

[0089] The step of performing depth estimation on the blurred image to be processed to obtain scene depth information specifically includes:

[0090] Acquire the image viewing angle of the blurred image to be processed, wherein the image viewing angle includes a single viewing angle and a multi-viewing angle;

[0091] If the image perspective is a single perspective, feature extraction is performed on the blurred image to be processed through a deep convolutional network to obtain target features, and regional depth inference is performed based on the target features to obtain scene depth information, wherein the target features include texture features, illumination features, and object size features;

[0092] If the image viewing angle is multi-view, the disparity calculation is performed on the blurred image to be processed by a stereoscopic vision algorithm to obtain a position difference result, and the depth calculation is performed according to the position difference result to obtain scene depth information.

[0093] The step of performing depth estimation on the blurred image to be processed to obtain scene depth information specifically includes:

[0094] Acquire the image viewing angle of the blurred image to be processed, wherein the image viewing angle includes a single viewing angle and a multi-viewing angle;

[0095] If the image perspective is a single perspective, feature extraction is performed on the blurred image to be processed through a deep convolutional network to obtain target features, and regional depth inference is performed based on the target features to obtain scene depth information, wherein the target features include texture features, illumination features, and object size features;

[0096] If the image viewing angle is multi-view, the disparity calculation is performed on the blurred image to be processed by a stereoscopic vision algorithm to obtain a position difference result, and the depth calculation is performed according to the position difference result to obtain scene depth information.

[0097] The step of optimizing the restoration strategy of the blurred image to be processed according to the target image information to obtain the target restoration strategy specifically includes:

[0098] Acquire a restoration strategy for the blurred image to be processed, infer the exposure time of the blurred image to be processed according to the time information to obtain the image exposure time, and obtain a filter adjustment parameter of the blurred image to be processed according to the image scene information;

[0099] The restoration strategy is optimized according to the image exposure time, the filter adjustment parameters and the blurring conditions of all regions to obtain a target restoration strategy.

[0100] The step of performing restoration processing on the blurred image to be processed according to the target blur kernel estimation scheme and the target restoration strategy to obtain a deblurred image specifically includes:

[0101] Deblurring the blurred image to be processed according to the target blur kernel estimation scheme and the target restoration strategy to obtain an initial clear image;

[0102] Performing error correction processing on the initial clear image to obtain a deblurred image, and performing quality evaluation on the deblurred image according to a preset image quality to obtain an evaluation result;

[0103] If the evaluation result is that the quality of the deblurred image meets the preset image quality, the deblurred image is output.

[0104] The step of optimizing the quality of the deblurred image to obtain a target image specifically includes:

[0105] Performing a first optimization process on the quality of the deblurred image to obtain an optimized image, wherein the first optimization process includes contrast enhancement process, sharpening process and color correction process;

[0106] The optimized image is subjected to a second optimization process to obtain a target image, wherein the second optimization process includes noise suppression, artifact suppression and image edge refinement.

[0107] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an image repair program based on auxiliary information, and when the image repair program based on auxiliary information is executed by a processor, the steps of the image repair method based on auxiliary information as described above are implemented.

[0108] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0109] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0110] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. An image restoration method based on auxiliary information, characterized in that: The image restoration method based on auxiliary information comprises: Acquire a blurred image to be processed, and perform information analysis on the blurred image to be processed to obtain target image information; Optimizing the blur kernel estimation scheme of the blurred image to be processed according to the target image information to obtain a target blur kernel estimation scheme, and optimizing the restoration strategy of the blurred image to be processed according to the target image information to obtain a target restoration strategy; The blurred image to be processed is repaired according to the target blur kernel estimation scheme and the target repair strategy to obtain a deblurred image, and the quality of the deblurred image is optimized to obtain a target image.

2. The image restoration method based on auxiliary information according to claim 1, characterized in that: The target image information includes time information, image scene information and image depth information; The step of obtaining a blurred image to be processed and performing information analysis on the blurred image to be processed to obtain target image information specifically includes: Acquire a blurred image to be processed, read metadata of the blurred image to be processed, acquire a corresponding image capture timestamp according to the metadata, and obtain the time information according to the image capture timestamp; Performing scene classification processing on the blurred image to be processed to obtain a classification result, performing feature extraction on the blurred image to be processed according to the classification result to obtain scene features, and obtaining the image scene information according to the scene features; Depth estimation is performed on the blurred image to be processed to obtain scene depth information, and smoothing is performed on the scene depth information to obtain image depth information.

3. The image restoration method based on auxiliary information according to claim 2, characterized in that: The performing depth estimation on the blurred image to be processed to obtain scene depth information specifically includes: Acquire the image viewing angle of the blurred image to be processed, wherein the image viewing angle includes a single viewing angle and a multi-viewing angle; If the image perspective is a single perspective, feature extraction is performed on the blurred image to be processed through a deep convolutional network to obtain target features, and regional depth inference is performed based on the target features to obtain scene depth information, wherein the target features include texture features, illumination features, and object size features; If the image viewing angle is multi-view, the disparity calculation is performed on the blurred image to be processed by a stereoscopic vision algorithm to obtain a position difference result, and the depth calculation is performed according to the position difference result to obtain scene depth information.

4. The image restoration method based on auxiliary information according to claim 2, characterized in that: The optimizing process of the blur kernel estimation scheme of the blurred image to be processed according to the target image information to obtain the target blur kernel estimation scheme specifically includes: Obtaining a blur kernel estimation scheme for the blurred image to be processed, and inferring the illumination condition of the blurred image to be processed according to the time information to obtain the image illumination condition; According to the image scene information, the blurred image to be processed is respectively identified and processed to obtain object information and background information, and fuzzy inference is performed according to the object information and the background information to obtain the target blur source; Performing regional distance judgment on the blurred image to be processed according to the image depth information to obtain distance information of multiple different regions, and obtaining blur conditions of corresponding regions according to the distance information of all regions; The blur kernel estimation scheme is optimized according to the image illumination condition, the target blur source and the blur conditions of all regions to obtain a target blur kernel estimation scheme.

5. The image restoration method based on auxiliary information according to claim 4, characterized in that: The optimizing process of the restoration strategy of the blurred image to be processed according to the target image information to obtain the target restoration strategy specifically includes: Acquire a restoration strategy for the blurred image to be processed, infer the exposure time of the blurred image to be processed according to the time information to obtain the image exposure time, and obtain a filter adjustment parameter of the blurred image to be processed according to the image scene information; The restoration strategy is optimized according to the image exposure time, the filter adjustment parameters and the blurring conditions of all regions to obtain a target restoration strategy.

6. The image restoration method based on auxiliary information according to claim 1, characterized in that: The repairing process of the blurred image to be processed according to the target blur kernel estimation scheme and the target repair strategy to obtain a deblurred image specifically includes: Deblurring the blurred image to be processed according to the target blur kernel estimation scheme and the target restoration strategy to obtain an initial clear image; Performing error correction processing on the initial clear image to obtain a deblurred image, and performing quality evaluation on the deblurred image according to a preset image quality to obtain an evaluation result; If the evaluation result is that the quality of the deblurred image meets the preset image quality, the deblurred image is output.

7. The image restoration method based on auxiliary information according to claim 1, characterized in that: The optimizing process of the quality of the deblurred image to obtain the target image specifically includes: Performing a first optimization process on the quality of the deblurred image to obtain an optimized image, wherein the first optimization process includes contrast enhancement process, sharpening process and color correction process; The optimized image is subjected to a second optimization process to obtain a target image, wherein the second optimization process includes noise suppression, artifact suppression and image edge refinement.

8. An image restoration system based on auxiliary information, characterized in that: The image restoration system based on auxiliary information comprises: An image analysis module is used to obtain a fuzzy image to be processed, and perform information analysis on the fuzzy image to be processed to obtain target image information; A scheme optimization module is used to optimize the blur kernel estimation scheme of the blurred image to be processed according to the target image information to obtain a target blur kernel estimation scheme, and optimize the restoration strategy of the blurred image to be processed according to the target image information to obtain a target restoration strategy; The image restoration module is used to restore the blurred image to be processed according to the target blur kernel estimation scheme and the target restoration strategy to obtain a deblurred image, and optimize the quality of the deblurred image to obtain a target image.

9. A terminal, characterized in that: The terminal includes a memory, a processor, and a program stored in the memory and executable on the processor. When the program is executed by the processor, the steps of the image restoration method based on auxiliary information as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, wherein the computer-readable storage medium stores an image restoration program based on auxiliary information, and when the image restoration program based on auxiliary information is executed by a processor, the steps of the image restoration method based on auxiliary information as described in any one of claims 1-7 are implemented.

Images