Image deblurring processing method, device and equipment and computer readable storage medium

Abstract

The application provides an image deblurring processing method, relates to the field of Internet of Vehicles and the field of artificial intelligence technology, and comprises the following steps: performing cascade coding processing on a first image based on M scales, and sequentially obtaining M scale coding images; performing cascade refining processing on the second scale coding image to the M scale coding image based on N scales, and sequentially obtaining N scale refining images; performing cascade decoding processing on the second scale coding image to the M scale coding image and the N scale refining image based on N scales, and sequentially obtaining N scale decoding images; and performing image prediction processing on the first scale coding image, the N scale decoding image in the N scale decoding image of the N scale decoding image, and the N scale refining image in the N scale refining image of the N scale refining image, so as to obtain a second image with higher definition than the first image. Through the application, the deblurring effect of multimedia images can be significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of Internet of Vehicles and the field of artificial intelligence, and in particular to an image deblurring processing method and device, an electronic device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] Artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is the design principle and implementation method of various intelligent machines, so that the machine has the functions of perception, reasoning and decision-making.

[0003] In the prior art image deblurring processing method, due to the need to rely on manual features and experience knowledge, the deblurring effect of the image is not ideal. For how to improve the deblurring effect of the image and obtain a deblurred image with higher clarity, the related art has no effective solution. SUMMARY

[0004] The embodiments of the present application provide an image deblurring processing method, device, equipment, computer readable storage medium and computer program product, which can improve the deblurring effect of the image and obtain a deblurred image with higher clarity.

[0005] The technical solution of the embodiments of the present application is as follows:

[0006] The embodiments of the present application provide an image deblurring processing method, which comprises:

[0007] performing encoding processing of M scales in cascade based on the first image to obtain encoding images of M scales in sequence; wherein M is an integer greater than or equal to 2, and the M scales are sequentially decreased;

[0008] performing refining processing of N scales in cascade based on the encoding images of the second scale to the Mth scale of the M scales to obtain refining images of N scales in sequence; wherein N=M-1, and the N scales are obtained by sequentially arranging the second scale to the Mth scale of the M scales;

[0009] performing decoding processing of N scales in cascade based on the encoding images of the second scale to the Mth scale of the M scales and the refining images of the N scales to obtain decoding images of N scales in sequence;

[0010] perform image prediction processing based on the first scale of the M scales of encoded images, the Nth scale of the N scales of decoded images, and the Nth scale of the N scales of extracted images, to obtain a second image with higher definition than the first image.

[0011] The embodiment of the present application provides an image deblurring processing device, comprising:

[0012] The encoding module is configured to perform cascaded encoding processing on the first image based on M scales, to sequentially obtain M scales of encoded images; wherein M is an integer greater than or equal to 2, and the M scales are sequentially decreased;

[0013] The extraction module is configured to perform cascaded extraction processing on the second scale to the Mth scale of the M scales of encoded images, to sequentially obtain N scales of extracted images; wherein N=M-1, and the N scales are obtained by sequentially increasing the second scale to the Mth scale of the M scales;

[0014] The decoding module is configured to perform cascaded decoding processing on the second scale to the Mth scale of the M scales of encoded images and the N scales of extracted images, to sequentially obtain N scales of decoded images;

[0015] The prediction module is configured to perform image prediction processing based on the first scale of the M scales of encoded images, the Nth scale of the N scales of decoded images, and the Nth scale of the N scales of extracted images, to obtain a second image with higher definition than the first image.

[0016] In the above scheme, the extraction module is further configured to perform the following processing by iteration n: performing the n th scale of extraction processing based on the n th scale of extraction input image, to obtain the n th scale of extracted image, and n is in the range of 1≤n≤N; wherein when n=1, the n th scale of extraction input image is the Mth scale of encoded image of the M scales of encoded images; wherein when 2≤n≤N, the n th scale of extraction input image is formed by the following manner after the (n-1) th scale of decoding processing: performing fusion processing based on the (n-1) th scale of decoded image, the M-(n-1) th scale of encoded image of the M scales of encoded images, and the (n-1) th scale of extracted image, to obtain the n th scale of extraction input image.

[0017] In the scheme, the refining module is further configured to perform n-scale bottleneck convolution processing on the n-scale refined input image to obtain an n-scale first bottleneck convolution image; and perform n-scale prediction convolution processing on the n-scale first bottleneck convolution image to obtain an n-scale refined image.

[0018] In the scheme, the refining module is further configured to perform pixel-by-pixel addition processing on the (n-1)-scale decoded image and an M-(n-1)-scale encoded image in the M-scale encoded images to obtain an (n-1)-scale added image; perform residual convolution processing on the (n-1)-scale added image to obtain an (n-1)-scale residual convolution image; perform up-sampling processing on the (n-1)-scale refined image, and perform bottleneck convolution processing on the obtained (n-1)-scale up-sampled image to obtain an (n-1)-scale second bottleneck convolution image; and perform channel-based splicing processing on the (n-1)-scale residual convolution image and the (n-1)-scale second bottleneck convolution image, and take the obtained (n-1)-scale spliced image as the n-scale refined input image.

[0019] In the scheme, the decoding module is further configured to perform the following processing iteratively n times: performing n-scale decoding processing based on an n-scale decoding input image to obtain an n-scale decoded image, where n ranges from 1 to N; when n is 1, the n-scale decoding input image is an M-scale encoded image in the M-scale encoded images; when 2≤n≤N, the n-scale decoding input image is formed after (n-1)-scale refining processing in the following manner: performing fusion processing based on an (n-1)-scale decoded image, an M-(n-1)-scale encoded image in the M-scale encoded images, and an (n-1)-scale refined image to obtain the n-scale decoding input image.

[0020] In the scheme, the decoding module is further configured to perform pixel-by-pixel addition processing on the (n-1)-scale decoded image and an M-(n-1)-scale encoded image in the M-scale encoded images, perform residual convolution processing on the obtained (n-1)-scale added image to obtain an (n-1)-scale residual convolution image, perform channel-based splicing processing on the (n-1)-scale residual convolution image and the (n-1)-scale refined image, and take the obtained (n-1)-scale spliced image as the n-scale decoding input image.

[0021] In the scheme, the prediction module is further configured to perform fusion processing on the first scale of the M scale encoded images, the N scale of the N scale decoded images, and the N scale of the N scale extracted images to obtain an N scale fused image; and perform prediction convolution processing based on the N scale fused image to obtain a second image with higher definition than the first image.

[0022] In the scheme, the prediction module is further configured to perform pixel-by-pixel addition processing on the first scale of the M scale encoded images and the N scale of the N scale decoded images to obtain an N scale addition image; perform residual convolution processing on the N scale addition image to obtain an N scale residual convolution image; and perform channel-based splicing processing on the N scale residual convolution image and the N scale of the N scale extracted images to obtain an N scale splicing image as the N scale fused image.

[0023] In the scheme, the prediction module is further configured to perform up-sampling processing on the N scale of the N scale extracted images to obtain an N scale up-sampling image; perform N scale bottleneck convolution processing on the N scale up-sampling image to obtain an N scale second bottleneck convolution image; and perform channel-based splicing processing on the N scale residual convolution image and the N scale second bottleneck convolution image to obtain the N scale splicing image.

[0024] In the scheme, the decoding module is further configured to perform the following processing iteratively m times: performing m scale encoding processing based on an m scale encoding input image to obtain an m scale encoded image, where m is in the range of 1≤m≤M; when m=1, the m scale encoding input image is the first image; when 2≤m≤M, the m scale encoding input image is an (m-1) scale encoded image.

[0025] In the scheme, the decoding module is further configured to perform general convolution processing on the m scale encoding input image to obtain an m scale general convolution image; and perform at least one residual convolution processing on the m scale general convolution image to obtain the m scale encoded image.

[0026] In the scheme, the image deblurring processing apparatus further comprises a training module configured to train the encoding network, the decoding network and the refining network by: calling the encoding network, the decoding network and the refining network to perform deblurring processing based on a blurred image sample, to obtain a predicted clear image with a clarity greater than the blurred image sample and N-scale predicted refined image samples; calling a first loss function to perform prediction processing based on the predicted clear image and a real clear image sample corresponding to the blurred image sample, to obtain a first loss value; performing the following processing for each of the N-scale predicted refined image samples: calling the first loss function to perform prediction processing based on an nth-scale predicted refined image sample and an nth-scale real refined image sample, to obtain a second loss value, where n is in a range of 1≤n≤N; performing weighted sum processing on the second loss values corresponding to the N-scale predicted refined image samples, to obtain a third loss value; performing weighted sum processing on the first loss value and the third loss value, to obtain a joint loss value; and updating the encoding network, the decoding network and the refining network based on the joint loss value.

[0027] The embodiment of the present application provides an image deblurring processing apparatus, comprising:

[0028] a memory configured to store executable instructions;

[0029] a processor configured to execute the executable instructions stored in the memory, so as to implement the image deblurring processing method provided by the embodiment of the present application.

[0030] The embodiment of the present application provides a computer readable storage medium, which stores executable instructions, and is used to cause a processor to execute the image deblurring processing method provided by the embodiment of the present application.

[0031] The embodiment of the present application provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the image deblurring processing method provided by the embodiment of the present application.

[0032] The embodiment of the present application has the following beneficial effects:

[0033] By incorporating the N scale refined images and the 2nd scale to the Mth scale encoded images of the M scales into the decoding process, the N scale refined images can assist the decoding process of the image at the N scales, so that the first image can be recovered from low resolution to high resolution at each scale, the image deblurring effect can be significantly improved, and a clearer deblurring image, i.e., the second image, can be obtained. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a structural schematic diagram of an image deblurring processing system architecture provided by an embodiment of the present application;

[0035] Figure 2 is a structural schematic diagram of an image deblurring processing apparatus provided by an embodiment of the present application;

[0036] Figures 3A-3D is a flow schematic diagram of an image deblurring processing method provided by an embodiment of the present application;

[0037] Figures 4A-4D is a principle schematic diagram of image deblurring processing provided by an embodiment of the present application;

[0038] Figures 5A-5C is an effect schematic diagram of an image deblurring processing method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application will be further described in detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application, all other embodiments obtained by those skilled in the art without creative labor shall belong to the scope of protection of the present application.

[0040] In the following description, "some embodiments" are related to a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subset of all possible embodiments, and can be combined with each other without conflict.

[0041] In the following description, the terms "first\second\third" are only to distinguish similar objects, and do not represent a specific order of the objects, and it can be understood that "first\second\third" can be interchanged with a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to be limiting of this application.

[0043] It can be understood that, in the embodiments of the present application, data related to user information and the like are involved, and when the embodiments of the present application are applied to specific products or technologies, user permission or consent needs to be obtained, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of countries and regions.

[0044] 1) Image deblurring: also known as image deblurring processing, refers to a technical means of enhancing the definition of a blurred image, that is, a process of deblurring a blurred image to obtain a clear image corresponding to the blurred image.

[0045] 2) Cascade: in the process of image encoding processing, cascade refers to the dependency relationship between the encoding processing of the previous scale and the encoding processing of the next scale, that is, the encoding processing of the previous scale depends on the encoding processing of the next scale. In the process of image decoding processing, cascade refers to the dependency relationship between the decoding processing of the previous scale and the decoding processing of the next scale, that is, the decoding processing of the previous scale depends on the decoding processing of the next scale.

[0046] 3) Clarity: refers to the degree of clarity of each detail and its boundary on the image.

[0047] 4) Up sampling: also known as image interpolation, usually refers to interpolating between pixels of an image, that is, inserting new elements between pixel points based on the original image pixels using a suitable interpolation algorithm.

[0048] 5) Pixel-wise addition processing: a processing process of adding each corresponding pixel point in two two-dimensional matrices of images with the same dimensions and corresponding equal rows and columns.

[0049] 6) Residual convolution processing: a process of processing an image through a residual convolution structure. The residual convolution structure has the characteristics of easy optimization and can improve accuracy by increasing the depth. The residual block inside uses a jump connection to alleviate the problem of gradient disappearance caused by increasing the depth in a deep neural network.

[0050] 7) Bottleneck Convolution processing: a special residual convolution processing, the bottleneck convolution processing is a processing mode for adjusting the number of image channels through a bottleneck convolution structure with a convolution kernel size of 1*1, wherein the bottleneck convolution structure is a special residual convolution structure, the number of input image and output image channels of the bottleneck convolution structure is large, the shape of the bottleneck convolution structure is similar to the shape of a bottleneck or funnel, which is narrow at the top and wide at the bottom or wide at the top and narrow at the bottom, thereby changing the number of output image channels.

[0051] 8) General convolution processing: a process of processing an image through a general convolution structure, the general convolution structure is a kind of feedforward neural network containing convolution calculation and having a deep structure, and the size of the convolution kernel of the general convolution structure is different from that of the bottleneck convolution structure.

[0052] 9) Convolutional Neural Networks (CNN): a kind of feedforward neural network containing convolution calculation and having a deep structure, which is one of the representative algorithms of deep learning, the convolutional neural network has a representation learning ability, and the convolutional neural network can not only be used for image processing but also be used for image understanding.

[0053] 10) Convolutional Layer: a basic operator in convolutional neural network, which can extract high-dimensional features of images in the field of computer vision.

[0054] 11) Deconvolutional Layer: a basic operator in convolutional neural network, which can generate low-dimensional and high-dimensional resolution features according to high-dimensional and low-dimensional resolution features in the field of computer vision.

[0055] In the implementation process of the embodiments of the present application, the applicant finds that the related art has the following problems:

[0056] For image deblurring processing, the deblurring processing of the blurred image is often performed under the condition that the blur kernel is unknown. In the related art, more manual features and experience knowledge are often needed to guess the blur kernel, and then the image deblurring is realized. In the related art, since manual features and experience knowledge are needed, the deblurring effect of the image is not ideal.

[0057] The embodiment of the present application provides an image deblurring processing method, device, equipment, computer readable storage medium and computer program product, which can improve the deblurring effect of an image, obtain a deblurring image with higher clarity, and the following describes an exemplary application of the image deblurring processing device provided by the embodiment of the present application. The device provided by the embodiment of the present application can be implemented as a notebook computer, a tablet computer, a desktop computer, a set-top box, a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated message device, a portable game device), a vehicle-mounted terminal and various types of user terminals, and can also be implemented as a server.

[0058] Referring to Figure 1 , Figure 1 is an architecture schematic diagram of the image deblurring processing system 100 provided by the embodiment of the present application. The terminal (exemplarily shown as the terminal 400) is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0059] The terminal 400 is used for a user to use the client 410 to display a blurred image and a clear image obtained by deblurring processing in the graphical interface 410-1 (exemplarily shown as the graphical interface 410-1). For example, a blurred image in an electronic album APP and a clear image obtained by deblurring processing on the blurred image in the electronic album APP are displayed in the graphical interface 410-1. For another example, the terminal 400 (for example, a vehicle-mounted terminal) runs a navigation application APP. When a map image in the navigation application APP is a blurred image, the blurred image can be deblurred, and a clear map image is displayed in the graphical interface 410-1. The terminal 400 and the server 200 are connected to each other through a wired or wireless network.

[0060] In some embodiments, the server 200 can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms. The terminal 400 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the embodiment of the present application.

[0061] In some embodiments, the image deblurring processing method provided by the embodiment of the present application can also be applied to an image deblurring processing scene related to a vehicle network service (for example, navigation, a reverse image auxiliary parking, etc.).

[0062] For example, when the terminal 400 (such as a vehicle terminal) is used for map navigation, generally, there is a high requirement for real-time performance of the map navigation. According to the operation of the user on the graphical interface 410-1, the terminal 400 sends a map acquisition request to the server 200 through the network 300. The server 200 sends a map image to the terminal 400 in response to the map acquisition request. If part of the data of the map image is lost, it will cause the terminal 400 to display a blurred map image. The terminal 400 performs deblurring processing on the blurred map image by using the image deblurring processing method provided in the embodiments of the present application, and displays the clear map image after deblurring processing in the graphical interface 410-1.

[0063] For another example, when the terminal 400 (such as a vehicle terminal) is used for reverse image assisted parking, when the terminal 400 receives a reverse image captured by a camera at the back of the vehicle, the received reverse image is often a blurred image due to the shaking of the vehicle during movement. In this case, after receiving the reverse image captured by the camera at the back of the vehicle, the terminal 400 performs deblurring processing on the blurred reverse image by using the image deblurring processing method provided in the embodiments of the present application, and displays the clear reverse image after deblurring processing in the graphical interface 410-1.

[0064] In some embodiments, the terminal 400 performs deblurring processing on the blurred image according to the operation of the user on the graphical interface 410-1, and displays the clear image after deblurring processing in the graphical interface 410-1.

[0065] In some embodiments, the terminal 400 performs deblurring processing on the blurred image according to the operation of the user on the graphical interface 410-1, and displays the clear image after deblurring processing in the graphical interface 410-1.

[0066] The electronic device for performing image deblurring processing is taken as an example of the server, referring to Figure 2 , Figure 2 is a structural schematic diagram of the server 200 provided by the embodiments of the present application for performing image deblurring processing, Figure 2 The server 200 shown in FIG. 15 includes at least one processor 410, a memory 450, and at least one network interface 420. The various components in the server 200 are coupled together by a bus system 440. It can be understood that the bus system 440 is used to realize the connection and communication between the components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the purpose of clear illustration, only the data bus is shown in the figure.Figure 2 Various buses are shown as bus system 440.

[0067] The processor 410 can be an integrated circuit chip that has a processing capability of signals, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc., and the general-purpose processor can be a microprocessor or any conventional processor.

[0068] The memory 450 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical drives, etc. The memory 450 optionally includes one or more storage devices remotely located from the processor 410.

[0069] The memory 450 includes volatile memory or non-volatile memory, and can also include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), and the volatile memory can be random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0070] In some embodiments, the memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, which are exemplarily illustrated below.

[0071] The operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0072] The network communication module 452 is used to communicate with other computing devices via one or more (wired or wireless) network interfaces 420, and exemplary network interfaces 420 include Bluetooth, wireless fidelity (WiFi), and universal serial bus (USB), etc.

[0073] In some embodiments, the image deblurring processing apparatus provided by the embodiments of the present application can be realized in a software manner, Figure 2An image deblurring processing apparatus 455 stored in the memory 450 is shown, which can be software in the form of programs and plug-ins, etc., including the following software modules: an encoding module 4551, an extraction module 4552, a decoding module 4553, and a prediction module 4554, which are logical, and thus can be combined or further split according to the implemented functions. The functions of the various modules will be described below.

[0074] In some embodiments, the image deblurring processing apparatus provided by the embodiments of the present application can be implemented in a hardware manner. As an example, the image deblurring processing apparatus provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image deblurring processing method provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can use one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic elements.

[0075] In some embodiments, the terminal or server can implement the image deblurring processing method provided by the embodiments of the present application by running a computer program. For example, the computer program can be a native program in the operating system (e.g., a dedicated image deblurring program) or a software module, for example, a deblurring module that can be embedded in any program (such as an instant messaging client, a photo album program, an electronic map client, a navigation client); for example, it can be a native (Native) application (APP), i.e., a program that needs to be installed in the operating system to run. In summary, the above computer program can be any form of application, module or plug-in.

[0076] The image deblurring processing method provided by the embodiments of the present application will be described in conjunction with exemplary applications and implementations of the terminal provided by the embodiments of the present application.

[0077] The image deblurring processing method provided by the embodiments of the present application is a blind image deblurring processing task in the field of computer vision. Generally, the blind image deblurring processing task can be represented as:

[0078] B = k x I + n (1)

[0079] Wherein, B represents a blurred image (i.e. the first image described below), k represents a blur kernel, I represents a clear image (i.e. the second image described below), and n represents noise.

[0080] That is, the blind image deblurring task is a process of calculating the clear image I under the condition that the blurred image B is known, and the blur kernel k and the noise n are unknown.

[0081] In some embodiments, referring to Figure 4A The image deblurring processing method provided in the embodiments of the present application can be implemented through an image processing network. The image processing network comprises an encoding network, a decoding network and an extraction network. The encoding network is used to extract high-dimensional and low-resolution features from an input blurred image (i.e. the first image described below). The decoding network is used to generate low-dimensional and high-resolution features layer by layer from the high-dimensional features extracted by the encoding network. At the same time, the low-dimensional features in the encoding network are directly short-circuited to the corresponding decoding network, so that the original information is not lost in the recovery process. The extraction network is used to assist the decoding network in deblurring processing.

[0082] Referring to Figure 3A , Figure 3A is a flowchart of the image deblurring processing method provided in the embodiments of the present application, which will be described in combination with steps 101 to 104 shown in Figure 3A .

[0083] In step 101, a cascaded encoding processing of M scales is performed based on a first image, and M encoded images of M scales are obtained in sequence; wherein M is an integer greater than or equal to 2, and the M scales decrease one by one.

[0084] In some embodiments, the scale of each scale encoded image is positively correlated with the resolution ratio of the input image and the output image at the scale, wherein the expression of the resolution ratio of the input image and the output image is:

[0085] stride = S x width (2)

[0086] Wherein, stride represents the resolution ratio of the input image and the output image, S represents the number of bytes occupied by each pixel, and width represents the number of pixels in a row.

[0087] For example, when stride = 1, the scale of the input image is equal to the scale of the output image; when stride = 2, the scale of the output image is equal to the scale of the input image divided by 2.

[0088] As an example, referring to Figure 4B , Figure 4B is a schematic diagram of the principle of the image deblurring processing provided in the embodiments of the present application, which will be described in combination with Figure 4BWhen M=4, the encoding processing based on the first image is performed on four scales in cascade, and four scale encoded images (i.e., a first scale encoded image 1, a second scale encoded image 2, a third scale encoded image 3, and a fourth scale encoded image 4) are sequentially obtained, and the four scales are sequentially reduced (the scale of the first scale encoded image 1 is greater than the scale of the second scale encoded image 2, the scale of the second scale encoded image 2 is greater than the scale of the third scale encoded image 3, and the scale of the third scale encoded image 3 is greater than the scale of the fourth scale encoded image 4).

[0089] In some embodiments, the encoding processing in step 101 can be implemented by performing the following processing iteratively for m: performing m-th scale encoding processing based on an m-th scale encoded input image to obtain an m-th scale encoded image, where m is in the range of 1≤m≤M; when m=1, the m-th scale encoded input image is the first image; and when 2≤m≤M, the m-th scale encoded input image is an (m-1)-th scale encoded image.

[0090] As an example, see Figure 4B , in Figure 4B , M=4, and m is in the range of 1≤m≤4. The following describes Figure 4B the cases of m=1 and m=2 in detail:

[0091] As an example, see Figure 4B , when m=1, perform first scale encoding processing based on a first scale encoded input image (the first scale encoded input image is the first image) to obtain a first scale encoded image (i.e., the first scale encoded image 1).

[0092] As an example, see Figure 4B , when m=2, perform second scale encoding processing based on a second scale encoded input image (the second scale encoded input image is the first scale encoded image, i.e., the first scale encoded image 1) to obtain a second scale encoded image (i.e., the second scale encoded image 2).

[0093] As an example, see Figure 4B , in some embodiments, the encoding processing includes general convolution processing and at least one residual convolution processing. The m-th scale encoding processing based on the m-th scale encoded input image to obtain the m-th scale encoded image can be implemented by performing general convolution processing on the m-th scale encoded input image to obtain an m-th scale general convolution image, and performing at least one residual convolution processing on the m-th scale general convolution image to obtain the m-th scale encoded image.

[0094] In some embodiments, the general convolution processing is a process of processing an image through a general convolution structure, which is a kind of feedforward neural network containing convolution calculation and having a deep structure, and the convolution kernel size of the general convolution structure is different from that of the bottleneck convolution structure. The residual convolution processing is a process of processing an image through a residual convolution structure, which is characterized by easy optimization and the ability to improve accuracy by increasing the depth, and the residual block inside uses a jump connection to alleviate the problem of gradient disappearance caused by increasing the depth in the deep neural network.

[0095] As an example, Figure 4B The general convolution processing shown can be a convolution processing implemented through a general convolution structure with a convolution kernel size of 5*5, and the residual convolution processing can be a convolution processing implemented through a residual convolution structure with a convolution kernel size of 3*3.

[0096] As an example, see Figure 4B For the case of m=1, the first scale encoding input image (first image) is processed by general convolution processing to obtain a first scale general convolution image; the first scale general convolution image is processed by at least one (three times in the example) residual convolution processing to obtain a first scale encoding image (i.e., the first scale encoding image 1). Figure 4B

[0097] In step 102, N scale refining processing is performed based on the concatenation of the second scale to the Mth scale of the M scale encoding images, and N scale refined images are obtained in turn; wherein N=M-1, and the N scales are obtained by arranging the second scale to the Mth scale of the M scales in ascending order.

[0098] For example, when M=2, N=1; when M=3, N=2; when M=4, N=3, i.e., the N scales are obtained by arranging the second scale to the fourth scale of the M scales in ascending order.

[0099] ​In some embodiments, the refinement processing at N scales and the decoding processing at N scales are performed synchronously at the same scale, and the decoded image at the previous scale is used as auxiliary refinement information for the refinement processing at the next scale. The refinement processing includes predictive convolution processing and bottleneck convolution processing. Predictive convolution processing is a process that implements prediction functionality through a predictive convolution structure. Bottleneck convolution processing is a special type of residual convolution processing. It is a processing method that adjusts the number of image channels through a bottleneck convolution structure. The bottleneck convolution structure is a special residual convolution structure where the number of channels in the input and output images differs significantly. The shape of the bottleneck convolution structure resembles a bottleneck or funnel, narrower at the top and wider at the bottom, or vice versa, thereby changing the number of channels in the output image.

[0100] In some embodiments, bottleneck convolution processing is implemented using a bottleneck convolution structure with a kernel size of 1*1, while the general convolution processing described above is implemented using a general convolution structure with a kernel size of 5*5. That is, the kernel sizes of the bottleneck convolution structure corresponding to bottleneck convolution processing and the general convolution structure corresponding to general convolution processing are different, and the kernel size of the bottleneck convolution structure corresponding to bottleneck convolution processing is smaller than the kernel size of the general convolution structure corresponding to general convolution processing. In other words, the kernel size of the general convolution structure is greater than 1*1.

[0101] As an example, see Figure 4B , Figure 4B In this context, M=4, N=3, i.e., N=M-1, and a concatenated three-scale refinement process is performed on the coded images at the second to fourth scales (second-scale coded image 2, third-scale coded image 3, and fourth-scale coded image 4) of the coded images at four scales (see [link]). Figure 4B (By "short-circuiting"), we can obtain refined images at three scales in sequence.

[0102] In some embodiments, the refinement process in step 102 above can be implemented in the following way: iterate n and perform the following processing: based on the refined input image at the nth scale, perform refinement processing at the nth scale to obtain the refined image at the nth scale, where the value of n is 1≤n≤N; where, when n=1, the refined input image at the nth scale is the encoded image at the Mth scale among the encoded images at M scales; where, when 2≤n≤N, the refined input image at the nth scale is formed after the decoding processing at the (n-1)th scale and before the decoding processing at the nth scale by the following way: perform fusion processing based on the decoded image at the (n-1)th scale, the encoded image at the M-(n-1)th scale among the encoded images at M scales, and the refined image at the (n-1)th scale to obtain the refined input image at the nth scale.

[0103] In some embodiments, the fusion processing can include splicing processing, pixel-by-pixel addition processing, residual convolution processing, and up-sampling processing, wherein the splicing processing can be a processing procedure of splicing images in the channel dimension, for example, splicing processing of an image with a channel number of 10 and an image with a channel number of 10 can obtain an image with a channel number of 20; the pixel-by-pixel addition processing can be a processing procedure of corresponding addition of pixel values of each pixel in an image, that is, a processing procedure of adding each corresponding pixel point in two-dimensional matrices of two images with the same dimension and corresponding equal rows and columns.

[0104] As an example, refer to Figure 4B , Figure 4B N = 3 in the above equation, that is, the value range of n is 1 ≤ n ≤ 3, and the cases of iterations n = 1 and n = 2 are schematically described below. Figure 4B

[0105] Refer to Figure 3B When n = 1, based on the first scale refining input image, the first scale refining processing is performed to obtain the first scale refining image 5, and the first scale refining input image is the fourth scale encoded image (i.e., the fourth scale encoded image 4 mentioned above) in the four scale encoded images. Here, the first scale refining input image can be used as auxiliary refining information for the first scale refining processing.

[0106] Refer to Figure 3B When n = 2, based on the second scale refining input image, the second scale refining processing is performed to obtain the second scale refining image 6, wherein the second scale refining input image is formed after the first scale decoding processing by the following manner: based on the first scale decoded image, the third scale encoded image (i.e., the third scale encoded image 3 mentioned above) in the four scale encoded images, and the first scale refining image 5, the fusion processing is performed to obtain the second scale refining input image.

[0107] ​In some embodiments, when 2≤n≤N, the formation of the refined input image of the nth scale can be achieved by: performing pixel-by-pixel addition processing on the decoded image of the n-1th scale and the encoded image of the M-(n-1)th scale among the M scale encoded images to obtain an added image of the n-1th scale; performing residual convolution processing on the added image of the n-1th scale to obtain a residual convolution image of the n-1th scale; performing up-sampling processing on the refined image of the n-1th scale, and performing bottleneck convolution processing on the obtained up-sampled image of the n-1th scale to obtain a second bottleneck convolution image of the n-1th scale; and performing channel splicing processing on the residual convolution image of the n-1th scale and the second bottleneck convolution image of the n-1th scale to obtain a spliced image of the n-1th scale as the refined input image of the nth scale.

[0108] As an example, refer to Figure 3B , Figure 4B In the example of FIG. 1, N=3, i.e., the value range of n is 2≤n≤3. The following will illustrate the case of n=2 in detail. Figure 4B

[0109] As an example, refer to Figure 4B When n=2, perform pixel-by-pixel addition processing on the decoded image 8 of the 1st scale and the encoded image of the 3rd scale among the 4 scale encoded images (i.e., the third scale encoded image 3 mentioned above) to obtain an added image 9 of the 1st scale; perform residual convolution processing on the added image 9 of the 1st scale to obtain a residual convolution image 10 of the 1st scale; perform up-sampling processing on the refined image 5 of the 1st scale, and perform bottleneck convolution processing on the obtained up-sampled image 5-2 of the 1st scale to obtain a second bottleneck convolution image 5-3 of the 1st scale; and perform channel splicing processing on the residual convolution image 10 of the 1st scale and the second bottleneck convolution image 5-3 of the 1st scale to obtain a spliced image of the 1st scale as the refined input image of the 2nd scale.

[0110] In some embodiments, refer to Figure 4B , Figure 4B FIG. 1 is a flowchart of an image deblurring method according to an embodiment of the present application, Figure 4B The step 102 shown in FIG. 1 can be implemented by steps 1021 to 1022.

[0111] In step 1021, the bottleneck convolution processing of the nth scale is performed on the refined input image of the nth scale to obtain a first bottleneck convolution image of the nth scale.

[0112] For example, the bottleneck convolution processing can be implemented by a bottleneck convolution structure with a convolution kernel size of 1*1.

[0113] ​As an example, refer to Figure 4B The first bottleneck convolution processing is performed on the input image of the first scale (the input image of the first scale is the encoded image of the fourth scale in the M encoded images, i.e., the fourth scale encoded image 4 mentioned above), to obtain a first bottleneck convolution image of the first scale 5-1.

[0114] In step 1022, the prediction convolution processing of the nth scale is performed on the first bottleneck convolution image of the nth scale, to obtain the refined image of the nth scale.

[0115] For example, the prediction convolution processing can be implemented by a prediction convolution structure with a convolution kernel size of 5*5, while the bottleneck convolution structure has a convolution kernel size of 1*1, i.e., the prediction convolution processing and the bottleneck convolution processing are convolution processing implemented by convolution structures with different convolution kernel sizes.

[0116] As an example, refer to Figure 4B The prediction convolution processing of the first scale is performed on the first bottleneck convolution image 5-1 of the first scale, to obtain the refined image 5 of the first scale.

[0117] In this way, by performing the bottleneck convolution processing of the nth scale on the input image of the nth scale, the deblurring information of different scales is adjusted to a unified number of channels, and then the prediction convolution processing of the nth scale is performed on the first bottleneck convolution image of the nth scale, to restore the refined image of the nth scale. By restoring the refined image of the nth scale, the image restoration of the nth scale decoding processing is assisted, so that the deblurring effect of the image can be significantly improved, and a deblurred image with higher clarity is obtained.

[0118] In step 103, the N-scale decoding processing based on the encoded images of the second scale to the Mth scale in the M-scale encoded images and the N-scale refined images is performed in cascade, to obtain the N-scale decoded images in sequence.

[0119] In some embodiments, the N-scale refining processing and the N-scale decoding processing are performed synchronously according to the same scale, and the refined image of the previous scale is used as auxiliary decoding information for the decoding processing of the next scale.

[0120] As an example, refer to Figure 4B In step 1021, the bottleneck convolution processing of the nth scale is performed on the input image of the nth scale, to obtain a first bottleneck convolution image of the nth scale. Figure 4BIn this model, M=4 and N=3. Based on the coded images at the second to fourth scales and the refined images at the third scale, a cascaded three-scale decoding process is performed to obtain three decoded images at different scales. The three-scale refinement and three-scale decoding processes are executed synchronously at the same scale, and the refined image from the previous scale is used as auxiliary decoding information for the next scale's decoding process.

[0121] In some embodiments, the decoding process in step 103 above can be implemented in the following way: iterating n times and performing the following processing: based on the decoded input image at the nth scale, perform decoding processing at the nth scale to obtain the decoded image at the nth scale, where the value of n is in the range of 1≤n≤N; where, when n=1, the decoded input image at the nth scale is the encoded image at the Mth scale among the encoded images at M scales; where, when 2≤n≤N, the decoded input image at the nth scale is formed after the refinement processing at the (n-1)th scale (or before the refinement processing at the nth scale) in the following way: based on the decoded image at the (n-1)th scale, the encoded image at the M-(n-1)th scale among the encoded images at M scales, and the refined image at the (n-1)th scale, perform fusion processing to obtain the decoded input image at the nth scale.

[0122] Here, the refined image at the (n-1)th scale can be used as auxiliary decoding information at the (n-1)th scale, which can enable the decoding process at the (n-1)th scale to achieve better decoding results.

[0123] Thus, see Figure 3B The second-scale decoding input image is obtained by fusing the first-scale decoding image (8), the third-scale encoded image from the four-scale encoded images (i.e., the third-scale encoded image 3 mentioned above), and the first-scale refined image. Compared to directly using the first-scale decoding image as the second-scale decoding input image, the second-scale decoding input image incorporates features from the first-scale refined image, allowing the next-scale decoding process to incorporate features from the previous-scale refined image. This means that multiple scales of refined images can better assist the decoding network's decoding process, enabling the encoding and decoding networks to gradually clarify the first image from blurry to sharp. Therefore, compared to directly using the first-scale encoded image as the second-scale decoding input image, incorporating features from the first-scale refined image into the second-scale decoding input image results in better image deblurring.

[0124] As an example, see Figure 3B , Figure 3B In this case, N = 3, meaning the range of n is 1 ≤ n ≤ 3. The following is combined with...Figure 4B The case of iteration n=1 and n=2 is schematically explained as follows:

[0125] Referring to Figure 4B When n=1, the decoding input image of the first scale is the coded image of the fourth scale in the four coded images (i.e., the fourth scale coded image 4 mentioned above).

[0126] Referring to Figure 4B When n=2, a fusion process is performed based on the decoding image of the first scale 8, the coded image of the third scale in the four coded images (i.e., the third scale coded image 3 mentioned above), and the refined image of the first scale to obtain the decoding input image of the second scale.

[0127] In some embodiments, referring to Figure 4B , Figure 4B is a flowchart of the image deblurring method provided by the embodiments of the present application, Figure 4B The step 103 shown in the figure can be implemented by steps 1031 to 1032.

[0128] In step 1031, the decoding image of the (n-1)th scale and the coded image of the (M-(n-1))th scale in the M coded images are added pixel by pixel, and the obtained added image of the (n-1)th scale is subjected to residual convolution processing to obtain the residual convolution image of the (n-1)th scale.

[0129] As an example, referring to Figure 4B , Figure 4B In the figure, N=3, i.e., the value range of n is 2≤n≤3, and the following figure is combined with Figure 3C The case of n=2 is schematically explained as follows:

[0130] Referring to Figure 3C When n=2, the decoding image of the first scale 8 and the coded image of the third scale in the four coded images (i.e., the third scale coded image 3 mentioned above) are added pixel by pixel, and the obtained added image 9 of the first scale is subjected to residual convolution processing to obtain the residual convolution image 10 of the first scale.

[0131] In step 1032, the residual convolution image of the (n-1)th scale and the refined image of the (n-1)th scale are spliced according to the channel, and the obtained spliced image of the (n-1)th scale is taken as the decoding input image of the nth scale.

[0132] As an example, referring to Figure 3C , Figure 4B In the figure, N=3, i.e., the value range of n is 2≤n≤3, and the following figure is combined with Figure 4BThe case of n = 2 is described as an example.

[0133] Referring to Figure 4B When n = 2, the residual convolution image 10 of the first scale and the refined image 5 of the first scale are spliced by channel, and the obtained spliced image of the first scale is taken as the decoding input image of the second scale.

[0134] In some embodiments, the step 1032 can be implemented by the following manner: the refined image of the n-1th scale is up-sampled, the obtained up-sampled image of the n-1th scale is bottleneck-convoluted to obtain the second bottleneck-convoluted image of the n-1th scale; the residual convolution image of the n-1th scale and the second bottleneck-convoluted image of the n-1th scale are spliced by channel, and the obtained spliced image of the n-1th scale is taken as the decoding input image of the n th scale.

[0135] As an example, referring to Figure 4B When n = 2, the refined image 5 of the first scale is up-sampled, the obtained up-sampled image 5-2 of the first scale is bottleneck-convoluted to obtain the second bottleneck-convoluted image 5-3 of the first scale; the residual convolution image 10 of the first scale and the second bottleneck-convoluted image 5-3 of the first scale are spliced by channel, and the obtained spliced image of the first scale is taken as the decoding input image of the second scale.

[0136] In step 104, image prediction processing is performed based on the encoding image of the first scale among the M-scale encoding images, the decoding image of the Nth scale among the N-scale decoding images, and the refined image of the Nth scale among the N-scale refined images, to obtain a second image with a higher definition than the first image.

[0137] In some embodiments, the first image and the second image can be images of any source, and the image contents of the first image and the second image are completely the same, but the definitions of the first image and the second image are different, and the definition of the first image is less than that of the second image, that is, the first image is a blurred image corresponding to the second image, and the second image is a clear image corresponding to the first image.

[0138] In some embodiments, the image prediction processing and the prediction convolution processing mentioned above can be processed in the same way, that is, the image prediction processing and the prediction convolution processing are both implemented by the prediction convolution structure, and the image prediction processing and the prediction convolution processing share the parameters of the same type of prediction convolution structure.

[0139] Thus, by sharing the parameters of the same type of prediction convolution structure between the image prediction processing and the prediction convolution processing, the consistency of the features learned in the decoding process and the distillation process is ensured.

[0140] In some embodiments, referring to Figure 4B , Figure 4B is a flowchart of an image deblurring method provided by the embodiments of the present application, Figure 3D The step 104 shown can be implemented by steps 1041 to 1042.

[0141] In step 1041, a fusion process is performed based on the first scale of the M scale encoded images, the N scale of the N scale decoded images, and the N scale of the N scale distilled images to obtain the N scale of the fusion image.

[0142] As an example, referring to Figure 3D , Figure 3D In the example of M=4, a fusion process is performed based on the first scale of the four scale encoded images (i.e., the first scale encoded image 1 mentioned above), the third scale of the three scale decoded images, and the third scale of the three scale distilled images to obtain the fourth scale of the fusion image.

[0143] In some embodiments, the above step 1041 can be implemented by the following manner: performing pixel-by-pixel addition processing on the first scale of the M scale encoded images and the N scale of the N scale decoded images to obtain the N scale of the sum image; performing residual convolution processing on the N scale of the sum image to obtain the N scale of the residual convolution image; and performing channel-based splicing processing on the N scale of the residual convolution image and the N scale of the N scale distilled image to obtain the N scale of the splicing image as the N scale of the fusion image.

[0144] As an example, referring to Figure 4A , Figure 4A In the example of M=4, the first scale of the four scale encoded images (i.e., the first scale encoded image 1 mentioned above) and the third scale of the three scale decoded images 11 are subjected to pixel-by-pixel addition processing to obtain the third scale of the sum image 12; the third scale of the sum image 12 is subjected to residual convolution processing to obtain the third scale of the residual convolution image 13; and the third scale of the residual convolution image 13 and the third scale of the three scale distilled images 7 are subjected to channel-based splicing processing to obtain the third scale of the splicing image as the third scale of the fusion image.

[0145] In some embodiments, the channel-wise splicing of the Nth-scale residual convolutional image and the Nth-scale extracted image in the N-scale extracted images can be achieved by upsampling the Nth-scale extracted image in the N-scale extracted images to obtain an Nth-scale upsampled image, performing Nth-scale bottleneck convolution on the Nth-scale upsampled image to obtain an Nth-scale second bottleneck convolutional image, and channel-wise splicing the Nth-scale residual convolutional image and the Nth-scale second bottleneck convolutional image to obtain an Nth-scale spliced image.

[0146] As an example, referring to Figure 4A , the 3rd-scale extracted image 7 is upsam pled to obtain a 3rd-scale upsampled image 7-1, the 3rd-scale upsampled image 7-1 is subjected to 3rd-scale bottleneck convolution to obtain a 3rd-scale second bottleneck convolutional image 7-2, and the 3rd-scale residual convolutional image 13 and the 3rd-scale second bottleneck convolutional image 7-2 are channel-wise spliced to obtain a 3rd-scale spliced image.

[0147] In step 1042, the Nth-scale fused image is subjected to predictive convolution to obtain a second image with higher definition than the first image.

[0148] As an example, referring to Figure 4A , the 3rd-scale fused image is subjected to predictive convolution to obtain a second image with higher definition than the first image.

[0149] In this way, by incorporating the N-scale extracted images and the 2nd-scale to Mth-scale encoded images in the M-scale encoded images into the decoding process, the N-scale extracted images can assist the decoding process of the images at the N scales, so that the first image can be recovered from low resolution to high resolution at each scale, the deblurring effect of the images can be significantly improved, and a second image with higher definition can be obtained.

[0150] In some embodiments, the encoding process is implemented by an encoding network, the decoding process is implemented by a decoding network, and the extraction process is implemented by an extraction network.

[0151] In some embodiments, referring to Figure 4B , Figure 4B is a flowchart of an image deblurring method provided by an embodiment of the present application, which will be described in combination with steps 201 to 206 shown in Figure 5A .

[0152] In step 201, the encoding network, the decoding network and the abstraction network are called based on the blurred image sample to perform deblurring processing, to obtain a predicted clear image with higher definition than the blurred image sample, and N-scale predicted abstraction image samples.

[0153] As an example, refer to Figure 5A , Figure 5A is a schematic diagram of the image deblurring processing provided by the embodiments of the present application, the encoding network, the decoding network and the abstraction network are called based on the blurred image sample to perform deblurring processing, to obtain a predicted clear image with higher definition than the blurred image sample.

[0154] In step 202, the first loss function is called based on the predicted clear image and the real clear image sample corresponding to the blurred image sample to perform prediction processing, to obtain a first loss value.

[0155] In some embodiments, the first loss function is a loss function corresponding to the encoding network and the decoding network, and the expression of the first loss function L2 can be:

[0156]

[0157] Wherein, I represents the predicted clear image, Ig represents the real clear image sample, and L2 represents the first loss function.

[0158] As an example, refer to Figure 5A , the first loss function is called based on the predicted clear image and the real clear image sample corresponding to the blurred image sample to perform prediction processing, to obtain a first loss value (i.e. L2 loss in Figure 5A ).

[0159] In step 203, for each scale of the N-scale predicted abstraction image samples, the following processing is performed: the first loss function is called based on the nth scale of the predicted abstraction image sample and the nth scale of the real abstraction image sample to perform prediction processing, to obtain a second loss value, wherein n takes a value in the range of 1≤n≤N.

[0160] As an example, refer to Figure 5B , the first loss function is called based on the first scale of the predicted abstraction image sample 5 and the first scale of the real abstraction image sample to perform prediction processing, to obtain the first scale of the second loss value; the first loss function is called based on the second scale of the predicted abstraction image sample 6 and the second scale of the real abstraction image sample to perform prediction processing, to obtain the second scale of the second loss value; the first loss function is called based on the third scale of the predicted abstraction image sample 7 and the third scale of the real abstraction image sample to perform prediction processing, to obtain the third scale of the second loss value.

[0161] In step 204, the second loss values corresponding to the prediction refinement image samples of the N scales are weighted and summed to obtain a third loss value.

[0162] In some embodiments, the expression for weighting and summing the second loss values to obtain the third loss value can be:

[0163]

[0164] where Ii represents the prediction clear image at the i-th scale, Igi represents the real clear image sample at the i-th scale, wri represents the weight in the summation, and w ri = 2 -i L1 represents the L1 refinement loss function corresponding to the third loss value.

[0165] As an example, referring to Figure 5B the second loss value corresponding to the prediction refinement image sample 5 at the first scale, the second loss value corresponding to the prediction refinement image sample 6 at the second scale, and the second loss value corresponding to the prediction refinement image sample 6 at the third scale are weighted and summed to obtain the third loss value.

[0166] In step 205, the first loss value and the third loss value are weighted and summed to obtain a joint loss value.

[0167] In some embodiments, the expression for weighting and summing the first loss value and the third loss value to obtain the joint loss value can be:

[0168] L = L2 + λ r L1 (5)

[0169] where λ r represents the weight of the L1 refinement loss function, and λ r may be set to 0.1.

[0170] In step 206, the encoding network, the decoding network, and the refinement network are updated by the joint loss value.

[0171] In some embodiments, the above step 206 can be implemented by updating the parameters of the encoding network, the decoding network, and the refinement network by the joint loss value to obtain updated encoding network, decoding network, and refinement network.

[0172] In this way, by training the encoding network, the decoding network, and the refinement network, the trained encoding network, decoding network, and refinement network have better deblurring effect when applied, which can significantly improve the deblurring effect of the image and obtain a clearer deblurring image.

[0173] Below, an exemplary application of the embodiments of the present application in an actual application scenario will be described.

[0174] The embodiments of the present application can have the following application scenarios, for example, when monitoring a pan-tilt, a smart mobile terminal, and various devices with a photographing function, the collected image is often a blurred image due to a long photographing exposure time or shaking during photographing. In this case, in addition to the known collected blurred image, the original image of the blurred image, the size of the blur kernel, and the noise are unknown. Then, by using the image deblurring processing method provided by the embodiments of the present application, the blurred image is deblurred, the blurred image can be more efficiently and intelligently restored, and a restored clear image is obtained.

[0175] In some embodiments, Figure 5B is an effect diagram of the image deblurring processing method provided by the embodiments of the present application. Referring to Figure 5B , the image processing network designed by the embodiments of the present application is an encoder-decoder structure based on a convolutional neural network, and a multi-scale refining branch (i.e., the refining network described above) is designed, so that the image processing network can gradually restore the blurred image from low resolution to high resolution, thereby obtaining a clear image corresponding to the blurred image, which is more conducive to the learning of the image processing network. The embodiments of the present application are driven by sample data, and in the case of sufficient sample data, a powerful image processing network with deblurring capability can be trained. Figure 5C The left image inis a collected blurred image, and the image processing network can gradually restore the blurred image from low resolution to high resolution, and after processing by the image processing network, a restored clear image (the right image in) is obtained. Figure 5C Figure 5C It can be observed that the clarity of the restored clear image is obviously higher than that of the blurred image.

[0176] In some embodiments, referring to Figure 5C , Figure 4D is an effect diagram of the image deblurring processing method provided by the embodiments of the present application, Figure 4D The left image in is a detail part of a collected blurred image, and after processing by the image processing network, a restored clear image (the right image in ) of the detail part is obtained. Figure 4D

[0177] In some embodiments, referring to Figure 4C , Figure 4C is an effect diagram of the image deblurring processing method provided by the embodiments of the present application, Figure 4CThe left image in FIG. 1 is another detail part of the blurred image collected, and the right image in FIG. 1 is a clear image of the detail part after the image processing network is processed. Figure 4C

[0178] The image processing network provided by the embodiments of the present application mainly includes two modules: the first module is a "funnel type main network" (i.e., the encoding network and the decoding network described above), which is a convolutional neural network (CNN) constructed by using a convolution network (Conv) and a deconvolution network (Deconv) and has the main functions of image encoding, decoding and restoration. The second module is a "multi-scale refining branch network" (i.e., the refining network described above), which is responsible for assisting the main network to restore the image in a multi-scale manner, so that the network can restore the blurred image from low resolution to high resolution layer by layer, which is better than directly restoring a high-resolution picture.

[0179] The first module in the image processing network provided by the embodiments of the present application is the funnel type main network (i.e., the encoding network and the decoding network described above), which includes four basic parameter-containing operators, such as a 3*3 residual convolution block (i.e., the residual convolution processing described above), a 5*5 convolution layer (i.e., the general convolution processing described above), a 5*5 deconvolution layer (i.e., the decoding processing described above), and a 5*5 prediction convolution layer (i.e., the prediction convolution processing described above), and two parameter-free operators, such as "element-wise addition" (i.e., the pixel-by-pixel addition processing described above) and "concatenate in channel dimension" (i.e., the channel-based concatenation processing described above).

[0180] In some embodiments, the encoding network includes a 3*3 residual convolution block (i.e., the residual convolution processing described above), a 5*5 convolution layer (i.e., the general convolution processing described above), and the decoding network includes a 5*5 deconvolution layer (i.e., the decoding processing described above), and a 5*5 prediction convolution layer (i.e., the prediction convolution processing described above).

[0181] Figure 4C FIG. 1 is a schematic diagram of the principle of the image deblurring processing method provided by the embodiments of the present application, referring to Figure 2 , in Figure 3A In the residual convolution block shown in FIG. 1, 1*1 convolution is used before and after 3*3 convolution, so that the dimension is reduced and then restored, thereby significantly reducing the error rate.

[0182] ​The second module in the image processing network provided by the embodiment of the present application is a multi-scale refining branch network (i.e., the refining network described above), which includes 1*1 bottleneck convolution layers (i.e., the bottleneck convolution processing described above), 5*5 prediction convolution layers (i.e., the prediction convolution processing described above), and four basic parameter-containing operators, and a "2x upsampling" (i.e., the upsampling processing described above) parameter-free operator.

[0183] The multi-scale refining branch network (i.e., the refining network described above) is mainly added to the decoding network of the funnel-shaped backbone network. Referring to ​ , ​ is a schematic diagram of the principle of the image deblurring processing method provided by the embodiment of the present application. In the decoding network stage of the funnel-shaped backbone network, the dimension of the deblurring feature is gradually reduced, and the resolution is gradually increased layer by layer, wherein each layer represents different scale information. The multi-scale refining branch network is used to adjust the deblurring information of different scales to a uniform channel number through a layer of 1*1 bottleneck convolution layers (i.e., the bottleneck convolution processing described above), and then restore the multi-scale clear image (i.e., the second image described above) through a layer of 5*5 prediction convolution layers (i.e., the prediction convolution processing described above). Finally, the restored multi-scale clear image will also pass through a layer of 1*1 bottleneck convolution layers (i.e., the bottleneck convolution processing described above), so that the clear information at this scale is restored to the funnel-shaped backbone network (i.e., the encoding network and the decoding network described above), to assist the generation of deblurring information at the next scale, thereby realizing the "from coarse to fine" deblurring process.

[0184] Referring to ​ , the 5*5 prediction convolution layers (i.e., the prediction convolution processing described above) appear in the funnel-shaped backbone network (i.e., the encoding network and the decoding network described above) and the multi-scale refining branch network (i.e., the refining network described above) at the same time, and both play a role in generating a clear image.

[0185] In the embodiment of the present application, in order to improve the identity of the features learned by the funnel-shaped backbone network (i.e., the prediction convolution processing described above) and the multi-scale refining branch network (i.e., the refining network described above), the mechanism of "shared prediction convolution layers" is used, so that the deblurring performance of the algorithm is more optimal, and ​ In the embodiment of the present application, all 5*5 prediction convolution layers (i.e., the prediction convolution processing described above) share parameters, that is, the same predictor is used for all multi-scale clear image prediction and final clear image prediction.

[0186] In the embodiment of the present application, referring to ​, the mechanism of "deblurring joint loss function" is used, so that the deblurring performance of the algorithm is better, the funnel-shaped main network (i.e. the encoding network and the decoding network described above) and the multi-scale refining branch network (i.e. the refining network described above) have different loss functions respectively, wherein the loss function corresponding to the funnel-shaped main network (i.e. the encoding network and the decoding network described above) is: L2 loss, and the loss function corresponding to the multi-scale refining branch network (i.e. the refining network described above) is: L1 refining loss.

[0187] In some embodiments, the L2 loss is the loss function (i.e. the first loss function described above) corresponding to the funnel-shaped main network (i.e. the encoding network and the decoding network described above), and the expression of the L2 loss is:

[0188]

[0189] wherein I represents the predicted clear image (i.e. the predicted clear image described above) in the embodiments of the present application, and Ig represents the real clear image (i.e. the real clear image sample described above).

[0190] In some embodiments, since the multi-scale refining branch network (i.e. the encoding network and the decoding network described above) will output a series of multi-scale clear images, the L1 refining loss (the third loss value) is the weighted sum of multiple L2 losses, and the expression of the L1 refining loss is:

[0191]

[0192] wherein under the i-th scale, Ii represents the clear image (i.e. the predicted clear image under the i-th scale described above) predicted by the image processing network, Igi represents the real clear image (i.e. the real clear image sample under the i-th scale described above), and wri represents the weight when summing, wri ri = 2 -i .

[0193] In some embodiments, the deblurring joint loss function (i.e. the joint loss value L described above) is used to integrate the L1 refining loss and the L2 loss, and is used for the end-to-end (input to output of the image processing network) training of the image processing network, and the expression of the deblurring joint loss function (i.e. the joint loss value L described above) is:

[0194] L = L2 + λ r L1 (8)

[0195] wherein λ r represents the weight of the L1 refining loss (i.e. the weight of the L1 refining loss function described above), and λ r may be set to 0.1.

[0196] The image deblurring processing method provided by the embodiments of the present application belongs to a supervised deep learning algorithm, and needs to rely on a dataset containing blurred images (i.e., the blurred image samples described above) and clear images (i.e., the real clear image samples described above) for training. A public training set can be selected as the training set of the image deblurring processing method provided by the embodiments of the present application.

[0197] In the embodiments of the present application, an optimizer (for example, an Adam optimizer) can be used to optimize the deblurring joint loss function, wherein the Adam optimizer can comprehensively optimize the first moment estimation (First Moment Estimation, i.e., the mean of the gradient) and the second moment estimation (Second Moment Estimation, i.e., the uncentered variance of the gradient) of the gradient, calculate the update step, and the learning rate can be set to 0.0001, i.e., in the entire training process, it will exponentially decay to 0.000001 at a rate of 0.3. The maximum number of training iterations can be set to 1000, the batch size (Batch Size) is set to 16, and the total training time on a single graphics card is 48 hours.

[0198] In this way, by assisting the decoding processing of the image at each scale with the refined images of multiple scales, the blurred image can be recovered from low resolution to high resolution at each scale, which can significantly improve the deblurring effect of the image and obtain a clearer deblurred image, i.e., a clear image.

[0199] The following continues to describe an exemplary structure of the implementation of the image deblurring processing apparatus 455 provided by the embodiments of the present application as a software module. In some embodiments, as shown in FIG. 6, the image deblurring processing apparatus 455 provided by the embodiments of the present application can include a deblurring model 601, a deblurring model training module 602, a deblurring model inference module 603, and a deblurring model updating module 604. ​As shown, the software modules stored in the image deblurring processing apparatus 455 in the memory 440 can include: an encoding module 4551, configured to perform encoding processing of M scales in cascade based on the first image, to sequentially obtain M scale encoding images; where M is an integer greater than or equal to 2, and the M scales are sequentially decreased; an extraction module 4552, configured to perform extraction processing of N scales in cascade based on the encoding images of the second scale to the M scale among the M scale encoding images, to sequentially obtain N scale extraction images; where N = M-1, and the N scales are obtained by sequentially arranging the second scale to the M scale among the M scales in ascending order; a decoding module 4553, configured to perform decoding processing of N scales in cascade based on the encoding images of the second scale to the M scale among the M scale encoding images and the N scale extraction images, to sequentially obtain N scale decoding images; and a prediction module 4554, configured to perform image prediction processing based on the encoding image of the first scale among the M scale encoding images, the decoding image of the N scale among the N scale decoding images, and the extraction image of the N scale among the N scale extraction images, to obtain a second image with higher definition than the first image.

[0200] In some embodiments, the extraction module 4552 is further configured to perform the following processing iteratively n times: performing extraction processing of the nth scale based on an extraction input image of the nth scale, to obtain an extraction image of the nth scale, where n is in the range of 1≤n≤N; where when n = 1, the extraction input image of the nth scale is the encoding image of the Mth scale among the M scale encoding images; and where when 2≤n≤N, the extraction input image of the nth scale is formed after the decoding processing of the (n-1)th scale by: performing fusion processing based on the decoding image of the (n-1)th scale, the encoding image of the M-(n-1)th scale among the M scale encoding images, and the extraction image of the (n-1)th scale, to obtain the extraction input image of the nth scale.

[0201] In some embodiments, the extraction module 4552 is further configured to perform nth scale bottleneck convolution processing on the extraction input image of the nth scale, to obtain a first bottleneck convolution image of the nth scale; and perform nth scale prediction convolution processing on the first bottleneck convolution image of the nth scale, to obtain the extraction image of the nth scale.

[0202] In some embodiments, the refining module 4552 is further configured to perform pixel-wise addition processing on the decoded image of the n-1th scale and the encoded image of the M-(n-1)th scale among the M encoded images, to obtain a summed image of the n-1th scale; perform residual convolution processing on the summed image of the n-1th scale, to obtain a residual convolution image of the n-1th scale; perform up-sampling processing on the refined image of the n-1th scale, and perform bottleneck convolution processing on the up-sampled image of the n-1th scale obtained, to obtain a second bottleneck convolution image of the n-1th scale; and perform channel-wise splicing processing on the residual convolution image of the n-1th scale and the second bottleneck convolution image of the n-1th scale, to obtain a spliced image of the n-1th scale as the refined input image of the nth scale.

[0203] In some embodiments, the decoding module 4553 is further configured to perform the following processing iteratively for n times: based on the decoded input image of the nth scale, perform decoding processing of the nth scale to obtain a decoded image of the nth scale, where n ranges from 1 to N; when n = 1, the decoded input image of the nth scale is the encoded image of the Mth scale among the M encoded images; when 2 ≤ n ≤ N, the decoded input image of the nth scale is formed after the refining processing of the n-1th scale by performing fusion processing based on the decoded image of the n-1th scale, the encoded image of the M-(n-1)th scale among the M encoded images, and the refined image of the n-1th scale, to obtain the decoded input image of the nth scale.

[0204] In some embodiments, the decoding module 4553 is further configured to perform pixel-wise addition processing on the decoded image of the n-1th scale and the encoded image of the M-(n-1)th scale among the M encoded images, and perform residual convolution processing on the summed image of the n-1th scale obtained, to obtain a residual convolution image of the n-1th scale; perform channel-wise splicing processing on the residual convolution image of the n-1th scale and the refined image of the n-1th scale, to obtain a spliced image of the n-1th scale as the decoded input image of the nth scale.

[0205] In some embodiments, the prediction module 4554 is further configured to perform fusion processing based on the encoded image of the 1st scale among the M encoded images, the decoded image of the Nth scale among the N decoded images, and the refined image of the Nth scale among the N refined images, to obtain a fusion image of the Nth scale; and perform prediction convolution processing based on the fusion image of the Nth scale, to obtain a second image with a higher definition than the first image.

[0206] In some embodiments, the prediction module 4554 is further configured to perform pixel-by-pixel addition processing on the coded image at the first scale of the M scales and the decoded image at the Nth scale of the N scales to obtain a summed image at the Nth scale; perform residual convolution processing on the summed image at the Nth scale to obtain a residual convolution image at the Nth scale; and perform channel-based splicing processing on the residual convolution image at the Nth scale and the extracted image at the Nth scale of the N extracted images to obtain a spliced image at the Nth scale as the fusion image at the Nth scale.

[0207] In some embodiments, the prediction module 4554 is further configured to perform upsampling processing on the extracted image at the Nth scale of the N extracted images to obtain an upsampled image at the Nth scale; perform bottleneck convolution processing at the Nth scale on the upsampled image at the Nth scale to obtain a second bottleneck convolution image at the Nth scale; and perform channel-based splicing processing on the residual convolution image at the Nth scale and the second bottleneck convolution image at the Nth scale to obtain a spliced image at the Nth scale.

[0208] In some embodiments, the decoding module 4553 is further configured to perform the following processing iteratively m times: performing coding processing at the mth scale based on the coded input image at the mth scale to obtain a coded image at the mth scale, where m ranges from 1 to M; when m = 1, the coded input image at the mth scale is the first image; when 2 ≤ m ≤ M, the coded input image at the mth scale is the coded image at the (m-1)th scale.

[0209] In some embodiments, the decoding module 4553 is further configured to perform general convolution processing on the coded input image at the mth scale to obtain a general convolution image at the mth scale; and perform residual convolution processing at least once on the general convolution image at the mth scale to obtain the coded image at the mth scale.

[0210] In some embodiments, the image deblurring processing apparatus further comprises a training module configured to train the encoding network, the decoding network and the refining network by: calling the encoding network, the decoding network and the refining network to perform deblurring processing based on a blurred image sample to obtain a predicted clear image with a higher definition than the blurred image sample and a predicted refining image sample of N scales; calling the first loss function to perform prediction processing based on the predicted clear image and a real clear image sample corresponding to the blurred image sample to obtain a first loss value; performing the following processing for each of the predicted refining image samples of the N scales: calling the first loss function to perform prediction processing based on the predicted refining image sample of the nth scale and a real refining image sample of the nth scale to obtain a second loss value, where n is in the range of 1≤n≤N; performing weighted sum processing on the second loss values corresponding to the predicted refining image samples of the N scales respectively to obtain a third loss value; performing weighted sum processing on the first loss value and the third loss value to obtain a joint loss value; and updating the encoding network, the decoding network and the refining network based on the joint loss value.

[0211] The embodiment of the present application provides an electronic device, including: a memory configured to store executable instructions; and a processor configured to execute the executable instructions stored in the memory to implement the image deblurring processing method provided by the embodiment of the present application.

[0212] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the image deblurring processing method provided by the embodiment of the present application.

[0213] The embodiment of the present application provides a computer readable storage medium storing executable instructions, wherein the executable instructions are stored in the computer readable storage medium, and when the executable instructions are executed by a processor, the processor will execute the image deblurring processing method provided by the embodiment of the present application, for example, the image deblurring processing method shown in the above embodiment. ​ The embodiment of the present application provides an image deblurring processing method.

[0214] In some embodiments, the computer readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or various devices including one or any combination of the above memories.

[0215] In some embodiments, the executable instructions can take the form of programs, software, software modules, scripts, or code, written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and they can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0216] By way of example, the executable instructions can, but need not, correspond to a file in a file system, can be stored in a part of a file that holds other programs or data, e.g., one or more scripts stored in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code.

[0217] By way of example, the executable instructions can be deployed to be executed on one computer device, or on multiple computer devices that are located at one site, or that are distributed across multiple sites and interconnected through a communication network.

[0218] In summary, the embodiments of the present application have the following beneficial effects:

[0219] (1) By incorporating the second scale to the Mth scale of the N scale refined images and the M scale encoding images obtained in sequence into the decoding process, the N scale refined images can assist the decoding process of the image at N scales, so that the first image can be recovered from low resolution to high resolution at each scale, which can significantly improve the image deblurring effect and obtain a second image with higher clarity.

[0220] (2) By performing the n scale bottleneck convolution processing on the n scale refined input image, the deblurring information of different scales is adjusted to a unified channel number, and then the n scale first bottleneck convolution image is subjected to n scale prediction convolution processing, thereby recovering the n scale refined image. By recovering the n scale refined image, the n scale decoding process is assisted to recover the image, thereby significantly improving the image deblurring effect and obtaining a deblurring image with higher clarity.

[0221] (3) By making the image prediction processing and the prediction convolution processing share the same type of prediction convolution layer parameters, the uniformity of the features learned in the decoding process and the refining process is ensured.

[0222] (4) Through training of the encoding network, the decoding network and the refining network, the trained encoding network, the decoding network and the refining network have better deblurring effect when applied, can more significantly improve the deblurring effect of the image, and obtain a deblurred image with higher clarity.

[0223] The above merely describes the embodiments of the present application, but is not intended to limit the protection scope of the present application. Any modification, equivalent replacement and improvement within the spirit and scope of the present application shall be included in the protection scope of the present application.

Claims

1. An image deblurring processing method characterized by, The method comprises: performing a cascade M-scale encoding process based on the first image to obtain M-scale encoded images in sequence; wherein M is an integer greater than or equal to 2, and the M scales are sequentially decreased; performing a cascade N-scale refining process based on the M-scale encoded images of the second scale to the M scale to obtain N-scale refined images in sequence, wherein the N-scale refining process comprises: performing the following process in iteration n: performing an n-scale refining process based on an n-scale refining input image to obtain an n-scale refined image, wherein n is in the range of 1≤n≤N; wherein when n=1, the n-scale refining input image is the M-scale encoded image of the M scale in the M-scale encoded images; wherein when 2≤n≤N, the n-scale refining input image is formed after the (n-1)-scale decoding process by performing a fusion process based on the (n-1)-scale decoded image, the M-(n-1)-scale encoded image in the M-scale encoded images, and the (n-1)-scale refined image to obtain the n-scale refining input image; wherein N=M-1, and the N scales are obtained by sequentially arranging the second scale to the M scale in the M scales in an increasing order; performing an N-scale decoding process based on the M-scale encoded images of the second scale to the M scale and the N-scale refined images to obtain N-scale decoded images in sequence; performing an image prediction process based on the first-scale encoded image in the M-scale encoded images, the N-scale decoded image in the N-scale decoded images, and the N-scale refined image in the N-scale refined images to obtain a second image with a higher resolution than the first image.

2. The method of claim 1, wherein the N-scale refining process and the N-scale decoding process are synchronously performed according to the same scale, and the decoded image of a previous scale is used as auxiliary refining information for the refining process of a next scale.

3. The method of claim 2, wherein the performing of the n-scale refining process based on the n-scale refining input image to obtain the n-scale refined image comprises: performing an n-scale bottleneck convolution process on the n-scale refining input image to obtain an n-scale first bottleneck convolution image; performing an n-scale prediction convolution process on the n-scale first bottleneck convolution image to obtain the n-scale refined image.

4. The method of claim 2, wherein the performing of the fusion process based on the (n-1)-scale decoded image, the M-(n-1)-scale encoded image in the M-scale encoded images, and the (n-1)-scale refined image to obtain the n-scale refining input image comprises: performing pixel-by-pixel addition processing on the decoded image of the n-1th scale and the encoded image of the M-(n-1)th scale among the M encoded images to obtain an added image of the n-1th scale; performing residual convolution processing on the added image of the n-1th scale to obtain a residual convolution image of the n-1th scale; performing up-sampling processing on the refined image of the n-1th scale, and performing bottleneck convolution processing on the obtained up-sampled image of the n-1th scale to obtain a second bottleneck convolution image of the n-1th scale; performing channel-based splicing processing on the residual convolution image of the n-1th scale and the second bottleneck convolution image of the n-1th scale, and taking the obtained spliced image of the n-1th scale as the refined input image of the n th scale.

5. The method of claim 1, wherein the N-scale refining processing and the N-scale decoding processing are performed synchronously according to the same scale, and the refined image of a previous scale is used as auxiliary decoding information for decoding processing of a next scale. the N-scale decoding processing based on the encoded images of the 2nd scale to the Mth scale among the M encoded images and the N-scale refined images is performed in sequence to obtain N-scale decoded images, including: iteratively performing the following processing based on the decoded input image of the n th scale to obtain a decoded image of the n th scale, where n is in the range of 1≤n≤N; when n = 1, the decoded input image of the n th scale is the encoded image of the Mth scale among the M encoded images; when 2≤n≤N, the decoded input image of the n th scale is formed after the refining processing of the n-1th scale by performing fusion processing based on the decoded image of the n-1th scale, the encoded image of the M-(n-1)th scale among the M encoded images, and the refined image of the n-1th scale to obtain the decoded input image of the n th scale. the fusion processing based on the decoded image of the n-1th scale, the encoded image of the M-(n-1)th scale among the M encoded images, and the refined image of the n-1th scale to obtain the decoded input image of the n th scale includes:

6. The method of claim 5, wherein, performing pixel-by-pixel addition processing on the decoded image of the n-1th scale and the encoded image of the M-(n-1)th scale among the M encoded images to obtain an added image of the n-1th scale; performing channel-based splicing processing on the residual convolution image of the n-1th scale and the refined image of the n-1th scale, and taking the obtained spliced image of the n-1th scale as the decoded input image of the n th scale. ​ 7. The method of claim 1, wherein, The image prediction processing is performed based on the first scale of the M scale encoded images, the N scale decoded image, and the N scale abstract image, to obtain a second image with higher definition than the first image, including: The fusion processing is performed based on the first scale of the M scale encoded images, the N scale decoded image, and the N scale abstract image, to obtain an N scale fusion image; The prediction convolution processing is performed based on the N scale fusion image, to obtain a second image with higher definition than the first image.

8. The method of claim 7, wherein, The fusion processing is performed based on the first scale of the M scale encoded images, the N scale decoded image, and the N scale abstract image, to obtain an N scale fusion image, including: The pixel-by-pixel addition processing is performed on the first scale of the M scale encoded images and the N scale decoded image, to obtain an N scale addition image; The residual convolution processing is performed on the N scale addition image, to obtain an N scale residual convolution image; The channel-based splicing processing is performed on the N scale residual convolution image and the N scale abstract image, and the obtained N scale splicing image is taken as the N scale fusion image.

9. The method of claim 8, wherein, The channel-based splicing processing is performed on the N scale residual convolution image and the N scale abstract image, including: The up-sampling processing is performed on the N scale abstract image, to obtain an N scale up-sampling image; The N scale bottleneck convolution processing is performed on the N scale up-sampling image, to obtain an N scale second bottleneck convolution image; The channel-based splicing processing is performed on the N scale residual convolution image and the N scale second bottleneck convolution image, to obtain the N scale splicing image.

10. The method of claim 1, wherein, The M scale encoding processing based on the first image is cascaded, to obtain M scale encoded images, including: The m-th scale encoding processing is performed based on the m-th scale encoding input image, to obtain the m-th scale encoded image, where m is in the range of 1≤m≤M; When m=1, the m-th scale encoding input image is the first image; When 2≤m≤M, the m-th scale encoding input image is the (m-1)-th scale encoded image.

11. The method of claim 10, wherein The m-th scale encoding processing is performed based on the m-th scale encoding input image, to obtain the m-th scale encoded image, including: perform general convolution processing on the mth scale coded input image to obtain a general convolution image of the mth scale; perform at least one residual convolution processing on the mth scale general convolution image to obtain the mth scale coded image.

12. The method of claim 1, wherein, the encoding processing is implemented by an encoding network, the decoding processing is implemented by a decoding network, and the refining processing is implemented by a refining network; the method further comprises training the encoding network, the decoding network, and the refining network by: calling the encoding network, the decoding network, and the refining network based on a blurred image sample to perform deblurring processing, to obtain a predicted clear image with a higher clarity than the blurred image sample, and N scale predicted refined image samples; calling a first loss function based on the predicted clear image and a real clear image sample corresponding to the blurred image sample to perform prediction processing, to obtain a first loss value; for each of the N scale predicted refined image samples, performing the following processing: calling the first loss function based on an nth scale predicted refined image sample and an nth scale real refined image sample to perform prediction processing, to obtain a second loss value, wherein n takes a value in the range of 1≤n≤N; performing weighted sum processing on the second loss values corresponding to the N scale predicted refined image samples respectively, to obtain a third loss value; performing weighted sum processing on the first loss value and the third loss value, to obtain a joint loss value; updating the encoding network, the decoding network, and the refining network through the joint loss value.

13. An image deblurring processing apparatus characterized by comprising: The apparatus comprises: an encoding module configured to perform cascaded encoding processing on a first image based on M scales, to sequentially obtain M scale coded images; wherein M is an integer greater than or equal to 2, and the M scales decrease one by one; a refining module configured to perform cascaded refining processing on the M scale coded images based on the second to Mth scale coded images, to sequentially obtain N scale refined images; the refining module is further configured to perform the following processing iteratively n times: based on an nth scale refined input image, performing nth scale refining processing to obtain an nth scale refined image, wherein n takes a value in the range of 1≤n≤N; when n=1, the nth scale refined input image is the Mth scale coded image in the M scale coded images; when 2≤n≤N, the nth scale refined input image is formed after nth-1 scale decoding processing by: based on an nth-1 scale decoded image, an M-(n-1)th scale coded image in the M scale coded images, and an nth-1 scale refined image, performing fusion processing to obtain an nth scale refined input image; wherein N=M-1, and the N scales are obtained by arranging the second to Mth scales in the M scales in ascending order. a decoding module, configured to perform decoding processing on the N scales based on the encoded images of the M scales and the extracted images of the N scales, to obtain decoded images of the N scales in sequence; a prediction module, configured to perform image prediction processing based on the encoded image of the first scale among the M scales, the decoded image of the Nth scale among the decoded images of the N scales, and the extracted image of the Nth scale among the extracted images of the N scales, to obtain a second image with higher definition than the first image.

14. The apparatus of claim 13, wherein, The N-scale extraction processing and the N-scale decoding processing are synchronously performed according to the same scale, and the decoded image of a previous scale is used as auxiliary extraction information for the extraction processing of a next scale.

15. The apparatus of claim 14, wherein the extraction module is further configured to perform n-scale bottleneck convolution processing on the n-scale extracted input image, to obtain an n-scale first bottleneck convolution image; perform n-scale prediction convolution processing on the n-scale first bottleneck convolution image, to obtain an n-scale extracted image.

16. The apparatus of claim 14, wherein the decoding module is further configured to perform pixel-by-pixel addition processing on the (n-1)-scale decoded image and an M-(n-1)-scale encoded image among the M-scale encoded images, to obtain an (n-1)-scale added image; perform residual convolution processing on the (n-1)-scale added image, to obtain an (n-1)-scale residual convolution image; perform up-sampling processing on the (n-1)-scale extracted image, and perform bottleneck convolution processing on the obtained (n-1)-scale up-sampled image, to obtain an (n-1)-scale second bottleneck convolution image; perform channel-based splicing processing on the (n-1)-scale residual convolution image and the (n-1)-scale second bottleneck convolution image, and use the obtained (n-1)-scale spliced image as the n-scale extracted input image.

17. The apparatus of claim 13, wherein, The N-scale extraction processing and the N-scale decoding processing are synchronously performed according to the same scale, and the extracted image of a previous scale is used as auxiliary decoding information for the decoding processing of a next scale; the decoding module is further configured to perform the following processing iteratively n times: performing n-scale decoding processing based on an n-scale decoded input image, to obtain an n-scale decoded image, where n takes a value in a range of 1≤n≤N; when n=1, the n-scale decoded input image is an M-scale encoded image among the M-scale encoded images. When 2≤n≤N, the decoded input image of the n-th scale is formed after the abstraction processing of the (n-1)-th scale by performing a fusion process based on the decoded image of the (n-1)-th scale, the encoded image of the M-(n-1)-th scale among the M encoded images, and the abstracted image of the (n-1)-th scale, to obtain the decoded input image of the n-th scale.

18. An electronic device, comprising: The electronic device comprises: a memory for storing executable instructions; a processor for executing the executable instructions stored in the memory to implement the image deblurring processing method of any one of claims 1 to 12.

19. A computer-readable storage medium storing executable instructions, the method comprising: The executable instructions are executed by the processor to implement the image deblurring processing method of any one of claims 1 to 12.

20. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions are executed by the processor to implement the image deblurring processing method of any one of claims 1 to 12.

Images