An image deblurring method, device, computer device, and storage medium

Abstract

The present application provides an image deblurring method, apparatus, computer device, and storage medium. The method includes: obtaining a to-be-processed image of a target object; configuring a trained convolutional neural model, where the convolutional neural model includes a convolutional unit, a plurality of dense residual units, and an attention mechanism unit. The convolutional unit is used to extract shallow features of the blurred image, the plurality of dense residual units are used to extract deep features of the blurred image based on the shallow features, and the attention mechanism unit is used to increase the weight of the region of interest of the deep features; inputting the to-be-processed image into the convolutional neural model to obtain a deblurred target image. In the embodiments of the present application, the deblurring effect of the image is improved in the above manner, achieving the purpose of obtaining a clear image, thereby ensuring that subsequent research conducted by recognizing the image is more accurate.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular, to an image deblurring method, an image deblurring device, a computer device, and a readable storage medium. Background Art

[0002] Digital Subtraction Angiography (DSA) is a powerful technique that visualizes blood vessels in a sequence of X-ray images for the diagnosis and treatment of vascular diseases. During the diagnosis and treatment process, it is often necessary to clearly display blood vessels to determine the location and morphology of lesions, such as aneurysms, vascular stenoses, and vascular malformations. Image blurring is usually caused by various degradation factors, such as object movement, focal size, and focal jitter. Therefore, the complexity of degradation factors makes it uncertain to map blurred images to the clear space, increasing the difficulty of image restoration.

[0003] Moreover, compared with natural images, DSA images have more complex tissue structures and different image qualities. Therefore, directly transplanting the solutions for processing natural images is full of challenges. Summary of the Invention

[0004] In view of this, the present application provides an image deblurring method, an image deblurring device, a computer device, and a readable storage medium, which improve the image deblurring effect and achieve the purpose of obtaining clear images.

[0005] In a first aspect, an embodiment of the present application provides an image deblurring method, including: obtaining a to-be-processed image of a target object; configuring a trained convolutional neural model, where the convolutional neural model includes a convolutional unit, a plurality of dense residual units, and an attention mechanism unit, the convolutional unit is used to extract shallow features of the blurred image, the plurality of dense residual units are used to extract deep features of the blurred image based on the shallow features, and the attention mechanism unit is used to increase the weight of the region of interest of the deep features; inputting the to-be-processed image into the convolutional neural model to obtain a deblurred target image.

[0006] According to the above image deblurring method of the embodiment of the present application, the following additional technical features may also be provided:

[0007] In the above technical solution, optionally, before configuring the trained convolutional neural model, the method further includes: acquiring a clear image of the sampling object under the exposure condition of the first focus, and acquiring a blurred image of the sampling object under the exposure condition of the second focus, where the size of the first focus is smaller than the size of the second focus; inputting the blurred image into the convolutional neural model for model training to output a deblurred image; ending the model training and saving the convolutional neural model when the accuracy of the target loss function between the deblurred image and the clear image is greater than or equal to a preset accuracy threshold, or the number of iterations of the model training is greater than or equal to a preset iteration threshold.

[0008] In any of the above technical solutions, optionally, inputting the blurred image into the convolutional neural model for model training to output a deblurred image includes: extracting shallow features of the blurred image using a convolutional unit; using the shallow features as input and extracting deep features of the blurred image in a way of cascading multiple dense residual units; using an attention mechanism unit to increase the weight of the region of interest of the deep features; adding the shallow features and the deep features pixel by pixel to obtain a deblurred image.

[0009] In any of the above technical solutions, optionally, inputting the blurred image into the convolutional neural model for model training to output a deblurred image includes: inputting the blurred image into the convolutional neural model, setting the learning rate, a preset iteration threshold, and initializing the weights and biases of each unit of the convolutional neural model, and performing model training to output a deblurred image.

[0010] In any of the above technical solutions, optionally, the target loss function loss = λ1×L1 + λ2×L2, where L1 is the first loss function, L2 is the second loss function, λ1 is the weight coefficient of the first loss function, and λ2 is the weight coefficient of the second loss function; the first loss function L1 = ∑|I deblurred -I label |, the second loss function L2 = ∑(f(I deblurred ) - f(I label )) 2 ), where I deblurred is the deblurred image output by the convolutional neural model, I label is the clear image, f(I deblurred ) is the feature map after high-dimensional feature extraction of the deblurred image output by the convolutional neural model, and f(I label ) is the feature map after high-dimensional feature extraction of the clear image.

[0011] In any of the above technical solutions, optionally, the number of dense residual units is n, each dense residual unit includes m convolutional layers, and the input of the jth dense residual unit is F j-1 , and the output is F j, where j is an integer greater than or equal to 1 and less than or equal to n. When j equals 1, the input of the j-th dense residual unit is the shallow feature. When j is greater than 1, the input of the j-th dense residual unit is the output of the (j - 1)-th dense residual unit; for the j-th dense residual unit, the first convolutional layer takes F j-1 as the input and outputs the feature f1. The i-th convolutional layer takes F j-1 and the features output by the previous i - 1 convolutional layers as the input and outputs the feature f i . Then, F j-1 , the feature f1, and all the features f i are concatenated to obtain f Cj . Finally, f Cj is added to F j-1 pixel by pixel to obtain the output F j of the j-th dense residual unit, where i is an integer greater than 1 and less than or equal to m.

[0012] In any of the above technical solutions, optionally, the attention mechanism unit includes a first convolutional layer, a rectified linear unit, a second convolutional layer, and a Sigmoid function unit; wherein, the deep feature is used as the input of the attention mechanism unit, processed by the first convolutional layer, activated by the rectified linear unit, then processed by the second convolutional layer, and activated by the Sigmoid function unit to obtain the target weight coefficient. Finally, the target weight coefficient is multiplied by the deep feature pixel by pixel to obtain the output of the attention mechanism unit.

[0013] In a second aspect, an embodiment of the present application provides an image deblurring device, including: an acquisition module for acquiring a to-be-processed image of a target object; a configuration module for configuring a trained convolutional neural model, where the convolutional neural model includes a convolutional unit, a plurality of dense residual units, and an attention mechanism unit. The convolutional unit is used to extract the shallow features of the blurred image, the plurality of dense residual units are used to extract the deep features of the blurred image based on the shallow features, and the attention mechanism unit is used to increase the weight of the region of interest of the deep feature; a deblurring module for inputting the to-be-processed image into the convolutional neural model to obtain a deblurred target image.

[0014] In a third aspect, an embodiment of the present application provides a computer device, which includes a processor and a memory. The memory stores a program or instruction that can run on the processor. When the program or instruction is executed by the processor, the steps of the image deblurring method as in the first aspect are implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by the processor, the steps of the image deblurring method as in the first aspect are implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor, and the processor is configured to run programs or instructions to implement the image deblurring method as described in the first aspect.

[0017] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the image deblurring method as described in the first aspect.

[0018] In the embodiment of the present application, first, an image to be processed of a target object is obtained, and then a trained convolutional neural model is determined. The convolutional neural model includes a convolutional unit, a plurality of dense residual units, and an attention mechanism unit. For the above-mentioned image to be processed, the convolutional unit can be used for feature extraction processing, and then a plurality of dense residual units are connected in series to further extract features. After that, the attention mechanism unit is used to increase the weight of the region of interest of the extracted features. Finally, the trained convolutional neural model is used to perform deblurring processing on the image to be processed, and a clearer target image after deblurring is obtained. In the embodiment of the present application, the deblurring effect of the image is improved through the above method, achieving the purpose of obtaining a clear image, thereby ensuring that subsequent research conducted by identifying the image is more accurate.

[0019] The above description is only an overview of the technical solution of the present application. In order to be able to understand the technical means of the present application more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features, and advantages of the present application more obvious and understandable, the specific embodiments of the present application are specifically described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0021] Figure 1 A flowchart showing the image deblurring method according to an embodiment of the present application is shown;

[0022] Figure 2 A network structure of a convolutional neural network according to an embodiment of the present application is shown;

[0023] Figure 3 A network structure of a dense residual unit according to an embodiment of the present application is shown;

[0024] Figure 4 A network structure of an attention mechanism unit according to an embodiment of the present application is shown;

[0025] Figure 5The structural schematic block diagram of the image deblurring device according to the embodiment of the present application is shown;

[0026] Figure 6 The structural schematic block diagram of the computer device according to the embodiment of the present application is shown. Specific embodiments

[0027] Next, the technical solutions in the embodiments of the present application will be clearly described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are usually of the same type, and the number of objects is not limited. For example, the first object can be one or multiple. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally represents an "or" relationship between the associated objects before and after.

[0029] In the related art, traditional deconvolution methods are more applicable to scenarios where the blur kernel is known. However, in real scenarios, the image blur kernel is not known, and various constraints and priors need to be used to infer the blur kernel, and then the image is deblurred. This process involves the evaluation of the blur kernel, and the accuracy of the evaluation directly affects the effect of image deblurring, including a large number of parameter tunings, and the process is relatively cumbersome. For deblurring methods based on deep learning, early research mainly focused on the estimation of the blur kernel. However, due to the extremely complexity of the blur kernel in real scenarios, it is difficult to be applicable to all scenarios through the training of a single blur kernel.

[0030] The present application proposes an image deblurring method based on deep learning. By building a training data set for images, adopting a network structure that combines dense connection and residual connection, using an attention mechanism unit, and using a method jointly supervised by multiple loss functions, image deblurring is achieved.

[0031] Next, in conjunction with the accompanying drawings, the image deblurring method, image deblurring device, computer device, and readable storage medium provided by the embodiments of the present application will be described in detail through specific embodiments and their application scenarios.

[0032] In this embodiment, an image deblurring method is provided, as Figure 1 shown, the method includes:

[0033] Step 101: Obtain the image to be processed of the target object.

[0034] In this step, the image to be processed of the target object is obtained. Herein, the obtaining method of the image to be processed of the target object is not specifically limited. For example, it can be acquired in real time or obtained through certain channels (such as shared by other parties).

[0035] Step 102: Configure the trained convolutional neural model. The convolutional neural model includes a convolutional unit, multiple dense residual units, and an attention mechanism unit. The convolutional unit is used to extract the shallow features of the blurred image, the multiple dense residual units are used to extract the deep features of the blurred image based on the shallow features, and the attention mechanism unit is used to increase the weight of the region of interest of the deep features.

[0036] In this step, the trained convolutional neural model is determined. The convolutional neural model includes a convolutional unit, multiple dense residual units, and an attention mechanism unit. For the above-mentioned image to be processed, the convolutional unit can be used for feature extraction processing, and then the multiple dense residual units can be connected in series to further extract features. After that, the attention mechanism unit is used to increase the weight of the region of interest of the extracted features.

[0037] It should be noted that there may also be iterative optimization of the model in this step.

[0038] Step 103: Input the image to be processed into the convolutional neural model to obtain the de-blurred target image.

[0039] In this step, the trained convolutional neural model is used to perform de-blurring processing on the image to be processed to obtain a clearer target image after de-blurring.

[0040] In the embodiment of the present application, the de-blurring effect of the image is improved through the above method, achieving the purpose of obtaining a clear image, thereby ensuring that the subsequent research conducted by identifying the image is more accurate.

[0041] In an embodiment of the present application, the above-mentioned target object can be a patient, and the image to be processed of the target object can be the DSA image of the patient. The present application provides a method for de-blurring DSA images, improving the de-blurring effect of DSA images, and enabling more accurate identification of the location and morphology of vascular lesions through DSA images.

[0042] It should be noted that compared with natural images in fields such as video surveillance and remote sensing images, the tissue structure of DSA images is more complex and the image quality is also different. Therefore, it requires creative labor to de-blur DSA images using the deep learning-based image de-blurring method.

[0043] In one embodiment of the present application, before configuring the trained convolutional neural model, the method further includes:

[0044] (1) Acquire a clear image of the sampling object under the exposure condition of the first focus, and acquire a blurred image of the sampling object under the exposure condition of the second focus, where the size of the first focus is smaller than the size of the second focus.

[0045] In this step, build a dataset for model training. Specifically, by taking a series of corresponding clear-blurred image pairs of the sampling object, the construction of the dataset can be completed. Among them, the sampling object may include but is not limited to DSA phantoms, patients, etc. The clear image is a clear DSA image of the DSA phantom or the patient, and the blurred image is a blurred DSA image of the DSA phantom or the patient.

[0046] (2) Set the convolutional neural model and the target loss function.

[0047] In this step, use a convolutional neural network as the processing model for image deblurring, and set the convolutional neural network to include a convolutional unit, multiple dense residual units, and an attention mechanism unit. Use the convolutional unit for feature extraction processing, and then use the method of connecting multiple dense residual units in series to further extract features. After that, use the attention mechanism unit to increase the weight of the region of interest in the extracted features.

[0048] And set a multi-supervised target loss function for determining the convergence of the model for subsequent model training.

[0049] (3) Input the blurred image into the convolutional neural model for model training, and output a deblurred image.

[0050] In this step, input the acquired blurred image into the set convolutional neural model, and output a deblurred image.

[0051] (4) When the accuracy of the target loss function between the deblurred image and the clear image is greater than or equal to the preset accuracy threshold, or the number of iterations of model training is greater than or equal to the preset iteration threshold, end the model training and save the convolutional neural model.

[0052] In this step, during the training process, obtain the accuracy of the target loss function between the deblurred image and the clear image, and judge the accuracy and the number of iterations of model training. When the accuracy of the target loss function is greater than or equal to the preset accuracy threshold, or the number of iterations of model training is greater than or equal to the preset iteration threshold, it indicates that the model converges. At this time, end the model training and save the obtained convolutional neural model.

[0053] In the embodiments of the present application, a convolutional neural model for accurately deblurring images is established through the above method, thereby improving the deblurring effect of images.

[0054] The following takes the construction of a convolutional neural model for deblurring DSA images as an example to give a specific implementation solution of the embodiments of the present application:

[0055] Step 1: Build a dataset for model training. Specifically, clear images of the sampling object are collected under the exposure conditions of the first focal point, and blurred images of the sampling object are collected under the exposure conditions of the second focal point, where the size of the first focal point is smaller than that of the second focal point.

[0056] In this embodiment, clear images of the sampling object are collected. This data is obtained by a DSA image acquisition device (for example, Neusoft Angio30C) under the exposure conditions of a small focal point (i.e., the first focal point) based on a human phantom or a patient. The shooting content involves various parts of the sampling object, such as the head, chest (including the heart), abdomen, upper and lower limbs, etc.

[0057] Blurred images corresponding to the above clear images are collected. The acquisition method of this data is similar to that of the above clear images. The only difference is that a large focal point (i.e., the second focal point) is used for exposure, and other exposure conditions (such as bed height, phantom position, voltage, current, etc.) remain the same.

[0058] After the above steps, the construction of the dataset is completed by shooting a series of clear-blurred image pairs corresponding to small-large focal points.

[0059] In the embodiments of the present application, by converting the focal point setting during exposure, a dataset for model training is built, making the images more targeted and avoiding problems such as scene discomfort caused by directly transplanting open-source natural image sets.

[0060] Step 2: Use the blurred images in the above image pairs as inputs and the corresponding clear images as target (true value) images, and process them using a convolutional neural network. The network structure of the convolutional neural network is as Figure 2 shown, including a convolutional unit 201 and a deep feature extraction module, where the deep feature extraction module includes a plurality of dense residual units 202 and an attention mechanism unit 203.

[0061] Taking the blurred images taken with the large focal point as the input of the convolutional neural network, first use a single-layer convolutional unit to directly extract the shallow features F0 of the blurred images. The size of this convolutional unit is 3×3×64.

[0062] Use the shallow feature F0 as the input of the deep feature extraction module, and extract deep features in the way of concatenating the residual connection and n groups of dense residual units 202. The deep feature output by each dense residual unit 202 is denoted as F j , where j = 1, 2, …, n. To fully fuse the deep features, concatenate the deep features F j output by all dense residual units 202 to obtain the deep feature F C .

[0063] Input the deep feature F C into the attention mechanism unit 203, and obtain F A after feature extraction.

[0064] Add the shallow feature F0 output by the convolutional unit 201 and F A output by the attention mechanism unit 203 pixel by pixel to obtain the final output image, that is, the deblurred image.

[0065] Specifically, the network structure of the dense residual unit is as Figure 3 shown, adopting the combination of dense connection and residual connection. Each dense residual unit contains m groups of convolutional layers, and the convolutional kernel size is 3×3×64.

[0066] For the first dense residual unit (i.e., j = 1), its first convolutional layer takes the shallow feature F0 as the input and outputs the feature f1. After that, each layer takes the shallow feature F0 and the features output by all previous layers as the input. That is, the i-th convolutional layer takes F j-1 and the features output by the previous i - 1 convolutional layers as the input and outputs the feature f i , where i = 2, …, m, thus increasing the feature reuse. Then, concatenate the feature f m extracted by the last layer with its input (i.e., f1 to f m-1 ) and the shallow feature F0 to obtain f C1 . Then, add f C1 and F0 pixel by pixel to obtain the output F1 of the first dense residual unit.

[0067] For any dense residual unit among the second to the n-th dense residual units (i.e., j = 2, …, n), its first convolutional layer takes the output F j-1 of its previous dense residual unit as the input and outputs the feature f1. After that, each layer takes the output F j-1 of its previous dense residual unit and the features output by all previous layers as the input, thus increasing the feature reuse. Then, concatenate the feature f m extracted by the last layer with its input (i.e., f1 to f m-1 ) and the output Fj-1 Perform cascading to obtain f Cj , and then use f Cj to add to F j-1 pixel by pixel to obtain the output F of this dense residual unit j .

[0068] The network structure of the attention mechanism unit is as shown in Figure 4 , including a first convolutional layer, a rectified linear unit (ReLU), a second convolutional layer, and a Sigmoid function unit. The deep feature F after cascading the features of multiple dense residual units C is used as the input of the attention mechanism unit. The attention mechanism unit adopts the attention mechanism, and by enabling the convolutional neural model to learn different weights for different positions of the feature map, that is, increasing the weights of the regions of interest of the deep feature F C , and at the same time reducing the weights of the non-interested regions, so that the convolutional neural model learns more important information and improves the accuracy of the training output.

[0069] The specific principle of the attention mechanism unit is as follows: First, use the first convolutional layer for processing, with a convolution kernel size of 3×3, and then activate it with ReLU; then, use the second convolutional layer for processing, at this time the convolution kernel size is 1×1, and then use the Sigmoid function for activation to map the variable to the range of (0, 1) and convert it into the target weight coefficient; finally, multiply the output target weight coefficient with the deep feature F C pixel by pixel to obtain the output F of the attention mechanism unit A .

[0070] By the above method, a network structure combining dense connection and residual connection is adopted, and the attention mechanism unit is used to increase feature reuse, so that the shallow features and deep features are fully fused, and at the same time the weights of the regions of interest are increased to improve the accuracy of model convergence.

[0071] Step 3, adopt a multi-supervised objective loss function.

[0072] To improve the edge information of the deblurred image as much as possible, a method of simultaneously supervising with multiple loss functions is adopted to reduce the error between the deblurred image I deblurred output by the convolutional neural network and the clear image I label . The specific implementation method is as follows:

[0073] Use the first loss function (that is, the L1 loss function) L1 = ∑|I deblurred - I label |;

[0074] Use the second loss function (that is, the perceptual loss function) L2 = ∑(f(I deblurred ) - f(Ilabel )) 2 , where I deblurred and I label both use the first 16 convolutional layers of the Visual Geometry Group (VGG) network for high-dimensional feature extraction to obtain feature maps, and the outputs are f(I deblurred ) and f(I label ) respectively. Then, the mean square error between the two is calculated, which is the second loss function.

[0075] Different weights are set for the two loss functions and summed up to obtain the objective loss function loss = λ1×L1 + λ2×L2, where λ1 is the weight coefficient of the first loss function and λ2 is the weight coefficient of the second loss function.

[0076] Through the above method, using the L1 loss function and the perceptual loss function for joint supervision, the problem of excessive smoothing caused by simply using the L1 loss function is avoided, and the expression ability of high-frequency information such as edges is improved.

[0077] Step Four, training of the model.

[0078] After completing the above steps, the training of the model can be carried out. The blurred image is fed into the convolutional neural model, the learning rate and the preset iteration number threshold are set, and the weights and biases of each unit are randomly initialized. When the accuracy of the objective loss function between the output result of the convolutional neural model and the clear image reaches the preset requirement or the training reaches the preset iteration number threshold, the training ends and the trained model is saved.

[0079] Step Five, testing of the model.

[0080] In the testing stage, the blurred image to be processed is input into the trained convolutional neural model, and the output is the de-blurred clear image.

[0081] In the embodiment of the present application, by building a dataset for DSA images, adopting a network structure combining dense connection and residual connection, using the attention mechanism unit to increase the weight of the region of interest, and at the same time using the L1 loss function and the perceptual loss function for joint supervision, the de-blurring effect of DSA images is improved.

[0082] As a specific implementation of the above image de-blurring method, the embodiment of the present application provides an image de-blurring device. As Figure 5 shown, the image de-blurring device 500 includes: an acquisition module 501, a configuration module 502, and a de-blurring module 503.

[0083] Among them, the acquisition module 501 is used to acquire the image to be processed of the target object; the configuration module 502 is used to configure the trained convolutional neural model. The convolutional neural model includes a convolutional unit, multiple dense residual units, and an attention mechanism unit. The convolutional unit is used to extract the shallow features of the blurred image, the multiple dense residual units are used to extract the deep features of the blurred image based on the shallow features, and the attention mechanism unit is used to increase the weight of the region of interest of the deep features; the deblurring module 503 is used to input the image to be processed into the convolutional neural model to obtain the deblurred target image.

[0084] In this embodiment, first, the image to be processed of the target object is acquired, and then the trained convolutional neural model is determined. The convolutional neural model includes a convolutional unit, multiple dense residual units, and an attention mechanism unit. For the above-mentioned image to be processed, the convolutional unit can be used for feature extraction processing, and then the multiple dense residual units are connected in series to further extract features. After that, the attention mechanism unit is used to increase the weight of the region of interest of the extracted features. Finally, the trained convolutional neural model is used to perform deblurring processing on the image to be processed to obtain a clearer target image after deblurring.

[0085] In the embodiment of the present application, the deblurring effect of the image is improved through the above method, and the purpose of obtaining a clear image is achieved, so as to ensure that the subsequent research carried out by identifying the image is more accurate.

[0086] Further, the device further includes: an acquisition module, which is used to acquire a clear image of the sampling object under the exposure condition of the first focal point and a blurred image of the sampling object under the exposure condition of the second focal point, where the size of the first focal point is smaller than the size of the second focal point; a training module, which is used to: input the blurred image into the convolutional neural model for model training and output a deblurred image; when the accuracy of the target loss function between the deblurred image and the clear image is greater than or equal to the preset accuracy threshold, or the number of iterations of the model training is greater than or equal to the preset iteration number threshold, end the model training and save the convolutional neural model.

[0087] Further, the training module is specifically used to: extract the shallow features of the blurred image by using the convolutional unit; use the shallow features as the input and extract the deep features of the blurred image in a way that multiple dense residual units are connected in series; use the attention mechanism unit to increase the weight of the region of interest of the deep features; add the shallow features and the deep features pixel by pixel to obtain a deblurred image.

[0088] Further, the training module is specifically used to: input the blurred image into the convolutional neural model, set the learning rate, the preset iteration number threshold, and initialize the weights and biases of each unit of the convolutional neural model, perform model training, and output a deblurred image.

[0089] Further, the target loss function loss = λ1 × L1 + λ2 × L2, where L1 is the first loss function, L2 is the second loss function, λ1 is the weight coefficient of the first loss function, and λ2 is the weight coefficient of the second loss function; the first loss function L1 = ∑|I deblurred - I label |, and the second loss function L2 = ∑(f(I deblurred ) - f(I label )) 2 , where I deblurred is the deblurred image output by the convolutional neural model, I label is the clear image, f(I deblurred ) is the feature map after high-dimensional feature extraction of the deblurred image output by the convolutional neural model, and f(I label ) is the feature map after high-dimensional feature extraction of the clear image.

[0090] Further, the number of dense residual units is n, each dense residual unit includes m convolutional layers, the input of the j-th dense residual unit is F j-1 , and the output is F j , where j is an integer greater than or equal to 1 and less than or equal to n. When j equals 1, the input of the j-th dense residual unit is the shallow feature. When j is greater than 1, the input of the j-th dense residual unit is the output of the (j - 1)-th dense residual unit; for the j-th dense residual unit, the first convolutional layer takes F j-1 as the input and outputs the feature f1. The i-th convolutional layer takes F j-1 and the features output by all the previous i - 1 convolutional layers as the input and outputs the feature f i . Then, F j-1 , the feature f1, and all the features f i are concatenated to obtain f Cj . Finally, f Cj and F j-1 are added pixel by pixel to obtain the output F j of the j-th dense residual unit, where i is an integer greater than 1 and less than or equal to m.

[0091] Further, the attention mechanism unit includes a first convolutional layer, a rectified linear unit, a second convolutional layer, and a Sigmoid function unit; among them, the deep feature is used as the input of the attention mechanism unit, processed by the first convolutional layer, activated by the rectified linear unit, then processed by the second convolutional layer, and activated by the Sigmoid function unit to obtain the target weight coefficient. Finally, the target weight coefficient is multiplied pixel by pixel with the deep feature to obtain the output of the attention mechanism unit.

[0092] The image deblurring device 500 in the embodiments of the present application may be a computer device or a component in a computer device, such as an integrated circuit or a chip. The computer device may be a terminal or other devices other than terminals. Exemplarily, the computer device may be a tablet computer, a notebook computer, a handheld computer, an in-vehicle computer device, a Mobile Internet Device (MID), a robot, an Ultra-Mobile Personal Computer (UMPC), a netbook, or a Personal Digital Assistant (PDA), etc., and may also be a server, a Network Attached Storage (NAS), a Personal Computer (PC), etc. The embodiments of the present application do not make specific limitations.

[0093] The image deblurring device 500 in the embodiments of the present application may be a device with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems. The embodiments of the present application do not make specific limitations.

[0094] The image deblurring device 500 provided in the embodiments of the present application can implement Figure 1 and Figure 2 each process implemented by the image deblurring method embodiments. To avoid repetition, it will not be elaborated here.

[0095] The embodiments of the present application further provide a computer device. As Figure 6 shown, the computer device 600 includes a processor 601 and a memory 602. A program or instruction that can run on the processor 601 is stored on the memory 602. When the program or instruction is executed by the processor 601, it implements each step of the above image deblurring method embodiments and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0096] It should be noted that the computer devices in the embodiments of the present application include mobile computer devices and non-mobile computer devices.

[0097] The memory 602 can be used to store software programs and various data. The memory 602 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data. Among them, the first storage area may store an operating system, application programs or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 602 may include a volatile memory or a non-volatile memory, or the memory 602 may include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synch link dynamic random access memory (SLDRAM), and a direct rambus random access memory (DRRAM). The memory 602 in the embodiments of the present application includes but is not limited to these and any other suitable types of memories.

[0098] The processor 601 may include one or more processing units; optionally, the processor 601 integrates an application processor and a modem processor. Among them, the application processor mainly processes operations related to the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above modem processor may not be integrated into the processor 601 either.

[0099] The embodiments of the present application further provide a readable storage medium. A program or instruction is stored on the readable storage medium. When the program or instruction is executed by a processor, it implements each process of the above embodiments of the image deblurring method and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0100] An embodiment of the present application also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is configured to run programs or instructions to implement each process of the above embodiment of the image deblurring method, and can achieve the same technical effects. To avoid repetition, details are not described herein again.

[0101] It should be understood that the chip mentioned in the embodiment of the present application may also be referred to as a system-on-chip, system chip, chip system, or system-on-chip, etc.

[0102] An embodiment of the present application also provides a computer program product. The program product is stored in a storage medium and is executed by at least one processor to implement each process of the above embodiment of the image deblurring method, and can achieve the same technical effects. To avoid repetition, details are not described herein again.

[0103] It should be noted that in this document, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device including a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "including a..." does not exclude the presence of additional identical elements in the process, method, article or device including such element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed. It may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0104] The embodiments of the present application have been described above with reference to the accompanying drawings. However, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the spirit and scope protected by the present application and the claims, and all of them fall within the protection scope of the present application.

Claims

1. An image deblurring method, characterized in that, Including: Obtain a to-be-processed image of a target object, where the to-be-processed image is a blurred image, the target object includes a patient, and the to-be-processed image includes an angiogram image of the patient; Configure a trained convolutional neural model. Among them, the convolutional neural model includes a convolutional unit, a plurality of dense residual units, and an attention mechanism unit. The convolutional unit is used to extract shallow features of the blurred image, the plurality of dense residual units are used to extract deep features of the blurred image based on the shallow features, and the attention mechanism unit is used to increase the weight of the region of interest of the deep features; Input the to-be-processed image into the convolutional neural model to obtain a de-blurred target image, where the target image is used to identify the location and morphology of vascular lesions. Among them, use the convolutional unit of the convolutional neural model to extract shallow features of the to-be-processed image, use the shallow features as input, use the plurality of dense residual units of the convolutional neural model connected in series to extract deep features of the to-be-processed image, use the attention mechanism unit of the convolutional neural model to increase the weight of the region of interest of the deep features, and add the shallow features and the deep features pixel by pixel to obtain a de-blurred image; The number of the dense residual units is n, each of which includes m convolutional layers, and the input of the jth dense residual unit is F j-1 , the output is F j , where j is an integer greater than or equal to 1 and less than or equal to n, when j is equal to 1, the input of the j-th dense residual unit is the shallow feature, and when j is greater than 1, the input of the j-th dense residual unit is the output of the j-1-th dense residual unit; For the j-th dense residual unit, the first convolutional layer takes F j-1 as input and outputs a feature f1. The i-th convolutional layer takes F j-1 and the features output by the previous i - 1 convolutional layers as input and outputs a feature f i . Then, F j-1 , the feature f1, and all the features f i are concatenated to obtain f Cj . Finally, f Cj is added to F j-1 pixel by pixel to obtain the output F j of the j-th dense residual unit, where i is an integer greater than 1 and less than or equal to m; The attention mechanism unit includes a first convolutional layer, a rectified linear unit, a second convolutional layer, and a Sigmoid function unit; Among them, the deep features are used as the input of the attention mechanism unit, processed by the first convolutional layer, then activated by the rectified linear unit, then processed by the second convolutional layer, and then activated by the Sigmoid function unit to obtain a target weight coefficient. Finally, multiply the target weight coefficient and the deep features pixel by pixel to obtain the output of the attention mechanism unit.

2. The method according to claim 1, wherein Before configuring the trained convolutional neural model, it further includes: Collect a clear image of a sampling object under the exposure condition of a first focal point, and collect a blurred image of the sampling object under the exposure condition of a second focal point, where the size of the first focal point is smaller than the size of the second focal point; Input the blurred image into the convolutional neural model for model training, and output a de-blurred image; When the accuracy of the target loss function between the de-blurred image and the clear image is greater than or equal to a preset accuracy threshold, or the number of iterations of model training is greater than or equal to a preset iteration number threshold, end the model training and save the convolutional neural model.

3. The method according to claim 2, wherein The step of inputting the blurred image into the convolutional neural model for model training and outputting a de-blurred image includes: Use the convolutional unit to extract shallow features of the blurred image; Use the shallow features as input, and use the plurality of dense residual units connected in series to extract deep features of the blurred image; Use the attention mechanism unit to increase the weight of the region of interest of the deep features; Add the shallow features and the deep features pixel by pixel to obtain the de-blurred image.

4. The method according to claim 2, wherein The step of inputting the blurred image into the convolutional neural model for model training and outputting a de-blurred image includes: Input the blurred image into the convolutional neural model, set the learning rate, the preset iteration number threshold, and initialize the weights and biases of each unit of the convolutional neural model, and perform model training to output the de-blurred image.

5. The method according to claim 2, wherein the target loss function loss = λ1×L1 + λ2×L2, where L1 is the first loss function, L2 is the second loss function, λ1 is the weight coefficient of the first loss function, and λ2 is the weight coefficient of the second loss function; The first loss function L1 = ∑|I deblurred - I label |, and the second loss function L2 = ∑(f(I deblurred ) - f(I label )) 2 , where I deblurred is the deblurred image output by the convolutional neural model, I label is the clear image, f(I deblurred ) is the feature map after high-dimensional feature extraction of the deblurred image output by the convolutional neural model, and f(I label ) is the feature map after high-dimensional feature extraction of the clear image.

6. An image deblurring device, characterized in that, including: an acquisition module, configured to acquire a to-be-processed image of a target object, the to-be-processed image being a blurred image, the target object including a patient, and the to-be-processed image including an angiography image of the patient; a configuration module, configured to configure a trained convolutional neural model, where the convolutional neural model includes a convolutional unit, a plurality of dense residual units, and an attention mechanism unit, the convolutional unit is configured to extract shallow features of the blurred image, the plurality of dense residual units are configured to extract deep features of the blurred image based on the shallow features, and the attention mechanism unit is configured to increase the weight of the region of interest of the deep features; a de-blurring module, configured to input the to-be-processed image into the convolutional neural model to obtain a de-blurred target image, the target image being used to identify the location and shape of a vascular lesion; wherein, use the convolutional unit of the convolutional neural model to extract shallow features of the to-be-processed image, use the shallow features as input, use the plurality of dense residual units of the convolutional neural model connected in series to extract deep features of the to-be-processed image, use the attention mechanism unit of the convolutional neural model to increase the weight of the region of interest of the deep features, and add the shallow features and the deep features pixel by pixel to obtain a de-blurred image; The number of the dense residual units is n, each of the dense residual units includes m convolutional layers, and the input of the j-th dense residual unit is F j-1 , and the output is F j , where j is an integer greater than or equal to 1 and less than or equal to n. When j is equal to 1, the input of the j-th dense residual unit is the shallow feature. When j is greater than 1, the input of the j-th dense residual unit is the output of the (j - 1)-th dense residual unit; For the j-th dense residual unit, the first convolutional layer takes F j-1 as the input and outputs the feature f1. The i-th convolutional layer takes F j-1 and the features output by the previous i - 1 convolutional layers as the input and outputs the feature f i . Then, F j-1 , the feature f1, and all the features f i are concatenated to obtain f Cj . Finally, f Cj is added to F j-1 pixel by pixel to obtain the output F j of the j-th dense residual unit, where i is an integer greater than 1 and less than or equal to m; the attention mechanism unit includes a first convolutional layer, a rectified linear unit, a second convolutional layer, and a Sigmoid function unit; wherein, the deep features are used as the input of the attention mechanism unit, processed by the first convolutional layer, then activated by the rectified linear unit, then processed by the second convolutional layer, and then activated by the Sigmoid function unit to obtain a target weight coefficient, and finally multiply the target weight coefficient and the deep features pixel by pixel to obtain the output of the attention mechanism unit.

7. A computer device, characterized in that, including a processor and a memory, the memory stores a program or instruction running on the processor, and when the program or instruction is executed by the processor, the steps of the image de-blurring method according to any one of claims 1 to 5 are implemented.

8. A readable storage medium, on which a program or instructions are stored, characterized in that, When the program or instruction is executed by the processor, the steps of the image de-blurring method according to any one of claims 1 to 5 are implemented.

Images