Image blind deblurring method and device, computer equipment, storage medium and computer program product

Abstract

The invention relates to an image blind deblurring method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a semantic mask of a to-be-processed image; extracting image features of the to-be-processed image through a trained image blind deblurring model, and performing fusion processing on the image features and the semantic mask to obtain fusion features; the trained image blind deblurring model is obtained by training a to-be-trained image blind deblurring model by using a pre-trained aesthetic attribute distribution prediction model; performing multiple fusion processing on the fusion feature and the image feature of the fusion feature to obtain a target fusion feature of the to-be-processed image; and performing image reconstruction processing on the image features of the target fusion features to obtain a deblurred image of the to-be-processed image. By adopting the method, the definition and aesthetic quality of the deblurred image can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer device, computer-readable storage medium and computer program product for blind image deblurring. Background Art

[0002] At present, in order to improve the overall visual quality of images, it is crucial to accurately deblur images.

[0003] In traditional technology, in the process of deblurring an image, the method generally adopted is to improve the clarity of the image; however, this method is likely to cause problems such as missing texture details, color deviation, and visual over-sharpening in the image, which in turn causes the aesthetic quality of the deblurred image to be low. Summary of the invention

[0004] Based on this, it is necessary to provide an image blind deblurring method, apparatus, computer device, computer-readable storage medium and computer program product that can improve the clarity and aesthetic quality of the deblurred image in response to the above technical problems.

[0005] In a first aspect, the present application provides an image blind deblurring method, comprising:

[0006] Get the semantic mask of the image to be processed;

[0007] The trained image blind deblurring model is used to extract the image features of the image to be processed, and the image features and the semantic mask are fused to obtain fused features; the trained image blind deblurring model is obtained by training the image blind deblurring model to be trained using a pre-trained aesthetic attribute distribution prediction model;

[0008] Performing multiple fusion processes on the fusion feature and the image feature of the fusion feature to obtain a target fusion feature of the image to be processed;

[0009] Perform image reconstruction processing on the image features of the target fusion features to obtain a deblurred image of the image to be processed.

[0010] In one embodiment, fusing the image feature and the semantic mask to obtain the fused feature includes:

[0011] Performing spatial attention mechanism processing on the image feature to obtain a first processed feature, performing channel attention mechanism processing on the first processed feature to obtain a first key feature corresponding to the image feature, and extracting a second key feature corresponding to the semantic mask;

[0012] Performing splicing processing on the first key feature and the second key feature to obtain a splicing feature;

[0013] The image features of the splicing features are activated to obtain the fusion features.

[0014] In one embodiment, the performing of spatial attention mechanism processing on the image feature to obtain a first processed feature includes:

[0015] Determining an attention weight matrix corresponding to the image feature;

[0016] The image features are grouped to obtain a plurality of sub-image features, and the attention weight matrix is ​​grouped to obtain a sub-attention weight matrix corresponding to each sub-image feature;

[0017] Each sub-image feature and the sub-attention weight matrix corresponding to each sub-image feature are concatenated to obtain the first processed feature.

[0018] In one embodiment, performing a channel attention mechanism on the first processed feature to obtain a first key feature corresponding to the image feature includes:

[0019] Performing feature extraction processing on the first processed features to obtain second processed features;

[0020] Performing dimensionality reduction processing on the second processed features to obtain global features;

[0021] Activating the global feature to obtain an activated feature;

[0022] The second processed feature and the activated feature are fused to obtain the first key feature.

[0023] In one embodiment, the trained image blind deblurring model is trained in the following manner:

[0024] Obtaining a first sample semantic mask of the first sample image;

[0025] The sample image features of the first sample image are extracted by the image blind deblurring model to be trained, and the sample image features and the first sample semantic mask are fused to obtain sample fusion features; the image blind deblurring model to be trained includes a pre-trained encoding module and a reconstruction module to be trained; the pre-trained encoding module is a pre-trained encoding module in the pre-trained aesthetic attribute distribution prediction model;

[0026] Performing multiple fusion processes on the sample fusion feature and the sample image feature of the sample fusion feature to obtain a target sample fusion feature of the first sample image;

[0027] Performing image reconstruction processing on the sample image features of the target sample fusion features to obtain a first predicted deblurred image of the first sample image;

[0028] A first actual deblurred image of the first sample image is obtained, and the image blind deblurring model to be trained is iteratively trained according to the first predicted deblurred image and the first actual deblurred image to obtain the trained image blind deblurring model.

[0029] In one embodiment, the iterative training of the image blind deblurring model to be trained according to the first predicted deblurred image and the first actual deblurred image to obtain the trained image blind deblurring model includes:

[0030] By using the pre-trained aesthetic attribute distribution prediction model, based on the first sample semantic mask, a first aesthetic quality score value and a first aesthetic attribute distribution value corresponding to the first predicted deblurred image, as well as a second aesthetic quality score value and a second aesthetic attribute distribution value corresponding to the first actual deblurred image are obtained; the pre-trained aesthetic attribute distribution prediction model includes the pre-trained encoding module and the trained multi-layer perceptron;

[0031] obtaining a first aesthetic quality score loss value according to a difference between the first aesthetic quality score value and the second aesthetic quality score value, obtaining a first aesthetic attribute distribution loss value according to a difference between the first aesthetic attribute distribution value and the second aesthetic attribute distribution value, and obtaining a mean square error loss value based on a difference between the first predicted deblurred image and the first actual deblurred image;

[0032] Performing weighted sum processing on the first aesthetic quality score loss value, the first aesthetic attribute distribution loss value, and the mean square error loss value to obtain a first target loss value;

[0033] According to the first target loss value, the image blind deblurring model to be trained is iteratively trained to obtain the trained image blind deblurring model; the weight of the pre-trained encoding module in the pre-trained aesthetic attribute distribution prediction model and the weight of the pre-trained encoding module in the image blind deblurring model to be trained are updated synchronously.

[0034] In one embodiment, the pre-trained aesthetic attribute distribution prediction model is trained in the following manner:

[0035] Obtaining a second sample semantic mask, a second predicted deblurred image, and a second actual deblurred image of the second sample image;

[0036] obtaining, by means of an aesthetic attribute distribution prediction model to be trained and based on the second sample semantic mask, a third aesthetic quality score value and a third aesthetic attribute distribution value corresponding to the second predicted deblurred image, and a fourth aesthetic quality score value and a fourth aesthetic attribute distribution value corresponding to the second actual deblurred image; the aesthetic attribute distribution prediction model to be trained includes a coding module to be trained and a multi-layer perceptron to be trained;

[0037] obtaining a second aesthetic quality score loss value according to a difference between the third aesthetic quality score value and the fourth aesthetic quality score value, and obtaining a second aesthetic attribute distribution loss value according to a difference between the third aesthetic attribute distribution value and the fourth aesthetic attribute distribution value;

[0038] Performing a weighted sum processing on the second aesthetic quality score loss value and the second aesthetic attribute distribution loss value to obtain a second target loss value;

[0039] According to the second target loss value, the aesthetic attribute distribution prediction model to be trained is iteratively trained to obtain the pre-trained aesthetic attribute distribution prediction model.

[0040] In a second aspect, the present application also provides an image blind deblurring device, comprising:

[0041] A mask acquisition module, used to acquire the semantic mask of the image to be processed;

[0042] A feature processing module is used to extract the image features of the image to be processed through the trained image blind deblurring model, and fuse the image features with the semantic mask to obtain fused features; the trained image blind deblurring model is obtained by training the image blind deblurring model to be trained using a pre-trained aesthetic attribute distribution prediction model;

[0043] A feature fusion module, used for fusing the fusion feature and the image feature of the fusion feature for multiple times to obtain a target fusion feature of the image to be processed;

[0044] The image reconstruction module is used to perform image reconstruction processing on the image features of the target fusion features to obtain a deblurred image of the image to be processed.

[0045] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0046] Get the semantic mask of the image to be processed;

[0047] The trained image blind deblurring model is used to extract the image features of the image to be processed, and the image features and the semantic mask are fused to obtain fused features; the trained image blind deblurring model is obtained by training the image blind deblurring model to be trained using a pre-trained aesthetic attribute distribution prediction model;

[0048] Performing multiple fusion processes on the fusion feature and the image feature of the fusion feature to obtain a target fusion feature of the image to be processed;

[0049] Perform image reconstruction processing on the image features of the target fusion features to obtain a deblurred image of the image to be processed.

[0050] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0051] Get the semantic mask of the image to be processed;

[0052] The trained image blind deblurring model is used to extract the image features of the image to be processed, and the image features and the semantic mask are fused to obtain fused features; the trained image blind deblurring model is obtained by training the image blind deblurring model to be trained using a pre-trained aesthetic attribute distribution prediction model;

[0053] Performing multiple fusion processes on the fusion feature and the image feature of the fusion feature to obtain a target fusion feature of the image to be processed;

[0054] Perform image reconstruction processing on the image features of the target fusion features to obtain a deblurred image of the image to be processed.

[0055] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0056] Get the semantic mask of the image to be processed;

[0057] The trained image blind deblurring model is used to extract the image features of the image to be processed, and the image features and the semantic mask are fused to obtain fused features; the trained image blind deblurring model is obtained by training the image blind deblurring model to be trained using a pre-trained aesthetic attribute distribution prediction model;

[0058] Performing multiple fusion processes on the fusion feature and the image feature of the fusion feature to obtain a target fusion feature of the image to be processed;

[0059] Perform image reconstruction processing on the image features of the target fusion features to obtain a deblurred image of the image to be processed.

[0060] The above-mentioned image blind deblurring method, device, computer equipment, storage medium and computer program product first use a pre-trained aesthetic attribute distribution prediction model to train the image blind deblurring model to be trained, obtain the trained image blind deblurring model, and obtain the semantic mask of the image to be processed, extract the image features of the image to be processed through the trained image blind deblurring model, then fuse the image features and the semantic mask to obtain the fused features, then fuse the fused features and the image features of the fused features multiple times to obtain the target fused features of the image to be processed, and finally, reconstruct the image features of the target fused features to obtain the deblurred image of the image to be processed. In this way, in the process of deblurring the image, the image features and semantic masks of the image to be processed are fused through the image blind deblurring model, so that features that are more in line with the real semantic information of the image to be processed can be captured, and through multiple fusion processes, the prior information of the semantic mask can be fully utilized, so that the deblurred image obtained after image reconstruction is more accurate, which is beneficial to improving the clarity of the deblurred image; moreover, the image blind deblurring model is trained using a pre-trained aesthetic attribute distribution prediction model, so the aesthetic quality of the deblurred image can be improved through the image blind deblurring model; in addition, the entire deblurring process avoids the lack of aesthetic constraints in traditional technologies while improving clarity, resulting in problems such as missing texture details, color deviation, and visual over-sharpening in the image, which in turn causes the defect of low aesthetic quality of the deblurred image, further improving the clarity and aesthetic quality of the deblurred image. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 1 is a flow chart of a blind image deblurring method according to an embodiment;

[0063] Figure 2 is a flow chart of an image blind deblurring method based on aesthetic quality guidance in one embodiment;

[0064] Figure 3 A schematic flow chart of the steps of obtaining fusion features in one embodiment;

[0065] Figure 4 It is a structural block diagram of a feature fusion module SFF based on SAM large model prior in one embodiment;

[0066] Figure 5 is a structural block diagram of a spatial attention submodule of a feature fusion module in one embodiment;

[0067] Figure 6 is a structural block diagram of a general attention submodule of a feature fusion module in one embodiment;

[0068] Figure 7 is a structural block diagram of an aesthetic attribute distribution prediction module based on encoding module reuse in one embodiment;

[0069] Figure 8 is a flow chart of an image blind deblurring method according to another embodiment;

[0070] Fig. 9 is a structural block diagram of an image blind deblurring device in one embodiment;

[0071] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0073] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0074] In an exemplary embodiment, Figure 1As shown, a method for blind deblurring an image is provided. This embodiment uses the method applied to a server as an example for illustration; it can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be but is not limited to various personal computers, laptops, smart phones and tablet computers; the server can be implemented with an independent server or a server cluster composed of multiple servers. In this embodiment, the method includes the following steps:

[0075] Step S101, obtaining a semantic mask of an image to be processed.

[0076] The image to be processed refers to an image that needs to be processed. In actual scenarios, the image to be processed refers to a blurred image.

[0077] The semantic mask is used to represent the result representation obtained after semantic segmentation of the image to be processed. In actual scenarios, the semantic mask refers to the semantic mask map predicted by the SAM (Segment Anything Model, an image segmentation model) large model.

[0078] It should be noted that the SAM large model is an image segmentation model that can automatically segment any object in an image without labeling or additional training, and is robust to damaged images.

[0079] Exemplarily, the server obtains multiple candidate images and the clarity of each candidate image; then, the server filters out candidate images with clarity less than a preset clarity from each candidate image as images to be processed; then, the server inputs the images to be processed into the image segmentation model, and predicts the semantic mask map corresponding to the images to be processed through the image segmentation model as the semantic mask of the images to be processed.

[0080] For example, see Figure 2 ,The server inputs the blurred image into the SAM large model, and obtains the semantic mask of the blurred image through the SAM large model.

[0081] Step S102, extracting image features of the image to be processed through the trained image blind deblurring model, fusing the image features and the semantic mask to obtain fused features; the trained image blind deblurring model is obtained by training the image blind deblurring model to be trained using the pre-trained aesthetic attribute distribution prediction model.

[0082] The image blind deblurring model refers to a network model that can perform deblurring on the image to be processed. In actual scenarios, the image blind deblurring model refers to a deblurring network, such as DeblurGAN, DeblurGAN v2, Uformer, etc. It should be noted that the image blind deblurring model includes but is not limited to the models mentioned above.

[0083] The image feature of the image to be processed refers to feature information obtained by performing feature extraction processing on the image to be processed.

[0084] The fused feature refers to the feature information obtained by fusing the image feature and the semantic mask.

[0085] The aesthetic attribute distribution prediction model refers to a network model that can obtain the aesthetic quality score value corresponding to the image and the distribution value of the aesthetic attribute based on the semantic mask of the image. It should be noted that the aesthetic attribute distribution prediction model is also called an aesthetic attribute distribution prediction module.

[0086] Exemplarily, the server uses a pre-trained aesthetic attribute distribution prediction model to train the image blind deblurring model to be trained to obtain a trained image blind deblurring model, and inputs the image to be processed into the trained image blind deblurring model, and performs feature extraction processing on the image to be processed through the trained image blind deblurring model to obtain image features of the image to be processed; then, the server fuses the image features and the semantic mask to obtain fused features.

[0087] For example, see Figure 2 , the server inputs the image to be processed into the trained deblurring network, and obtains the image features of the image to be processed through the first feature extraction layer in the trained deblurring network; then, the server fuses the image features and the semantic mask through the first SFF (Select Feature Fusion) module to obtain the fused features.

[0088] Step S103, performing multiple fusion processing on the fusion features and the image features of the fusion features to obtain target fusion features of the image to be processed.

[0089] The image feature of the fused feature refers to feature information obtained by performing feature extraction processing on the fused feature.

[0090] The target fusion feature refers to feature information obtained by fusing the fusion feature and the image feature of the fusion feature multiple times.

[0091] Exemplarily, the server judges the fused feature; when the fused feature is not the fused feature output by the last feature fusion module, the server performs feature extraction processing on the fused feature to obtain the image feature of the fused feature; then, the server fuses the fused feature and the image feature of the fused feature until the obtained fused feature is the fused feature output by the last feature fusion module, and the server uses the fused feature as the target fused feature.

[0092] For example, see Figure 2 , the server inputs the fused features into the trained deblurring network, and obtains the image features of the fused features through the second feature extraction layer in the trained deblurring network; then, the server fuses the fused features and the image features of the fused features through the second SFF module (the second SFF module is the last SFF module at this time) to obtain the target fused features.

[0093] Step S104, performing image reconstruction processing on the image features of the target fusion features to obtain a deblurred image of the image to be processed.

[0094] The image feature of the target fusion feature refers to feature information obtained by performing feature extraction processing on the target fusion feature.

[0095] The deblurred image refers to an image obtained by performing image reconstruction processing on the image features of the target fusion features.

[0096] Exemplarily, the server preprocesses the image features of the target fusion features to obtain preprocessed image features; then, the server inputs the preprocessed image features into the trained image blind deblurring model, and performs multiple deconvolution processes on the preprocessed image features through the trained image blind deblurring model to obtain a deblurred image of the image to be processed.

[0097] For example, see Figure 2 , the image features of the target fusion features are input into the trained deblurring network, and the image features of the target fusion features are subjected to multiple deconvolution processes through the reconstruction module in the trained deblurring network to obtain the deblurred image of the image to be processed.

[0098] In the above-mentioned image blind deblurring method, a pre-trained aesthetic attribute distribution prediction model is first used to train the image blind deblurring model to be trained to obtain a trained image blind deblurring model, and a semantic mask of the image to be processed is obtained. The image features of the image to be processed are extracted by the trained image blind deblurring model, and then the image features and the semantic mask are fused to obtain fused features. Subsequently, the fused features and the image features of the fused features are fused multiple times to obtain the target fused features of the image to be processed. Finally, the image features of the target fused features are reconstructed to obtain a deblurred image of the image to be processed. In this way, in the process of deblurring the image, the image features and semantic masks of the image to be processed are fused through the image blind deblurring model, so that features that are more in line with the real semantic information of the image to be processed can be captured, and through multiple fusion processes, the prior information of the semantic mask can be fully utilized, so that the deblurred image obtained after image reconstruction is more accurate, which is beneficial to improving the clarity of the deblurred image; moreover, the image blind deblurring model is trained using a pre-trained aesthetic attribute distribution prediction model, so the aesthetic quality of the deblurred image can be improved through the image blind deblurring model; in addition, the entire deblurring process avoids the lack of aesthetic constraints in traditional technologies while improving clarity, resulting in problems such as missing texture details, color deviation, and visual over-sharpening in the image, which in turn causes the defect of low aesthetic quality of the deblurred image, further improving the clarity and aesthetic quality of the deblurred image.

[0099] In an exemplary embodiment, Figure 3 As shown, the above step S102, fusing the image features and the semantic mask to obtain the fused features, specifically includes the following steps:

[0100] Step S301, perform spatial attention mechanism processing on the image feature to obtain a first processed feature, perform channel attention mechanism processing on the first processed feature to obtain a first key feature corresponding to the image feature, and extract a second key feature corresponding to the semantic mask.

[0101] Step S302, performing splicing processing on the first key feature and the second key feature to obtain a spliced ​​feature.

[0102] Step S303, activating the image features of the spliced ​​features to obtain fused features.

[0103] Among them, the first processed feature refers to the feature information obtained by performing spatial attention mechanism processing on the image feature.

[0104] The first key feature is used to represent important areas and key features in the image to be processed.

[0105] The second key feature is used to represent important features related to the content and semantics of the image to be processed.

[0106] The splicing feature refers to feature information obtained by splicing the first key feature and the second key feature.

[0107] Exemplarily, the server performs spatial attention mechanism processing on the image feature to obtain the feature processed by the spatial attention mechanism as the first processed feature; then, the server performs channel attention mechanism processing on the first processed feature to obtain the feature processed by the channel attention mechanism as the first key feature corresponding to the image feature; then, the server performs feature extraction processing on the semantic mask to obtain the image feature of the semantic mask, and performs activation processing and feature extraction processing again on the image feature of the semantic mask to obtain the second key feature corresponding to the semantic mask; then, the server performs splicing processing on the first key feature and the second key feature in a preset splicing order to obtain the splicing feature; then, the server performs feature extraction processing on the splicing feature to obtain the image feature of the splicing feature, and performs activation processing on the image feature of the splicing feature to obtain the activated feature as the fusion feature.

[0108] For example, see Figure 4 , the server inputs the image features into the SAM large model, and processes the image features with the spatial attention mechanism through the spatial attention module in the first SFF module, and obtains the features processed by the spatial attention mechanism as the first processed features; then, the server processes the first processed features with the channel attention mechanism through the channel attention module in the first SFF module, and obtains the features processed by the channel attention mechanism as the first key features corresponding to the image features; then, the server performs feature extraction processing on the semantic mask through the convolution layer in the first SFF module to obtain the image features of the semantic mask, and performs activation processing and feature extraction processing again on the image features of the semantic mask through the activation layer and convolution layer in the first SFF module to obtain the second key features corresponding to the semantic mask; then, the server performs splicing processing on the first key feature and the second key feature according to the preset splicing order to obtain the splicing feature; then, the server performs feature extraction processing on the splicing feature through the convolution layer in the first SFF module to obtain the image features of the splicing feature, and performs GELU (Gaussian Error Linear Unit, Gaussian error linear unit) activation layer, activates the image features of the spliced ​​features, and obtains the activated features as the fusion features.

[0109] In this embodiment, the image features are processed by spatial attention mechanism and channel attention mechanism to obtain the first key feature corresponding to the image feature, and combined with the second key feature corresponding to the semantic mask, so as to obtain a more comprehensive fusion feature, thereby improving the accuracy of determining the fusion feature and providing an accurate data basis for subsequent data analysis.

[0110] In an exemplary embodiment, the above step S201 performs spatial attention mechanism processing on the image features to obtain the first processed features, which specifically includes the following contents: determining the attention weight matrix corresponding to the image features; grouping the image features to obtain multiple sub-image features, and grouping the attention weight matrix to obtain a sub-attention weight matrix corresponding to each sub-image feature; and splicing each sub-image feature and the sub-attention weight matrix corresponding to each sub-image feature to obtain the first processed feature.

[0111] Among them, the attention weight matrix is ​​used to represent the numerical matrix corresponding to the weights of image features.

[0112] The sub-image feature refers to a part of the image feature obtained by grouping the image features.

[0113] Among them, the sub-attention weight matrix refers to the partial attention weight matrix obtained by grouping the attention weight matrix, which can be the matrix information of each column in the attention weight matrix.

[0114] Exemplarily, the server performs dimensionality reduction, feature extraction, and activation processing on the image features to obtain an attention weight matrix corresponding to the image features; then, the server groups the image features to obtain multiple sub-image features, and groups the attention weight matrix to obtain a sub-attention weight matrix corresponding to each sub-image feature; then, the server separately splices each sub-image feature and the sub-attention weight matrix corresponding to each sub-image feature, and performs feature extraction on the spliced ​​features to obtain a first processed feature.

[0115] For example, see Figure 5, the server inputs the image features into the first SFF module, and performs global average pooling (GAP, Global Average Pooling) processing, feature extraction processing and activation processing on the image features through the global average pooling module, the first convolution module and the Tanh (HyperbolicTangent, hyperbolic tangent) module in the spatial attention module in the first SFF module to obtain the attention weight matrix corresponding to the image features; then, the server groups the image features through the channel grouping module in the first SFF module to obtain multiple sub-image features, and groups the attention weight matrix to obtain the sub-attention weight matrix corresponding to each sub-image feature; then, the server splices each sub-image feature and the sub-attention weight matrix corresponding to each sub-image feature, and extracts the spliced ​​features through the second convolution module in the spatial attention module to obtain the first processed feature.

[0116] For example, the server can obtain the attention weight matrix through the following formula:

[0117] , formula (1)

[0118] in, refers to the attention weight matrix, It refers to the weight matrix of the first convolution module in the spatial attention module. * indicates the convolution operation. Refers to image features.

[0119] For example, the server can obtain the first processed feature through the following formula:

[0120] , formula (2)

[0121] in, refers to the first processed feature, refers to the sub-attention weight matrices of different groups, , refers to different groups of sub-image features, , Refers to the splicing process, Refers to the weight matrix of the second convolutional module in the spatial attention module.

[0122] In this embodiment, by determining the attention weight matrix corresponding to the image features, and through grouping and splicing processing, the image features and the attention weight matrix can be more effectively integrated, thereby making the first processed features more comprehensive, which is beneficial to improving the determination accuracy of the first processed features.

[0123] In an exemplary embodiment, the above step S201, performs a channel attention mechanism on the first processed feature to obtain a first key feature corresponding to the image feature, which specifically includes the following contents: performing feature extraction on the first processed feature to obtain a second processed feature; performing dimensionality reduction on the second processed feature to obtain a global feature; performing activation on the global feature to obtain an activated feature; and performing fusion processing on the second processed feature and the activated feature to obtain the first key feature.

[0124] The second processed feature refers to feature information obtained by performing feature extraction processing on the first processed feature.

[0125] The global feature refers to feature information obtained by performing dimensionality reduction processing on the second processed feature.

[0126] The activated feature refers to the feature information obtained by activating the global feature.

[0127] Exemplarily, the server performs feature extraction processing on the first processed feature to obtain the second processed feature; then, the server performs dimensionality reduction processing on the second processed feature to obtain the reduced dimensionality feature as the global feature; then, the server performs activation processing on the global feature to obtain the activated feature; then, the server obtains the weight of the second processed feature and the weight of the activated feature, and fuses the second processed feature and the activated feature according to the weight of the second processed feature and the weight of the activated feature to obtain the first key feature.

[0128] For example, see Figure 6 , the server inputs the first processed feature into the first SFF module, and performs feature extraction on the first processed feature through the convolution module in the channel attention module in the first SFF module to obtain the second processed feature; then, the server performs dimensionality reduction on the second processed feature through the global average pooling module in the channel attention module to obtain the reduced dimensionality feature as the global feature; then, the server activates the global feature through the activation module (such as the sigmoid activation function) in the channel attention module to obtain the activated feature; then, the server fuses the second processed feature and the activated feature to obtain the first key feature.

[0129] For example, the server can obtain the first key feature through the following formula:

[0130] , formula (3)

[0131] in, refers to the first key feature, It refers to the convolution module in the channel attention module, which is the weight matrix of the 1×1 convolution layer. Represents sigmoid activation processing.

[0132] In this embodiment, the second processed feature is obtained by performing feature extraction processing on the first processed feature, and is fused with the activated feature obtained by performing dimensionality reduction processing and activation processing on the second processed feature, thereby achieving an organic combination of two types of different characteristic features, which is beneficial to improving the determination accuracy of the first key feature.

[0133] In an exemplary embodiment, the image blind deblurring method provided by the present application also includes a training step of a trained image blind deblurring model, which specifically includes the following contents: obtaining a first sample semantic mask of a first sample image; extracting sample image features of the first sample image through the image blind deblurring model to be trained, and fusing the sample image features and the first sample semantic mask to obtain sample fusion features; the image blind deblurring model to be trained includes a pre-trained encoding module and a reconstruction module to be trained; the pre-trained encoding module is a pre-trained encoding module in a pre-trained aesthetic attribute distribution prediction model; fusing the sample fusion features and the sample image features of the sample fusion features multiple times to obtain a target sample fusion feature of the first sample image; performing image reconstruction processing on the sample image features of the target sample fusion features to obtain a first predicted deblurred image of the first sample image; obtaining a first actual deblurred image of the first sample image, and iteratively training the image blind deblurring model to be trained based on the first predicted deblurred image and the first actual deblurred image to obtain a trained image blind deblurring model.

[0134] The first sample image refers to an image used to train the image blind deblurring model to be trained.

[0135] The first sample semantic mask is used to represent the result representation obtained after semantic segmentation of the first sample image.

[0136] The sample image feature of the first sample image refers to feature information obtained by performing feature extraction processing on the first sample image.

[0137] The sample fusion feature refers to feature information obtained by fusing the sample image feature and the first sample semantic mask.

[0138] The encoding module refers to a network module that performs encoding processing on images.

[0139] The reconstruction module refers to a network module that reconstructs images.

[0140] It should be noted that the image blind deblurring model to be trained includes a pre-trained encoding module and a reconstruction module to be trained, wherein the pre-trained encoding module is the same as the pre-trained encoding module in the pre-trained aesthetic attribute distribution prediction model, and both are reused encoding modules.

[0141] The sample image feature of the sample fusion feature refers to feature information obtained by performing feature extraction processing on the sample fusion feature.

[0142] The target sample fusion feature refers to feature information obtained by fusing the sample fusion feature and the sample image feature of the sample fusion feature multiple times.

[0143] The sample image feature of the target sample fusion feature refers to feature information obtained by performing feature extraction processing on the target sample fusion feature.

[0144] The first predicted deblurred image refers to a predicted value of a deblurred image corresponding to the first sample image, and is also called a deblurred image corresponding to the first sample image.

[0145] The first actual deblurred image refers to an actual value of a deblurred image corresponding to the first sample image, and is also called a clear image corresponding to the first sample image.

[0146] Exemplarily, in response to a model training instruction for an image blind deblurring model to be trained, the server determines an image to be trained on the image blind deblurring model to be trained as a first sample image; then, the server inputs the first sample image into an image segmentation model, and predicts a semantic mask map corresponding to the first sample image through the image segmentation model as a first sample semantic mask; then, the server inputs the first sample image into the image blind deblurring model to be trained, and extracts sample image features of the first sample image through a pre-trained encoding module in the image blind deblurring model to be trained; then, the server fuses the sample image features and the first sample semantic mask. The sample fusion feature is obtained; then, the server performs multiple fusion processing on the sample fusion feature and the sample image feature of the sample fusion feature to obtain the target sample fusion feature of the first sample image; then, the server performs image reconstruction processing on the sample image feature of the target sample fusion feature through the reconstruction module to be trained in the image blind deblurring model to be trained to obtain a first predicted deblurred image of the first sample image; then, the server obtains the first actual deblurred image of the first sample image from the database, and iteratively trains the image blind deblurring model to be trained according to the first predicted deblurred image and the first actual deblurred image to obtain a trained image blind deblurring model.

[0147] In this embodiment, by pre-training the image blind deblurring model, it is convenient to predict the deblurred image of the image to be processed after obtaining the image to be processed in practical applications; moreover, the image blind deblurring model receives new data in each round of iteration, and performs internal improvements and optimizations on the model, so that predictions can be made more effectively, which is beneficial to improving the prediction accuracy of the image blind deblurring model.

[0148] In an exemplary embodiment, an image blind deblurring model to be trained is iteratively trained according to a first predicted deblurred image and a first actual deblurred image to obtain a trained image blind deblurring model, which specifically includes the following contents: a first aesthetic quality score value and a first aesthetic attribute distribution value corresponding to the first predicted deblurred image, as well as a second aesthetic quality score value and a second aesthetic attribute distribution value corresponding to the first actual deblurred image are obtained through a pre-trained aesthetic attribute distribution prediction model based on a first sample semantic mask; the pre-trained aesthetic attribute distribution prediction model includes a pre-trained encoding module and a trained multi-layer perceptron; the first aesthetic quality score value and the second aesthetic attribute distribution value are obtained according to the difference between the first aesthetic quality score value and the second aesthetic quality score value. A quality score loss value is obtained according to the difference between the first aesthetic attribute distribution value and the second aesthetic attribute distribution value, and a mean square error loss value is obtained based on the difference between the first predicted deblurred image and the first actual deblurred image; the first aesthetic quality score loss value, the first aesthetic attribute distribution loss value and the mean square error loss value are weighted and summed to obtain a first target loss value; according to the first target loss value, the image blind deblurring model to be trained is iteratively trained to obtain a trained image blind deblurring model; the weights of the pre-trained encoding module in the pre-trained aesthetic attribute distribution prediction model and the weights of the pre-trained encoding module in the image blind deblurring model to be trained are updated synchronously.

[0149] Among them, the aesthetic quality score is generally used for aesthetic quality evaluation, focusing on the image's visual effects, composition, color, light and shadow, creativity and other aesthetic factors, and is related to whether the image is eye-catching, harmonious or artistic. It should be noted that the aesthetic quality score is also called the aesthetic score.

[0150] Among them, aesthetic attributes refer to a series of features that are strongly related to image quality and appearance, which are used to help the model predict the attractiveness or aesthetic score of the image in aesthetic evaluation, usually including image texture, clarity, contrast, color harmony, blur, etc.

[0151] The first aesthetic quality score value refers to the aesthetic quality score value corresponding to the first predicted deblurred image.

[0152] The first aesthetic attribute distribution value refers to the distribution value of the aesthetic attribute corresponding to the first predicted deblurred image.

[0153] The second aesthetic quality score value refers to the aesthetic quality score value corresponding to the first actual deblurred image.

[0154] The second aesthetic attribute distribution value refers to the distribution value of the aesthetic attribute corresponding to the first actual deblurred image.

[0155] Among them, the multi-layer perceptron refers to a network module used to determine the aesthetic score and aesthetic attributes corresponding to the image.

[0156] The first aesthetic quality score loss value refers to an aesthetic quality score loss value obtained according to the difference between the first aesthetic quality score value and the second aesthetic quality score value.

[0157] The first aesthetic attribute distribution loss value refers to an aesthetic attribute distribution loss value obtained according to the difference between the first aesthetic attribute distribution value and the second aesthetic attribute distribution value.

[0158] The mean square error loss value (MSE) refers to a loss value obtained based on the difference between the first predicted deblurred image and the first actual deblurred image.

[0159] Among them, the first target loss value refers to the loss value obtained by weighted summing the first aesthetic quality score loss value, the first aesthetic attribute distribution loss value and the mean square error loss value.

[0160] Exemplarily, the server inputs the first predicted deblurred image into the trained aesthetic attribute distribution prediction model, and obtains a first aesthetic quality score value and a first aesthetic attribute distribution value corresponding to the first predicted deblurred image based on the first sample semantic mask through the trained aesthetic attribute distribution prediction model; then, the server inputs the first actual deblurred image into the trained aesthetic attribute distribution prediction model, and obtains a second aesthetic quality score value and a second aesthetic attribute distribution value corresponding to the first actual deblurred image based on the first sample semantic mask through the trained aesthetic attribute distribution prediction model; then, the server obtains a first aesthetic quality score loss value based on the difference between the first aesthetic quality score value and the second aesthetic quality score value, and obtains a first aesthetic attribute distribution loss value based on the difference between the first aesthetic attribute distribution value and the second aesthetic attribute distribution value. The server calculates the loss value, and obtains the mean square error loss value based on the difference between the first predicted deblurred image and the first actual deblurred image; then, the server performs weighted summation processing on the first aesthetic quality score loss value, the first aesthetic attribute distribution loss value and the mean square error loss value according to the weight of the first aesthetic quality score loss value, the weight of the first aesthetic attribute distribution loss value and the weight of the mean square error loss value to obtain a first target loss value; then, the server adjusts the model parameters of the image blind deblurring model to be trained according to the first target loss value; then, the server re-trains the image blind deblurring model after the model parameters are adjusted until the first target loss value obtained by the trained image blind deblurring model is less than the first loss value threshold, then stops training, and uses the trained image blind deblurring model as the trained image blind deblurring model.

[0161] For example, see Figure 7 , the server inputs the deblurred image into the trained aesthetic attribute distribution prediction module, obtains the image features corresponding to the deblurred image and the first sample semantic mask based on the first sample semantic mask through the encoding module and the SFF module in the trained aesthetic attribute distribution prediction module, and obtains the aesthetic score and aesthetic attributes corresponding to the deblurred image through a multilayer perceptron (MLP); then, the server inputs the clear image into the trained aesthetic attribute distribution prediction module, obtains the image features corresponding to the clear image and the first sample semantic mask based on the first sample semantic mask through the encoding module and the SFF module in the trained aesthetic attribute distribution prediction module, and obtains the aesthetic score and aesthetic attributes corresponding to the clear image through a multilayer perceptron (MLP). It should be noted that the weights of the pre-trained encoding module in the pre-trained aesthetic attribute distribution prediction model and the weights of the pre-trained encoding module in the image blind deblurring model to be trained are updated synchronously.

[0162] For example, the server can obtain the aesthetic quality score loss value through the following formula:

[0163] , formula (4)

[0164] Where N is the total number of sample images for training the aesthetic attribute distribution prediction module, represents the aesthetic quality score of the i-th deblurred image, represents the aesthetic quality score of the i-th clear image.

[0165] For example, the server can obtain the aesthetic attribute distribution loss value through the following formula:

[0166] , formula (5)

[0167] Where N is the total number of sample images for training the aesthetic attribute distribution prediction module, M is the number of aesthetic attributes selected for each image, is the loss of a certain aesthetic attribute of each image, represents the distribution value of the jth aesthetic attribute of the i-th deblurred image, represents the distribution value of the jth aesthetic attribute of the i-th clear image, represents the empirical weight of the j-th aesthetic attribute.

[0168] It should be noted that the distribution value of the aesthetic attribute may be numerical or binary.

[0169] When the distribution of aesthetic attributes is expressed numerically, the distribution value of the aesthetic attribute can be calculated using the following formula:

[0170] , formula (6)

[0171] When the distribution of aesthetic attributes uses binary, the distribution value of the aesthetic attribute can be calculated by the following formula:

[0172] , formula (7)

[0173] For example, the server can obtain the mean square error loss value through the following formula:

[0174] , formula (8)

[0175] Where N is the total number of sample images for training the aesthetic attribute distribution prediction module, represents the i-th deblurred image The pixel value of the position, Represents the i-th clear image The pixel value at the location.

[0176] For example, the server can obtain the first target loss value through the following formula:

[0177] , formula (9)

[0178] in, Refers to the weight of the first aesthetic quality score loss value, which is used to represent the parameter for weighing the loss of the aesthetic quality score. It refers to the weight of the loss value of the first aesthetic attribute distribution, which is used to represent the parameter for weighing the loss of the aesthetic distribution.

[0179] In this embodiment, by using the aesthetic attribute distribution prediction model, the aesthetic quality score value and the aesthetic attribute distribution value of the deblurred image are obtained at the same time, so that the overall visual quality of the image can be improved from an aesthetic perspective, avoiding the problem that in the process of deblurring the image, only the method of improving the clarity of the image is used, which is easy to cause the image to have problems such as missing texture details, color deviation, visual over-sharpening, and thus resulting in the defect of low accuracy of image deblurring processing.

[0180] In an exemplary embodiment, the image blind deblurring method provided by the present application also includes a training step of a trained aesthetic attribute distribution prediction model, which specifically includes the following contents: obtaining a second sample semantic mask of the second sample image, a second predicted deblurred image, and a second actual deblurred image; obtaining a third aesthetic quality score value and a third aesthetic attribute distribution value corresponding to the second predicted deblurred image, and a fourth aesthetic quality score value and a fourth aesthetic attribute distribution value corresponding to the second actual deblurred image based on the second sample semantic mask through the aesthetic attribute distribution prediction model to be trained; the aesthetic attribute distribution prediction model to be trained includes a coding module to be trained and a multi-layer perceptron to be trained; obtaining a second aesthetic quality score loss value according to the difference between the third aesthetic quality score value and the fourth aesthetic quality score value, and obtaining a second aesthetic attribute distribution loss value according to the difference between the third aesthetic attribute distribution value and the fourth aesthetic attribute distribution value; performing weighted sum processing on the second aesthetic quality score loss value and the second aesthetic attribute distribution loss value to obtain a second target loss value; and iteratively training the aesthetic attribute distribution prediction model to be trained according to the second target loss value to obtain a pre-trained aesthetic attribute distribution prediction model.

[0181] The second sample image refers to an image used to train the aesthetic attribute distribution prediction model to be trained.

[0182] The second sample semantic mask is used to represent the result representation obtained after semantic segmentation of the second sample image.

[0183] The second predicted deblurred image refers to a predicted value of a deblurred image corresponding to the second sample image, and is also called a deblurred image corresponding to the second sample image.

[0184] The second actual deblurred image refers to the actual value of the deblurred image corresponding to the second sample image, and is also called the clear image corresponding to the second sample image.

[0185] The third aesthetic quality score value refers to the aesthetic quality score value corresponding to the second predicted deblurred image.

[0186] The third aesthetic attribute distribution value refers to the distribution value of the aesthetic attribute corresponding to the second predicted deblurred image.

[0187] The fourth aesthetic quality score value refers to the aesthetic quality score value corresponding to the second actual deblurred image.

[0188] The fourth aesthetic attribute distribution value refers to the distribution value of the aesthetic attribute corresponding to the second actual deblurred image.

[0189] The second aesthetic quality score loss value refers to the aesthetic quality score loss value obtained according to the difference between the third aesthetic quality score value and the fourth aesthetic quality score value.

[0190] The second aesthetic attribute distribution loss value refers to an aesthetic attribute distribution loss value obtained according to the difference between the third aesthetic attribute distribution value and the fourth aesthetic attribute distribution value.

[0191] The second target loss value refers to the loss value obtained by weighted summing the second aesthetic quality score loss value and the second aesthetic attribute distribution loss value.

[0192] Exemplarily, in response to a model training instruction for the aesthetic attribute distribution prediction model to be trained, the server determines an image to be trained for the aesthetic attribute distribution prediction model to be trained as a second sample image; then, the server inputs the second sample image into the image segmentation model, and predicts a semantic mask map corresponding to the second sample image through the image segmentation model as a second sample semantic mask; then, the server obtains a second predicted deblurred image and a second actual deblurred image of the second sample image from a database; then, the server inputs the second predicted deblurred image into the aesthetic attribute distribution prediction model to be trained, and obtains a third aesthetic quality score value and a third aesthetic attribute distribution value corresponding to the second predicted deblurred image based on the second sample semantic mask through the encoding module to be trained and the multi-layer perceptron to be trained in the aesthetic attribute distribution prediction model to be trained; then, the server inputs the second actual deblurred image into the aesthetic attribute distribution prediction model to be trained, and obtains a third aesthetic quality score value and a third aesthetic attribute distribution value corresponding to the second predicted deblurred image through the encoding module to be trained and the multi-layer perceptron to be trained in the aesthetic attribute distribution prediction model to be trained based on the second sample semantic mask; semantic mask, obtain a fourth aesthetic quality score value and a fourth aesthetic attribute distribution value corresponding to the second actual deblurred image; then, the server obtains a second aesthetic quality score loss value according to the difference between the third aesthetic quality score value and the fourth aesthetic quality score value, and obtains a second aesthetic attribute distribution loss value according to the difference between the third aesthetic attribute distribution value and the fourth aesthetic attribute distribution value; then, the server determines a preset weight, and performs weighted summation processing on the second aesthetic quality score loss value and the second aesthetic attribute distribution loss value according to the preset weight, to obtain a second target loss value; then, the server adjusts the model parameters of the aesthetic attribute distribution prediction model to be trained according to the second target loss value; then, the server re-trains the aesthetic attribute distribution prediction model after the model parameters are adjusted, until the second target loss value obtained by the trained aesthetic attribute distribution prediction model is less than the second loss value threshold, then stops training, and uses the trained aesthetic attribute distribution prediction model as the pre-trained aesthetic attribute distribution prediction model.

[0193] For example, the server can obtain the second target loss value through the following formula:

[0194] , formula (10)

[0195] in, Refers to the preset weights.

[0196] In this embodiment, by pre-training the aesthetic attribute distribution prediction model, it is convenient to directly predict the aesthetic quality score value and the aesthetic attribute distribution value of the image after acquiring the image during the training of the image blind deblurring model, thereby reducing the training time of the image blind deblurring model and facilitating improving the training efficiency of the image blind deblurring model.

[0197] In an exemplary embodiment, Figure 8 As shown, another image blind deblurring method is provided, which is described by taking the application of the method to a server as an example, and includes the following steps:

[0198] Step S801, obtaining a semantic mask of the image to be processed.

[0199] Step S802, extracting image features of the image to be processed through the trained image blind deblurring model; the trained image blind deblurring model is obtained by training the image blind deblurring model to be trained using a pre-trained aesthetic attribute distribution prediction model.

[0200] Step S803, determine the attention weight matrix corresponding to the image feature; group the image features to obtain multiple sub-image features, and group the attention weight matrix to obtain a sub-attention weight matrix corresponding to each sub-image feature.

[0201] Step S804, concatenating each sub-image feature and the sub-attention weight matrix corresponding to each sub-image feature to obtain a first processed feature.

[0202] Step S805, performing feature extraction processing on the first processed features to obtain second processed features; performing dimensionality reduction processing on the second processed features to obtain global features.

[0203] Step S806, performing activation processing on the global feature to obtain an activated feature; performing fusion processing on the second processed feature and the activated feature to obtain a first key feature.

[0204] Step S807: extract the second key feature corresponding to the semantic mask.

[0205] Step S808, performing splicing processing on the first key feature and the second key feature to obtain a spliced ​​feature.

[0206] Step S809, activating the image features of the spliced ​​features to obtain fused features.

[0207] Step S810, performing multiple fusion processing on the fusion features and the image features of the fusion features to obtain the target fusion features of the image to be processed.

[0208] Step S811, performing image reconstruction processing on the image features of the target fusion features to obtain a deblurred image of the image to be processed.

[0209] In the above-mentioned image blind deblurring method, in the process of deblurring the image, the image features and semantic masks of the image to be processed are fused through the image blind deblurring model, so that features that are more in line with the real semantic information of the image to be processed can be captured, and through multiple fusion processes, the prior information of the semantic mask can be fully utilized, so that the deblurred image obtained after the image reconstruction process is more accurate, which is beneficial to improving the clarity of the deblurred image; moreover, the image blind deblurring model is trained using a pre-trained aesthetic attribute distribution prediction model, so the aesthetic quality of the deblurred image can be improved through the image blind deblurring model; in addition, the entire deblurring process avoids the lack of aesthetic constraints in traditional technologies while improving clarity, resulting in problems such as missing texture details, color deviation, and visual over-sharpening in the image, which in turn causes the defect of low aesthetic quality of the deblurred image, further improving the clarity and aesthetic quality of the deblurred image.

[0210] In an exemplary embodiment, in order to more clearly illustrate the image blind deblurring method provided in the embodiment of the present application, the image blind deblurring method is specifically described below with a specific embodiment. Figure 2 As shown, the present application also provides an image blind deblurring method based on aesthetic quality guidance. In the process of deblurring the image, the pre-trained aesthetic attribute distribution prediction model is first used to train the image blind deblurring model to be trained, and the trained image blind deblurring model is obtained, and the semantic mask of the image to be processed is obtained. The image features of the image to be processed are extracted by the trained image blind deblurring model, and the image features and the semantic mask are fused to obtain fused features. Then, the fused features and the image features of the fused features are fused multiple times to obtain the target fused features of the image to be processed. Finally, the image features of the target fused features are reconstructed to obtain the deblurred image of the image to be processed. Specifically, it includes the following contents:

[0211] This embodiment proposes an image blind deblurring method based on aesthetic quality guidance. Based on an end-to-end deblurring network, the method consists of two main modules: a feature fusion module (SFF) based on the SAM large model prior and an aesthetic attribute distribution prediction module based on encoding module reuse.

[0212] Figure 2The overall framework of the algorithm is shown. This method maps the blurred image to the feature space through an end-to-end network, and then reconstructs the deblurred image using the features fused with prior information. The feature fusion module based on the SAM large model prior takes the blurred image features and fused features of the previous feature extraction layer (the input of the first feature fusion module is the semantic mask map predicted by SAM) as input, outputs the fused features as the input of the subsequent feature fusion module, and is further processed by the next feature extraction layer in the deblurring network, thereby effectively promoting the deep information interaction and optimization between features. The aesthetic attribute distribution prediction module based on the reuse of the encoding module inputs the deblurred image and the clear image into the encoding module for feature extraction based on the encoding module reuse strategy. Subsequently, the extracted features are processed by a multi-layer perceptron (MLP) module to predict the aesthetic score and aesthetic attribute distribution of the image. This process can quantify the aesthetic quality difference between the deblurred image and the clear image from the overall subjective perception level to the fine-grained visual information level, providing a more comprehensive aesthetic quality evaluation.

[0213] 1. Feature fusion module based on SAM large model prior:

[0214] like Figure 4 As shown in the figure, the feature fusion module based on the SAM large model prior first passes the features of the blurred image through the spatial attention module and the channel attention module to highlight the important areas and key features in the image; then the SAM semantic features (the first feature fusion module input is the semantic mask map predicted by SAM) are filtered out through a multi-layer convolutional network (with a RELU activation layer in the middle) to filter out important features related to the image content and semantics; then, the two feature information are concatenated, followed by feature mapping through the convolutional layer, and processed by the GELU activation function, and finally the fused features are output.

[0215] (1) Spatial Attention Submodule:

[0216] The first step is to calculate the attention weight matrix:

[0217] Blurred image features First, global average pooling (GAP) is used to generate global features, then a convolutional layer is used to adjust the channel dimension, and then a hyperbolic tangent activation function is used to suppress pixels with negative effects, and finally the attention weight matrix is ​​output. .

[0218] , formula (1)

[0219] in, It is the weight matrix of the convolution layer through which the global feature passes, and * represents the convolution operation.

[0220] The second step is to calculate the spatial attention weighted blurred image features:

[0221] Blurred image features The channel grouping module is divided into K groups in the channel dimension. Each group shares the same attention weight matrix in the channel and spatial dimensions. The output of the spatial attention submodule for:

[0222] , formula (2)

[0223] in, are the weight matrices of different groups, , are the characteristics of different groups, , Indicates a connection operation. is the weight matrix of the connected convolutional layer.

[0224] (2) Channel attention submodule:

[0225] First, the spatial attention weighted blurred image features The number of channels is adjusted through a 1×1 convolution layer, and then the global features are generated through global average pooling (GAP). The weight information of each channel is obtained through sigmoid, and the features after channel attention are obtained by multiplying them with the original features. .

[0226] , formula (3)

[0227] in, represents the weight matrix of the 1×1 convolutional layer, Represents the sigmoid operation.

[0228] Studies have shown that the semantic prior information obtained by segmentation can effectively improve the performance of image deblurring tasks. This kind of prior information contains key information such as texture and gradient features of different objects in the image. The algorithm proposed in this paper uses the large image segmentation model SAM trained with large-scale data. It has strong zero-sample generalization ability and can extract high-level semantic priors that are difficult for mainstream networks to obtain on images that do not conform to the distribution of training data, and shows good robustness on damaged images. SAM provides more refined semantic prior information than instance or category labels, and divides the image content in a fine-grained manner. Using this rich semantic prior information in the deblurring process can significantly enhance the effect of the deblurring task.

[0229] The feature fusion module based on the SAM large model prior integrates the semantic prior information of SAM at the feature level. Compared with the fixed or preset semantic prior used by traditional methods, it can dynamically provide fine-grained and high-resolution semantic features, automatically adapt to different image contents, and accurately capture the object boundaries, detail areas and global semantic information in the image. This flexible prior guidance improves the model's understanding of complex scenes, enhances spatial consistency and contextual relevance, makes the image deblurring task have stronger performance, and can restore finer texture details.

[0230] 2. Aesthetic attribute distribution prediction module based on encoding module reuse:

[0231] like Figure 7 As shown in the figure, the aesthetic attribute distribution prediction module based on encoding module reuse reuses the encoding module of the deblurring network so that the extracted features of the deblurred image and the clear image can better reflect the information that the deblurring network pays attention to. It is worth noting that when reusing the encoding module, the aesthetic attribute distribution prediction module also needs to integrate the SAM semantic mask features to improve the reuse accuracy. Then, the features of the deblurred image and the clear image extracted by the encoding module are input into a 3-layer MLP network to predict the aesthetic quality score and aesthetic attribute distribution of the image. Finally, the loss of the aesthetic quality score and the aesthetic attribute distribution is calculated, and the network weights of the aesthetic attribute distribution prediction module are updated based on the loss.

[0232] In addition to requiring the generated image to be clear, the deblurring task also requires that the image has aesthetic fidelity in the fields of digital photographic images, visual creative images, portraits, landscapes, etc. Considering the influence of each aesthetic attribute on the deblurred image, the candidate aesthetic attributes are selected as follows: brightness, structure, contrast, clarity, blur, color fidelity, color harmony, color richness, element balance, depth of field (expressed by the difference in clarity between foreground and background), etc. Important aesthetic attributes can be adaptively selected according to the actual application scenario to guide the encoding module to learn aesthetic prior knowledge.

[0233] The loss of aesthetic quality score is as follows:

[0234] , formula (4)

[0235] Where N is the total number of sample images for training the aesthetic attribute distribution prediction module, represents the aesthetic quality score of the i-th deblurred image, represents the aesthetic quality score of the i-th clear image.

[0236] The loss of aesthetic distribution is as follows:

[0237] , formula (5)

[0238] Where N is the total number of sample images for training the aesthetic attribute distribution prediction module, M is the number of aesthetic attributes selected for each image, is the loss of a certain aesthetic attribute of each image, represents the distribution value of the jth aesthetic attribute of the i-th deblurred image, represents the distribution value of the jth aesthetic attribute of the i-th clear image, represents the empirical weight of the jth aesthetic attribute. The distribution value of the aesthetic attribute can be either numerical or binary.

[0239] When the distribution of aesthetic attributes is expressed numerically, the distribution value of the aesthetic attribute can be calculated using the following formula:

[0240] , formula (6)

[0241] When the distribution of aesthetic attributes is binary, the distribution value of the aesthetic attribute can be calculated by the following formula:

[0242] , formula (7)

[0243] The aesthetic attribute distribution prediction module based on encoding module reuse extracts the features of the deblurred image and the clear image by reusing the encoding module of the deblurring network, thereby obtaining the key information that the encoding module focuses on. These feature information is then used to evaluate whether the encoding module has effectively learned the features that reflect the aesthetic quality. The encoding module reuse strategy implicitly utilizes semantic priors, which improves the interactivity between semantic priors and deblurring tasks. In the process of aesthetic quality prediction, the proposed aesthetic attribute distribution prediction module not only considers the aesthetic quality score representing the overall quality of the image, but also further introduces the aesthetic attribute distribution with higher granularity. The hierarchical information from the whole to the details provides a richer expression for the understanding of image aesthetics. In addition, this processing process is consistent with the aesthetic reasoning mode of the human visual system from rough to detailed, which improves the model's comprehensive analysis ability of image aesthetics.

[0244] 3. Determine the total loss function:

[0245] This algorithm is applicable to a variety of end-to-end deblurring networks, such as DeblurGAN, DeblurGAN v2, Uformer, etc. The embodiment of the present application uses the generator of DeblurGAN as the baseline of the end-to-end deblurring network, and introduces in detail the specific implementation steps of an image blind deblurring method based on aesthetic quality guidance.

[0246] The end-to-end deblurring network of this embodiment includes an encoding module and a reconstruction module. The encoding module includes A convolution blocks (the convolution size is 3×3, the step size is 2, and the channel is ), B residual blocks (convolution size is 3×3, stride is 1, channels are ), each residual block consists of a convolution layer, an instance normalization layer, and a RELU activation layer, and the reconstruction module includes A transposed convolution blocks. For the first B-1 residual blocks, the output features of each residual block and the SAM semantic mask features are input into the feature fusion module SFF, and the output of the SFF module is used as the input of the next residual block and one of the inputs of the next SFF module.

[0247] The end-to-end deblurring network generates a deblurred image. First, the MSE loss between the deblurred image and the clear image is calculated to constrain the pixel fidelity of the deblurred image. The MSE loss in the pixel domain is calculated as follows:

[0248] , formula (8)

[0249] Where N is the total number of images in each batch for training the end-to-end deblurring network, represents the i-th deblurred image The pixel value of the position, Represents the i-th clear image The pixel value at the location.

[0250] Then, the deblurred image and the clear image are input into the aesthetic attribute distribution prediction module to obtain the aesthetic quality score and aesthetic attribute distribution of the image, respectively, and then the loss of the aesthetic quality score and the loss of the aesthetic distribution are calculated.

[0251] Finally, the end-to-end deblurring network is optimized using the total loss function. The total loss function is:

[0252] , formula (9)

[0253] in, and is a parameter that weighs the loss of aesthetic quality score and the loss of aesthetic distribution.

[0254] 4. Pre-training aesthetic attribute distribution prediction module:

[0255] Step 1, input pre-training image:

[0256] Input image from the Aesthetic Quality Assessment Dataset ,This embodiment uses the AADB dataset as a pre-training dataset, which contains 10,000 images and 8 aesthetic attributes.

[0257] Step 2: Get the SAM semantic mask:

[0258] Use the automatic mode of the SAM large model to obtain the semantic mask of the image .

[0259] Step 3: Feature extraction and feature fusion:

[0260] Extract the image using the first feature extraction layer of the reused encoding module Features .

[0261] The features of the first feature extraction layer and semantic mask Input the first feature fusion module , get the fusion features , and the fusion features As the input of the second feature extraction layer.

[0262] Use the second feature extraction layer of the reused encoding module to obtain features .

[0263] The features of the second feature extraction layer and fusion features Input the second feature fusion module , get the fusion features , and the fusion features As the input of the third feature extraction layer.

[0264] Repeat the previous step until the features of the last feature extraction layer are output .

[0265] Step 4, prediction of aesthetic quality score and aesthetic attribute distribution:

[0266] The characteristics It is fed into a 3-layer MLP module to predict the aesthetic quality score and aesthetic attribute distribution of the image.

[0267] Step 5: Calculate the loss function of the quality score, attribute distribution and label predicted by the pre-trained network:

[0268] , formula (10)

[0269] Step 6, network training:

[0270] The standard stochastic gradient descent (SGD) algorithm is used for training to update the network weights.

[0271] Step 7, weight freezing:

[0272] Freeze the weights of the MLP module during formal training.

[0273] 5. Formal training process:

[0274] Step 1: Input the formal training image pair:

[0275] Input a blurred image from the deblurring dataset , clear image ,This embodiment uses the RSBlur dataset as the formal training dataset, which contains 13,358 real blurred images of 697 scenes and their corresponding clear images.

[0276] Step 2: Initialize the encoding module weights:

[0277] The weights of the encoding module pre-trained by the aesthetic attribute distribution prediction module are used as the initial weights of the deblurring network encoding module during formal training.

[0278] Step 3: Get the SAM semantic mask:

[0279] Using the automatic mode of the SAM large model to obtain semantic masks of blurred images .

[0280] Step 4: Feature extraction and feature fusion:

[0281] Use the first feature extraction layer of the reused encoding module to extract the blurred image Features .

[0282] The features of the first feature extraction layer and semantic mask Input the first feature fusion module , get the fusion features , and the fusion features As the input of the second feature extraction layer.

[0283] Use the second feature extraction layer of the reused encoding module to obtain features .

[0284] The features of the second feature extraction layer and fusion features Input the second feature fusion module , get the fusion features , and the fusion features As the input of the third feature extraction layer.

[0285] Repeat the previous step until the features of the last feature extraction layer are output .

[0286] Step 5, generate the deblurred image:

[0287] Blurred image features Input to the reconstruction module to generate a deblurred image.

[0288] Step 6, total loss function calculation:

[0289] Freeze the MLP weights of the aesthetic attribute distribution prediction module and calculate the total loss function of the deblurred image and the clear image .

[0290] Step 7, end-to-end training:

[0291] The Adam algorithm is used for end-to-end training to update the weights of the end-to-end deblurring network and the weights of the encoding module of the aesthetic attribute distribution prediction module (updated as the weights of the encoding module of the deblurring network are updated).

[0292] 6. Reasoning process:

[0293] Step 1: Remove the aesthetic attribute distribution prediction module:

[0294] In the inference stage, the domain aesthetic attribute distribution prediction module is removed, and only the end-to-end deblurring network, feature fusion module SFF and SAM network are retained for image deblurring.

[0295] Step 2: Input the test blurred image .

[0296] Step 3: Get the SAM semantic mask:

[0297] Use the automatic mode of the SAM large model to obtain the semantic mask of the test blurred image .

[0298] Step 4: Feature extraction and feature fusion:

[0299] Use the first feature extraction layer of the reused encoding module to extract the blurred image Features .

[0300] The features of the first feature extraction layer and semantic mask Input the first feature fusion module , get the fusion features , and the fusion features As the input of the second feature extraction layer.

[0301] Use the second feature extraction layer of the reused encoding module to obtain features .

[0302] The features of the second feature extraction layer and fusion features Input the second feature fusion module , get the fusion features , and the fusion features As the input of the third feature extraction layer.

[0303] Repeat the previous step until the features of the last feature extraction layer are output .

[0304] Step 5, generate the deblurred image:

[0305] Blurred image features Input to the reconstruction module to generate a deblurred image.

[0306] The above embodiments can achieve the following technical effects:

[0307] (1) In the process of deblurring the image, the image features and semantic masks of the image to be processed are fused through the image blind deblurring model, which can capture features that are more in line with the real semantic information of the image to be processed. Through multiple fusion processes, the prior information of the semantic mask is fully utilized, so that the deblurred image obtained after image reconstruction is more accurate, which is beneficial to improving the clarity of the deblurred image. Moreover, the image blind deblurring model is trained using a pre-trained aesthetic attribute distribution prediction model. Therefore, the aesthetic quality of the deblurred image can be improved through the image blind deblurring model. In addition, the entire deblurring process avoids the problem that the traditional technology lacks aesthetic constraints while improving clarity, resulting in problems such as missing texture details, color deviation, and visual over-sharpening, which in turn causes the low aesthetic quality of the deblurred image, further improving the clarity and aesthetic quality of the deblurred image.

[0308] (2) A blind image deblurring method based on aesthetic quality guidance is proposed, including a feature fusion method based on the SAM large model prior and an aesthetic attribute distribution prediction method based on the reuse of coding modules. The feature fusion method based on the SAM large model prior introduces the image segmentation large model SAM with rich semantic knowledge, and integrates the SAM image segmentation prior information into the coding module of the deblurring network through the feature fusion module, making full use of SAM's understanding of semantic concepts and context, and providing more accurate feature-level information prompts for the deblurring network. The aesthetic attribute distribution prediction method based on the reuse of coding modules introduces a coding module reuse strategy, which implicitly predicts the aesthetic attribute distribution features. Learning aesthetic quality priors enhances the interactivity between aesthetic quality priors and the deblurring task. In addition, through fine-grained image aesthetic attributes and aesthetic quality scores that reflect the overall aesthetics and quality of the image, the multi-dimensional aesthetic characteristics of the image are captured comprehensively from the overall to the details. By jointly utilizing feature-level prior information and label-level aesthetic quality constraints, the advantages of the SAM large model's prior knowledge of massive image data are fully utilized. At the same time, by reusing the encoding module to predict the distribution of aesthetic attributes and aesthetic quality scores, the capture of aesthetic priors at multiple levels and dimensions is achieved. It not only has a significant effect in improving the visual clarity and detail fidelity of the deblurred image, but also effectively enhances the overall aesthetic performance of the image.

[0309] (3) A blind image deblurring method guided by aesthetic quality is proposed. Through the proposed feature fusion module based on the SAM large model prior and the aesthetic attribute distribution prediction module based on the reuse of the encoding module, the semantic prior information reflecting the image texture, edge, contour, etc. provided by the SAM large model is effectively combined with the aesthetic prior information reflecting the image color, clarity, visual harmony, etc. provided by the aesthetic attributes, thus breaking through the limitations of the traditional single feature method. By jointly utilizing the feature-level prior information and the label-level aesthetic quality constraints, the deblurred image has higher fidelity in both objective quality and aesthetic quality.

[0310] (4) A method combining the semantic prior of the SAM large model and the prediction of the distribution of aesthetic attributes is proposed, so that image deblurring not only relies on semantic information, but also further optimizes the visual effect and overall aesthetic quality of the image through aesthetic quality scores and aesthetic attributes (such as color, visual harmony, etc.); through multi-dimensional information fusion, the aesthetic performance of the image is significantly enhanced while improving the clarity at the physical level; a coding module reuse strategy is introduced, which deeply and interactively learns the aesthetic prior during the training process by predicting the distribution of aesthetic attributes and quality scores. This mechanism not only strengthens the semantic understanding of the image deblurring process, but also enables the deblurring network to take into account the aesthetic quality of the image while optimizing the image details, thereby achieving all-round optimization of image quality while deblurring.

[0311] (5) Not only is the image restored through its pixel-level information, but an optimization mechanism guided by aesthetic quality is also introduced. By combining the semantic prior information of the SAM large model with the strategy of reusing the encoding module, the image is deblurred and aesthetically optimized from more levels (including semantic, pixel, and aesthetic attributes). By reusing the encoding module to predict the distribution of aesthetic attributes and the aesthetic quality score, a deep interaction between the aesthetic quality prior and the deblurring task is achieved. Not only is the physical clarity of the image optimized by combining the pixel-level loss, but the deblurring network is also guided to learn richer aesthetic information in a nonlinear and interactive manner through the reuse strategy.

[0312] (6) By predicting the aesthetic quality score and aesthetic attribute distribution based on the prior knowledge of the SAM large model and the reused coding module, and combining the feature-level and label-level prior information, a multi-level and multi-dimensional fusion of image deblurring and aesthetic optimization is achieved; compared with traditional methods, the advantages of the SAM model in semantic understanding and contextual reasoning are fully utilized to provide richer and more accurate feature-level prompts, thereby improving the deblurring effect and the overall image quality; different from the existing deblurring method that relies solely on physical clarity, the dual guidance of aesthetic quality score loss and aesthetic attribute distribution loss is introduced to ensure that the deblurred image not only improves the clarity, but also optimizes the image's aesthetic features such as color and visual harmony, thereby enhancing the image's visual effect and aesthetic feeling; through fine-grained aesthetic attribute prediction, the multi-dimensional aesthetic characteristics of the image from the overall to the details can be fully captured, further improving the visual quality of the image. Compared with the traditional deblurring method, this refined aesthetic optimization improves the overall aesthetic perception of the image while improving the fidelity of details.

[0313] (7) The image segmentation prior information of the SAM large model is integrated with the encoding module of the deblurring network to achieve deep interaction at the feature level of semantic information. Through this integration, the deblurring network can not only improve the physical clarity, but also enhance the semantic understanding, improve the texture details and overall visual effect of the image content. The proposed encoding module reuse strategy makes the prediction of aesthetic quality score and aesthetic attribute distribution and the deblurring task intertwined. In this way, the deblurring network can dynamically adjust the aesthetic prior information during the training process. This interactive learning method breaks through the limitations of the traditional single optimization task and allows the aesthetic optimization and deblurring tasks to evolve co-evolved, thereby effectively improving the comprehensive performance of the image. By combining the prior information of the SAM large model and the encoding module reuse strategy, multi-level and multi-dimensional learning of image content prior and aesthetic prior is achieved, avoiding the singleness and limitations of prior information in traditional methods. This multi-level prior learning method ensures the improvement of the overall aesthetic quality of the image while improving the image clarity.

[0314] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0315] Based on the same inventive concept, the embodiment of the present application also provides an image blind deblurring device for implementing the above-mentioned image blind deblurring method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image blind deblurring device embodiments provided below can refer to the limitations of the image blind deblurring method above, and will not be repeated here.

[0316] In an exemplary embodiment, Fig. 9 As shown, an image blind deblurring device is provided, comprising: a mask acquisition module 901, a feature processing module 902, a feature fusion module 903 and an image reconstruction module 904, wherein:

[0317] The mask acquisition module 901 is used to acquire the semantic mask of the image to be processed.

[0318] The feature processing module 902 is used to extract the image features of the image to be processed through the trained image blind deblurring model, and fuse the image features and the semantic mask to obtain the fused features; the trained image blind deblurring model is obtained by training the image blind deblurring model to be trained using the pre-trained aesthetic attribute distribution prediction model.

[0319] The feature fusion module 903 is used to perform multiple fusion processes on the fusion features and the image features of the fusion features to obtain the target fusion features of the image to be processed.

[0320] The image reconstruction module 904 is used to perform image reconstruction processing on the image features of the target fusion features to obtain a deblurred image of the image to be processed.

[0321] In an exemplary embodiment, the feature processing module 902 is also used to perform spatial attention mechanism processing on the image feature to obtain a first processed feature, perform channel attention mechanism processing on the first processed feature to obtain a first key feature corresponding to the image feature, and extract a second key feature corresponding to the semantic mask; perform splicing processing on the first key feature and the second key feature to obtain a spliced ​​feature; perform activation processing on the image feature of the spliced ​​feature to obtain a fusion feature.

[0322] In an exemplary embodiment, the feature processing module 902 is also used to determine the attention weight matrix corresponding to the image feature; group the image features to obtain multiple sub-image features, and group the attention weight matrix to obtain a sub-attention weight matrix corresponding to each sub-image feature; and splice each sub-image feature and the sub-attention weight matrix corresponding to each sub-image feature to obtain a first processed feature.

[0323] In an exemplary embodiment, the feature processing module 902 is also used to perform feature extraction processing on the first processed feature to obtain the second processed feature; perform dimensionality reduction processing on the second processed feature to obtain the global feature; perform activation processing on the global feature to obtain the activated feature; and perform fusion processing on the second processed feature and the activated feature to obtain the first key feature.

[0324] In an exemplary embodiment, the image blind deblurring device also includes a first training module, which is used to obtain a first sample semantic mask of a first sample image; extracting sample image features of the first sample image through the image blind deblurring model to be trained, and fusing the sample image features and the first sample semantic mask to obtain sample fusion features; the image blind deblurring model to be trained includes a pre-trained encoding module and a reconstruction module to be trained; the pre-trained encoding module is a pre-trained encoding module in a pre-trained aesthetic attribute distribution prediction model; fusing the sample fusion features and the sample image features of the sample fusion features multiple times to obtain a target sample fusion feature of the first sample image; performing image reconstruction on the sample image features of the target sample fusion features to obtain a first predicted deblurred image of the first sample image; obtaining a first actual deblurred image of the first sample image, and iteratively training the image blind deblurring model to be trained based on the first predicted deblurred image and the first actual deblurred image to obtain a trained image blind deblurring model.

[0325] In an exemplary embodiment, the first training module is further used to obtain a first aesthetic quality score value and a first aesthetic attribute distribution value corresponding to the first predicted deblurred image, and a second aesthetic quality score value and a second aesthetic attribute distribution value corresponding to the first actual deblurred image, based on the first sample semantic mask through a pre-trained aesthetic attribute distribution prediction model; the pre-trained aesthetic attribute distribution prediction model includes a pre-trained encoding module and a trained multi-layer perceptron; according to the difference between the first aesthetic quality score value and the second aesthetic quality score value, a first aesthetic quality score loss value is obtained, and according to the first aesthetic attribute distribution value and the second aesthetic attribute distribution value, a first aesthetic quality score loss value is obtained. The difference between the values ​​is used to obtain a first aesthetic attribute distribution loss value, and the mean square error loss value is obtained based on the difference between the first predicted deblurred image and the first actual deblurred image; the first aesthetic quality score loss value, the first aesthetic attribute distribution loss value and the mean square error loss value are weighted and summed to obtain a first target loss value; according to the first target loss value, the image blind deblurring model to be trained is iteratively trained to obtain a trained image blind deblurring model; the weights of the pre-trained encoding modules in the pre-trained aesthetic attribute distribution prediction model and the weights of the pre-trained encoding modules in the image blind deblurring model to be trained are updated synchronously.

[0326] In an exemplary embodiment, the image blind deblurring device also includes a second training module, which is used to obtain a second sample semantic mask of the second sample image, a second predicted deblurred image, and a second actual deblurred image; through the aesthetic attribute distribution prediction model to be trained, based on the second sample semantic mask, a third aesthetic quality score value and a third aesthetic attribute distribution value corresponding to the second predicted deblurred image, and a fourth aesthetic quality score value and a fourth aesthetic attribute distribution value corresponding to the second actual deblurred image are obtained; the aesthetic attribute distribution prediction model to be trained includes a coding module to be trained and a multi-layer perceptron to be trained; according to the difference between the third aesthetic quality score value and the fourth aesthetic quality score value, a second aesthetic quality score loss value is obtained, and according to the difference between the third aesthetic attribute distribution value and the fourth aesthetic attribute distribution value, a second aesthetic attribute distribution loss value is obtained; the second aesthetic quality score loss value and the second aesthetic attribute distribution loss value are weightedly summed to obtain a second target loss value; according to the second target loss value, the aesthetic attribute distribution prediction model to be trained is iteratively trained to obtain a pre-trained aesthetic attribute distribution prediction model.

[0327] Each module in the above-mentioned image blind deblurring device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0328] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as semantic masks and image features. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for blind deblurring of an image is implemented.

[0329] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0330] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0331] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0332] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0333] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0334] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0335] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A blind deblurring method for an image, characterized in that: The method comprises: Get the semantic mask of the image to be processed; The trained image blind deblurring model is used to extract the image features of the image to be processed, and the image features and the semantic mask are fused to obtain fused features; the trained image blind deblurring model is obtained by training the image blind deblurring model to be trained using a pre-trained aesthetic attribute distribution prediction model; Performing multiple fusion processes on the fusion feature and the image feature of the fusion feature to obtain a target fusion feature of the image to be processed; Perform image reconstruction processing on the image features of the target fusion features to obtain a deblurred image of the image to be processed.

2. The method according to claim 1, characterized in that The fusing the image feature and the semantic mask to obtain a fused feature includes: Performing spatial attention mechanism processing on the image feature to obtain a first processed feature, performing channel attention mechanism processing on the first processed feature to obtain a first key feature corresponding to the image feature, and extracting a second key feature corresponding to the semantic mask; Performing splicing processing on the first key feature and the second key feature to obtain a splicing feature; The image features of the splicing features are activated to obtain the fusion features.

3. The method according to claim 2, characterized in that The performing of spatial attention mechanism processing on the image feature to obtain a first processed feature includes: Determining an attention weight matrix corresponding to the image feature; The image features are grouped to obtain a plurality of sub-image features, and the attention weight matrix is ​​grouped to obtain a sub-attention weight matrix corresponding to each sub-image feature; Each sub-image feature and the sub-attention weight matrix corresponding to each sub-image feature are concatenated to obtain the first processed feature.

4. The method according to claim 2, characterized in that: The performing channel attention mechanism processing on the first processed feature to obtain a first key feature corresponding to the image feature includes: Performing feature extraction processing on the first processed features to obtain second processed features; Performing dimensionality reduction processing on the second processed features to obtain global features; Activating the global feature to obtain an activated feature; The second processed feature and the activated feature are fused to obtain the first key feature.

5. The method according to any one of claims 1 to 4, characterized in that: The trained image blind deblurring model is obtained by training in the following way: Obtaining a first sample semantic mask of the first sample image; The sample image features of the first sample image are extracted by the image blind deblurring model to be trained, and the sample image features and the first sample semantic mask are fused to obtain sample fusion features; the image blind deblurring model to be trained includes a pre-trained encoding module and a reconstruction module to be trained; the pre-trained encoding module is a pre-trained encoding module in the pre-trained aesthetic attribute distribution prediction model; Performing multiple fusion processes on the sample fusion feature and the sample image feature of the sample fusion feature to obtain a target sample fusion feature of the first sample image; Performing image reconstruction processing on the sample image features of the target sample fusion features to obtain a first predicted deblurred image of the first sample image; A first actual deblurred image of the first sample image is obtained, and the image blind deblurring model to be trained is iteratively trained according to the first predicted deblurred image and the first actual deblurred image to obtain the trained image blind deblurring model.

6. The method according to claim 5, characterized in that The iterative training of the image blind deblurring model to be trained according to the first predicted deblurred image and the first actual deblurred image to obtain the trained image blind deblurring model includes: By using the pre-trained aesthetic attribute distribution prediction model, based on the first sample semantic mask, a first aesthetic quality score value and a first aesthetic attribute distribution value corresponding to the first predicted deblurred image, as well as a second aesthetic quality score value and a second aesthetic attribute distribution value corresponding to the first actual deblurred image are obtained; the pre-trained aesthetic attribute distribution prediction model includes the pre-trained encoding module and the trained multi-layer perceptron; obtaining a first aesthetic quality score loss value according to a difference between the first aesthetic quality score value and the second aesthetic quality score value, obtaining a first aesthetic attribute distribution loss value according to a difference between the first aesthetic attribute distribution value and the second aesthetic attribute distribution value, and obtaining a mean square error loss value based on a difference between the first predicted deblurred image and the first actual deblurred image; Performing weighted sum processing on the first aesthetic quality score loss value, the first aesthetic attribute distribution loss value, and the mean square error loss value to obtain a first target loss value; According to the first target loss value, the image blind deblurring model to be trained is iteratively trained to obtain the trained image blind deblurring model; the weight of the pre-trained encoding module in the pre-trained aesthetic attribute distribution prediction model and the weight of the pre-trained encoding module in the image blind deblurring model to be trained are updated synchronously.

7. The method according to claim 6, characterized in that The pre-trained aesthetic attribute distribution prediction model is trained in the following way: Obtaining a second sample semantic mask, a second predicted deblurred image, and a second actual deblurred image of the second sample image; obtaining, by means of an aesthetic attribute distribution prediction model to be trained and based on the second sample semantic mask, a third aesthetic quality score value and a third aesthetic attribute distribution value corresponding to the second predicted deblurred image, and a fourth aesthetic quality score value and a fourth aesthetic attribute distribution value corresponding to the second actual deblurred image; the aesthetic attribute distribution prediction model to be trained includes a coding module to be trained and a multi-layer perceptron to be trained; obtaining a second aesthetic quality score loss value according to a difference between the third aesthetic quality score value and the fourth aesthetic quality score value, and obtaining a second aesthetic attribute distribution loss value according to a difference between the third aesthetic attribute distribution value and the fourth aesthetic attribute distribution value; Performing a weighted sum processing on the second aesthetic quality score loss value and the second aesthetic attribute distribution loss value to obtain a second target loss value; According to the second target loss value, the aesthetic attribute distribution prediction model to be trained is iteratively trained to obtain the pre-trained aesthetic attribute distribution prediction model.

8. An image blind deblurring device, characterized in that: The device comprises: A mask acquisition module, used to acquire the semantic mask of the image to be processed; A feature processing module is used to extract the image features of the image to be processed through the trained image blind deblurring model, and fuse the image features with the semantic mask to obtain fused features; the trained image blind deblurring model is obtained by training the image blind deblurring model to be trained using a pre-trained aesthetic attribute distribution prediction model; A feature fusion module, used for fusing the fusion feature and the image feature of the fusion feature for multiple times to obtain a target fusion feature of the image to be processed; The image reconstruction module is used to perform image reconstruction processing on the image features of the target fusion features to obtain a deblurred image of the image to be processed.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images