Camera blur detection method and related device, equipment and storage medium

Abstract

The present application discloses a camera blur detection method and related devices, equipment and storage media, wherein the camera blur detection method includes: inputting an image to be detected into a DDPM model to obtain the coding features output by the image coding network in the DDPM model and the decoding features output by the decoder in the noise prediction network in the DDPM model; fusing the coding features and the decoding features to obtain the fusion features; predicting based on the fusion features to obtain the detection result of the image to be detected; wherein the detection result at least includes whether there is imaging blur. The above scheme can improve the accuracy of camera blur detection.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a camera blur detection method and related devices, equipment and storage media. Background Art

[0002] Computer vision technology has been rapidly applied in target tracking, industrial quality inspection, and other fields. However, the effectiveness of computer vision algorithms depends on the image quality of the input image. In some harsh environments, camera imaging quality often faces a series of challenges. For example, in outdoor scenes such as surfing and skiing, the camera lens is exposed to interference such as water and snow, which has a serious impact on the camera imaging quality. In industrial scenes, dust, oil vapor, etc. may also cause blurred images.

[0003] In view of the above factors, detecting the camera imaging state is particularly important for computer vision technology. Existing technologies mainly use traditional algorithms such as Laplace transform to detect imaging blur. However, these technologies will interfere with imaging blur detection when the image contrast is not high in outdoor scenes or industrial scenes. In addition, some imaging detection is based on deep learning, but the usability is poor. In view of this, how to improve the accuracy of camera blur detection has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a camera blur detection method and related devices, equipment and storage media, which can improve the accuracy of camera blur detection.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a camera blur detection method, including: inputting an image to be detected into a DDPM model to obtain encoding features output by an image encoding network in the DDPM model and decoding features output by a decoder in a noise prediction network in the DDPM model; fusing the encoding features and the decoding features to obtain fused features; making predictions based on the fused features to obtain detection results of the image to be detected; wherein the detection results at least include whether imaging blur exists.

[0006] In order to solve the above-mentioned technical problems, the second aspect of the present application provides a camera blur detection device, including: a feature acquisition module, a feature fusion module and a result prediction module, the feature acquisition module is used to input the image to be detected into the DDPM model to obtain the encoding features output by the image encoding network in the DDPM model and the decoding features output by the decoder in the noise prediction network in the DDPM model; the feature fusion module is used to fuse the encoding features and the decoding features to obtain the fused features; the result prediction module is used to predict based on the fused features to obtain the detection result of the image to be detected; wherein the detection result at least includes whether there is imaging blur.

[0007] In order to solve the above technical problems, the third aspect of the present application provides an electronic device, which at least includes a memory and a processor coupled to each other, wherein the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the camera blur detection method in the above first aspect.

[0008] In order to solve the above technical problem, the fourth aspect of the present application provides a computer-readable storage medium, which stores program instructions that can be executed by a processor, and the program instructions are used to implement the camera blur detection method of the first aspect.

[0009] The above scheme inputs the image to be detected into the DDPM model to obtain the coding features output by the image coding network in the DDPM module and the decoding features output by the decoder in the noise prediction network in the DDPM model, and then fuses the coding features and the decoding features to obtain the fused features, so as to make a prediction based on the fused features to obtain the detection result of the image to be detected, and the detection result at least includes whether there is imaging blur, so that the DDPM model can be reused in the camera blur detection process. On the one hand, the blur is regarded as noise to refer to the noise prediction network in the DDPM model The perception ability of noise and borrow the decoding features output by the decoder; on the other hand, the image coding network in the DDPM model takes into account The modeling ability of the input image and borrows the coding features output by the image coding network, and then combines the coding features and the decoding features to jointly predict whether there is imaging blur, which can improve the accuracy of camera blur detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flowchart of an embodiment of a camera blur detection method of the present application; Figure 2 This is a process diagram of an embodiment of a camera blur detection method of the present application; Figure 3 It is a schematic diagram of the framework of an embodiment of a camera blur detection device of the present application; Figure 4 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application; Figure 5 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0011] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0012] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0013] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the fragment " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0014] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the camera blur detection method of the present application. Specifically, it may include the following steps: Step S11: input the image to be detected into the DDPM model to obtain the encoding features output by the image encoding network in the DDPM model and the decoding features output by the decoder in the noise prediction network in the DDPM model.

[0015] It should be noted that the DDPM model is the Denoising Diffusion Probabilistic Model. Figure 2 , Figure 2 FIG. 1 is a schematic diagram of a process of an embodiment of the camera blur detection method of the present application. Figure 2 As shown in Figure 1, the DDPM model can include an image coding network and a noise prediction network. The image coding network is used to encode the input image to obtain the coding feature x 0 , the sampler uniformly samples in {1, 2, ..., T} to obtain the sampling step number t, where T represents the maximum sampling step number, and further generates noise features based on the sampling step number t, such as randomly sampling Gaussian noise , and then the noise feature and the encoding feature x 0 Superposition, get the noise feature x t , and is input into the noise prediction network together with the sampling step number t. As a possible example, the noise prediction network can use an encoder-decoder architecture such as U-Net to predict the noise characteristics. In other words, the noise prediction network has a t , obtain the prediction ability of noise features. Of course, the above description is only a principle description of the DDPM model. For the specific process of steps such as generating noise features, please refer to the technical details of DDPM, which will not be repeated here.

[0016] In one implementation scenario, the image to be detected may be different images depending on the application scenario of camera blur detection. For example, taking an outdoor scene as an example, the image to be detected may be an image taken by a sports camera, and the image to be detected may or may not contain interferences such as water, snow, mud, sand, etc. that cause blurred imaging; or, taking an industrial scene as an example, the image to be detected may be an image taken by a quality inspection camera, and the image to be detected may or may not contain interferences such as dust, oil, etc. that cause blurred imaging. Of course, the above examples are only a few possible examples of images to be detected, and the images to be detected in other application scenarios can be deduced by analogy, and no examples are given one by one here.

[0017] In one implementation scenario, the image coding network may include but is not limited to convolutional neural networks, autoencoders, etc., and the network structure of the image coding network is not limited here. It should be noted that the image coding network can be pre-trained. As a possible example, the sample image can be encoded using the image coding network to obtain sample coding features, and then the sample coding features can be decoded using the image decoding network to obtain a predicted image, so as to adjust the network parameters of both the image coding network and the image decoding network based on the difference between the sample image and the predicted image, so that the image coding network has the ability to accurately encode images, so that the image coding network can take into account various image feature information as much as possible when encoding. Similarly, the above training can also enable the image decoding network to accurately decode images. Of course, the above example is only a possible example of pre-training the image coding network, and does not limit other possible ways of the image coding network, and will not be given one by one here.

[0018] In one implementation scenario, as mentioned above, the noise prediction network can adopt an encoder-decoder network architecture such as U-Net. In this case, the decoder in the noise prediction network can include multiple decoding layers, and the decoded features output by the decoder can specifically include the decoded features output by at least one decoding layer. As a possible example, the decoding features output by each decoding layer can be obtained; or, as another possible example, the decoding features output by some of the decoding layers can also be obtained, which is not limited here.

[0019] In one implementation scenario, as a possible example, before the DDPM model is applied to the camera blur detection of the embodiment of the present disclosure, the DDPM model can also be pre-trained, and the DDPM model can be specifically pre-trained based on the positive sample image. It should be noted that in the embodiment of the present disclosure, the positive sample image representation does not have imaging blur, and conversely, the negative sample image representation has imaging blur. Specifically, the first sample feature output by the image coding network processing the positive sample image can be obtained, and the sample noise feature can be generated based on the target sampling step number, and the first sample feature and the sample noise feature can be spliced ​​in the channel dimension to obtain the sample noise feature, and then the predicted noise feature output by the noise prediction network based on the sample noise feature and the target sampling step number can be obtained, so that the network parameters of the noise prediction network can be adjusted based on the difference between the sample noise feature and the predicted noise feature, and the network parameters of the image coding network can be fixed. It should be noted that the method for obtaining the first sample feature can refer to the aforementioned description of the encoding feature, the method for obtaining the target sampling step number can refer to the aforementioned description of the sampling step number, the method for obtaining the sample noise feature can refer to the aforementioned description of the noise feature, and other related contents can also be combined with the technical details of the DDPM model, which will not be repeated here. In the above method, the DDPM model is pre-trained through positive sample images, and the network parameters of the noise prediction network are adjusted during the pre-training process to constrain the sample noise features and the predicted noise features to be as close as possible, and the network parameters of the image coding network are fixed, so that the noise prediction network can learn to model the noise feature information in the input image as accurately as possible.

[0020] In a specific implementation scenario, the positive sample image may include at least the originally collected first positive sample image. Exemplarily, according to the application scenario of camera blur detection, a sample image without imaging blur can be selected from the image set of the application scenario as the first positive sample image. For example, when camera blur detection is applied to outdoor scenes, sample images without imaging blur can be selected from the image set of outdoor scenes as the first positive sample images for camera blur detection in outdoor scenes; or, when camera blur detection is applied to industrial scenes, sample images without imaging blur can be selected from the image set of industrial scenes as the first positive sample images for camera blur detection in industrial scenes. Of course, the above examples are only several possible ways to obtain the first positive sample image in actual application, and other possible acquisition methods are not limited thereto, so they are not given one by one here.

[0021] In a specific implementation scenario, the positive sample image may include not only the first positive sample image that was originally collected, but also the second positive sample image that was not originally collected. Exemplarily, image enhancement may be performed based on the first positive sample image to obtain the second positive sample image, and the image enhancement may include: at least one of image cropping, color gamut enhancement, contrast enhancement, and histogram equalization. In other words, the above-mentioned image enhancement methods may be randomly applied to the first positive sample image, that is, each image enhancement may be performed with independent probability, and multiple image enhancements may also be superimposed on the same first positive sample image.

[0022] In a specific implementation scenario, the difference between the sample noise feature and the predicted noise feature can be measured by a loss function such as mean square error. For example, the difference between the sample noise feature and the predicted noise feature can be expressed as: …… (1) In the above formula (1), represents the sample noise characteristics, represents the sample noise feature, represents the number of sampling steps, represents the network parameters of the noise prediction network, represents the predicted noise characteristics output by the noise prediction network, Of course, the above example is only one possible measurement method for the difference between the sample noise feature and the predicted noise feature, and the measurement method for the difference between the sample noise feature and the predicted noise feature is not limited here.

[0023] Step S12: Fusing the encoding features and the decoding features to obtain fused features.

[0024] In an implementation scenario, as a possible example, the encoding features and the decoding features can be sampled to the same scale and then concatenated to achieve fusion, that is, to obtain the fused features.

[0025] In a specific implementation scenario, as mentioned above, the decoding features output by the decoder include the decoding features output by each network layer (i.e., the decoding layer) in the decoder, and the decoding features output by each network layer can be upsampled to the same scale as the encoding features. Exemplarily, the resolution of the encoding feature can be 64*64, and the resolution of the decoding features output by each network layer can include but is not limited to: 32*32, 16*16, 8*8, and the above decoding features can be upsampled separately: the decoding features with a resolution of 32*32 are upsampled by 2 times, the decoding features with a resolution of 16*16 are upsampled by 4 times, and the decoding features with a resolution of 8*8 are upsampled by 8 times. Of course, the above examples are only several possible examples of unified scales for encoding features and decoding features, and other possible situations are not given examples one by one here.

[0026] In a specific implementation scenario, after unifying the scale, the encoded features and decoded features after unifying the scale can be concatenated in the channel dimension to obtain fused features.

[0027] In another implementation scenario, as another possible example, different from the aforementioned implementation, after the coding features and the decoding features are sampled to the same scale, the coding features and the decoding features sampled to the same scale can be spliced ​​to obtain spliced ​​features. Exemplarily, the coding features and the decoding features sampled to the same scale can be spliced ​​in the channel dimension to obtain spliced ​​features. After obtaining the spliced ​​features, a convolution operation can be performed based on the spliced ​​features to obtain a reduced dimension feature. Exemplarily, the spliced ​​features can be convoluted through multiple convolutional layers to obtain a reduced dimension feature. Thereafter, the reduced dimension feature can be processed based on a spatial attention mechanism to obtain weight values ​​at each position in the reduced dimension feature. Specifically, after the spatial attention mechanism, the weight values ​​at each pixel point of the reduced dimension feature can be obtained. On this basis, the reduced dimension feature can be weighted based on the weight value to obtain a fused feature. Exemplarily, the feature values ​​of the corresponding pixel points of the reduced dimension feature can be weighted based on the weight values ​​at each pixel point of the reduced dimension feature to obtain a fused feature.

[0028] It should be noted that the above examples are only several possible examples of the fusion of encoding features and decoding features, and other possible fusion methods are not limited thereto. Exemplarily, fusion can also be performed based on a channel attention mechanism. The specific methods of fusion of encoding features and decoding features are not given one by one here.

[0029] Step S13: Predict based on the fused features to obtain the detection result of the image to be detected.

[0030] In the embodiment of the present disclosure, the detection result at least includes whether the imaging blur exists. That is, the detection result includes: whether the imaging blur exists or not.

[0031] In an implementation scenario, the detection result can be obtained by detecting a blur detection model. The blur detection model can be trained based on a sample image set after the DDPM model pre-training converges. The sample image set can include positive sample images with imaging blur and negative sample images with imaging blur. In other words, the blur detection model is trained based on both positive sample images and negative sample images. In addition, the DDPM model can be pre-trained based on positive sample images. The specific process can be referred to the aforementioned related description, which will not be repeated here.

[0032] In a specific implementation scenario, the negative sample image may include a first negative sample image with imaging blur in a local area. It should be noted that the negative sample image is obtained by adding noise to the positive sample image, and the first negative sample image can be obtained by adding noise locally to the positive sample image. Specifically, the first negative sample image can be obtained in the following manner: first, a mask image is obtained based on the Perlin noise generated by the image size of the positive sample image, and the pixel points in the mask image with a pixel value of a first value (such as 1) represent that the pixel points at the corresponding position in the positive sample image need to be noised, while the pixel points in the mask image with a pixel value of a second value (such as 0) represent that the pixel points at the corresponding position in the positive sample image do not need to be noised. On this basis, the positive sample image can be noised based on the mask image to obtain the first negative sample image. Exemplarily, for a connected domain formed by pixel points with a pixel value of a first value in the mask image, a perturbation noise can be performed in the positive sample image corresponding to the connected domain, so that the positive sample image has imaging blur in the corresponding area, thereby forming a first negative sample image with local imaging blur. It should be noted that the process of obtaining the mask image can refer to the technical details of Perlin noise, which will not be repeated here. In addition, after the first negative sample image is generated, the first negative sample image can be annotated with a real result representing the presence of imaging blur. In the above method, a mask image is obtained based on the Perlin noise generated by the image size of the positive sample image, and the pixel points with a first value in the mask image represent that the pixel points at the corresponding position in the positive sample image need to be noised, and the pixel points with a second value in the mask image represent that the pixel points at the corresponding position in the positive sample image do not need to be noised. Then, the positive sample image is noised based on the mask image to obtain the first negative sample image. This can generate a first negative sample image with imaging blur in the local area, and there is no need to manually obtain and annotate the negative sample image, which helps to reduce time costs and reduce resource requirements.

[0033] In a specific implementation scenario, the negative sample image may also include a second negative sample image in which the image has global imaging blur. It should be noted that the negative sample image is obtained based on the positive sample image by adding noise, and the second negative sample image can be obtained based on the positive sample image by global noise addition. Specifically, the second negative sample image can be obtained by the following method: Gaussian blurring is performed on the positive sample image to obtain the second negative sample image. It should be noted that the specific process of Gaussian blurring can refer to the technical details of Gaussian blurring, which will not be repeated here. In addition, after the second negative sample image is generated, the second negative sample image can be annotated with a real result representing the presence of imaging blur. The above method, based on the positive sample image, Gaussian blurring is performed to obtain the second negative sample image, which can generate a second negative sample image in which the image has global imaging blur, and there is no need to manually obtain and annotate the negative sample image, which helps to reduce time costs and reduce resource requirements.

[0034] In a specific implementation scenario, in order to balance the positive sample image and the negative sample image, the positive sample image may specifically include the first positive sample image originally acquired, and the second positive sample image obtained by image enhancement based on the first positive sample image. The specific process of image enhancement can refer to the above-mentioned related description, which will not be repeated here. In addition, both the first positive sample image and the second positive sample image can be annotated with a real result representing the absence of imaging blur.

[0035] In a specific implementation scenario, after obtaining a sample image set including a first positive sample image, a second positive sample image, a first negative sample image, and a second negative sample image, the blur detection model can be trained. Specifically, the sample image can be input into the DDPM model to obtain the second sample image output by the image coding network and the sample decoding features output by the decoder, and the sample image is any image in the sample image set. In other words, the sample image can be either the first positive sample image, the second positive sample image, the first negative sample image, or the second negative sample image, which is not limited here. In addition, the sample image is annotated with a real result representing whether there is imaging blur. On this basis, the second sample feature and the sample decoding feature can be fused to obtain a sample fusion feature, and then a prediction can be made based on the sample fusion feature to obtain the detection result of the sample image, so that the network parameters of the blur detection model can be adjusted based on the difference between the detection result and the real result of the sample image. It should be noted that the specific process of obtaining the second sample image can refer to the related description of obtaining the encoding feature, the specific process of obtaining the sample decoding feature can refer to the related description of obtaining the decoding feature, and the specific process of obtaining the sample fusion feature can refer to the related description of obtaining the fusion feature, which will not be repeated here. As a possible example, the difference between the detection result and the true result of the sample image can be measured by a loss function such as cross entropy, which can be expressed as: …… (2) In the above formula (2), They represent the probability value of the sample image being detected as having no imaging blur and the probability value of the sample image being detected as having imaging blur, respectively. They represent the true labels of the sample image being labeled as having no imaging blur or having imaging blur. For example, when the sample image is truly labeled as having no imaging blur, Can be set to 1, Can be set to 0. Of course, the above example is only a possible example in the actual application process, and other possible ways of loss measurement will not be given one by one here. In the above method, a sample image is input into the DDPM model to obtain the second sample feature output by the image coding network and the sample decoding feature output by the decoder, and the sample image is any image in the sample image set, and the sample image is annotated with a real result representing whether there is imaging blur, and then based on the second sample feature and the sample decoding feature, a fusion is performed to obtain a sample fusion feature, and then a prediction is made based on the sample fusion feature to obtain the detection result of the sample image, and then based on the difference between the detection result of the sample image and the real result, the network parameters of the blur detection model are adjusted. Therefore, training the blur detection model with the sample image set can enable the blur detection model to learn to perform blur detection using the relevant features obtained from the DDPM model, which helps to improve the accuracy of blur detection.

[0036] In one implementation scenario, the blur detection model may specifically include a feature fusion network and a classification prediction network, and the feature fusion network is used to fuse the encoding features and the decoding features, and the classification prediction network is used to classify and predict the fused features to determine whether there is imaging blur. As a possible example, the feature fusion network may specifically include a convolution layer and an attention mechanism layer (such as a spatial attention mechanism). For details, please refer to the relevant description of the aforementioned feature fusion, which will not be repeated here. In addition, the classification prediction network may specifically include multiple linear layers to achieve binary classification prediction (i.e., the presence of imaging blur and the absence of imaging blur). Of course, the above example is only a possible example of the network structure of the blur detection model, and the network structure of the blur detection model is not limited here. That is to say, after obtaining the fused features in the inference stage, the fused features can be input into the classification prediction network to obtain a binary classification detection result, such as a probability value of the absence of imaging blur and a probability value of the presence of imaging blur. On this basis, whether there is imaging blur can be determined based on the larger probability value. If the probability value of no imaging blur is larger than the probability value of imaging blur, it can be determined that there is no imaging blur. On the contrary, if the probability value of imaging blur is larger than the probability value of no imaging blur, it can be determined that there is imaging blur. Alternatively, the probability value of imaging blur and the probability value of no imaging blur can be compared with the probability threshold respectively, and whether there is imaging blur can be determined based on whether the probability value is higher than the probability threshold. If the probability value of no imaging blur is greater than the probability threshold, it can be determined that there is no imaging blur. On the contrary, if the probability value of imaging blur is greater than the probability threshold, it can be determined that there is imaging blur. Of course, if both the probability value of imaging blur and the probability value of no imaging blur are not greater than the probability threshold, it can be considered that it is impossible to determine whether there is imaging blur in the image to be detected.

[0037] The above scheme inputs the image to be detected into the DDPM model to obtain the coding features output by the image coding network in the DDPM module and the decoding features output by the decoder in the noise prediction network in the DDPM model, and then fuses the coding features and the decoding features to obtain the fused features, so as to make a prediction based on the fused features to obtain the detection result of the image to be detected, and the detection result at least includes whether there is imaging blur, so that the DDPM model can be reused in the camera blur detection process. On the one hand, the blur is regarded as noise to refer to the noise prediction network in the DDPM model The perception ability of noise and borrow the decoding features output by the decoder; on the other hand, the image coding network in the DDPM model takes into account The modeling ability of the input image and borrows the coding features output by the image coding network, and then combines the coding features and the decoding features to jointly predict whether there is imaging blur, which can improve the accuracy of camera blur detection.

[0038] See also Figure 3 , Figure 3 It is a schematic diagram of the framework of an embodiment of the camera blur detection device of the present application. The camera blur detection device 30 includes: a feature acquisition module 31, a feature fusion module 32 and a result prediction module 33. The feature acquisition module 31 is used to input the image to be detected into the DDPM model to obtain the encoding features output by the image encoding network in the DDPM model and the decoding features output by the decoder in the noise prediction network in the DDPM model; the feature fusion module 32 is used to fuse based on the encoding features and the decoding features to obtain the fused features; the result prediction module 33 is used to predict based on the fused features to obtain the detection result of the image to be detected; wherein the detection result at least includes whether there is imaging blur.

[0039] In the above scheme, the camera blur detection device 30 inputs the image to be detected into the DDPM model to obtain the coding features output by the image coding network in the DDPM module and the decoding features output by the decoder in the noise prediction network in the DDPM model, and then fuses the coding features and the decoding features to obtain the fused features, so as to make a prediction based on the fused features to obtain the detection result of the image to be detected, and the detection result at least includes whether there is imaging blur, so that the DDPM model can be reused in the camera blur detection process. On the one hand, the blur is regarded as noise to refer to the noise prediction network in the DDPM model to perceive noise and borrow the decoding features output by the decoder. On the other hand, the image coding network in the DDPM model takes into account the modeling ability of the input image and borrows the coding features output by the image coding network. Then, the coding features and the decoding features are combined to jointly predict whether there is imaging blur, which can improve the accuracy of camera blur detection.

[0040] In some disclosed embodiments, the feature fusion module 32 includes an image sampling submodule for sampling the encoding features and the decoding features to the same scale; the feature fusion module 32 includes a feature splicing submodule for splicing the encoding features and the decoding features sampled to the same scale to obtain spliced ​​features; the feature fusion module 32 includes a convolution dimension reduction submodule for performing convolution operations based on the spliced ​​features to obtain reduced dimension features; the feature fusion module 32 includes a weight prediction submodule for processing the reduced dimension features based on the spatial attention mechanism to obtain the weight values ​​at each position in the reduced dimension features; the feature fusion module 32 includes a feature weighting submodule for weighting the reduced dimension features based on the weight values ​​to obtain fused features.

[0041] In some disclosed embodiments, the decoding features output by the decoder include the decoding features output by each network layer in the decoder, and the image sampling submodule is specifically used to upsample the decoding features output by each network layer to the same scale as the encoding features.

[0042] In some disclosed embodiments, the detection result is obtained by detecting a blur detection model, which is trained based on a sample image set after the DDPM model pre-training converges. The sample image set includes positive sample images without imaging blur and negative sample images with imaging blur. The DDPM model is pre-trained based on the positive sample images, and the negative sample images are obtained by adding noise to the positive sample images.

[0043] In some disclosed embodiments, the camera blur detection device 30 includes a sample processing module for obtaining a first sample feature output by an image coding network processing a positive sample image, and generating a sample noise feature based on a target sampling step number; the camera blur detection device 30 includes a sample noise addition module for splicing the first sample feature and the sample noise feature in the channel dimension to obtain a sample noise addition feature; the camera blur detection device 30 includes a noise prediction module for obtaining a predicted noise feature output by a noise prediction network based on the sample noise addition feature and the target sampling step number; the camera blur detection device 30 includes a parameter optimization module for adjusting the network parameters of the noise prediction network based on the difference between the sample noise feature and the predicted noise feature, and fixing the network parameters of the image coding network.

[0044] In some disclosed embodiments, the camera blur detection device 30 includes a sample acquisition module, which is used to input a sample image into the DDPM model to obtain a second sample feature output by the image encoding network and a sample decoding feature output by the decoder; wherein the sample image is any image in the sample image set, and the sample image is annotated with a real result representing whether there is imaging blur; the camera blur detection device 30 includes a sample fusion module, which is used to fuse the second sample feature and the sample decoding feature to obtain a sample fusion feature; the camera blur detection device 30 includes a sample prediction module, which is used to predict based on the sample fusion feature to obtain a detection result of the sample image; the camera blur detection device 30 includes a parameter adjustment module, which is used to adjust the network parameters of the blur detection model based on the difference between the detection result of the sample image and the real result.

[0045] In some disclosed embodiments, the negative sample image includes a first negative sample image in which imaging blur exists in a local area, and the camera blur detection device 30 includes a mask generation module for generating Perlin noise based on the image size of the positive sample image to obtain a mask image; wherein, the pixel points in the mask image whose pixel values ​​are the first value represent that the pixel points at the corresponding positions in the positive sample image need to be noised, and the pixel points in the mask image whose pixel values ​​are the second value represent that the pixel points at the corresponding positions in the positive sample image do not need to be noised; the camera blur detection device 30 includes a local noise addition module for adding noise to the positive sample image based on the mask image to obtain a first negative sample image.

[0046] In some disclosed embodiments, the negative sample image includes a second negative sample image in which global imaging blur exists, and the camera blur detection device 30 includes a global noise addition module for performing Gaussian blur based on the positive sample image to obtain the second negative sample image.

[0047] In some disclosed embodiments, the positive sample image includes at least an originally collected first positive sample image, and the camera blur detection device 30 includes an image enhancement module for performing image enhancement based on the first positive sample image to obtain a second positive sample image; wherein the positive sample image also includes the second positive sample image, and the image enhancement includes: at least one of image cropping, color gamut enhancement, contrast enhancement, and histogram equalization.

[0048] In some disclosed embodiments, the blur detection model includes a feature fusion network and a classification prediction network, and the feature fusion network is used to fuse encoding features and decoding features, and the classification prediction network is used to perform classification prediction on the fused features to determine whether imaging blur exists.

[0049] See also Figure 4 , Figure 4 : is a schematic diagram of the framework of an embodiment of an electronic device of the present application. The electronic device 40 includes at least a memory 41 and a processor 42 coupled to each other, the memory 41 stores at least program instructions, and the processor 42 is used to execute the program instructions to implement the steps in any of the above-mentioned camera blur detection method embodiments. For details, please refer to the aforementioned disclosed embodiments, which will not be repeated here. As a possible example, the electronic device 40 may include but is not limited to a server, a camera, a smart phone, a tablet computer, etc., and the specific type of the electronic device 40 is not limited here.

[0050] Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps in any of the above-mentioned camera blur detection method embodiments. The processor 42 can also be called a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip with signal processing capabilities. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 42 can be implemented by an integrated circuit chip.

[0051] In the above scheme, the electronic device 40 inputs the image to be detected into the DDPM model to obtain the coding features output by the image coding network in the DDPM module and the decoding features output by the decoder in the noise prediction network in the DDPM model, and then fuses the coding features and the decoding features to obtain the fused features, so as to make a prediction based on the fused features to obtain the detection result of the image to be detected, and the detection result at least includes whether there is imaging blur, so that the DDPM model can be reused in the camera blur detection process. On the one hand, the blur is regarded as noise to refer to the noise prediction network in the DDPM model for the perception of noise and borrow the decoding features output by the decoder. On the other hand, the image coding network in the DDPM model takes into account the modeling ability of the input image and borrows the coding features output by the image coding network. Then, the coding features and the decoding features are combined to jointly predict whether there is imaging blur, which can improve the accuracy of camera blur detection.

[0052] See also Figure 5 , Figure 5 1 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 50 of the present application. The computer-readable storage medium 50 stores program instructions 51 that can be executed by a processor, and the program instructions 51 are used to implement the steps in any of the above-mentioned camera blur detection method embodiments.

[0053] In the above scheme, the computer-readable storage medium 50 inputs the image to be detected into the DDPM model to obtain the coding features output by the image coding network in the DDPM module and the decoding features output by the decoder in the noise prediction network in the DDPM model, and then fuses the coding features and the decoding features to obtain the fused features, so as to make a prediction based on the fused features to obtain the detection result of the image to be detected, and the detection result at least includes whether there is imaging blur, so that the DDPM model can be reused in the camera blur detection process. On the one hand, the blur is regarded as noise to refer to the noise prediction network in the DDPM model The perception ability of noise and borrow the decoding features output by the decoder; on the other hand, the image coding network in the DDPM model takes into account the modeling ability of the input image and borrows the coding features output by the image coding network, and then combines the coding features and the decoding features to jointly predict whether there is imaging blur, which can improve the accuracy of camera blur detection.

[0054] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0055] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0056] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0057] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0058] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0059] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0060] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A camera blur detection method, characterized in that: include: Inputting the image to be detected into the DDPM model to obtain the encoding features output by the image encoding network in the DDPM model and the decoding features output by the decoder in the noise prediction network in the DDPM model; Fusing the encoding feature and the decoding feature to obtain a fused feature; Prediction is performed based on the fusion features to obtain a detection result of the image to be detected; wherein the detection result at least includes whether imaging blur exists.

2. The method according to claim 1, characterized in that The fusing the encoding feature and the decoding feature to obtain the fused feature includes: Sampling the encoding feature and the decoding feature to the same scale; Concatenate the encoding features and decoding features sampled to the same scale to obtain concatenated features; Performing a convolution operation based on the concatenated features to obtain a dimension reduction feature; Processing the reduced-dimensionality features based on a spatial attention mechanism to obtain weight values ​​at various positions in the reduced-dimensionality features; The dimension reduction features are weighted based on the weight values ​​to obtain the fusion features.

3. The method according to claim 2, characterized in that The decoding features output by the decoder include decoding features respectively output by each network layer in the decoder, and sampling the encoding features and the decoding features to the same scale includes: The decoded features output by each network layer are upsampled to have the same scale as the encoded features.

4. The method according to claim 1, characterized in that: The detection result is obtained by detecting a blur detection model, and the blur detection model is trained based on a sample image set after the DDPM model pre-training converges, and the sample image set includes positive sample images without imaging blur and negative sample images with imaging blur, and the DDPM model is pre-trained based on the positive sample images, and the negative sample images are obtained by adding noise to the positive sample images.

5. The method according to claim 4, characterized in that The pre-training steps of the DDPM model include: Acquire a first sample feature output by the image coding network processing the positive sample image, and generate a sample noise feature based on a target sampling step number; Concatenate the first sample feature and the sample noise feature in the channel dimension to obtain a sample noise-added feature; Obtaining a predicted noise feature output by the noise prediction network based on the sample noise feature and the target sampling step number; Based on the difference between the sample noise feature and the predicted noise feature, the network parameters of the noise prediction network are adjusted, and the network parameters of the image encoding network are fixed.

6. The method according to claim 4, characterized in that The training steps of the fuzzy detection model include: Input a sample image to the DDPM model to obtain a second sample feature output by the image encoding network and a sample decoding feature output by the decoder; wherein the sample image is any image in the sample image set, and the sample image is annotated with a real result indicating whether there is imaging blur; Fusing the second sample feature and the sample decoding feature to obtain a sample fusion feature; Predicting based on the sample fusion features to obtain a detection result of the sample image; Based on the difference between the detection result of the sample image and the true result, the network parameters of the blur detection model are adjusted.

7. The method according to claim 4, characterized in that The negative sample image includes a first negative sample image with imaging blur in a local area, and the step of acquiring the first negative sample image includes: A mask image is obtained based on the Perlin noise generated based on the image size of the positive sample image; wherein a pixel point in the mask image whose pixel value is a first value indicates that a pixel point at a corresponding position in the positive sample image needs to be noised, and a pixel point in the mask image whose pixel value is a second value indicates that a pixel point at a corresponding position in the positive sample image does not need to be noised; Adding noise to the positive sample image based on the mask image to obtain the first negative sample image; And / or, the negative sample image includes a second negative sample image with global imaging blur, and the step of acquiring the second negative sample image includes: Gaussian blur is performed on the positive sample image to obtain the second negative sample image.

8. The method according to claim 4, characterized in that The positive sample image at least includes a first positive sample image originally collected, and the method further includes: Image enhancement is performed based on the first positive sample image to obtain a second positive sample image; wherein the positive sample image also includes the second positive sample image, and the image enhancement includes: at least one of image cropping, color gamut enhancement, contrast enhancement, and histogram equalization.

9. The method according to claim 4, characterized in that The blur detection model includes a feature fusion network and a classification prediction network, and the feature fusion network is used to fuse the encoding features and the decoding features, and the classification prediction network is used to perform classification prediction on the fused features to determine whether imaging blur exists.

10. A camera blur detection device, characterized in that: include: A feature acquisition module, used to input the image to be detected into the DDPM model to obtain the encoding features output by the image encoding network in the DDPM model and the decoding features output by the decoder in the noise prediction network in the DDPM model; A feature fusion module, used for fusing the encoding feature and the decoding feature to obtain a fusion feature; A result prediction module is used to make predictions based on the fusion features to obtain a detection result of the image to be detected; wherein the detection result at least includes whether imaging blur exists.

11. An electronic device, characterized in that: The method comprises at least a memory and a processor coupled to each other, wherein the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the camera blur detection method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the camera blur detection method according to any one of claims 1 to 9.

Images