Image processing method and device, electronic equipment and storage medium

Abstract

The invention relates to an image processing method and device, electronic equipment and a storage medium, and the image processing method comprises the steps: obtaining an original image collected by an image sensor and light quantity information corresponding to the original image, the light quantity information being used for describing the light intake quantity when the image sensor collects the original image; acquiring image features of the original image; and carrying out smudginess detection on the image sensor based on the light quantity information and the image features, so that the imaging quality of the original image is judged by utilizing the image features and the prior information of the light quantity information, thereby accurately judging whether the image sensor has smudginess or not, and improving the accuracy of the image sensor. The reliability of smudginess detection of the image sensor is improved, so that the purpose of ensuring the surface cleanness of the image sensor before the image sensor is used for imaging work is achieved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to an image processing method and device, electronic equipment, and storage medium. Background Art

[0002] In pursuit of higher quality images, people are paying more and more attention to factors that affect camera image quality, such as mold and dust. Mold growing on the outside of the camera is easier to detect, but many molds grow from the inside of the camera, such as on the image sensor, forming large areas of mold. This kind of mold is generally difficult to detect with the naked eye, and if not handled in time, it may shorten the life of the camera.

[0003] At present, there are corresponding detection methods for dirt on camera lenses (such as mildew and dust), but the dirt on camera sensors is difficult to observe with the naked eye, and there is currently a lack of simple and effective detection methods. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides an image processing method and device, an electronic device, and a storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including:

[0006] Acquire an original image captured by an image sensor and light quantity information corresponding to the original image, wherein the light quantity information is used to describe the amount of light taken in when the image sensor captures the original image;

[0007] Acquire image features of the original image;

[0008] Based on the light amount information and the image feature, dirt detection is performed on the image sensor.

[0009] In some embodiments, the performing dirt detection on the image sensor based on the light amount information and the image feature includes:

[0010] The light quantity information and the image features are input into a pre-trained perceptron model to obtain an image quality result output by the perceptron model; the image quality result is used to indicate the probability that the image sensor is dirty.

[0011] In some embodiments, the method further comprises:

[0012] Performing grayscale processing on the original image to obtain a grayscale image;

[0013] The image features of the grayscale image are extracted using a feature extraction network.

[0014] In some embodiments, the extracting the image features of the grayscale image using a feature extraction network includes:

[0015] Acquiring brightness information of the grayscale image;

[0016] Based on the brightness information, determining whether the grayscale image is an image with abnormal illumination;

[0017] If the grayscale image is not an image with abnormal brightness, the feature extraction network is used to extract image features of the grayscale image.

[0018] In some embodiments, grayscale processing is performed on the original image to obtain a grayscale image, including:

[0019] Performing white balance processing on the original image to obtain a gain image;

[0020] The gain image is grayscaled to obtain the grayscale image.

[0021] In some embodiments, the method further comprises:

[0022] When it is determined that the image sensor is dirty, a prompt message is output.

[0023] In some embodiments, the light amount information includes at least one of the following information:

[0024] Aperture size information, shutter speed information, and exposure time information.

[0025] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, comprising at least:

[0026] A first acquisition module is configured to acquire the original image captured by the image sensor and light quantity information corresponding to the original image; wherein the light quantity information is used to describe the amount of light taken in when the image sensor captures the original image;

[0027] A second acquisition module is configured to acquire image features of the original image;

[0028] The first processing module is configured to perform dirt detection on the image sensor based on the light quantity information and the image feature.

[0029] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising at least:

[0030] processor;

[0031] a memory for storing processor-executable instructions;

[0032] Wherein, the processor is configured to execute the image processing method as described in the first aspect above.

[0033] According to a fourth aspect of an embodiment of the present disclosure, there is provided a storage medium, including:

[0034] When the instructions in the storage medium are executed by a processor of a terminal device, the terminal device is enabled to execute the image processing method in the first aspect described above.

[0035] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0036] In the disclosed embodiment, the original image captured by the image sensor and the light quantity information corresponding to the original image are obtained, wherein the light quantity information is used to describe the light quantity taken in when the image sensor captures the original image, and then the image features of the original image are obtained, and the image sensor is detected for dirt based on the light quantity information and the image features. In this way, by using the image features and the light quantity information as a priori information, the imaging quality of the original image is judged, so that it is possible to accurately judge whether the image sensor is dirty, thereby improving the reliability of the image sensor dirt detection, so as to achieve the purpose of ensuring the surface of the image sensor is clean before using the image sensor for imaging.

[0037] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0039] Figure 1 is a schematic diagram of a process of an image processing method according to an exemplary embodiment Figure 1 ;

[0040] Figure 2 is a diagram of a practical application scenario of an image sensor according to an exemplary embodiment;

[0041] Figure 3 is a schematic diagram showing a comparison between a dirty image sensor and an image sensor without dirt according to an exemplary embodiment;

[0042] Figure 4 is a schematic diagram showing a comparison of imaging effects when the image sensor is dirty and when the image sensor is not dirty when shooting the same scene according to an exemplary embodiment;

[0043] Figure 5 is a schematic diagram of the structure of a three-layer perceptron according to an exemplary embodiment;

[0044] Figure 6 is a schematic diagram showing a comparison of imaging effects of a dirty image sensor at different apertures according to an exemplary embodiment;

[0045] Figure 7 is a schematic diagram of a process of an image processing method according to an exemplary embodiment Figure 2 ;

[0046] Figure 8 is a block diagram of an image processing device according to an exemplary embodiment;

[0047] Fig. 9 is a hardware structure block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0048] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0049] The present disclosure provides an image processing method. Figure 1 is a schematic diagram of a process of an image processing method according to an exemplary embodiment Figure 1 ,like Figure 1 The method mainly comprises the following steps:

[0050] In step 101, an original image captured by an image sensor and light quantity information corresponding to the original image are obtained, wherein the light quantity information is used to describe the amount of light taken in when the image sensor captures the original image;

[0051] In step 102, image features of the original image are obtained;

[0052] In step 103, based on the light amount information and the image features, dirt detection is performed on the image sensor.

[0053] It should be noted that the image processing method proposed in the present disclosure can be applied to electronic devices or servers. When applied to a server, the electronic device can send the collected original image and the light quantity information corresponding to the original image to the server, and the server processes the original image and the light quantity information using steps 101-103 to obtain a dirt detection result for the image sensor and send it to the electronic device.

[0054] Taking the application of the image processing method to electronic devices as an example, the electronic devices may include: terminal devices, such as mobile terminals or fixed terminals. Among them, mobile terminals may include: mobile phones, tablet computers or laptop computers and other devices. Fixed terminals may include: desktop computers, smart TVs, etc.

[0055] It should be noted that the image sensor disclosed in the present invention, or photosensitive element, is a device that converts optical images into electronic signals and is widely used in digital cameras or other electronic optical devices. It mainly includes a lens through which light enters, a photosensitive element and its surrounding circuits and a processing chip. When light passes through the lens and enters the image sensor, it hits the photosensitive unit on the photosensitive element. The photons in the photosensitive unit cause the release of electrons and gather at the output end of the chip to form a voltage signal. These voltage signals are converted into digital signals by the processing chip for generating images.

[0056] Figure 2 is a diagram showing an actual application scenario of an image sensor according to an exemplary embodiment. Figure 2 As shown; the image sensor includes an acquisition module, a processing module and a display module. After the acquisition module captures the natural scene signal, it can be sent to the processing module for processing, and the processed image is then displayed through the display module.

[0057] In the disclosed embodiment, after the image sensor collects light signals in the natural scene, it converts them into electrical signals and sends them to the processing module. The processing module performs style migration processing, and the processed image is then displayed through the display module.

[0058] According to the product type, the type of the image sensor may be a CCD image sensor (charged coupled device) or a CMOS image sensor (complementary metal oxide semiconductor), which is not limited in the embodiments of the present disclosure.

[0059] The original image captured by the image sensor is obtained. It should be noted that the original image can be an original image that is photographed, captured or generated without subsequent modification, or it can be a grayscale image with enhanced image effect that is directly obtained. The embodiments of the present disclosure are not limited to this.

[0060] Here, light quantity information is used to describe the amount of light taken in by the image sensor when capturing the original image, that is, the amount of light signal captured by the image sensor when capturing the original image; it should be explained that the less light signal captured by the image sensor, the more obvious the dirt on the image sensor will be in the original image captured by the image sensor. Therefore, it is necessary to combine light quantity information to judge the imaging quality.

[0061] Based on the original image, the image features of the original image are obtained. It can be understood that the image features may include color features, texture features, shape features, and space features of the image.

[0062] Here, image features can be extracted by Fourier transform algorithm or edge extraction algorithm;

[0063] In some embodiments, in the field of image processing, time domain analysis can only analyze the change of the amplitude of the signal over time, and it is difficult to clearly reveal the frequency composition of the signal and the size of each frequency component, while the frequency domain signal can represent the size of the signal at different frequency components, and can provide a more intuitive and richer signal than the time domain. As a signal analysis method, Fourier transform can convert time domain signals into frequency domain signals and concentrate the energy of the signal. For any image, after Fourier transform, the area near the center point is white, representing a relatively low-frequency signal, and the edge position is black, representing a relatively high-frequency signal. The low-frequency signal represents the average value, that is, the contour information of the image; the high-frequency signal represents the mutation, that is, the details and texture of the image. If a Fourier transformed image is filtered out with a high-frequency filter to remove the low-frequency signal, the details and texture information of the image can be obtained. Correspondingly, if a Fourier transformed image is filtered out with a low-frequency filter to remove the high-frequency signal, the contour information of the image can be obtained.

[0064] In other embodiments, edge features of an image can be extracted based on an edge extraction operator. Edges can be positive or negative, such as from dark to bright is positive, and from bright to dark is negative. The image is Gaussian filtered, and the grayscale values ​​of the pixels to be filtered and their domain points are weighted averaged according to certain parameter rules. The Sobel operator is used to detect horizontal edges, vertical edges, and diagonal edges in the image, and then the local maximum values ​​in the image pixels are found. The grayscale values ​​corresponding to non-maximum points are set to 0, and the gradient amplitude of the image is non-maximum suppressed to obtain a binary image. Finally, the binary image is subjected to double threshold detection and edge connection to eliminate the influence of noise. The embodiments disclosed herein are not limited to this.

[0065] In some embodiments, the original image may be preprocessed before image feature extraction;

[0066] Here, the preprocessing may include: cropping processing, size adjustment processing, etc.; the embodiments of the present disclosure are not limited to this.

[0067] It can be understood that by preprocessing the original image, interference factors in the original image can be reduced and feature information can be highlighted, so that effective image feature extraction can be performed on the preprocessed image.

[0068] In some embodiments, after the image features of the original image are obtained, the image features may be subjected to dimensionality reduction processing, and based on the image features after dimensionality reduction processing and light quantity information, dirt detection may be performed on the image sensor.

[0069] It can be understood that by performing dimensionality reduction processing on the extracted image features, features with small information content can be excluded and the amount of calculation can be reduced, so as to improve the efficiency of image sensor dirt detection.

[0070] Based on the light quantity information and the image features, the image sensor is detected for dirt. It can be understood that the present disclosure determines whether the image sensor is dirty based on the light quantity information of the original image and the extracted image features.

[0071] Figure 3 FIG. 1 is a schematic diagram showing a comparison between an image sensor with dirt and an image sensor without dirt according to an exemplary embodiment. Figure 3 As shown; reference numeral 201 is an image sensor with dirt, and reference numeral 202 is an image sensor without dirt.

[0072] Figure 4 FIG. 1 is a schematic diagram showing a comparison of imaging effects when the image sensor is dirty and when the image sensor is not dirty when shooting the same scene according to an exemplary embodiment. Figure 4 As shown; reference numeral 301 is an image captured by an image sensor without dirt, and reference numeral 302 is an image captured by an image sensor with dirt.

[0073] For the same image region in the two images (ie, the image regions indicated by reference numerals a1 and a2 ), the image clarity of the image region indicated by reference numeral a2 is significantly lower than the image clarity of the image region indicated by reference numeral a1 .

[0074] From the above content, it can be seen that the dirtiness of the image sensor will significantly affect the imaging accuracy of the image. Therefore, in order to ensure the imaging quality of the image sensor, it is necessary to detect the dirtiness of the image sensor. It should be explained that the image quality of the original image generated by the dirty image sensor will be affected by the amount of light taken in when the image sensor collects the original image; if the light amount information indicates that the image sensor takes in less light when collecting the original image, the dirty area of ​​the image sensor (such as the location of the mildew spot) will have a greater impact on the image quality of the original image.

[0075] Based on this, when performing dirt detection on the image sensor, the image features of the original image can be used to determine the image quality of the original image, and based on the image quality and the light amount information corresponding to the original image, it can be comprehensively determined whether there is dirt in the image sensor.

[0076] The imaging effect of the image is judged based on the light information and image features, which improves the accuracy of image quality analysis and makes the detection of dirt on the image sensor more precise. The user can clean the surface of the image sensor according to the detection results, thereby improving the imaging accuracy of the target object. At the same time, the image sensor is prevented from being affected by dirt, thereby extending the service life of the image sensor.

[0077] In the disclosed embodiment, the original image captured by the image sensor and the light quantity information corresponding to the original image are obtained, wherein the light quantity information is used to describe the light quantity taken in when the image sensor captures the original image, and then the image features of the original image are obtained, and the image sensor is detected for dirt based on the light quantity information and the image features. In this way, by using the image features and the light quantity information as a priori information, the imaging quality of the original image is judged, so that it is possible to accurately judge whether the image sensor is dirty, thereby improving the reliability of the image sensor dirt detection, so as to achieve the purpose of ensuring that the surface of the image sensor is clean before using the image sensor for imaging.

[0078] In some embodiments, the performing dirt detection on the image sensor based on the light amount information and the image feature includes:

[0079] The light quantity information and the image features are input into a pre-trained perceptron model to obtain an image quality result output by the perceptron model, where the image quality result is used to indicate the probability that the image sensor is dirty.

[0080] Here, the perceptron model may be a single-layer perceptron model or a multi-layer perceptron model, which is not limited in the embodiments of the present disclosure.

[0081] Take the multi-layer perceptron as an example. The multi-layer perceptron is a machine learning model based on a neural network. It uses multiple neuron layers to achieve complex nonlinear mapping of input data, thereby achieving tasks such as data classification and regression. The multi-layer perceptron mainly consists of an input layer, a hidden layer, and an output layer. The input layer receives the original data, the hidden layer maps the input data to a high-dimensional space through a series of nonlinear transformations, and the output layer performs tasks such as classification or regression based on the output of the hidden layer.

[0082] In a multilayer perceptron, each neuron has a weight and a bias term, which are used to adjust the weight and bias of the input data, respectively, to achieve nonlinear transformation of the data. Each neuron in the hidden layer performs nonlinear transformation on the input data through activation functions such as sigmoid, relu, etc. The training process of a multilayer perceptron usually uses a back propagation algorithm to continuously optimize the performance of the model. During the training process, it is necessary to select a suitable loss function to measure the performance of the model, such as the cross entropy loss function, the mean square error function, etc.

[0083] Multilayer perceptron can process high-dimensional data, has strong nonlinear fitting ability and the ability to automatically extract image features, so it is widely used in the field of image classification.

[0084] In the disclosed embodiment, the training process of the multi-layer perception model mainly includes: a data collection phase, a training phase and a testing phase.

[0085] The data collection stage of the multi-layer perceptron includes: using the same camera to collect images with and without the influence of dirt. For images with the influence of dirt, real mold cultured in a culture dish is attached to the image sensor for real-time data collection.

[0086] During the collection process, images with and without the effects of dirt will be collected for different scenes (such as blue sky, green land, human faces, animals, and buildings) and different light conditions (for example, different apertures (F1.8, F4, F11, F16, and F22) and different shutter speeds (1 / 100, 1 / 30, and 1 / 10)). Images with the effects of dirt will be labeled as 1, and images without the effects of dirt will be labeled as 0.

[0087] The training phase of the multi-layer perceptron includes: extracting the image features of the original image through the feature extraction network. Downsampling the acquired image features to a one-dimensional vector of fixed length N1, merging the acquired light quantity information N2 with the one-dimensional vector of fixed length N1 to obtain a one-dimensional vector of length N1+N2. Then inputting the one-dimensional vector of length N1+N2 into a multi-layer perceptron.

[0088] For example, Figure 5 As shown, Figure 5 It is a schematic diagram of the structure of a three-layer perceptron according to an exemplary embodiment; the three-layer perceptron 400 is composed of three fully connected layers; the image features of the original image can be downsampled to a one-dimensional vector with a length of 1000, and the light amount information and the one-dimensional vector with a length of 1000 are merged to obtain a one-dimensional vector with a length of 1001; the one-dimensional vector with a length of 1001 is input into the three-layer perceptron model.

[0089] Among them, the first fully connected layer (shown by label 401) of the three-layer perceptron model maps a vector of length 1001 to a vector of length 256; the second fully connected layer (shown by label 402) maps a vector of length 256 to a vector of length 10; the third fully connected layer (shown by label 403) maps a vector of length 10 to a probability value between 0 and 1.

[0090] For images affected by dirt, the purpose of training is to make the output probability value closer to 1. For images not affected by dirt, the purpose of training is to make the output probability value closer to 0.

[0091] In some embodiments, the loss function used in the training of the three-layer perceptron model may be a cross entropy loss function, and the training adopts an SGD optimizer, and the learning rate is set to 10e-4;

[0092] It should be noted that, based on the above training parameters, the three-layer perceptron model can reach convergence after training for about 100,000 iterations, and the classification accuracy of the three-layer perceptron model can reach 98%.

[0093] The testing phase of the multi-layer perceptron includes: encoding the acquired image features and the light quantity information corresponding to the image into a one-dimensional vector, inputting it into the above-mentioned trained multi-layer perceptron model, and outputting the judgment result of whether the image sensor is dirty.

[0094] In this way, according to the above process, a pre-trained multi-layer perceptron model can be obtained.

[0095] After the trained perceptron model is obtained, the light quantity information and the image features may be input into the pre-trained perceptron model.

[0096] It can be understood that the obtained image features are first downsampled to a one-dimensional vector of length N1, and then the light amount information N2 is merged with the one-dimensional vector of length N1 to obtain a one-dimensional vector of length N1+N2, and finally the one-dimensional vector of length N1+N2 is input into a pre-trained perceptron model to obtain the image quality result output by the perceptron model.

[0097] Here, the image quality result is used to indicate the probability that the image sensor is dirty. It can be understood that the trained perceptron model has image data with the influence of dirt (marked as 1) and image data without the influence of dirt (marked as 0). The one-dimensional vector with an input length of N1+N2 is vector mapped in the fully connected layer of the perceptron model, and its vector length is mapped to a probability value between 0 and 1, so as to obtain the detection result of whether the original image is an image with the influence of dirt or an image without the influence of dirt.

[0098] If the perceptron model outputs that the original image is an image affected by dirt, it means that the image sensor that collects the original image is dirty; if the perceptron model outputs that the original image is an image not affected by dirt, it means that the image sensor that collects the original image is not dirty.

[0099] The disclosed embodiment inputs light quantity information and image features into a pre-trained perceptron model, and uses the light quantity information as prior information to judge the image quality of the original image, thereby obtaining an image quality result output by the perceptron model. The image quality result is used to indicate the probability that the image sensor is dirty. The image quality detection result is used to determine whether the image sensor is dirty. Compared with directly detecting the dirt on the image sensor, the detection method disclosed in the present invention has simple steps and accurate detection results, thereby improving the detection efficiency of the image sensor and reducing the detection cost.

[0100] In some embodiments, the method further comprises:

[0101] Performing grayscale processing on the original image to obtain a grayscale image;

[0102] The image features of the grayscale image are extracted using a feature extraction network.

[0103] In the embodiment of the present disclosure, the original image may be firstly grayscaled to obtain a grayscale image corresponding to the original image;

[0104] It should be noted that compared with color images, grayscale images take up less memory, run faster, visually enhance image contrast, highlight the target area, and make image processing more convenient and efficient.

[0105] Here, the grayscale processing can be: maximum method, average method, minimum method or weighted average method. For example, the maximum method is to directly take the value of the component with the largest value among the three components R, B, and G, and the average method is to take the average of the values ​​of the three components R, B, and G. The embodiments of the present disclosure are not limited to this.

[0106] In some embodiments, the original image may be grayscaled based on a weighted average method.

[0107] The weighted average method is to average the pixels of the red, green and blue channels in the color image according to certain weights to obtain a grayscale value. Since the human eye is most sensitive to green and least sensitive to blue, a more reasonable grayscale image can be obtained by weighted averaging the three components of RGB according to the following formula:

[0108] Gray(i,j)=0.299*R(i,j)+0.578*G(i,j)+0.114*B(i,j)

[0109] It should be noted that the grayscale processing method based on the weighted average takes into account the influence of the image color on the image display effect, and can obtain a better grayscale image.

[0110] In the embodiment of the present disclosure, the R channel, G channel, and B channel of the original image are weighted averaged according to their respective weights to obtain a grayscale value, so that the R value, G value, and B value of each pixel in the original image are converted into the grayscale value of the pixel, thereby obtaining a grayscale image that enhances the image effect and facilitates image feature extraction.

[0111] Here, a feature extraction network is used to extract image features of the grayscale image. It should be explained that a convolutional neural network can be used as a feature extractor, and the grayscale image can be input into a trained convolutional neural network to extract image features.

[0112] Here, the convolutional neural network used is a classification network pre-trained in the ImageNet dataset, such as VGG-16, VGG-19 or ResNet-50. Since the ImageNet dataset is a dataset with millions of data points, the models trained in this dataset are often used to extract high-dimensional image features.

[0113] In some embodiments, the basic structure of the convolutional neural network consists of several convolutional layers, pooling layers, and fully connected layers. Among them, the convolutional layer performs discrete convolution operations on the input image by using a set of learnable convolution kernels to generate an output feature map. The convolution kernel can automatically extract the features of the input data, such as edges, corners, or textures. The pooling layer can shrink the output feature map and reduce the dimension of the feature map to improve computational efficiency. The fully connected layer can map the output feature map of the pooling layer to a specific category to complete the classification task.

[0114] In a convolutional neural network, each layer can extract different image features. High-level features are composed of low-level features. The first convolutional layer of a convolutional neural network can extract some basic edge features, such as vertical edges, horizontal edges, oblique lines or points, etc. Through a deeper network structure, the second convolutional layer can learn some higher-level features, such as corners, curves, etc. As the number of network layers increases, the size and number of convolution kernels will increase accordingly. After obtaining higher-level features, such as the third and fourth convolution kernels, more abstract features such as circles and triangles can be learned.

[0115] In the disclosed embodiment, the original image is grayed and converted into a grayscale image; based on a convolutional neural network, its input layer receives the grayscale image, the convolution layer extracts the features of the grayscale image, the pooling layer reduces the feature dimension extracted by the convolution layer, and the global average pooling uses the average value of all pixels in the image after dimensionality reduction as the feature value, and its output layer outputs the processing result of the input image, that is, outputs the classification result of each pixel in the grayscale image.

[0116] Here, the extracted image features of the grayscale image may be used to describe information such as shape, texture and / or brightness of the image.

[0117] The image features may include: gray level co-occurrence matrix features, gray level histogram features or gray level gradient features, etc.

[0118] Among them, the grayscale co-occurrence matrix feature refers to selecting a part of the area in the image and calculating the co-occurrence probability between the grayscale values ​​of two pixels in these areas, and extracting a series of features from them, including entropy, contrast and / or correlation; the grayscale histogram feature refers to counting the grayscale values ​​of all pixels according to the numerical size to obtain a histogram, from which the mean, variance and / or peak value are extracted; the grayscale gradient feature refers to calculating the grayscale gradient value at each pixel in the image, and extracting features such as edges, focus and / or textures by analyzing the gradient value.

[0119] The disclosed embodiments do not limit the image features, as long as the extracted image features can be used in the fields of image classification and target recognition.

[0120] The disclosed embodiment performs grayscale processing on the original image to obtain a grayscale image, then uses a convolutional neural network as a feature extractor, inputs the grayscale image into a pre-trained convolutional neural network, uses the middle layer of the convolutional neural network to extract the image features of the grayscale image, and outputs the pixel classification result of the grayscale image. On the one hand, grayscale processing of the original image visually enhances the image effect, highlights the target area of ​​the image, and facilitates the image feature extraction operation. On the other hand, using a neural network model to extract features from the grayscale image, the pixel features of the grayscale image are classified, which can effectively improve the accuracy of the image sensor's dirt detection, thereby improving the robustness and reliability of the image sensor's dirt detection.

[0121] In some embodiments, the extracting the image features of the grayscale image using a feature extraction network includes:

[0122] Acquiring brightness information of the grayscale image;

[0123] Based on the brightness information, determining whether the grayscale image is an image with abnormal illumination;

[0124] If the grayscale image is not an image with abnormal brightness, the feature extraction network is used to extract image features of the grayscale image.

[0125] Here, the brightness information of the grayscale image is obtained. It can be understood that the grayscale values ​​of all pixels in the grayscale image are obtained, and the brightness information of the grayscale image is determined based on the grayscale values ​​of all pixels.

[0126] Here, the brightness information of the grayscale image may be determined based on a maximum grayscale value, a minimum grayscale value, an average grayscale value, etc. of the grayscale image.

[0127] In the embodiment of the present disclosure, the brightness information may be the average grayscale value in the grayscale image. The average grayscale value of the grayscale image is obtained based on the following formula, where N represents the number of rows of pixels, M represents the number of columns of pixels, and Y i,j Represents the grayscale value corresponding to the pixel in the i-th row and j-th column. Represents the average gray value of all pixels.

[0128]

[0129] After obtaining the brightness information of the grayscale image, the brightness information may be compared with a preset brightness range to determine whether the grayscale image is an image with abnormal lighting.

[0130] It should be explained that the present invention can pre-set a brightness range according to the shooting environment of the original image, and the brightness range includes a maximum brightness threshold and a minimum brightness threshold, perform image brightness judgment on the acquired grayscale image, and screen out images with normal lighting, thereby improving the accuracy of image feature extraction.

[0131] If the calculated brightness information is greater than the preset maximum brightness threshold, the grayscale image is judged to be too bright; if the calculated brightness information is less than the preset minimum brightness threshold, the grayscale image is judged to be too dark. In other words, the images in these two cases are abnormal illumination images, and their image quality is not high, which is not conducive to extracting effective features based on the existing image content.

[0132] Thus, in the case where the image imaging effect is too bright or too dark, the present disclosure will not perform feature extraction operations on the grayscale image, and needs to reacquire the original image until an image with normal illumination is obtained.

[0133] If the calculated brightness information is within the preset brightness range, it means that the image is not an image with abnormal lighting and the image quality is high, which is conducive to extracting effective features in the image.

[0134] In this way, the present disclosure will perform a feature extraction operation on the grayscale image when the image imaging effect is good, thereby improving the accuracy of image feature extraction.

[0135] In some embodiments, if the brightness information of the grayscale image is: the maximum grayscale value of the grayscale image; the maximum grayscale value of the grayscale image can be compared with a preset minimum brightness threshold to determine whether the current grayscale image is an image with abnormal lighting.

[0136] If the maximum grayscale value of the acquired grayscale image is less than the preset minimum brightness threshold, it means that the imaging effect of the image is too dark and the current grayscale image is an image with abnormal lighting.

[0137] In other embodiments, if the brightness information of the grayscale image is: the minimum grayscale value of the grayscale image; the maximum grayscale value of the grayscale image can be compared with a preset maximum brightness threshold to determine whether the current grayscale image is an image with abnormal lighting.

[0138] If the minimum grayscale value of the obtained grayscale image is less than the preset maximum brightness threshold, it means that the imaging effect of the image is too bright and the current grayscale image is an image with abnormal lighting.

[0139] Thus, in the case where the image imaging effect is too bright or too dark, the present disclosure will not perform feature extraction operations on the grayscale image, and the original image needs to be reacquired until an image with normal illumination is obtained.

[0140] The disclosed embodiment obtains the brightness information of the grayscale image, and based on the brightness information, determines whether the current image is an image with abnormal lighting. When the grayscale image is not an image with abnormal lighting, a convolutional neural network is used as a feature extractor to extract image features of the grayscale image. In this way, an image with normal lighting is obtained by screening the brightness information, which is beneficial for the neural network model to extract effective image features based on the existing image, thereby further improving the accuracy of dirt detection of the image sensor.

[0141] In some embodiments, grayscale processing is performed on the original image to obtain a grayscale image, including:

[0142] Performing white balance processing on the original image to obtain a gain image;

[0143] The gain image is grayscaled to obtain the grayscale image.

[0144] Due to the different spectral characteristics of different lighting, objects will appear in different tones under different color temperature lighting. The higher the color temperature of the light source, the bluer the object will appear, and the lower the color temperature of the light source, the yellower the object will appear. Therefore, it is necessary to perform white balance correction on images taken under light sources with different color temperatures. The reflection of the red, green, and blue channels in a white scene is the same, so the channel values ​​of the red, green, and blue channels in a white scene are equal. Therefore, the color cast caused by light sources with different color temperatures can be corrected based on white.

[0145] In some embodiments, the white image blocks in the image are first found, and the average values ​​of all white image blocks in the red, green, and blue channels are calculated respectively. The red and blue channels in the image are corrected based on the green channel. Through the correction, the values ​​of all white images in the image in the red, green, and blue channels are equal.

[0146] Since the peak of the brightness response curve of the human eye is closer to the green frequency range, the G channel can be used to obtain brightness information close to what the human eye actually observes. At the same time, the spectral response intensity of the image sensor to the green band is generally stronger than that of the blue and red bands, and the response wavelength range is wider. Due to the strong spectral response to the green band, the image sensor obtains more energy in the green band, which is reflected in the RAW data of the image, meaning that the grayscale value of the G component in the RAW domain is larger than that of the R and B components.

[0147] Therefore, when performing white balance processing on the original image, the present disclosure uses the G component as a reference to perform significant gain adjustment on the R component and the B component.

[0148] Here, the original image is subjected to white balance processing to obtain a gain image. It can be understood that the gain coefficients corresponding to the R channel, B channel and G channel of the original image are obtained, and then the R channel, B channel and G channel of the original image are multiplied by their respective gains to obtain the gain image. Among them, the gain coefficient is obtained by combining the hardware characteristics when the original image is exposed. The gain image obtained by white balance processing corrects the image color temperature of the original image, restores the main color of the image, and makes the pictures taken under different light sources similar to the pictures viewed by the human eye.

[0149] In some embodiments, taking the original image in the Bayer format as an example, first average its two G channels, convert the Bayer format into a normal RGB image format, and obtain the gain coefficients of the three channels of the RGB image according to the spectral information of the image when it is exposed. Then multiply the R channel, B channel and G channel in the RGB image by their respective gain coefficients to obtain a gain image. Then, based on the weighted average method of the following formula, the R channel, B channel and G channel in the gain image are weighted averaged according to their respective weights to obtain a gray value, so that the R value, G value and B value of each pixel in the gain image are converted into the calculated gray value, and the gain image is converted into a gray image.

[0150] Y=0.298R+0.612G+0.117B

[0151] In the disclosed embodiment, white balance processing is performed on the original image to obtain a gain image, and then the gain image is grayscaled to obtain a grayscale image. This makes image processing more flexible, improves the resolution of the original image, corrects the image color temperature of the original image, avoids image color distortion, and lays a good foundation for subsequent image grayscale processing, thereby further improving the accuracy of image feature extraction.

[0152] In some embodiments, the method further comprises:

[0153] When it is determined that the image sensor is dirty, a prompt message is output.

[0154] In the disclosed embodiment, if it is determined that the image sensor is dirty, a prompt box will automatically pop up on the image display interface of the device, indicating that the image sensor may be dirty and prompting the user to go to a professional after-sales store to clean the image sensor.

[0155] When determining that the image sensor is dirty, the embodiment of the present disclosure outputs a prompt message, so that the user can be prompted in time so that the user can perform subsequent cleaning operations on the dirty image sensor, thereby preventing greater losses.

[0156] In some embodiments, the light amount information includes at least one of the following information:

[0157] Aperture size information, shutter speed information, and exposure time information.

[0158] The aperture is used to limit the amount of light entering the lens and adjust the brightness of the image. The larger the aperture, the more light enters the image sensor, and the smaller the aperture, the less light enters the image sensor.

[0159] Figure 6 FIG. 1 is a schematic diagram showing a comparison of imaging effects of a dirty image sensor at different apertures according to an exemplary embodiment. Figure 6 As shown, reference numeral 501 is a schematic diagram of the mapping of the mildew spot by the image sensor under a large aperture, and reference numeral 502 is a schematic diagram of the mapping of the mildew spot by the image sensor under a small aperture.

[0160] like Figure 6 As shown in the figure, the projection area of ​​the mildew on the sensor in the schematic diagram of the image sensor mapping the mildew under a large aperture (indicated by the label b1) is smaller than the projection area of ​​the mildew on the sensor in the schematic diagram of the image sensor mapping the mildew under a small aperture (indicated by the label b2). This is because when the aperture is small, the image sensor captures less light signal, and if the image sensor is dirty, it will be more obvious in the original image captured by the image sensor.

[0161] It is understandable that the faster the shutter speed, the more light enters the image sensor, and the slower the shutter speed, the less light enters the image sensor.

[0162] The shutter speed also represents the exposure time during shooting. That is, the faster the shutter speed, the longer the exposure time and the more light enters; the slower the shutter speed, the shorter the exposure time and the less light enters.

[0163] Therefore, the disclosed embodiment needs to combine the aperture size information, shutter speed information and image exposure time information to judge the imaging quality of the image. When the image sensor is dirty and the image sensor captures less light information, the projection display of the dirt on the original image will be more obvious, which is more conducive to the user to judge the quality of the image captured by the image sensor, thereby further improving the dirt detection of the image sensor.

[0164] Figure 7 is a schematic diagram of a process of an image processing method according to an exemplary embodiment Figure 2 ,like Figure 7 As shown, the image sensor can be contaminated based on the image features and the light quantity information corresponding to the image. The method mainly includes the following steps:

[0165] In step S601, a camera captures an image;

[0166] In some embodiments, the original image is captured by an image sensor inside the camera, and at the same time, the light quantity corresponding to the original image is obtained, wherein the light quantity information at least includes: aperture size information, shutter speed information and / or exposure time information.

[0167] In step S602, the image is preprocessed;

[0168] Here, the preprocessing of the original image includes: performing white balance processing on the original image to obtain a gain image; and then performing grayscale processing on the gain image to obtain a grayscale image.

[0169] In some embodiments, taking the G channel of the original image as a reference, the white balance gain coefficients corresponding to the R channel and the B channel of the original image are obtained, so that the R channel and the B channel of the original image are multiplied by their respective white balance gains to obtain a gain image; based on an image grayscale algorithm, such as using a grayscale weighted average algorithm, the R channel, G channel, and B channel of the gain image are weighted averaged according to their respective weights to obtain a grayscale value, so that the R value, G value, and B value of each pixel in the original image are converted into a calculated grayscale value to obtain a grayscale image.

[0170] In step S603, it is determined whether the current brightness meets the determination condition;

[0171] Here, by obtaining the brightness information of all pixels of the current grayscale image, based on the brightness information, it is determined whether the current grayscale image is an image with abnormal lighting. If the current grayscale image is an image with abnormal lighting, the operation of step S601 is restarted; if the current grayscale image is not an image with abnormal lighting, the image feature extraction operation is started.

[0172] In some embodiments, a brightness interval is pre-set, and the brightness interval includes: a maximum brightness threshold and a minimum brightness threshold; then the brightness information of all pixels of the current grayscale image is obtained, based on the formula Get the average grayscale value of the grayscale image.

[0173] If the calculated average brightness value is greater than the preset maximum brightness threshold, it is determined that the imaging effect of the grayscale image is too bright; if the calculated average brightness value is less than the preset minimum brightness threshold, it is determined that the imaging effect of the grayscale image is too dark. That is, the images in these two cases are abnormal illumination images, and their image quality is not high, which is not conducive to extracting effective features according to the existing image content. Therefore, it is necessary to restart steps S601-S602.

[0174] In other embodiments, if the calculated average grayscale value is within a preset brightness range, it means that the image is not an image with abnormal lighting and has high image quality, which is conducive to extracting effective features in the image. Therefore, subsequent detection can be performed.

[0175] In step S604, a pre-trained convolutional neural network is input to extract image features;

[0176] In some embodiments, a convolutional neural network is used as a feature extractor, and a grayscale image that meets the brightness judgment is input into the convolutional neural network. The middle layer of the convolutional neural network extracts features of the grayscale image, and the output layer outputs the pixel classification result of the grayscale image.

[0177] In step S605, the image features and light quantity information are combined and input into a multi-layer perceptron;

[0178] In some embodiments, when training a multi-layer perceptron, images affected by dirt and images not affected by dirt are collected respectively, and the images affected by dirt are marked as 1, and the images not affected by dirt are marked as 0, and through iterative learning, a trained multi-layer perceptron model is obtained.

[0179] The image features extracted by the convolutional neural network are downsampled to obtain a one-dimensional vector of length N1. The light amount information N2 corresponding to the original image is merged with the one-dimensional vector of length N1 to obtain a one-dimensional vector of length N1+N2. The one-dimensional vector of length N1+N2 is input into the pre-trained perceptron model, and vector mapping is performed in the fully connected layer of the perceptron model to map its vector length to a probability value between 0 and 1.

[0180] In step S606, the judgment result is output;

[0181] Here, the quality detection result of the grayscale image is output by a pre-trained multi-layer perceptron, and the quality detection result represents the probability that the image sensor is dirty.

[0182] In some embodiments, the multi-layer perceptron model outputs an image probability marked as 1, and the current grayscale image is an image affected by dirt;

[0183] In other embodiments, if the multi-layer perceptron model outputs an image probability marked as 0, the current grayscale image is an image that is not affected by dirt.

[0184] In step S607, it is determined whether the current sensor is dirty;

[0185] Here, whether the current image sensor is dirty is determined by the quality detection result of the grayscale image output by the multi-layer perceptron.

[0186] In some embodiments, if the current image output is an image that is not affected by dirt, then the current image sensor is not dirty, and steps S601 - S606 may be restarted.

[0187] In other embodiments, if the currently output image is an image affected by dirt, then the current sensor is dirty and the next operation can be performed.

[0188] In step S608, the terminal device outputs a user prompt;

[0189] In some embodiments, if the image sensor is dirty, a prompt box will automatically pop up on the image display interface of the terminal device, indicating that the image sensor may be dirty, and promptly reminding the user to clean the image sensor to avoid causing greater losses.

[0190] The present disclosure also provides an image processing device, Figure 8 is a block diagram of an image processing device according to an exemplary embodiment. Figure 8 As shown, the image processing device is applied to an electronic device, the image processing device 700, comprising:

[0191] A first acquisition module 701 is configured to acquire the original image captured by the image sensor and light quantity information corresponding to the original image; wherein the light quantity information is used to describe the amount of light taken in when the image sensor captures the original image;

[0192] A second acquisition module 702 is configured to acquire image features of the original image;

[0193] The first processing module 703 is configured to perform dirt detection on the image sensor based on the light quantity information and the image feature.

[0194] In some embodiments, the first processing module 703 is configured to:

[0195] Inputting the light quantity information and the image features into a pre-trained perceptron model to obtain an image quality result output by the perceptron model;

[0196] The image quality result is used to indicate the probability that the image sensor is dirty.

[0197] In some embodiments, the image processing module 700 further includes:

[0198] The second processing module is configured to perform grayscale processing on the original image based on an image grayscale algorithm to obtain a grayscale image;

[0199] Based on the brightness information, determining whether the grayscale image is an image with abnormal illumination;

[0200] If the grayscale image is not an image with abnormal brightness, the feature extraction network is used to extract image features of the grayscale image.

[0201] In some embodiments, the second processing module is further configured to:

[0202] Acquiring brightness information of the grayscale image;

[0203] Based on the brightness information, determining whether the grayscale image is an image with abnormal illumination;

[0204] If the grayscale image is not an image with abnormal brightness, the feature extraction network is used to extract image features of the grayscale image.

[0205] In some embodiments, the second processing module is further configured to:

[0206] Performing white balance processing on the original image to obtain a gain image;

[0207] The gain image is grayscaled to obtain the grayscale image.

[0208] In some embodiments, the image processing apparatus 700 further includes:

[0209] The output module is configured to output prompt information when it is determined that the image sensor is dirty.

[0210] In some embodiments, the first acquisition module 701 is further configured so that the light quantity information includes at least one of the following information:

[0211] Aperture size information, shutter speed information, and exposure time information.

[0212] Fig. 9 8 is a hardware structure block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a message transceiver device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0213] Reference Fig. 9 , device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0214] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0215] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0216] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0217] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0218] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0219] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0220] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0221] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 3G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0222] In an exemplary embodiment, the device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0223] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0224] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a network processing device, enables the network processing device to perform an image processing method, the method comprising:

[0225] Acquire an original image captured by an image sensor and light quantity information corresponding to the original image, wherein the light quantity information is used to describe the amount of light taken in when the image sensor captures the original image;

[0226] Acquire image features of the original image;

[0227] Based on the light amount information and the image feature, dirt detection is performed on the image sensor.

[0228] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present disclosure. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present disclosure, the size of the serial number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. The serial numbers of the embodiments of the present disclosure are for description only and do not represent the advantages and disadvantages of the embodiments.

[0229] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0230] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0231] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0232] In addition, all functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may be separately configured as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0233] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0234] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, It is characterized in that The method comprises: Acquire an original image captured by an image sensor and light quantity information corresponding to the original image, wherein the light quantity information is used to describe the amount of light taken in when the image sensor captures the original image; Acquire image features of the original image; Based on the light amount information and the image feature, dirt detection is performed on the image sensor.

2. The method according to claim 1, It is characterized in that The performing dirt detection on the image sensor based on the light amount information and the image feature includes: The light quantity information and the image features are input into a pre-trained perceptron model to obtain an image quality result output by the perceptron model; the image quality result is used to indicate the probability that the image sensor is dirty.

3. The method according to claim 1, It is characterized in that The method further comprises: Performing grayscale processing on the original image to obtain a grayscale image; The image features of the grayscale image are extracted using a feature extraction network.

4. The method according to claim 3, It is characterized in that The step of extracting the image features of the grayscale image using a feature extraction network includes: Acquiring brightness information of the grayscale image; Based on the brightness information, determining whether the grayscale image is an image with abnormal illumination; If the grayscale image is not an image with abnormal brightness, the feature extraction network is used to extract image features of the grayscale image.

5. The method according to claim 3, It is characterized in that The grayscale processing of the original image to obtain a grayscale image includes: Performing white balance processing on the original image to obtain a gain image; The gain image is grayscaled to obtain the grayscale image.

6. The method according to claim 1, It is characterized in that The method further comprises: When it is determined that the image sensor is dirty, a prompt message is output.

7. The method according to claim 1, It is characterized in that The light quantity information includes at least one of the following information: Aperture size information, shutter speed information, and exposure time information.

8. An image processing device, It is characterized in that The image processing device comprises: A first acquisition module is configured to acquire the original image captured by the image sensor and light quantity information corresponding to the original image; wherein the light quantity information is used to describe the amount of light taken in when the image sensor captures the original image; A second acquisition module is configured to acquire image features of the original image; The first processing module is configured to perform dirt detection on the image sensor based on the light quantity information and the image feature.

9. An electronic device, It is characterized in that include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: implement the steps in the image processing method according to any one of claims 1 to 7 when executing the executable instructions.

10. A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, the steps in the image processing method according to any one of claims 1 to 7 are implemented.

Images