Method, system, electronic device and storage medium for identifying oil blur in image

Abstract

The present invention discloses a method, system, electronic device and storage medium for identifying oil blur in an image, the identification method comprising: obtaining a number of sample image frames and corresponding category annotation information; processing each frame of the sample image to obtain a corresponding edge information image; calculating the oil blur information corresponding to each frame of the sample image according to the edge information image; training a target classifier for outputting whether there is oil blur in the image based on the oil blur information and category annotation information of each frame of the sample image; obtaining a target frame image and inputting the target classifier to obtain a classification result. The present invention can effectively distinguish the difference between oil smoke, water vapor rising blur and oil smoke adhesion blur, improve the recognition accuracy and efficiency of oil blur in images, and meet the higher usage requirements of users.

Description

Technical Field

[0001] The present invention relates to the technical field of smoke and stove scenes, and in particular to a method, system, electronic device and storage medium for identifying oily blur in an image. Background Art

[0002] With the advancement of intelligentization, it has become a trend to use cameras to realize automatic control of range hoods in range hood environments, such as automatically identifying the concentration of oil smoke and then controlling the range hood to switch gears; or identifying whether the stove is unattended for a long time when it is on fire, or whether it is in a dry burning state. In this application scenario, the camera is a key input source of perception signals, and its imaging quality will have a great impact on the real-time decision-making and judgment of the range hood. A large amount of oil smoke particles will be generated during the process of using the range hood. When the oil smoke particles adhere to the lens, the overall image will be blurred, and the image information such as edges and colors in the image will be weakened, which will have a negative impact on the subsequent performance of the range hood.

[0003] At present, the following image blur discrimination methods are mainly used to identify oil stains: (1) a discrimination method based on spectral information. This method grayscales the image and performs a two-dimensional discrete Fourier transform, performs statistical analysis on the spectrum of a specific area, and finally achieves the discrimination between a clear image and a blurred image. However, the rising oil smoke in a smoke stove environment will produce local image blur, which is an unstable state and will reduce the accuracy of the spectrum analysis. (2) a discrimination method combined with image salient region segmentation. This method uses a segmentation algorithm to extract the local area of ​​interest, and then calculates the eigenvalues ​​of the global image and the area of ​​interest for classification and judgment. However, in an actual smoke stove scene, smoke will cause image blur. This method cannot distinguish between oil smoke rising blur and lens oil blur. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the defect of low recognition accuracy of oily blur in images in the prior art, and to provide a method, system, electronic device and storage medium for recognizing oily blur in images.

[0005] The present invention solves the above technical problems through the following technical solutions:

[0006] The present invention provides a method for identifying oily blur in an image, the method comprising:

[0007] Acquire several frames of sample images and corresponding category annotation information; wherein the category annotation information is used to characterize whether oily blur exists in the image or not;

[0008] Processing each frame of the sample image to obtain a corresponding edge information image;

[0009] Obtaining the oil stain blur information corresponding to each frame of the sample image by calculation according to the edge information image;

[0010] Taking the oil stain blurriness information corresponding to each frame of the sample image as input and the corresponding category labeling information as output, training a target classifier for outputting whether there is oil stain blur in the image;

[0011] A target frame image is obtained and input into the target classifier to obtain a classification result.

[0012] Preferably, the step of processing each frame of the sample image to obtain a corresponding edge information image comprises:

[0013] Performing grayscale processing on each frame of the sample image to obtain a corresponding grayscale image;

[0014] Calculating the second-order derivatives of different pixel points in the grayscale image to obtain an intermediate result corresponding to each pixel point;

[0015] Each of the intermediate results is added together to obtain the edge information image corresponding to the sample image.

[0016] Preferably, the step of calculating the oil stain blurriness information corresponding to each frame of the sample image according to the edge information image comprises:

[0017] For each frame of the sample image, a first variance value of the edge information image is calculated based on the intermediate result corresponding to each pixel in the grayscale image, and the first variance value is used as the overall blurriness information of the current frame.

[0018] Preferably, the identification method further comprises:

[0019] Predetermine different fume generation areas;

[0020] The step of calculating the oil stain blurriness information corresponding to each frame of the sample image according to the edge information image also includes:

[0021] According to different fume generation areas, the edge information image is processed into blocks to obtain different image blocks;

[0022] For each frame of the sample image, a second variance value of each image block is calculated based on the intermediate result corresponding to each pixel in the image block, and the second variance value is used as the local blur information of the current frame;

[0023] The oil stain blurriness information of each frame of the sample image includes the overall blurriness information and local blurriness information.

[0024] Preferably, the step of calculating the oil stain blurriness information corresponding to each frame of the sample image according to the edge information image further includes:

[0025] Calculating the fuzziness difference of the oil stain fuzziness information of two frames of the sample image separated by a set number of frames;

[0026] The fuzziness difference is superimposed with the overall fuzziness information and the local fuzziness information of the current frame to obtain the oil fuzziness information of the current frame.

[0027] Preferably, the step of taking the oil stain blur information corresponding to each frame of the sample image as input and the corresponding category label information as output, and training to obtain a target classifier for whether there is oil stain blur in the image comprises:

[0028] The oil blur information corresponding to each frame of the sample image is used as input, and the corresponding category label information is used as output. A support vector machine (SVM) or a neural network algorithm is used to train to obtain a target classifier for outputting whether there is oil blur in the image.

[0029] Preferably, before training the target classifier, the recognition method further comprises:

[0030] Normalizing the oil pollution ambiguity information; and / or,

[0031] After the step of acquiring the target frame image and inputting the target classifier to obtain the classification result, the following steps are further included:

[0032] Calculating the confidence level corresponding to the classification result;

[0033] It is determined whether the confidence is greater than a confidence threshold, and if so, the classification result is determined to be valid; if not, the classification result is determined to be invalid.

[0034] Preferably, after the step of determining that the classification result is valid, the method further includes:

[0035] When the classification result indicates that there is oil blur in the image, a prompt message indicating that there is oil on the lens is generated; and / or a control instruction is generated to execute a lens cleaning operation.

[0036] The present invention also provides a system for identifying oily blur in an image, the system comprising:

[0037] A sample information acquisition module is used to acquire a number of sample image frames and corresponding category labeling information; wherein the category labeling information is used to characterize whether oily blur exists in the image or not;

[0038] An edge information image acquisition module, used for processing each frame of the sample image to acquire a corresponding edge information image;

[0039] A fuzziness information calculation module, used for calculating the oil fuzziness information corresponding to each frame of the sample image according to the edge information image;

[0040] A classifier training module, used to take the oil stain blur information corresponding to each frame of the sample image as input, take the corresponding category label information as output, and train to obtain a target classifier for outputting whether there is oil stain blur in the image;

[0041] The image classification module is used to obtain the target frame image and input it into the target classifier to obtain the classification result.

[0042] Preferably, the edge information image acquisition module includes:

[0043] A grayscale processing unit, used for performing grayscale processing on each frame of the sample image to obtain a corresponding grayscale image;

[0044] An intermediate result calculation unit, used to calculate the second-order derivatives of different pixel points in the grayscale image to obtain an intermediate result corresponding to each pixel point;

[0045] The edge information image acquisition unit is used to add each of the intermediate results to obtain the edge information image corresponding to the sample image.

[0046] Preferably, the ambiguity information calculation module includes:

[0047] The overall blur information acquisition unit is used to calculate the first variance value of the edge information image based on the intermediate result corresponding to each pixel in the grayscale image for each frame of the sample image, and use the first variance value as the overall blur information of the current frame.

[0048] Preferably, the identification system further comprises:

[0049] An area determination module is used to predetermine different oil smoke generation areas;

[0050] The ambiguity information calculation module also includes:

[0051] An image block processing unit, used for performing block processing on the edge information image according to different oil fume generation areas to obtain different image blocks;

[0052] A local blur information acquisition unit, configured to calculate, for each frame of the sample image, a second variance value of each image block based on the intermediate result corresponding to each pixel in the image block, and use the second variance value as the local blur information of the current frame;

[0053] The oil stain blurriness information of each frame of the sample image includes the overall blurriness information and local blurriness information.

[0054] Preferably, the ambiguity information calculation module further includes:

[0055] A fuzziness difference calculation unit, used for calculating a fuzziness difference of the oil stain fuzziness information of two frames of the sample image separated by a set number of frames;

[0056] The oil stain ambiguity information acquisition unit is used to superimpose the ambiguity difference with the overall ambiguity information and the local ambiguity information of the current frame to acquire the oil stain ambiguity information of the current frame.

[0057] Preferably, the classifier training module is used to take the oil blur information corresponding to each frame of the sample image as input, and the corresponding category labeling information as output, and use support vector machine SVM or neural network algorithm training to obtain a target classifier for outputting whether there is oil blur in the image.

[0058] Preferably, the identification system further comprises:

[0059] A normalization processing module, used for normalizing the oil pollution ambiguity information before training the target classifier; and / or,

[0060] A confidence calculation module, used to calculate the confidence corresponding to the classification result;

[0061] The judgment module is used to judge whether the confidence is greater than a confidence threshold. If so, the classification result is determined to be valid; if not, the classification result is determined to be invalid.

[0062] Preferably, the identification system further comprises:

[0063] a reminder information generating module, for generating a reminder information indicating that oil stains exist on the lens when the classification result indicates that the image is blurry due to oil stains; and / or,

[0064] The control instruction generating module is used to generate a control instruction to perform a lens cleaning operation when the classification result indicates that there is oil blur in the image.

[0065] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for identifying oil blur in an image when executing the computer program.

[0066] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for identifying oil blur in an image are implemented.

[0067] The positive and progressive effects of the present invention are:

[0068] (1) Convert several consecutive frames of sample images into grayscale images, calculate the second-order derivative of each pixel in each grayscale image to obtain the corresponding edge information image, and use the variance value of the edge information image as the overall blur; divide the edge information image into multiple image blocks based on the predetermined oil smoke generation area, and use the variance value corresponding to each image block as the local blur. Then, use the classification model to iteratively train based on the overall blur and local blur corresponding to each frame of the sample image and the pre-labeled category information to obtain a classifier for identifying whether there is oil blur in the image, thereby improving the recognition accuracy and efficiency of oil blur in the image and effectively avoiding the interference caused by oil smoke, water vapor rising, etc. in the smoke stove scene; (2) Use the instantaneous blur difference between fixed frames to effectively characterize the blur change between frames, and then distinguish the difference between oil smoke, water vapor rising blur and oil smoke adhesion blur, further improving the recognition accuracy and efficiency of oil blur in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is a schematic diagram of a smoke stove scene according to Example 1 of the present invention.

[0070] Figure 2 This is a flow chart of the method for identifying oil blur in an image according to Example 1 of the present invention.

[0071] Figure 3 This is a flow chart of the method for identifying oil blur in an image according to Example 1 of the present invention.

[0072] Figure 4 This is a schematic diagram of image blocks after a sample image is divided according to Embodiment 1 of the present invention.

[0073] Figure 5 This is a flow chart of a method for identifying oil blur in an image according to Embodiment 2 of the present invention.

[0074] Figure 6 This is a module diagram of a system for identifying oil blur in an image according to Embodiment 3 of the present invention.

[0075] Figure 7This is a schematic diagram of the structure of an electronic device for implementing a method for identifying oil blur in an image according to Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0076] The present invention is further described below by way of examples, but the present invention is not limited to the scope of the examples.

[0077] The method for identifying oil stain blur in an image of the present invention is applied in a smoke stove scene, and is mainly used to identify the situation where oil stains adhere to a camera (lens) and cause image blur.

[0078] like Figure 1 As shown, the camera is installed on the range hood, and the image it collects is facing the stove, so that the stove can be completely in the field of view.

[0079] Example 1

[0080] like Figure 2 and Figure 3 As shown, the method for identifying oily blur in an image in this embodiment includes:

[0081] S101, obtaining several frames of sample images and corresponding category annotation information;

[0082] The category labeling information is used to characterize whether there is oil blur or no oil blur in the image;

[0083] The video data set in the smoke stove environment is collected and stored in advance through the camera, and then the video data in the video data set is selected according to the continuous fixed number of frames N (N ≥ 2 and an integer) sample images, and the corresponding category L is pre-labeled according to the actual blur situation of each frame of the sample image, such as L1 non-oil blur L1, oil blur L2;

[0084] The presence of oily blur in the image indicates that oil is attached to the lens of the current camera; conversely, the absence of oily blur in the image indicates that no oil is attached to the lens of the current camera.

[0085] S102, processing each frame of sample image to obtain a corresponding edge information image;

[0086] Specifically, step S102 includes:

[0087] S1021, performing grayscale processing on each frame of sample image to obtain a corresponding grayscale image;

[0088] Among them, for each sample image I' in the continuous frames, grayscale processing is performed to convert it into a grayscale image I' gray , in order to reduce the impact of differences in lighting environments among different users, and ensure the accuracy of subsequent image recognition results.

[0089] Of course, other preprocessing operations can also be performed on the sample image according to actual needs to further improve the accuracy of subsequent image recognition results. The specific other preprocessing methods can be determined and adjusted according to actual needs.

[0090] S1022, calculating the second-order derivatives of different pixel points in the grayscale image to obtain an intermediate result corresponding to each pixel point;

[0091] Among them, for the grayscale image of each frame of sample image, the second-order derivative of the pixel points with a set proportion in the grayscale image can be calculated, which can reduce the amount of data calculation while ensuring the overall blur calculation of the sample image, improve the data processing speed, and then improve the overall recognition efficiency of oil smoke blur.

[0092] In order to achieve a better result for determining the image blur, the second-order derivatives of all pixels in the grayscale image are calculated to obtain intermediate results corresponding to all pixels.

[0093] S1023. Add each intermediate result to obtain an edge information image corresponding to the sample image.

[0094] Specifically, when oil stains are attached to the camera, the camera will be blurred as a whole, so I′ gray The second-order derivatives in the x and y directions are calculated for each pixel. And add the second-order directional derivatives of each pixel to obtain the edge information map I′ edge , and the variance value D of the edge information graph 0 A measure of the overall blurriness of the sample image of the current frame;

[0095] Variance value D of edge information graph 0 The calculation formula is as follows:

[0096]

[0097] Among them, Var represents the calculated variance value.

[0098] S103, calculating the oil stain blur information corresponding to each frame of the sample image according to the edge information image;

[0099] Specifically, step S103 includes:

[0100] S1031. For each frame of sample image, a first variance value of the edge information image is calculated based on an intermediate result corresponding to each pixel in the grayscale image, and the first variance value is used as the overall blur information of the current frame.

[0101] Considering that in the smoke and stove scene, the rising of oil smoke and steam during cooking will cause image blur in local areas, and these image blurs do not belong to the oil blur to be detected, so we can use the edge information image I′ edge The image is divided into blocks and the second-order derivative variance statistics are performed to obtain the local blur corresponding to each frame of the sample image, which effectively distinguishes the blur caused by oil smoke, water vapor rising blur and oil smoke adhesion blur, so as to ensure the accuracy and efficiency of identifying the oil blur in the image.

[0102] Specifically, firstly, a block division strategy needs to be formulated in advance to predetermine different oil fume generating areas; the oil fume generating areas can be determined based on prior knowledge, image object detection, detection of pots or burners on the stove, etc. Among them, the detection process of image object detection of pots or burners on the stove is a mature technology in this field, so it will not be repeated here.

[0103] S1032, dividing the edge information image into blocks according to different oil smoke generation areas to obtain different image blocks;

[0104] S1033, for each frame of the sample image, calculate a second variance value of each image block based on the intermediate result corresponding to each pixel in the image block, and use the second variance value as the local blur information of the current frame;

[0105] The oil stain blurriness information of each frame of sample image includes overall blurriness information and local blurriness information.

[0106] Specifically, to be applicable to different types of cookers (single-eye, double-eye, triple-eye), the cooker is set to be placed in the center of the camera's field of view. Figure 4 As shown in FIG. 15, 15 oil fume generating areas are predetermined, and the overall image is divided into blocks according to the oil fume generating areas. Figure 4 Among the 15 areas shown, the oil fume generation area of ​​the single-burner stove is set in area 8, and the widths of columns 6, 7, 9, and 10 are divided according to the principle of equal division; the oil fume generation area of ​​the double-burner stove is set in areas 7 and 9; and the oil fume generation area of ​​the three-burner stove is set in areas 7, 8, and 9. edge After the unified 15-region segmentation, the variance statistics of each region are performed to obtain the local blur information D corresponding to each image block. n .

[0107] Each local ambiguity information D n The calculation formula is as follows:

[0108]

[0109] Wherein, n = 1 ... 15. Of course, the number of divisions of the oil smoke generating area and the division rules can be reset and adjusted according to actual needs.

[0110] It can be seen that the sample image of each frame corresponds to the overall blur representation value D 0 and the local ambiguity characterization value D n , which constitutes the instantaneous blur representation value D of the current frame ori =[D 0 ,D 1 ,…,D n ,…,D 15 ].

[0111] S104, taking the oil stain blur information corresponding to each frame of the sample image as input, taking the corresponding category label information as output, and training to obtain a target classifier for outputting whether there is oil stain blur in the image;

[0112] A target classifier for outputting whether there is oil blur in the image is obtained by training a classification model, and the classification model includes but is not limited to a support vector machine SVM or a neural network algorithm.

[0113] Before training the target classifier, in order to eliminate the data magnitude differences between samples, the recognition method of this embodiment further includes: normalizing the oil pollution fuzziness information.

[0114] Steps S101-S104 are used to complete the training process of the offline model for outputting whether there is oil blur in the image.

[0115] S105 , obtaining a target frame image and inputting it into a target classifier to obtain a classification result.

[0116] After step S105, the following steps are also included:

[0117] S106, calculating the confidence level corresponding to the classification result;

[0118] S107, determining whether the confidence is greater than the confidence threshold, if so, determining that the classification result is valid, and executing S108; if not, determining that the classification result is invalid.

[0119] S108. When the classification result indicates that there is oil blur in the image, a prompt message indicating that there is oil on the lens is generated; and / or a control instruction is generated to execute a lens cleaning operation.

[0120] In the process of oil stain blur recognition in actual images, the camera transmits the video stream in real time, and the image data processing flow of each frame is the same as that of the image data of offline training, that is, grayscale is pre-converted to obtain I gray , calculate the second-order directional gradient of the whole image to obtain the edge information map Iedge , Statistics I edge The overall variance and block variance of are integrated into the instantaneous blur representation value D of the current frame ori , and normalize it to get D Norm . The processed data D Norm The image is sent to the classifier M to obtain the corresponding classification result and confidence level, and the confidence level is compared with the confidence level threshold to finally determine the oil fuzzy classification of the current image.

[0121] In addition, when oil blur is detected, the range hood starts the corresponding lens cleaning device, or sends a prompt message to the user to remind the user to clean manually; when the detection result is non-oil blur, the camera-based functional module in the range hood operates normally.

[0122] The recognition method of this embodiment can be applied before cooking, during cooking, and after cooking, which reduces the requirements for recognition scenarios, greatly improves the versatility of the recognition process, and meets the user's higher usage needs.

[0123] In this embodiment, the edge information image corresponding to each frame of the sample image is obtained, and then the corresponding oil blur information is obtained. Then, a classification model is used to train a classifier based on the oil blur information and category labeling information of each frame of the sample image. The target frame image collected by the camera in real time is input into the classifier to obtain the classification result, so as to timely know whether the camera is attached with oil, thereby improving the recognition accuracy and efficiency of oil blur in the image.

[0124] Example 2

[0125] like Figure 5 As shown, the method for identifying oily blur in an image in this embodiment is a further improvement on the embodiment 1, specifically:

[0126] After step S1033, the following steps are also included:

[0127] S1034. Calculate the blur difference of the oil stain blur information of two sample images separated by a set number of frames; the blur difference indicates the persistence of the oil stain blur on the image between frames. By using the instantaneous blur difference between fixed frames, the blur change between frames can be effectively characterized, and the difference between oil smoke, water vapor rising blur and oil smoke adhesion blur can be distinguished, thereby ensuring the accuracy of oil stain blur recognition in the image.

[0128] S1035. Superimpose the fuzziness difference with the overall fuzziness information and local fuzziness information of the current frame to obtain the oil fuzziness information of the current frame.

[0129] By calculating the interval D between images with a specific number of frames N ori The difference ΔD=[ΔD0 ,ΔD 1 ,…,ΔD n ,…,ΔD 15 ], ΔD is used to characterize the persistence of oil blur in the image between frames, and finally D ori Combined with ΔD to form the comprehensive representation value of the current image D = [D ori ,ΔD]; in order to eliminate the data magnitude differences between samples, D is normalized to obtain D Norm , the specific formula is as follows:

[0130]

[0131] Among them, min(…) and min(…) are the minimum and maximum functions respectively. So far, the characterization parameter D normalized to the range of [0,1] can be obtained. Norm .

[0132] In the process of oil stain blur recognition in actual images, the camera transmits the video stream in real time, and the image data processing flow of each frame is the same as that of the image data of offline training, that is, grayscale is pre-converted to obtain I gray , calculate the second-order directional gradient of the whole image to obtain the edge information map I edge , Statistics I edge The overall variance and block variance of are integrated into the instantaneous blur representation value D of the current frame ori ; To calculate D ori The inter-frame difference is calculated for the first N images after the camera is started. ori Value, starting from the N+1th frame, start calculating D ori The inter-frame difference ΔD is calculated by the current frame D ori Integrate into a comprehensive representation value D, and normalize to get D Norm ; The processed data D Norm The image is sent to the classifier M to obtain the corresponding classification result and confidence level, and the confidence level is compared with the confidence level threshold to finally determine the oil fuzzy classification of the current image.

[0133] In addition, when oil blur is detected, the range hood starts the corresponding lens cleaning device, or sends a prompt message to the user to remind the user to clean manually; when the detection result is non-oil blur, the camera-based functional module in the range hood operates normally.

[0134] In this embodiment, the instantaneous blur difference between fixed frames can be used to effectively characterize the blur change between frames, and then effectively distinguish the difference between the rising blur of oil smoke and water vapor and the blur of oil smoke adhesion, avoiding the interference of rising oil smoke and water vapor, and further improving the accuracy and efficiency of identifying oil blur in the image, ensuring the accuracy of range hood control, and meeting users' higher usage requirements.

[0135] Example 3

[0136] like Figure 6 As shown, the system for identifying oil blur in an image of this embodiment includes a sample information acquisition module 1, an edge information image acquisition module 2, a blur information calculation module 3, a classifier training module 4 and an image classification module 5.

[0137] The sample information acquisition module 1 is used to acquire a number of sample image frames and corresponding category annotation information; wherein the category annotation information is used to characterize whether there is oily blur or no oily blur in the image;

[0138] The video data set in the smoke stove environment is collected and stored in advance through the camera, and then the video data in the video data set is selected according to the continuous fixed number of frames N (N ≥ 2 and an integer) sample images, and the corresponding category L is pre-labeled according to the actual blur situation of each frame of the sample image, such as L1 non-oil blur L1, oil blur L2;

[0139] The presence of oily blur in the image indicates that there is oily stain attached to the lens of the current camera; conversely, the absence of oily blur in the image indicates that there is no oily stain attached to the lens of the current camera. The edge information image acquisition module 2 is used to process each frame of sample image to obtain the corresponding edge information image;

[0140] Specifically, the edge information image acquisition module 2 includes a grayscale processing unit, an intermediate result calculation unit and an edge information image acquisition unit.

[0141] The grayscale processing unit is used to perform grayscale processing on each frame of sample image to obtain a corresponding grayscale image;

[0142] Among them, for each sample image I' in the continuous frames, grayscale processing is performed to convert it into a grayscale image I' gray , in order to reduce the impact of differences in lighting environments among different users, and ensure the accuracy of subsequent image recognition results.

[0143] Of course, other preprocessing operations can also be performed on the sample image according to actual needs to further improve the accuracy of subsequent image recognition results. The specific other preprocessing methods can be determined and adjusted according to actual needs.

[0144] The intermediate result calculation unit is used to calculate the second-order derivatives of different pixel points in the grayscale image to obtain the intermediate result corresponding to each pixel point;

[0145] Among them, for the grayscale image of each frame of sample image, the second-order derivative of the pixel points with a set proportion in the grayscale image can be calculated, which can reduce the amount of data calculation while ensuring the overall blur calculation of the sample image, improve the data processing speed, and then improve the overall recognition efficiency of oil smoke blur.

[0146] In order to achieve a better result for determining the image blur, the second-order derivatives of all pixels in the grayscale image are calculated to obtain intermediate results corresponding to all pixels.

[0147] The edge information image acquisition unit is used to add each intermediate result to obtain an edge information image corresponding to the sample image.

[0148] Specifically, when oil stains are attached to the camera, the camera will be blurred as a whole, so I′ gray The second-order derivatives in the x and y directions are calculated for each pixel. And add the second-order directional derivatives of each pixel to obtain the edge information map I′ edge , and the variance value D of the edge information graph 0 A measure of the overall blurriness of the sample image of the current frame;

[0149] Variance value D of edge information graph 0 The calculation formula is as follows:

[0150]

[0151] Among them, Var represents the calculated variance value.

[0152] The fuzziness information calculation module 3 is used to calculate the oil fuzziness information corresponding to each frame of sample image according to the edge information image;

[0153] The recognition system of this embodiment further includes an area determination module for predetermining different oil fume generating areas.

[0154] Specifically, considering that in the smoke and stove scene, the rising of oil smoke and steam during cooking will cause image blur in local areas, and these image blurs do not belong to the oil blur to be detected, so the edge information image I′ can be used to detect the oil stain. edge The image is divided into blocks and the second-order derivative variance statistics are performed to obtain the local blur corresponding to each frame of the sample image, which effectively distinguishes the blur caused by oil smoke, water vapor rising blur and oil smoke adhesion blur, so as to ensure the accuracy and efficiency of identifying the oil blur in the image.

[0155] Specifically, firstly, a block division strategy needs to be formulated in advance to predetermine different oil fume generating areas; the oil fume generating areas can be determined based on prior knowledge, image object detection, detection of pots or burners on the stove, etc. Among them, the detection process of image object detection of pots or burners on the stove is a mature technology in this field, so it will not be repeated here.

[0156] The ambiguity information calculation module 3 includes an overall ambiguity information acquisition unit, an image block processing unit and a local ambiguity information acquisition unit.

[0157] The overall blur information acquisition unit is used to calculate the first variance value of the edge information image based on the intermediate result corresponding to each pixel point in the grayscale image for each frame of sample image, and use the first variance value as the overall blur information of the current frame.

[0158] The image block processing unit is used to perform block processing on the edge information image according to different oil smoke generation areas to obtain different image blocks;

[0159] The local blur information acquisition unit is used to calculate the second variance value of each image block based on the intermediate result corresponding to each pixel point in the image block for each frame of the sample image, and use the second variance value as the local blur information of the current frame;

[0160] The oil stain blurriness information of each frame of sample image includes overall blurriness information and local blurriness information.

[0161] Specifically, to be applicable to different types of cookers (single-eye, double-eye, triple-eye), the cooker is set to be placed in the center of the camera's field of view. Figure 4 As shown in FIG. 15, 15 oil fume generating areas are predetermined, and the overall image is divided into blocks according to the oil fume generating areas. Figure 4 Among the 15 areas shown, the oil fume generation area of ​​the single-burner stove is set in area 8, and the widths of columns 6, 7, 9, and 10 are divided according to the principle of equal division; the oil fume generation area of ​​the double-burner stove is set in areas 7 and 9; and the oil fume generation area of ​​the three-burner stove is set in areas 7, 8, and 9. edge After the unified 15-region segmentation, the variance statistics of each region are performed to obtain the local blur information D corresponding to each image block. n .

[0162] Each local ambiguity information D n The calculation formula is as follows:

[0163]

[0164] Wherein, n = 1 ... 15. Of course, the number of divisions of the oil smoke generating area and the division rules can be reset and adjusted according to actual needs.

[0165] It can be seen that the sample image of each frame corresponds to the overall blur representation value D 0 and the local ambiguity characterization value D n , which constitutes the instantaneous blur representation value D of the current frame ori =[D 0 ,D 1 ,…,D n ,…,D 15 ].

[0166] The classifier training module 4 is used to take the oil stain blur information corresponding to each frame of the sample image as input, take the corresponding category label information as output, and train to obtain a target classifier for outputting whether there is oil stain blur in the image;

[0167] A target classifier for outputting whether there is oil blur in the image is obtained by training a classification model, and the classification model includes but is not limited to a support vector machine SVM or a neural network algorithm.

[0168] In order to eliminate the difference in data magnitude between samples, the recognition system also includes a normalization processing module, which is used to normalize the oil pollution fuzziness information before training the target classifier.

[0169] The image classification module 5 is used to obtain the target frame image and input it into the target classifier to obtain the classification result.

[0170] The recognition system of this embodiment further includes a confidence calculation module 6 , a judgment module 7 , a reminder information generation module 8 and a control instruction generation module 9 .

[0171] The confidence calculation module 6 is used to calculate the confidence corresponding to the classification result;

[0172] The judgment module is used to judge whether the confidence is greater than the confidence threshold. If so, the classification result is determined to be valid; if not, the classification result is determined to be invalid.

[0173] The reminder information generating module 8 is used to generate a reminder information indicating that there is oil stain on the lens when it is determined that the classification result is valid and the classification result indicates that there is oil stain blur in the image;

[0174] The control instruction generating module 9 is used to generate a control instruction to perform a lens cleaning operation when it is determined that the classification result is valid and the classification result indicates that there is oil blur in the image.

[0175] In the process of oil stain blur recognition in actual images, the camera transmits the video stream in real time, and the image data processing flow of each frame is the same as that of the image data of offline training, that is, grayscale is pre-converted to obtain I gray , calculate the second-order directional gradient of the whole image to obtain the edge information map Iedge , Statistics I edge The overall variance and block variance of are integrated into the instantaneous blur representation value D of the current frame ori , and normalize it to get D Norm . The processed data D Norm The image is sent to the classifier M to obtain the corresponding classification result and confidence level, and the confidence level is compared with the confidence level threshold to finally determine the oil fuzzy classification of the current image.

[0176] In addition, when oil blur is detected, the range hood starts the corresponding lens cleaning device, or sends a prompt message to the user to remind the user to clean manually; when the detection result is non-oil blur, the camera-based functional module in the range hood operates normally.

[0177] The recognition system of this embodiment can be used before cooking, during cooking, and after cooking, which reduces the requirements for recognition scenarios, greatly improves the versatility of the recognition process, and meets the user's higher usage needs.

[0178] In this embodiment, the edge information image corresponding to each frame of the sample image is obtained, and then the corresponding oil blur information is obtained. Then, a classification model is used to train a classifier based on the oil blur information and category labeling information of each frame of the sample image. The target frame image collected by the camera in real time is input into the classifier to obtain the classification result, so as to timely know whether the camera is attached with oil, thereby improving the recognition accuracy and efficiency of oil blur in the image.

[0179] Example 4

[0180] The system for identifying oily blur in an image in this embodiment is a further improvement of the embodiment 3. Specifically:

[0181] The ambiguity information calculation module 3 of this embodiment further includes a ambiguity difference calculation unit and an oil pollution ambiguity information acquisition unit.

[0182] The blur difference calculation unit is used to calculate the blur difference of the oil blur information of two sample images separated by a set number of frames; the blur difference represents the persistence of the oil blur on the image between frames. By using the instantaneous blur difference between fixed frames, the blur change between frames can be effectively characterized, and the difference between oil smoke, water vapor rising blur and oil smoke adhesion blur can be distinguished, thereby ensuring the accuracy of oil blur recognition in the image.

[0183] The oil stain ambiguity information acquisition unit is used to perform superposition processing on the ambiguity difference with the overall ambiguity information and the local ambiguity information of the current frame to acquire the oil stain ambiguity information of the current frame.

[0184] By calculating the interval D between images with a specific number of frames Nori The difference ΔD=[ΔD 0 ,ΔD 1 ,…,ΔD n ,…,ΔD 15 ], ΔD is used to characterize the persistence of oil blur in the image between frames, and finally D ori Combined with ΔD to form the comprehensive representation value of the current image D = [D ori ,ΔD]; in order to eliminate the data magnitude differences between samples, D is normalized to obtain D Norm , the specific formula is as follows:

[0185]

[0186] Among them, min(…) and min(…) are the minimum and maximum functions respectively. So far, the characterization parameter D normalized to the range of [0,1] can be obtained. Norm .

[0187] In the process of oil stain blur recognition in actual images, the camera transmits the video stream in real time, and the image data processing flow of each frame is the same as that of the image data of offline training, that is, grayscale is pre-converted to obtain I gray , calculate the second-order directional gradient of the whole image to obtain the edge information map I edge , Statistics I edge The overall variance and block variance of are integrated into the instantaneous blur representation value D of the current frame ori ; To calculate D ori The inter-frame difference is calculated for the first N images after the camera is started. ori Value, starting from the N+1th frame, start calculating D ori The inter-frame difference ΔD is calculated by the current frame D ori Integrate into a comprehensive representation value D, and normalize to get D Norm ; The processed data D Norm The image is sent to the classifier M to obtain the corresponding classification result and confidence level, and the confidence level is compared with the confidence level threshold to finally determine the oil fuzzy classification of the current image.

[0188] In addition, when oil blur is detected, the range hood starts the corresponding lens cleaning device, or sends a prompt message to the user to remind the user to clean manually; when the detection result is non-oil blur, the camera-based functional module in the range hood operates normally.

[0189] In this embodiment, the instantaneous blur difference between fixed frames can be used to effectively characterize the blur change between frames, and then effectively distinguish the difference between the rising blur of oil smoke and water vapor and the blur of oil smoke adhesion, avoiding the interference of rising oil smoke and water vapor, and further improving the accuracy and efficiency of identifying oil blur in the image, ensuring the accuracy of range hood control, and meeting users' higher usage requirements.

[0190] Example 5

[0191] Figure 7 This is a schematic diagram of the structure of an electronic device provided in Example 5 of the present invention. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method for identifying oil blur in an image in Example 1 or 2 when executing the program. Figure 7 The electronic device 30 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0192] like Figure 7 As shown, the electronic device 30 may be in the form of a general-purpose computing device, for example, it may be a server device. The components of the electronic device 30 may include, but are not limited to: at least one processor 31, at least one memory 32, and a bus 33 connecting different system components (including the memory 32 and the processor 31).

[0193] The bus 33 includes a data bus, an address bus, and a control bus.

[0194] The memory 32 may include a volatile memory, such as a random access memory (RAM) 321 and / or a cache memory 322 , and may further include a read-only memory (ROM) 323 .

[0195] The memory 32 may also include a program / utility 325 having a set (at least one) of program modules 324, such program modules 324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0196] The processor 31 executes various functional applications and data processing by running the computer program stored in the memory 32, such as the method for identifying oil blur in an image in Embodiment 1 or 2 of the present invention.

[0197] The electronic device 30 may also communicate with one or more external devices 34 (e.g., keyboards, pointing devices, etc.). Such communication may be performed via an input / output (I / O) interface 35. Furthermore, the model generating device 30 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 36. Figure 7 As shown, the network adapter 36 communicates with other modules of the model-generated device 30 via the bus 33. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the model-generated device 30, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.

[0198] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0199] Example 6

[0200] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the method for identifying oil blur in an image in Embodiment 1 or 2 are implemented.

[0201] The readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device or any suitable combination of the above.

[0202] In a possible implementation, the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps in the method for identifying oil blur in an image in Example 1 or 2.

[0203] Among them, the program code for executing the present invention can be written in any combination of one or more programming languages, and the program code can be executed completely on the user device, partially on the user device, as an independent software package, partially on the user device and partially on a remote device, or completely on the remote device.

[0204] Although the specific embodiments of the present invention are described above, it should be understood by those skilled in the art that this is only for illustration and the protection scope of the present invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but these changes and modifications all fall within the protection scope of the present invention.

Claims

1. A method for identifying oily blur in an image, characterized in that: The identification method comprises: Acquire several frames of sample images and corresponding category annotation information; wherein the category annotation information is used to characterize whether oily blur exists in the image or not; Processing each frame of the sample image to obtain a corresponding edge information image; Obtaining the oil stain blur information corresponding to each frame of the sample image by calculation according to the edge information image; Taking the oil stain blurriness information corresponding to each frame of the sample image as input and the corresponding category labeling information as output, training a target classifier for outputting whether there is oil stain blur in the image; Acquire a target frame image and input it into the target classifier to obtain a classification result; The step of calculating the oil stain blurriness information corresponding to each frame of the sample image according to the edge information image comprises: For each frame of the sample image, a first variance value of the edge information image is calculated based on the intermediate result corresponding to each pixel in the grayscale image, and the first variance value is used as the overall blur information of the current frame; The identification method further comprises: Predetermine different fume generation areas; The step of calculating the oil stain blurriness information corresponding to each frame of the sample image according to the edge information image also includes: According to different fume generation areas, the edge information image is processed into blocks to obtain different image blocks; For each frame of the sample image, a second variance value of each image block is calculated based on the intermediate result corresponding to each pixel in the image block, and the second variance value is used as the local blur information of the current frame; The oil stain blurriness information of each frame of the sample image includes the overall blurriness information and local blurriness information.

2. The method for identifying oily blur in an image as claimed in claim 1, characterized in that: The step of processing each frame of the sample image to obtain a corresponding edge information image comprises: Performing grayscale processing on each frame of the sample image to obtain a corresponding grayscale image; Calculating the second-order derivatives of different pixel points in the grayscale image to obtain an intermediate result corresponding to each pixel point; Each of the intermediate results is added together to obtain the edge information image corresponding to the sample image.

3. The method for identifying oily blur in an image as claimed in claim 1, characterized in that: The step of calculating the oil stain blurriness information corresponding to each frame of the sample image according to the edge information image also includes: Calculating the fuzziness difference of the oil stain fuzziness information of two frames of the sample image separated by a set number of frames; The fuzziness difference is superimposed with the overall fuzziness information and the local fuzziness information of the current frame to obtain the oil fuzziness information of the current frame.

4. The method for identifying oil blur in an image according to any one of claims 1 to 3, characterized in that: The step of taking the oil stain blurriness information corresponding to each frame of the sample image as input and the corresponding category labeling information as output, and training to obtain a target classifier for whether there is oil stain blur in the image comprises: The oil blur information corresponding to each frame of the sample image is used as input, and the corresponding category label information is used as output. A support vector machine (SVM) or a neural network algorithm is used to train to obtain a target classifier for outputting whether there is oil blur in the image.

5. The method for identifying oil blur in an image according to any one of claims 1 to 3, characterized in that: Before training the target classifier, the recognition method further includes: Normalizing the oil pollution ambiguity information; and / or, After the step of acquiring the target frame image and inputting the target classifier to obtain the classification result, the following steps are further included: Calculating the confidence level corresponding to the classification result; It is determined whether the confidence is greater than a confidence threshold, and if so, the classification result is determined to be valid; if not, the classification result is determined to be invalid.

6. The method for identifying oil blur in an image as claimed in claim 5, characterized in that: After the step of determining that the classification result is valid, the method further includes: When the classification result indicates that there is oil blur in the image, a prompt message indicating that there is oil on the lens is generated; and / or a control instruction is generated to execute a lens cleaning operation.

7. A system for identifying oil blur in an image, characterized in that: The identification system comprises: A sample information acquisition module is used to acquire a number of sample image frames and corresponding category labeling information; wherein the category labeling information is used to characterize whether oily blur exists in the image or not; An edge information image acquisition module, used for processing each frame of the sample image to acquire a corresponding edge information image; A fuzziness information calculation module, used for calculating the oil fuzziness information corresponding to each frame of the sample image according to the edge information image; A classifier training module, used to take the oil stain blur information corresponding to each frame of the sample image as input, take the corresponding category label information as output, and train to obtain a target classifier for outputting whether there is oil stain blur in the image; An image classification module, used for acquiring a target frame image and inputting the image into the target classifier to obtain a classification result; The ambiguity information calculation module includes: an overall blur information acquisition unit, configured to calculate, for each frame of the sample image, a first variance value of the edge information image based on the intermediate result corresponding to each pixel in the grayscale image, and use the first variance value as the overall blur information of the current frame; The identification system further comprises: An area determination module is used to predetermine different oil smoke generation areas; The ambiguity information calculation module also includes: An image block processing unit, used for performing block processing on the edge information image according to different oil fume generation areas to obtain different image blocks; A local blur information acquisition unit, configured to calculate, for each frame of the sample image, a second variance value of each image block based on the intermediate result corresponding to each pixel in the image block, and use the second variance value as the local blur information of the current frame; The oil stain blurriness information of each frame of the sample image includes the overall blurriness information and local blurriness information.

8. The system for identifying oil blur in an image as claimed in claim 7, characterized in that: The edge information image acquisition module comprises: A grayscale processing unit, used for performing grayscale processing on each frame of the sample image to obtain a corresponding grayscale image; An intermediate result calculation unit, used to calculate the second-order derivatives of different pixel points in the grayscale image to obtain an intermediate result corresponding to each pixel point; The edge information image acquisition unit is used to add each of the intermediate results to obtain the edge information image corresponding to the sample image.

9. The system for identifying oil blur in an image as claimed in claim 7, characterized in that: The ambiguity information calculation module also includes: A fuzziness difference calculation unit, used for calculating a fuzziness difference of the oil stain fuzziness information of two frames of the sample image separated by a set number of frames; The oil stain ambiguity information acquisition unit is used to superimpose the ambiguity difference with the overall ambiguity information and the local ambiguity information of the current frame to acquire the oil stain ambiguity information of the current frame.

10. The system for identifying oil blur in an image according to any one of claims 7 to 9, characterized in that: The classifier training module is used to take the oil blur information corresponding to each frame of the sample image as input, and the corresponding category label information as output, and use support vector machine SVM or neural network algorithm training to obtain a target classifier for outputting whether there is oil blur in the image.

11. The system for identifying oil blur in an image according to any one of claims 7 to 9, characterized in that: The identification system further comprises: A normalization processing module, used for normalizing the oil pollution ambiguity information before training the target classifier; and / or, A confidence calculation module, used to calculate the confidence corresponding to the classification result; The judgment module is used to judge whether the confidence is greater than a confidence threshold. If so, the classification result is determined to be valid; if not, the classification result is determined to be invalid.

12. The system for identifying oil blur in an image according to any one of claims 7 to 9, characterized in that: The identification system further comprises: a reminder information generating module, for generating a reminder information indicating that oil stains exist on the lens when the classification result indicates that the image is blurry due to oil stains; and / or, The control instruction generating module is used to generate a control instruction to perform a lens cleaning operation when the classification result indicates that there is oil blur in the image.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for identifying oil blur in an image described in any one of claims 1-6 is implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for identifying oil blur in an image described in any one of claims 1 to 6 are implemented.

Images