A method and device for detecting electronic cigarette defects based on machine vision perception

Abstract

The present invention discloses an electronic cigarette defect detection method and device based on machine vision perception, which relates to the field of image processing technology; multiple feature extraction is performed on a target image to obtain multiple feature sets, and abnormal images are obtained according to the multiple feature sets; the abnormal image is brought into a backbone network to obtain a multi-level feature map, and a target attention feature map is obtained according to the multi-level feature map; the target attention feature map is used as the input of a preset abnormal perception model to obtain a target abnormal perception map; the target defect parameters are determined according to the multi-level feature map and the target abnormal perception map, and the target defect parameters are compared with the parameters in a preset database to obtain the defect type. Through multi-feature extraction to capture key information, the generated target saliency map can accurately highlight the potential abnormal area of ​​the electronic cigarette, and then the abnormal features of the abnormal area are strengthened by mixed attention, so as to improve the recognition of complex appearance defects, and then the defect type is finally determined according to the multi-level feature map and the target abnormal perception map, so as to enhance the detection accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an electronic cigarette defect detection method and device based on machine vision perception. Background Art

[0002] With the advancement of technology and the growing demand of consumers, the e-cigarette market has shown an explosive growth trend in the past few years. As a new type of electronic product, the appearance, function and safety of e-cigarettes have attracted great attention from consumers. Therefore, quality control and defect detection in the production process of e-cigarettes have become a crucial link.

[0003] Traditional manual visual inspection methods have always been the main means of detecting e-cigarette defects in the past. However, with the continuous expansion of production scale and the improvement of consumer requirements for product quality, manual visual inspection methods have gradually revealed their limitations. First, manual visual inspection is affected by many factors such as workers' eyesight, fatigue level, work experience, etc., and it is difficult to ensure the stability and consistency of inspection. Secondly, manual visual inspection is inefficient and cannot meet the needs of mass production. In addition, manual visual inspection is difficult to achieve high-precision inspection standards, especially in the detection of some minor defects, missed detection or false detection often occurs.

[0004] In order to overcome the limitations of traditional manual visual inspection methods, existing technologies have begun to use machine vision technology to detect electronic cigarette defects. Machine vision technology simulates the working principle of the human visual system and uses computers and image processing technology to collect, process and analyze images, thereby realizing the functions of electronic cigarette appearance recognition, positioning and fault detection. However, when using machine vision technology to detect electronic cigarette defects, although it performs well for electronic cigarettes with simple appearance, its accuracy of defect detection is challenged when facing electronic cigarettes with complex appearance. In particular, for electronic cigarettes with complex designs, diverse features and textures, existing machine vision systems are difficult to accurately identify and evaluate potential defects, resulting in inaccurate detection. Summary of the invention

[0005] The purpose of the present invention is to solve the problem that when faced with electronic cigarettes with complex appearance, the existing machine vision system is difficult to accurately identify and evaluate potential defects, resulting in inaccurate detection, and to propose an electronic cigarette defect detection method and device based on machine vision perception.

[0006] In a first aspect of the present invention, a method for detecting electronic cigarette defects based on machine vision perception is first proposed, the method comprising:

[0007] Performing multi-feature extraction on a target image to obtain multiple feature sets, generating a saliency map for each feature in the multiple feature sets, and fusing all saliency maps to obtain a target saliency map; the target image is a photographed image of an electronic cigarette device;

[0008] Determining an abnormal region in the target image according to the target saliency map, and segmenting the abnormal region from the target image to obtain an abnormal image;

[0009] Bringing the abnormal image into the backbone network to obtain a multi-level feature map, performing mixed attention processing on each level of the multi-level feature map to obtain multiple mixed attention feature maps, and fusing all the mixed attention feature maps to obtain a target attention feature map;

[0010] Using the target attention feature map as an input of a preset abnormality perception model to obtain a target abnormality perception map;

[0011] The multi-level feature map and the target abnormality perception map are brought into a preset defect detection model to obtain target defect parameters, and the target defect parameters are compared with the parameters in a preset database to obtain the defect type; the preset database is used to store defect parameters corresponding to different defect types.

[0012] Optionally, performing multi-feature extraction on the target image to obtain multiple feature sets, and generating a saliency map for each feature in the multiple feature sets includes:

[0013] After converting the target image from RGB to HSV color space, calculating the color significance score corresponding to each pixel of the target image through a color histogram, retaining pixels in the target image whose color significance score is greater than a first preset value to obtain a color significance map;

[0014] Extracting texture features of the target image through a filter to obtain a texture response atlas, determining a texture saliency score corresponding to each pixel in the target image according to a calculation statistic corresponding to each texture response map, and retaining pixels in the target image whose texture saliency score is greater than a second preset value to obtain a texture saliency map;

[0015] The target image is calculated by an edge detection algorithm to obtain an edge map, the edge strength of each pixel point in the edge map is calculated to determine the edge significance score corresponding to each pixel in the target image, and the pixels in the target image whose edge significance score is greater than a third preset value are retained to obtain an edge significance map.

[0016] Optionally, determining an abnormal region in the target image according to the target saliency map, and segmenting the abnormal region from the target image to obtain an abnormal image includes:

[0017] Segmenting the target image using a superpixel segmentation algorithm to obtain a superpixel block set;

[0018] The average value of the significance value of each superpixel block is calculated. If the average value of the superpixel block is greater than the preset significance value, the superpixel block is separated from the target image, and all separated superpixel blocks are mapped to the target image to obtain an abnormal image containing only the separated superpixel blocks.

[0019] Optionally, each level feature map is subjected to mixed attention processing to obtain multiple mixed attention feature maps, and all mixed attention feature maps are fused to obtain a target attention feature map including:

[0020] For any level feature map, calculate the importance weight of the channel and the importance weight of the position in the level feature map, and reshape the level feature map according to the importance weight and the importance weight to obtain a reshaped feature map;

[0021] The reshaped feature map is convolved with a 1×1 convolution kernel to obtain a target feature map, and all target feature maps are added together to obtain a target attention feature map.

[0022] Optionally, bringing the multi-level feature map and the target abnormality perception map into a preset defect detection model to obtain target defect parameters includes:

[0023] For each feature map, outer product calculation is performed with the target abnormal perception map to obtain multiple outer product matrices;

[0024] Performing average pooling processing on all outer product matrices to obtain a feature vector set, and performing weighted summation on all feature vectors of the feature vector set to obtain a target feature vector;

[0025] The target feature vector is input into the fully connected layer, and the target feature is processed by a linear activation function to obtain the target defect parameter.

[0026] In a second aspect of the present invention, an electronic cigarette defect detection device based on machine vision perception is proposed, comprising:

[0027] A target saliency map determination module is used to extract multiple features from a target image to obtain multiple feature sets, generate a saliency map for each feature in the multiple feature sets, and fuse all saliency maps to obtain a target saliency map; the target image is a captured image of an electronic cigarette device;

[0028] an abnormal image determination module, used for determining an abnormal region in the target image according to the target saliency map, and segmenting the abnormal region from the target image to obtain an abnormal image;

[0029] A target attention feature map determination module is used to bring the abnormal image into the backbone network to obtain a multi-level feature map, perform mixed attention processing on each level of the multi-level feature map to obtain multiple mixed attention feature maps, and fuse all the mixed attention feature maps to obtain a target attention feature map;

[0030] A target abnormality perception map determination module is used to obtain a target abnormality perception map by using the target attention feature map as an input of a preset abnormality perception model;

[0031] The defect type determination module is used to bring the multi-level feature map and the target abnormality perception map into the preset defect detection model to obtain the target defect parameters, and compare the target defect parameters with the parameters in the preset database to obtain the defect type; the preset database is used to save the defect parameters corresponding to different defect types.

[0032] Optionally, the target saliency map determination module includes:

[0033] A color saliency map determination module is used to convert the target image from RGB to HSV color space, calculate the color saliency score corresponding to each pixel of the target image through a color histogram, and retain pixels in the target image whose color saliency score is greater than a first preset value to obtain a first saliency map;

[0034] A texture saliency map determination module is used to extract texture features of the target image through a filter to obtain a texture response atlas, determine a texture saliency score corresponding to each pixel in the target image according to a calculation statistic corresponding to each texture response map, and retain pixels in the target image whose texture saliency score is greater than a second preset value to obtain a texture saliency map;

[0035] The edge significance map determination module is used to calculate the target image through an edge detection algorithm to obtain an edge map, calculate the edge strength of each pixel point in the edge map to determine the edge significance score corresponding to each pixel in the target image, and retain the pixels in the target image whose edge significance score is greater than a third preset value to obtain an edge significance map.

[0036] Optionally, the abnormal image determination module includes:

[0037] An image segmentation module, used for segmenting the target image by a superpixel segmentation algorithm to obtain a superpixel block set;

[0038] The abnormal image generation module is used to calculate the average value of the significance value of each superpixel block. If the average value of the superpixel block is greater than the preset significance value, the superpixel block is separated from the target image, and all separated superpixel blocks are mapped to the target image to obtain an abnormal image containing only the separated superpixel blocks.

[0039] Optionally, the target attention feature map determination module includes:

[0040] A reshaping feature map module is used to calculate the importance weights of the channels and the importance weights of the positions in any level feature map, and reshape the level feature map according to the importance weights and the importance weights to obtain a reshaped feature map;

[0041] The target feature map addition module is used to convolve the reshaped feature map through a 1×1 convolution kernel to obtain a target feature map, and add all target feature maps to obtain a target attention feature map.

[0042] Optionally, the defect type determination module includes:

[0043] An outer product matrix generation module is used to perform outer product calculations on each feature map and the target abnormal perception map to obtain multiple outer product matrices;

[0044] A target feature vector determination module is used to perform average pooling processing on all outer product matrices to obtain a feature vector set, and for the feature vector set, perform weighted summation on all feature vectors to obtain a target feature vector;

[0045] The feature processing module is used to input the target feature vector into the fully connected layer, and process the target feature through a linear activation function to obtain a target defect parameter.

[0046] Beneficial effects of the present invention:

[0047] The present invention proposes an electronic cigarette defect detection method based on machine vision perception, which performs multi-feature extraction on a target image to obtain multiple feature sets, generates a saliency map for each feature in the multiple feature sets, and fuses all the saliency maps to obtain a target saliency map; the target image is a photographed image of an electronic cigarette device; an abnormal area in the target image is determined according to the target saliency map, and the abnormal area is segmented from the target image to obtain an abnormal image; the abnormal image is brought into a backbone network to obtain a multi-level feature map, and for the multi-level feature map, mixed attention processing is performed on each level feature map to obtain multiple mixed attention feature maps, and all the mixed attention feature maps are fused to obtain a target attention feature map; the target attention feature map is used as the input of a preset abnormal perception model to obtain a target abnormal perception map; the multi-level feature map and the target abnormal perception map are brought into a preset defect detection model to obtain target defect parameters, and the target defect parameters are compared with parameters in a preset database to obtain defect types; the preset database is used to store defect parameters corresponding to different defect types. Through multi-feature extraction, multiple key features of the e-cigarette image are captured. The target saliency map obtained based on multiple key features can comprehensively and accurately highlight the potential abnormal areas in the e-cigarette, which provides assistance for subsequent image segmentation. Then, by performing mixed attention processing on the multi-level feature map of the abnormal image, the key features of the abnormal area are further highlighted, while the features of the non-abnormal area are suppressed, thereby improving the ability to recognize potential defects in the complex appearance of e-cigarettes. After obtaining the target defect parameters from the multi-level feature map and the target abnormality perception map, they are compared with the parameters in the preset database, so that the defect types in the e-cigarette can be more accurately identified, thereby improving the accuracy of defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The present invention will be further described below in conjunction with the accompanying drawings.

[0049] Figure 1 A flowchart of an electronic cigarette defect detection method based on machine vision perception is provided for an embodiment of the present invention;

[0050] Figure 2 A structural schematic diagram of an electronic cigarette defect detection device based on machine vision perception is provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0051] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0052] The embodiment of the present invention provides an electronic cigarette defect detection method based on machine vision perception. Figure 1 , Figure 1A flowchart of an electronic cigarette defect detection method based on machine vision perception provided by an embodiment of the present invention. The method comprises the following steps:

[0053] S101, performing multi-feature extraction on a target image to obtain multiple feature sets, generating a saliency map for each feature in the multiple feature sets, and fusing all saliency maps to obtain a target saliency map.

[0054] S102, determining an abnormal region in a target image according to the target saliency map, and segmenting the abnormal region from the target image to obtain an abnormal image.

[0055] S103, bringing the abnormal image into the backbone network to obtain a multi-level feature map, for the multi-level feature map, performing mixed attention processing on each level feature map to obtain multiple mixed attention feature maps, and fusing all the mixed attention feature maps to obtain a target attention feature map.

[0056] S104, using the target attention feature map as an input of a preset abnormality perception model to obtain a target abnormality perception map.

[0057] S105, bringing the multi-level feature map and the target abnormality perception map into the preset defect detection model to obtain the target defect parameters, and comparing the target defect parameters with the parameters in the preset database to obtain the defect type.

[0058] Among them, the target image is a photographed image of the electronic cigarette device; the preset database is used to save the defect parameters corresponding to different defect types.

[0059] An electronic cigarette defect detection method based on machine vision perception provided by an embodiment of the present invention captures multiple key features of an electronic cigarette image through multi-feature extraction. A target saliency map obtained based on the multiple key features can comprehensively and accurately highlight potential abnormal areas in the electronic cigarette, which provides assistance for subsequent image segmentation. Then, by performing mixed attention processing on the multi-level feature map of the abnormal image, the key features of the abnormal area are further highlighted, while the features of the non-abnormal area are suppressed, thereby improving the ability to recognize potential defects in the complex appearance of the electronic cigarette. After obtaining the target defect parameters from the multi-level feature map and the target abnormal perception map, the parameters are compared with the parameters in the preset database, so that the defect type in the electronic cigarette is more accurately identified, thereby improving the accuracy of defect detection.

[0060] In one implementation, all feature maps are indexed to obtain features Figure 1 ,feature Figure 2And feature map 3, for each salient point in each feature map, calculate the salient point density around the salient point. The size of the salient point is determined by the operator. If the salient point density is less than 0.3 times the maximum salient point density, the salient point is regarded as a noise point and removed. All denoised feature maps are fused and superimposed to obtain the target salient map. If there are 2 or 3 features marked as salient at the same position in the fused salient map, the position is defined as the foreground, otherwise it is recorded as the background.

[0061] In one implementation, all mixed attention feature maps are fused to obtain a target attention feature map, which can be achieved through methods such as element-by-element addition, weighted summation, fused convolution, and residual connection.

[0062] In one implementation, the preset abnormality perception model is a convolutional neural network, and the feature representation of the image is automatically learned through multiple convolutional layers and pooling layers of the convolutional neural network to obtain predicted abnormal values ​​and predicted normal values, and the predicted abnormal values ​​and predicted normal values ​​are back-propagated, and the gradient of each channel relative to the input feature map is calculated. The importance weight of each channel in the feature map is calculated using global average pooling, and the weighted feature map is converted into a response map of the abnormal area and the normal area through a ReLU activation function. The response map of the normal area is inverted and a logical intersection operation is performed with the abnormal response map to obtain the final target abnormality perception map.

[0063] In one implementation, through multi-feature extraction and saliency map generation, potential abnormal areas in electronic cigarette images can be accurately highlighted, thereby improving detection accuracy.

[0064] In one implementation, hybrid attention processing is performed on multi-level feature maps, which helps to extract more critical and discriminative features and further enhance the ability to identify abnormal areas.

[0065] In one implementation, by comparing the target defect parameters with the parameters in a preset database, the defect type in the electronic cigarette can be accurately identified, providing strong support for subsequent quality control and improvement measures.

[0066] In one embodiment, performing multi-feature extraction on a target image to obtain multiple feature sets, and generating a saliency map for each feature in the multiple feature sets includes the following steps:

[0067] Step 1: After converting the target image from RGB to HSV color space, the color saliency score corresponding to each pixel of the target image is calculated through the color histogram, and the pixels in the target image whose color saliency score is greater than a first preset value are retained to obtain a color saliency map;

[0068] Step 2: extracting texture features of the target image through a filter to obtain a texture response atlas, determining a texture saliency score corresponding to each pixel in the target image according to a calculated statistic corresponding to each texture response map, and retaining pixels in the target image whose texture saliency score is greater than a second preset value to obtain a texture saliency map;

[0069] Step 3: Calculate the target image through the edge detection algorithm to obtain an edge map, calculate the edge strength of each pixel in the edge map to determine the edge significance score corresponding to each pixel in the target image, and retain the pixels in the target image whose edge significance score is greater than the third preset value to obtain an edge significance map.

[0070] In one implementation, the color features in the HSV color space are calculated by using a color histogram, and the color significance score corresponding to each pixel in the target image is determined according to the color feature statistics corresponding to each pixel. The histograms of the H, S, and V channels are calculated respectively, the kurtosis of the histogram of the H channel is calculated, and the color contrast of the S channel is calculated. A significance score is assigned to each pixel in the H and S channels according to the kurtosis and contrast, where the significance score ranges from 0 to 100, and the higher the kurtosis and contrast, the closer it is to 100. The significance scores of the H and S channels are merged and averaged to obtain a final color significance map, and the significance map is subjected to threshold processing and morphological operations to optimize the significance map and visualize the final color significance map.

[0071] In one implementation, the filter may be a Gabor filter, a Sobel filter, a Prewitt filter, or the like.

[0072] In one implementation, for a texture response atlas, a calculation statistic corresponding to each texture response map is obtained, and a texture saliency score corresponding to each pixel in the target image is determined according to the calculation statistics corresponding to all texture response maps, wherein the energy of the texture response map can be calculated as a statistic, and the energy can be calculated by summing up each pixel value in the texture response map, and the formula is as follows: in is the pixel value of the texture response map at (i, j), E is the energy, and the energy of each image is normalized Where max(E) is the maximum energy value in the texture response map, and the normalized energy value is used as the texture saliency score of each pixel. In order to generate a texture saliency map, each normalized texture response map can be merged into one map. The merging method can be to take the maximum energy value of all texture response maps.

[0073] In one implementation, an edge detection algorithm outputs an edge strength value for each pixel. For any pixel point in the edge map, the pixel is used as the central window, the edge strength of all pixels in the central window is calculated, and the edge significance score corresponding to each pixel in the target image is determined according to the relative size of the edge strength of the pixel and the edge strength of all edge pixels. The pixels in the target image with edge significance scores greater than a third preset value are retained to obtain a third significance map; the edge pixels are pixels other than the pixel in the central window, and all edge strength values ​​are normalized to 0 to 100.

[0074] In one implementation, the first preset value, the second preset value, and the third preset value are all thresholds determined by maximizing the inter-class variance.

[0075] In one implementation, three different features, color, texture, and edge, are combined to generate a saliency map, which enables the model to more comprehensively capture the key information in the target image.

[0076] In one embodiment, determining an abnormal region in a target image according to a target saliency map, and segmenting the abnormal region from the target image to obtain an abnormal image comprises the following steps:

[0077] Step 1: Segment the target image using a superpixel segmentation algorithm to obtain a superpixel block set;

[0078] Step 2: Calculate the average value of the significance value for each superpixel block. If the average value of the superpixel block is greater than the preset significance value, separate the superpixel block from the target image, and map all separated superpixel blocks to the target image to obtain an abnormal image containing only the separated superpixel blocks.

[0079] In one implementation, the preset saliency value is an average value of saliency values ​​of the entire image.

[0080] In one implementation, superpixels can be used to reduce the amount of computation while retaining important structural information of the image, and by calculating the average saliency value of superpixel blocks, salient areas in the image can be identified more accurately.

[0081] In one embodiment, each level feature map is subjected to mixed attention processing to obtain multiple mixed attention feature maps, and all mixed attention feature maps are fused to obtain a target attention feature map, including the following steps:

[0082] Step 1: For any level feature map, calculate the importance weight of the channel and the importance weight of the position in the level feature map, and reshape the level feature map according to the importance weight and the importance weight to obtain a reshaped feature map;

[0083] Step 2: Convolve the reshaped feature map with a 1×1 convolution kernel to obtain the target feature map, and add all target feature maps to obtain the target attention feature map.

[0084] In one implementation, by calculating the importance weights of channels and positions in the feature map, features that are useful for target detection can be highlighted while irrelevant or redundant features can be suppressed, thereby improving the accuracy and efficiency of target detection.

[0085] In one implementation, a 1×1 convolution kernel is used to convolve the reshaped feature map, which can reduce the number of model parameters and computational complexity and improve processing speed without changing the spatial size of the feature map.

[0086] In one embodiment, the multi-level feature map and the target abnormality perception map are brought into a preset defect detection model to obtain target defect parameters, including the following steps:

[0087] Step 1: For each feature map, perform outer product calculations with the target abnormal perception map to obtain multiple outer product matrices;

[0088] Step 2: Perform average pooling on all outer product matrices to obtain a feature vector set. For the feature vector set, perform weighted summation on all feature vectors to obtain the target feature vector.

[0089] Step 3: Input the target feature vector into the fully connected layer, and process the target features through a linear activation function to obtain the target defect parameters.

[0090] In one implementation, by calculating the outer product of each feature map with the target anomaly perception map, the information of different feature maps can be fused to obtain a more comprehensive and rich feature representation, which helps to capture the complex patterns and structures in the target image and improve the accuracy of defect detection.

[0091] In one implementation, multiple outer product matrices obtained after the outer product calculation are processed by average pooling, which can further filter and compress feature information, remove redundancy and noise, and retain the most representative features, which helps to reduce the amount of calculation and improve processing efficiency.

[0092] In one implementation, the target feature vector is input into the fully connected layer and processed through a linear activation function to obtain the target defect parameter, which can maintain the linear relationship of the features and avoid errors and overfitting problems that may be introduced by nonlinear processing.

[0093] Based on the same inventive concept, the embodiment of the present invention also provides an electronic cigarette defect detection device based on machine vision perception. Figure 2 , Figure 2A schematic diagram of the structure of an electronic cigarette defect detection device based on machine vision perception provided by an embodiment of the present invention includes:

[0094] The target saliency map determination module is used to extract multiple features from the target image to obtain multiple feature sets, generate a saliency map for each feature in the multiple feature sets, and fuse all saliency maps to obtain a target saliency map; the target image is a captured image of an electronic cigarette device;

[0095] An abnormal image determination module is used to determine an abnormal area in a target image according to a target saliency map, and segment the abnormal area from the target image to obtain an abnormal image;

[0096] The target attention feature map determination module is used to bring the abnormal image into the backbone network to obtain a multi-level feature map. For the multi-level feature map, each level of the feature map is subjected to mixed attention processing to obtain multiple mixed attention feature maps, and all the mixed attention feature maps are fused to obtain the target attention feature map;

[0097] A target abnormality perception map determination module is used to obtain a target abnormality perception map by using the target attention feature map as an input of a preset abnormality perception model;

[0098] The defect type determination module is used to bring the multi-level feature map and the target abnormality perception map into the preset defect detection model to obtain the target defect parameters, and compare the target defect parameters with the parameters in the preset database to obtain the defect type; the preset database is used to save the defect parameters corresponding to different defect types.

[0099] An electronic cigarette defect detection device based on machine vision perception provided by an embodiment of the present invention captures multiple key features of an electronic cigarette image through multi-feature extraction. A target saliency map obtained based on the multiple key features can comprehensively and accurately highlight potential abnormal areas in the electronic cigarette, providing assistance for subsequent image cutting. Then, by performing mixed attention processing on the multi-level feature map of the abnormal image, the key features of the abnormal area are further highlighted, while the features of the non-abnormal area are suppressed, thereby improving the ability to recognize potential defects in the complex appearance of the electronic cigarette. After obtaining the target defect parameters from the multi-level feature map and the target abnormal perception map, the parameters are compared with the parameters in the preset database, so that the defect type in the electronic cigarette is more accurately identified, thereby improving the accuracy of defect detection.

[0100] In one embodiment, the target saliency map determination module includes:

[0101] A color saliency map determination module is used to convert the target image from RGB to HSV color space, calculate the color saliency score corresponding to each pixel of the target image through a color histogram, and retain pixels in the target image whose color saliency score is greater than a first preset value to obtain a first saliency map;

[0102] A texture saliency map determination module is used to extract texture features of a target image through a filter to obtain a texture response atlas, determine a texture saliency score corresponding to each pixel in the target image according to a calculated statistic corresponding to each texture response map, and retain pixels in the target image whose texture saliency score is greater than a second preset value to obtain a texture saliency map;

[0103] The edge saliency map determination module is used to calculate the target image through the edge detection algorithm to obtain the edge map, calculate the edge strength of each pixel point in the edge map to determine the edge saliency score corresponding to each pixel in the target image, and retain the pixels in the target image whose edge saliency score is greater than the third preset value to obtain the edge saliency map.

[0104] In one embodiment, the abnormal image determination module includes:

[0105] An image segmentation module is used to segment the target image by a superpixel segmentation algorithm to obtain a superpixel block set;

[0106] The abnormal image generation module is used to calculate the average value of the significance value of each superpixel block. If the average value of the superpixel block is greater than the preset significance value, the superpixel block is separated from the target image, and all separated superpixel blocks are mapped to the target image to obtain an abnormal image containing only the separated superpixel blocks.

[0107] In one embodiment, the target attention feature map determination module includes:

[0108] A reshaping feature map module is used to calculate the importance weights of the channels and the importance weights of the positions in any level feature map, and reshape the level feature map according to the importance weights and the importance weights to obtain a reshaped feature map;

[0109] The target feature map addition module is used to convolve the reshaped feature map with a 1×1 convolution kernel to obtain the target feature map, and add all the target feature maps to obtain the target attention feature map.

[0110] In one embodiment, the defect type determination module includes:

[0111] An outer product matrix generation module is used to perform outer product calculations on each feature map and the target abnormal perception map to obtain multiple outer product matrices;

[0112] A target feature vector determination module is used to perform average pooling processing on all outer product matrices to obtain a feature vector set, and for the feature vector set, perform weighted summation on all feature vectors to obtain a target feature vector;

[0113] The feature processing module is used to input the target feature vector into the fully connected layer and process the target feature through a linear activation function to obtain the target defect parameter.

[0114] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A method for detecting electronic cigarette defects based on machine vision perception, characterized in that: The method comprises: Performing multi-feature extraction on a target image to obtain multiple feature sets, generating a saliency map for each feature in the multiple feature sets, and fusing all saliency maps to obtain a target saliency map; the target image is a captured image of an electronic cigarette device; determining an abnormal region in the target image according to the target saliency map, and segmenting the abnormal region from the target image to obtain an abnormal image; Bringing the abnormal image into the backbone network to obtain a multi-level feature map, performing mixed attention processing on each level feature map to obtain multiple mixed attention feature maps, and fusing all the mixed attention feature maps to obtain a target attention feature map; Using the target attention feature map as an input of a preset anomaly perception model to obtain a target anomaly perception map; Bringing the multi-level feature map and the target abnormality perception map into a preset defect detection model to obtain target defect parameters, and comparing the target defect parameters with parameters in a preset database to obtain a defect type; The preset database is used to store defect parameters corresponding to different defect types; The multi-level feature map and the target abnormality perception map are brought into the preset defect detection model to obtain target defect parameters including: For each feature map, outer product calculation is performed with the target abnormal perception map to obtain multiple outer product matrices; Performing average pooling on all outer product matrices to obtain a set of eigenvectors, and performing weighted summation on all eigenvectors in the set to obtain a target eigenvector; The target feature vector is input into the fully connected layer, and the target feature is processed by a linear activation function to obtain the target defect parameter.

2. The electronic cigarette defect detection method based on machine vision perception according to claim 1 is characterized in that: Performing multi-feature extraction on the target image to obtain multiple feature sets, and generating a saliency map for each feature in the multiple feature sets includes: After converting the target image from RGB to HSV color space, calculating a color saliency score corresponding to each pixel of the target image using a color histogram, and retaining pixels in the target image having a color saliency score greater than a first preset value to obtain a color saliency map; Extracting texture features from the target image using a filter to obtain a texture response atlas, determining a texture saliency score corresponding to each pixel in the target image based on a calculated statistic corresponding to each texture response atlas, and retaining pixels in the target image having a texture saliency score greater than a second preset value to obtain a texture saliency map; The target image is calculated using an edge detection algorithm to obtain an edge map, the edge strength of each pixel point in the edge map is calculated to determine the edge significance score corresponding to each pixel in the target image, and pixels in the target image whose edge significance score is greater than a third preset value are retained to obtain an edge significance map.

3. The electronic cigarette defect detection method based on machine vision perception according to claim 1, characterized in that: Determining an abnormal region in the target image according to the target saliency map, and segmenting the abnormal region from the target image to obtain an abnormal image includes: Segmenting the target image using a superpixel segmentation algorithm to obtain a superpixel block set; The average value of the saliency value of each superpixel block is calculated. If the average value of the superpixel block is greater than the preset saliency value, the superpixel block is separated from the target image, and all separated superpixel blocks are mapped to the target image to obtain an abnormal image containing only the separated superpixel blocks.

4. The electronic cigarette defect detection method based on machine vision perception according to claim 1, characterized in that: Each level feature map is subjected to mixed attention processing to obtain multiple mixed attention feature maps, and all mixed attention feature maps are fused to obtain the target attention feature map including: For any level feature map, calculate the importance weight of the channel and the importance weight of the position in the level feature map, and reshape the level feature map according to the importance weight and the importance weight to obtain a reshaped feature map; The reshaped feature map is convolved with a 1×1 convolution kernel to obtain a target feature map, and all target feature maps are added together to obtain a target attention feature map.

5. An electronic cigarette defect detection device based on machine vision perception, characterized in that: The device comprises: A target saliency map determination module is configured to perform multi-feature extraction on a target image to obtain multiple feature sets, generate a saliency map for each feature in the multiple feature sets, and fuse all saliency maps to obtain a target saliency map; the target image is a captured image of an electronic cigarette device; an abnormal image determination module, configured to determine an abnormal region in the target image according to the target saliency map, and segment the abnormal region from the target image to obtain an abnormal image; A target attention feature map determination module is used to bring the abnormal image into the backbone network to obtain a multi-level feature map, perform mixed attention processing on each level feature map to obtain multiple mixed attention feature maps, and fuse all the mixed attention feature maps to obtain a target attention feature map; a target anomaly perception map determination module, configured to use the target attention feature map as input to a preset anomaly perception model to obtain a target anomaly perception map; A defect type determination module is used to bring the multi-level feature map and the target abnormality perception map into a preset defect detection model to obtain target defect parameters, and compare the target defect parameters with parameters in a preset database to obtain the defect type; The preset database is used to store defect parameters corresponding to different defect types; The defect type determination module includes: An outer product matrix generation module is used to perform outer product calculations on each feature map and the target anomaly perception map to obtain multiple outer product matrices; A target eigenvector determination module is configured to perform average pooling processing on all outer product matrices to obtain a eigenvector set, and to perform weighted summation of all eigenvectors in the eigenvector set to obtain a target eigenvector; The feature processing module is used to input the target feature vector into the fully connected layer and process the target feature through a linear activation function to obtain the target defect parameter.

6. The electronic cigarette defect detection device based on machine vision perception according to claim 5, characterized in that: The target saliency map determination module includes: a color saliency map determining module, configured to convert the target image from RGB to HSV color space, calculate a color saliency score corresponding to each pixel of the target image using a color histogram, and retain pixels in the target image having a color saliency score greater than a first preset value to obtain a first saliency map; a texture saliency map determination module, configured to extract texture features from the target image using a filter to obtain a texture response atlas, determine a texture saliency score corresponding to each pixel in the target image based on a calculated statistic corresponding to each texture response map, and retain pixels in the target image having a texture saliency score greater than a second preset value to obtain a texture saliency map; An edge saliency map determination module is configured to calculate the target image using an edge detection algorithm to obtain an edge map, calculate the edge strength of each pixel in the edge map to determine an edge saliency score corresponding to each pixel in the target image, and retain pixels in the target image whose edge saliency score is greater than a third preset value to obtain an edge saliency map.

7. The electronic cigarette defect detection device based on machine vision perception according to claim 5, characterized in that: The abnormal image determination module includes: An image segmentation module is used to segment the target image using a superpixel segmentation algorithm to obtain a superpixel block set; The abnormal image generation module is used to calculate the average value of the significance value of each superpixel block. If the average value of the superpixel block is greater than the preset significance value, the superpixel block is separated from the target image, and all separated superpixel blocks are mapped to the target image to obtain an abnormal image containing only the separated superpixel blocks.

8. The electronic cigarette defect detection device based on machine vision perception according to claim 5, characterized in that: The target attention feature map determination module includes: A reshaping feature map module is used to calculate the importance weights of the channels and the importance weights of the positions in any level feature map, and reshape the level feature map according to the importance weights and the importance weights to obtain a reshaped feature map; The target feature map addition module is used to convolve the reshaped feature map through a 1×1 convolution kernel to obtain a target feature map, and add all target feature maps to obtain a target attention feature map.

Images