An optoelectronic product detection method and system based on image processing

Abstract

The present application provides an optoelectronic product detection method and system based on image processing, which relates to the field of image recognition technology. By adaptively and dynamically scaling the multi-modal images on the surface of the optoelectronic product, complex defect inspection samples are obtained, and multi-scale feature vectors are extracted from the complex defect inspection samples. Furthermore, a fusion scale feature map of the optoelectronic product is constructed based on the multi-scale feature vectors and the light source mode, and the defect area is preliminarily detected from the fusion scale feature map. A defect subdivision model is constructed based on the feature entropy and local gradient information of the defect area, and then the defect area is secondarily segmented and the defect type is discriminated through the defect subdivision model to obtain a fine segmentation result. Furthermore, pixel-level intensity adjustment is performed on the defect area based on the defect probability map and the fine segmentation result of the defect area to obtain the defect detection result of the optoelectronic product. The present application can realize the optimized detection of fine defects on the surface of the optoelectronic product to improve the reliability of the optoelectronic product detection result.

Description

Technical Field

[0001] This application relates to the field of image recognition technology. More specifically, this application relates to a method and system for detecting optoelectronic products based on image processing. Background Art

[0002] Image recognition technology is an important branch in the field of computer vision, aiming to enable a computer to understand and interpret the content in digital images. Through processing and analyzing images, image recognition technology can identify objects, scenes, or features in images and is widely used in fields such as face recognition, autonomous driving, and medical image analysis.

[0003] In the detection of optoelectronic products, image processing-based methods have become the mainstream. By using a machine vision system to scan the product surface, defects existing in optoelectronic products, such as scratches, stains, or structural abnormalities, can be detected. However, existing optoelectronic product detection methods often rely on feature extraction at a single scale, making it difficult to comprehensively capture the diversity of complex defects, which may lead to missed detections or false detections. In the face of changes in different light source modes or product materials, existing methods lack an adaptive adjustment mechanism, affecting the accuracy and robustness of detection. In addition, in the segmentation of defect regions, traditional methods lack precision in detecting the exact defect boundaries, resulting in a decrease in the reliability of detection results. Therefore, how to achieve optimized detection of fine defects on the surface of optoelectronic products to improve the reliability of optoelectronic product detection results is a difficult problem faced by the current industry. Summary of the Invention

[0004] This application provides a method and system for detecting optoelectronic products based on image processing, which can achieve optimized detection of fine defects on the surface of optoelectronic products to improve the reliability of optoelectronic product detection results.

[0005] In a first aspect, this application provides a method and system for detecting optoelectronic products based on image processing. The optoelectronic product detection method includes the following steps:

[0006] Collect multi-modal images of the surface of an optoelectronic product and determine the light source mode during collection;

[0007] Perform adaptive dynamic scaling on the multi-modal images to obtain complex defect inspection samples, and extract multi-scale feature vectors from the complex defect inspection samples;

[0008] Construct a fusion scale feature map of the optoelectronic product based on the multi-scale feature vectors and the light source mode, and perform preliminary defect detection on the fusion scale feature map through a boundary recognition box optimized by a preset coordinate attention mechanism to obtain a defect region;

[0009] Construct a defect segmentation model based on the feature entropy and local gradient information of the defect area, and then perform secondary segmentation and defect type discrimination on the defect area through the defect segmentation model to obtain a fine segmentation result;

[0010] Extract the defect probability map of the defect area, and perform pixel-level intensity adjustment on the defect area based on the defect probability map and the fine segmentation result to obtain the defect detection result of the optoelectronic product.

[0011] In this embodiment, a multi-modal image acquisition device is used to acquire multi-modal images of the surface of the optoelectronic product.

[0012] In this embodiment, the specific steps of adaptively dynamically scaling the multi-modal image to obtain a complex defect inspection sample include:

[0013] Perform preliminary scaling processing on the multi-modal image through a preset dynamic scaling strategy to obtain a sequence of preliminary scaled images;

[0014] Generate local scaling factors based on the brightness, contrast, and texture statistical features of local regions in the sequence of preliminary scaled images;

[0015] Use the local scaling factors to perform dynamic scaling processing on the sequence of preliminary scaled images to obtain a complex defect inspection sample.

[0016] In this embodiment, the specific steps of extracting multi-scale feature vectors from the complex defect inspection sample include:

[0017] Use a multi-scale convolutional network to extract features from the complex defect inspection sample to generate local feature maps at multiple different resolutions;

[0018] Perform weighted fusion on all local feature maps to obtain a global feature map, and then extract multi-scale feature vectors from the global feature map.

[0019] In this embodiment, the specific steps of constructing a fusion scale feature map of the optoelectronic product based on the multi-scale feature vectors and the light source mode include:

[0020] Perform weighted fusion processing on the multi-scale feature vectors and the light source mode to obtain a weighted feature map;

[0021] Use a multi-scale feature vector fusion network with bidirectional cross-scale connections to perform hierarchical fusion on the weighted feature map to generate a preliminary fusion scale feature map;

[0022] Use a deformable attention mechanism to perform key feature screening and optimization on the preliminary fusion scale feature map to obtain a fusion scale feature map.

[0023] In this embodiment, the preset coordinate attention mechanism is a coordinate attention mechanism that fuses average pooling and max pooling.

[0024] In this embodiment, the defect segmentation model is a multi-scale feature vector fusion model based on a convolutional neural network.

[0025] In this embodiment, performing secondary segmentation and defect type discrimination on the defect region through the defect segmentation model to obtain a fine segmentation result specifically includes:

[0026] Using an improved semantic segmentation network to perform fine segmentation on the defect region to obtain defect boundary features and defect shape features;

[0027] Performing linear discriminant analysis on the defect boundary features and defect shape features to obtain a fine segmentation result.

[0028] In this embodiment, extracting the defect probability map of the defect region specifically includes:

[0029] Performing feature analysis on the defect region to obtain multiple defect feature values;

[0030] Generating a defect probability map based on all the defect feature values through a pre-trained defect probability model.

[0031] In a second aspect, the present application provides an optoelectronic product detection system based on image processing for performing an optoelectronic product detection method based on image processing. The optoelectronic product detection system includes:

[0032] An image acquisition module for acquiring multi-modal images of the surface of an optoelectronic product and determining the light source mode during acquisition;

[0033] A preprocessing module for adaptively dynamically scaling the multi-modal images to obtain complex defect inspection samples and extracting multi-scale feature vectors from the complex defect inspection samples;

[0034] A defect detection module for constructing a fusion scale feature map of an optoelectronic product based on the multi-scale feature vectors and the light source mode, and performing preliminary defect detection on the fusion scale feature map through a boundary recognition box optimized by a preset coordinate attention mechanism to obtain a defect region;

[0035] A defect segmentation module for constructing a defect segmentation model based on the feature entropy and local gradient information of the defect region, and then performing secondary segmentation and defect type discrimination on the defect region through the defect segmentation model to obtain a fine segmentation result;

[0036] An optimization detection module is used to extract the defect probability map of the defect area, and perform pixel-level intensity adjustment on the defect area based on the defect probability map and the fine segmentation result to obtain the defect detection result of the optoelectronic product.

[0037] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects:

[0038] By collecting multi-modal images on the surface of an optoelectronic product, determining the light source mode during collection; performing adaptive dynamic scaling on the multi-modal images to obtain complex defect inspection samples, and extracting multi-scale feature vectors from the complex defect inspection samples; constructing a fusion scale feature map of the optoelectronic product based on the multi-scale feature vectors and the light source mode, and performing preliminary defect detection on the fusion scale feature map through a boundary recognition box optimized by a preset coordinate attention mechanism to obtain a defect area; constructing a defect segmentation model based on the feature entropy and local gradient information of the defect area, and then performing secondary segmentation and defect type discrimination on the defect area through the defect segmentation model to obtain a fine segmentation result; extracting the defect probability map of the defect area, and performing pixel-level intensity adjustment on the defect area based on the defect probability map and the fine segmentation result to obtain the defect detection result of the optoelectronic product.

[0039] It can be seen that in this application, the optimized detection of fine defects on the surface of optoelectronic products can be realized. First, through the synchronous acquisition of multi-modal images and the intelligent switching of the dynamic light source mode, the problems of low defect contrast and missing features caused by reflection interference under a single light source can be solved. Among them, different light source modes can highlight the differential optical responses of defects such as cracks, bubbles, and scratches, enhancing the comprehensiveness of defect feature expression. Second, through the generation of dynamic scaling strategies and local scaling factors, combined with the hierarchical feature fusion of multi-scale convolutional networks, the problems of insufficient resolution of micro-defects and missing context information of large-size defects caused by traditional fixed scaling are solved. The scaling factor is generated based on local brightness, contrast, and texture features, which can enhance the resolution of the defect area and avoid edge blurring caused by global scaling. The global feature map is generated through weighted fusion to ensure the synchronous capture of features of micron-scale cracks and millimeter-scale scratches, which is conducive to improving the detection rate of complex defects. Then, the multi-scale feature vectors and the light source mode are weighted and fused, and the coordinate attention mechanism is used to optimize the boundary recognition box, solving the problems of weak correlation between features and light sources and inaccurate boundary positioning in traditional methods. Through the weighted fusion of the light source mode and multi-scale feature vectors, the distinction between the defect area and the background can be strengthened, and the reflection artifacts can be suppressed. A defect segmentation model is constructed based on feature entropy and local gradient information, which can solve the problem of low segmentation accuracy of fuzzy defects and overlapping defects in traditional single detection and improve the accuracy of defect classification. Finally, through the defect probability model, a pixel-level probability map is generated, and the intensity of the defect area is dynamically adjusted in combination with the fine segmentation result. Among them, pixel-level weights are generated through the probability map and the segmentation result to balance the brightness consistency between the defect and the background and avoid the loss of details caused by overcorrection.

[0040] In summary, the technical solution adopted in this application can realize the optimized detection of fine defects on the surface of optoelectronic products, so as to improve the reliability of the detection results of optoelectronic products. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to these drawings.

[0042] Figure 1 is a flowchart of an optoelectronic product detection method based on image processing provided by the present application;

[0043] Figure 2 is an exemplary flowchart for determining complex defect inspection samples provided by the present application;

[0044] Figure 3It is an exemplary flowchart for determining a defective area provided according to the present application;

[0045] Figure 4 It is a module structure diagram of an optoelectronic product detection system provided according to the present application. Detailed implementation manners

[0046] Next, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without making creative efforts belong to the scope of protection of the present application.

[0047] The embodiments of the present application provide an optoelectronic product detection method and system based on image processing. The core is to collect multimodal images on the surface of the optoelectronic product to determine the light source mode during collection; perform adaptive dynamic scaling on the multimodal images to obtain complex defect inspection samples, and extract multi-scale feature vectors from the complex defect inspection samples; construct a fusion scale feature map of the optoelectronic product based on the multi-scale feature vectors and the light source mode, and perform preliminary defect detection on the fusion scale feature map through a boundary recognition box optimized by a preset coordinate attention mechanism to obtain a defective area; construct a defect segmentation model based on the feature entropy and local gradient information of the defective area, and then perform secondary segmentation and defect type discrimination on the defective area through the defect segmentation model to obtain a fine segmentation result; extract the defect probability map of the defective area, and perform pixel-level intensity adjustment on the defective area based on the defect probability map and the fine segmentation result to obtain the defect detection result of the optoelectronic product.

[0048] Embodiment 1. To better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings of the specification and specific implementation manners. Refer to Figure 1 As shown, this figure is an exemplary flowchart of an optoelectronic product detection method based on image processing according to this embodiment of the present application. The optoelectronic product detection method includes the following steps:

[0049] In step S1, collect multimodal images on the surface of the optoelectronic product to determine the light source mode during collection.

[0050] In specific implementation, a multi-modal image acquisition device synchronously acquires multi-modal images on the surface of an optoelectronic product within a specified time period. The multi-modal images include visible light images, infrared images, depth images, etc. Among them, the multi-modal image acquisition device continuously captures image frames obtained by different sensors within a 3-5 second acquisition window. Each frame records the detailed information of the surface of the optoelectronic product under its respective spectrum, such as color texture, thermal distribution characteristics, and three-dimensional structure data, providing rich and multi-angle information input for subsequent image adaptive processing and defect detection.

[0051] It should be noted that the multi-modal images in this application refer to different modal image data synchronously acquired by multiple image sensors within the same time period, including: visible light images, which can provide color texture and detailed information on the product surface; infrared images, which can reflect the temperature distribution and thermal radiation characteristics of the product surface; depth images, which can record the three-dimensional structure and shape changes of the product surface. By capturing different physical characteristics of the surface of the optoelectronic product through multi-modal images, they complement each other, which is beneficial to improving the accuracy of overall detection. In addition, the light source mode refers to the lighting conditions and parameter settings adopted during image acquisition, including the color, brightness, irradiation angle, etc. of the light source. These parameters directly affect the acquisition effect and signal-to-noise ratio of different modal images. By analyzing the characteristics such as the light intensity distribution and edge information in the acquired images, the system can determine the current light source mode and provide accurate lighting references for subsequent adaptive dynamic scaling and defect detection, ensuring the effective fusion and utilization of each modal image data.

[0052] In step S2, perform adaptive dynamic scaling on the multi-modal images to obtain complex defect inspection samples, and extract multi-scale feature vectors from the complex defect inspection samples.

[0053] Preferably, in this embodiment, refer to Figure 2 As shown, this figure is an exemplary flowchart for determining complex defect inspection samples in an embodiment of this application. In this embodiment, performing adaptive dynamic scaling on the multi-modal images to obtain complex defect inspection samples can be specifically implemented by the following steps:

[0054] First, in step S21, perform preliminary scaling processing on the multi-modal images through a preset dynamic scaling strategy to obtain a preliminary scaled image sequence;

[0055] Then, in step S22, generate local scaling factors based on the brightness, contrast, and texture statistical characteristics of local regions in the preliminary scaled image sequence;

[0056] Finally, in step S23, perform dynamic scaling processing on the preliminary scaled image sequence using the local scaling factors to obtain complex defect inspection samples.

[0057] In specific implementation, first, the multi-modal image is preliminarily scaled using a preset dynamic scaling strategy, that is: the preset dynamic scaling strategy refers to the overall brightness and color saturation equalization adjustment of each modal image (such as visible light, infrared, depth map) by combining image normalization and automatic exposure correction technology. The cv2.normalize function in OpenCV can be used to normalize the multi-modal image, and the automatic exposure adjustment algorithm is combined to unify the brightness distribution of each image. The result obtained after normalization and pre-scaling is a sequence of preliminarily scaled images, which ensures a unified basis for the subsequent extraction of local features. Then, each preliminarily scaled image in the sequence of preliminarily scaled images is divided into regions using a 16×16 grid to obtain multiple local regions. For each local region, the grayscale histogram of the region is calculated using the calcHist function of OpenCV, and the local edge gradient is calculated using the Sobel operator (such as cv2.Sobel). The standard deviation of the grayscale values within the calculated region is used as the region contrast index. When the region contrast index ≥ 0.6, a lower local scaling factor (for example, 1.0 to 1.1) is set, indicating that the details in this region are relatively sufficient; when the contrast index is between 0.3 and 0.6, a medium scaling factor (for example, 1.2 to 1.4) is set; when the contrast index < 0.3, a higher scaling factor (for example, 1.5 to 1.8) is set to magnify the region with insufficient details. In other embodiments, the local scaling factor can also be set according to actual inspection requirements, which is not limited here. Finally, for each local region, the cv2.resize function of OpenCV is used to adjust the size of the local region according to the corresponding local scaling factor, and then the local regions are recombined into a complete image using the image stitching function of OpenCV, so that the combined image is used as a complex defect inspection sample.

[0058] It should be noted that the multi-modal image includes visible light images, infrared images, depth images, etc. Each modality respectively reflects different physical characteristics of the product surface, such as color texture, thermal distribution, and three-dimensional structure; the generation of the local scaling factor is based on the brightness, contrast, and texture statistical characteristics of the local region, which can ensure that the defect region can be magnified in detail during the image adaptive scaling process, thus helping to reduce the risk of missed detection and improve the detection accuracy; the dynamic scaling process adaptively adjusts the scaling ratio, so that high-detail regions (such as tiny scratches or cracks) have a more obvious magnification effect, while maintaining the overall visual continuity of low-detail regions, providing high-quality samples for the subsequent extraction of multi-scale feature vectors and defect detection.

[0059] In this embodiment, the multi-scale feature vectors can be extracted from the complex defect inspection sample in the following specific way, that is:

[0060] Use a multi-scale convolutional network to extract features from the complex defect inspection samples, generating local feature maps at multiple different resolutions;

[0061] Perform weighted fusion on all the local feature maps to obtain a global feature map, and then extract multi-scale feature vectors from the global feature map.

[0062] In specific implementation, first, by designing multiple convolutional layers, each layer using convolutional kernels of different sizes (such as 3×3, 5×5, 7×7, etc.) to capture the detailed information of local regions in the image at different resolutions, thus generating multiple local feature maps. This process can be implemented based on the OpenCV library after analysis using the Pandas library. The feature maps output by each convolutional layer can reflect the defect features at different scales in the samples. Secondly, perform weighted fusion on all the generated local feature maps. Pixel-by-pixel weighted summation can be used to integrate the normalized features in the feature maps of each scale, and calculate the mean square error and structural similarity before weighted fusion to obtain a comprehensive global feature map. Finally, the principal component analysis (PCA) can be used to process the global feature map to extract multi-scale feature vectors that contain both fine-grained local information and global description ability, providing accurate feature basis for the subsequent detection module.

[0063] It should be noted that in this embodiment, through the design of the multi-scale convolutional network and the weighted fusion strategy, the comprehensive utilization of multi-level and multi-resolution information in the complex defect inspection samples is realized, effectively separating the environmental interference factors from the real defect features, thereby improving the accuracy and robustness of defect detection and providing reliable data support for subsequent defect location and type discrimination.

[0064] In step S3, based on the multi-scale feature vector and the light source mode, construct a fusion scale feature map of the optoelectronic product, and perform preliminary defect detection on the fusion scale feature map through the boundary recognition box optimized by the preset coordinate attention mechanism to obtain the defect region.

[0065] In this embodiment, constructing the fusion scale feature map of the optoelectronic product based on the multi-scale feature vector and the light source mode can be specifically implemented in the following way, that is:

[0066] Perform weighted fusion processing on the multi-scale feature vector and the light source mode to obtain a weighted feature map;

[0067] Use a multi-scale feature vector fusion network with bidirectional cross-scale connection to perform hierarchical fusion on the weighted feature map to generate a preliminary fusion scale feature map;

[0068] Adopt a deformable attention mechanism to perform key feature screening and optimization on the preliminary fusion scale feature map to obtain the fusion scale feature map.

[0069] In specific implementation, first, the light source mode parameters (such as light source intensity, wavelength, irradiation angle, etc.) can be encoded as feature vectors and concatenated with the multi-scale feature vectors. A learnable weight matrix is introduced through machine learning techniques to perform element-wise weighted summation on the concatenated feature vectors, thereby generating a weighted feature map. By fusing the ambient light information and defect features, subsequent processing can automatically compensate for the feature differences caused by light changes, thus improving the overall detection robustness. Secondly, a top-down and bottom-up two-way transmission path is constructed based on the structure of the bidirectional feature pyramid network to perform hierarchical fusion on feature maps with different resolutions, and then perform normalization and weighted fusion processing on the features of each layer to obtain a preliminary fusion scale feature map, which is beneficial to enhancing the comprehensiveness and consistency of feature expression. Finally, the offset is determined through an autoregressive analysis algorithm, and the sampling points in the feature map are dynamically adjusted according to the offset, so that the attention module can adaptively focus on the key structures in the defect area, and perform element-wise multiplication operation (which can be implemented through the OpenCV library) on the key structures in the defect area and the preliminary fusion scale feature map to obtain a fusion scale feature map, which can strengthen the expression of important defect features, reduce environmental interference, and further improve the detection accuracy and stability.

[0070] It should be noted that in this embodiment, by performing weighted fusion on the multi-scale feature vector and the light source mode, and using a bidirectional cross-scale connection module to achieve hierarchical feature fusion, and finally adopting a deformable attention mechanism to optimize the key features, the multi-level comprehensive expression of complex defect information is effectively realized, providing accurate and robust feature support for the optoelectronic product detection system, thereby significantly improving the detection accuracy.

[0071] Preferably, in this embodiment, referring to Figure 3 As shown, this figure is an exemplary flowchart for determining the defect area in the embodiment of the present application. In this embodiment, the fusion scale feature map is initially defect-detected through a boundary recognition box optimized by a preset coordinate attention mechanism, and the defect area can be specifically implemented by the following steps:

[0072] First, in step S31, the fusion scale feature map is input into the boundary recognition box optimized by the preset coordinate attention mechanism to obtain a fusion scale feature map optimized by attention.

[0073] Then, in step S32, the attention-concentrated area in the fusion scale feature map optimized by attention is regionally segmented to obtain a potential defect area.

[0074] Finally, in step S33, the fused image of the potential defect area is extracted, and then morphological operations are performed on the fused image to obtain the defect area.

[0075] It should be noted that the preset coordinate attention mechanism is a coordinate attention mechanism that fuses average pooling and max pooling. By integrating global statistics and local significant information, it can more accurately focus on the key features of the defect area, providing efficient feature support for preliminary defect detection.

[0076] In specific implementation, first, average pooling is performed on each channel of the fused scale feature map to extract global statistical information, and max pooling is applied to the same feature map to capture local significant information. Then, the results of the two poolings are concatenated, and feature fusion is achieved through 1×1 convolution mapping to generate an attention weight map. Furthermore, the attention weight map is multiplied pixel by pixel with the original fused scale feature map to obtain an attention-optimized fused scale feature map, which can effectively suppress redundant background information and provide a more accurate data basis for subsequent region proposals. Then, threshold segmentation is performed on the high response values of the attention-optimized fused scale feature map to obtain the attention concentration area, and the clustering algorithm is used to integrate the attention concentration area to obtain the potential defect area. Finally, operations such as erosion and dilation can be used to remove noise, smooth the candidate region boundary, and extract the precise defect boundary through contour detection, thereby outputting the defect area after morphological optimization processing, making the detection result have high boundary accuracy and region coherence.

[0077] In step S4, a defect subdivision model is constructed based on the feature entropy and local gradient information of the defect area, and then the defect area is secondarily segmented and the defect type is discriminated through the defect subdivision model to obtain a fine segmentation result.

[0078] It should be noted that the defect subdivision model in this embodiment is a multi-scale feature vector fusion model based on a convolutional neural network. Using an improved semantic segmentation network can fully extract the detailed information of the defect area, and linear discriminant analysis is used to quantitatively discriminate the defect boundary and shape features, which can effectively separate environmental interference from real defect features, thereby realizing the refined segmentation of the defect area and the discrimination of the defect type, providing reliable and accurate defect segmentation data for subsequent optoelectronic product detection.

[0079] In specific implementation, first, the Sobel operator is used to calculate the gradient of the defect area, and the local gradient map can capture the edge changes and texture details in the defect area, providing significant features for subsequent segmentation. The feature entropy reflects the randomness and information complexity of the texture within the area, and can effectively distinguish environmental noise from real defect features. The information entropy formula can be used to calculate the histogram of the local gradient magnitude, and an entropy value map is output. The entropy value of each pixel in this entropy value map can represent the uncertainty of local information. Then, the convolutional neural network based on the U-Net architecture is improved through channel splicing, and the improved convolutional neural network is used as the defect refinement model. Among them, the multi-scale convolutional layer of the improved convolutional neural network extracts fused features. Through the fused feature map, the model can more comprehensively capture the edge, texture, and structure information of the defect area, thereby realizing the secondary segmentation of the defect area and the discrimination of defect types.

[0080] It should be noted that in this embodiment, by taking the local gradient information and feature entropy as key auxiliary features and inputting them into the defect refinement model, environmental interference factors and real defect features can be effectively separated, thereby improving the accuracy and robustness of defect segmentation and providing an accurate deviation correction basis for subsequent defect location and type discrimination.

[0081] In this embodiment, the defect refinement model is used to perform secondary segmentation and defect type discrimination on the defect area. The specific method for obtaining the refined segmentation result is as follows:

[0082] The improved semantic segmentation network is used to perform refined segmentation on the defect area to obtain defect boundary features and defect shape features.

[0083] Linear discriminant analysis is performed on the defect boundary features and defect shape features to obtain the refined segmentation result.

[0084] In specific implementation, the defect area is input into the improved semantic segmentation network. This network adopts a deep convolutional architecture such as U-Net and is optimized to improve the local detail capture ability. The improved semantic segmentation network extracts boundary features and shape features within the defect area at multiple scales, generating a high-resolution defect boundary map and defect shape map. Then, the defect boundary features are extracted from the defect boundary map through an existing feature extraction model, and the defect shape features are extracted from the defect shape map through the same existing feature extraction model. Then, linear discriminant analysis is used to reduce the dimension of the extracted defect boundary features and defect shape features, and then by calculating the discriminant indexes (mean square error and structural similarity) between different defect types, each defect area is finely classified to obtain the refined segmentation result.

[0085] It should be noted that the improved semantic segmentation network in this embodiment is obtained through multi-scale feature extraction and fine-grained structure optimization, and can accurately capture the boundaries and shapes of defect regions; through linear discriminant analysis, the ability of feature dimensionality reduction and classification can be utilized to efficiently integrate the defect boundary features and shape features, thereby providing an accurate basis for defect type discrimination.

[0086] In step S5, a defect probability map of the defect region is extracted, and pixel-level intensity adjustment is performed on the defect region based on the defect probability map and the fine segmentation result to obtain a defect detection result of the optoelectronic product.

[0087] In this embodiment, the defect probability map of the defect region can be extracted specifically in the following manner, that is:

[0088] Feature analysis is performed on the defect region to obtain a plurality of defect feature values;

[0089] A defect probability map is generated based on all the defect feature values through a pre-trained defect probability model.

[0090] Specifically, when implemented, the defect region is preprocessed through an edge detection algorithm, and a convolutional neural network is used to statistically analyze various features such as color, texture, shape, and brightness in the region to obtain a plurality of defect feature values; then, the defect feature values are formed into a defect feature value set, and the defect feature value set is comprehensively evaluated through a Bayesian network, and then the defect probability value of each pixel is output to obtain a defect probability map.

[0091] It should be noted that performing pixel-level intensity adjustment based on the defect probability map and the fine segmentation result can not only accurately highlight the defect region, reduce background noise interference, but also effectively balance local details and overall brightness, thereby significantly improving the reliability and detection accuracy of the optoelectronic product detection result; specifically, when implemented, the defect probability map can be used as a weight map and multiplied pixel by pixel with the original pixel intensity in the corresponding region of the fine segmentation result to achieve local enhancement or suppression. At the same time, an adaptive smoothing filtering algorithm is used to further optimize the processed image, the noise in the detection image can be eliminated, and finally the defect detection result of the optoelectronic product after pixel-level intensity adjustment is output, and the defect detection result of the optoelectronic product is visualized through the visualization library in OpenCV.

[0092] It can be seen that in this application, the optimized detection of fine defects on the surface of optoelectronic products can be realized. First, through the synchronous acquisition of multi-modal images and the intelligent switching of dynamic light source modes, the problems of low defect contrast and feature loss caused by reflection interference under a single light source can be solved. Among them, different light source modes can highlight the differential optical responses of defects such as cracks, bubbles, and scratches, enhancing the comprehensiveness of defect feature expression. Second, through the generation of dynamic scaling strategies and local scaling factors, combined with the hierarchical feature fusion of multi-scale convolutional networks, the problems of insufficient resolution of tiny defects and missing context information of large-size defects caused by traditional fixed scaling are solved. The scaling factors are generated based on local brightness, contrast, and texture features, which can enhance the resolution of the defect area and avoid edge blurring caused by global scaling. A global feature map is generated through weighted fusion to ensure the synchronous capture of features of micron-scale cracks and millimeter-scale scratches, which is beneficial to improving the detection rate of complex defects. Then, the multi-scale feature vectors are weighted and fused with the light source mode, and the coordinate attention mechanism is used to optimize the boundary recognition box, solving the problems of weak correlation between features and light sources and inaccurate boundary positioning in traditional methods. Through the weighted fusion of the light source mode and multi-scale feature vectors, the distinction between the defect area and the background can be strengthened, and reflection artifacts can be suppressed. A defect segmentation model is constructed based on feature entropy and local gradient information, which can solve the problem of low segmentation accuracy of fuzzy defects and overlapping defects in traditional single detection and improve the accuracy of defect classification. Finally, a pixel-level probability map is generated through a defect probability model, and the intensity of the defect area is dynamically adjusted in combination with the fine segmentation result. Among them, pixel-level weights are generated through the probability map and the segmentation result to balance the brightness consistency between the defect and the background and avoid detail loss caused by overcorrection.

[0093] In summary, the technical solution adopted in this application can realize the optimized detection of fine defects on the surface of optoelectronic products, so as to improve the reliability of the detection results of optoelectronic products.

[0094] Embodiment 2. This application provides an optoelectronic product detection system based on image processing. Refer to Figure 4 As shown in the figure, which is the module structure diagram of the optoelectronic product detection system according to this embodiment of this application, the optoelectronic product detection system includes:

[0095] An image acquisition module 100, configured to acquire multi-modal images of the surface of an optoelectronic product and determine the light source mode during acquisition;

[0096] A preprocessing module 200, configured to perform adaptive dynamic scaling on the multi-modal images to obtain complex defect inspection samples, and extract multi-scale feature vectors from the complex defect inspection samples;

[0097] The defect detection module 300 is used to construct a fused scale feature map of the optoelectronic product based on the multi-scale feature vector and the light source pattern, and perform preliminary defect detection on the fused scale feature map through a boundary recognition box optimized by a preset coordinate attention mechanism to obtain a defect area;

[0098] The defect subdivision module 400 is used to construct a defect subdivision model based on the feature entropy and local gradient information of the defect area, and then perform secondary segmentation and defect type discrimination on the defect area through the defect subdivision model to obtain a fine segmentation result;

[0099] The optimization detection module 500 is used to extract the defect probability map of the defect area, and perform pixel-level intensity adjustment on the defect area based on the defect probability map and the fine segmentation result to obtain the defect detection result of the optoelectronic product.

[0100] This application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one Figure 1 one or more flows and / or Figure 1 blocks or multiple blocks.

[0101] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. The storage medium includes read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc memories, magnetic disk memories, tape memories, or any other medium that can be used to carry or store data and is computer-readable.

[0102] It should also be noted that the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent in such process, method, commodity or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, commodity or device including the element.

Claims

1. An optoelectronic product detection method based on image processing, characterized in that, The optoelectronic product detection method includes the following steps: Collect multimodal images on the surface of the optoelectronic product and determine the light source mode during collection; Perform adaptive dynamic scaling on the multimodal images to obtain complex defect inspection samples, and extract multi-scale feature vectors from the complex defect inspection samples; Construct a fusion scale feature map of the optoelectronic product based on the multi-scale feature vectors and the light source mode, and perform preliminary defect detection on the fusion scale feature map through a boundary recognition box optimized by a preset coordinate attention mechanism to obtain a defect area; Construct a defect segmentation model based on the feature entropy and local gradient information of the defect area, and then perform secondary segmentation and defect type discrimination on the defect area through the defect segmentation model to obtain a fine segmentation result; Extract the defect probability map of the defect area, and perform pixel-level intensity adjustment on the defect area based on the defect probability map and the fine segmentation result to obtain the defect detection result of the optoelectronic product; Among them, performing adaptive dynamic scaling on the multimodal images to obtain complex defect inspection samples specifically includes: Perform preliminary scaling processing on the multimodal images through a preset dynamic scaling strategy to obtain a preliminary scaled image sequence; Generate local scaling factors according to the brightness, contrast, and texture statistical features of local regions in the preliminary scaled image sequence; Use the local scaling factors to perform dynamic scaling processing on the preliminary scaled image sequence to obtain complex defect inspection samples; Among them, performing preliminary defect detection on the fusion scale feature map through a boundary recognition box optimized by a preset coordinate attention mechanism to obtain a defect area specifically includes: Input the fusion scale feature map into a boundary recognition box optimized by a preset coordinate attention mechanism to obtain a fusion scale feature map with optimized attention; Perform region segmentation on the attention-concentrated regions in the fusion scale feature map with optimized attention to obtain potential defect regions; Extract the fusion images of the potential defect regions, and then perform morphological operations on the fusion images to obtain defect regions; Among them, the preset coordinate attention mechanism is a coordinate attention mechanism that fuses average pooling and max pooling; Among them, performing secondary segmentation and defect type discrimination on the defect area through the defect segmentation model to obtain a fine segmentation result specifically includes: Use an improved semantic segmentation network to perform refined segmentation on the defect area to obtain defect boundary features and defect shape features; Perform linear discriminant analysis on the defect boundary features and defect shape features to obtain a fine segmentation result; Among them, the improved semantic segmentation network is obtained through multi-scale feature extraction and fine-grained structure optimization.

2. The optoelectronic product detection method based on image processing according to claim 1, characterized in that, Collect multimodal images on the surface of the optoelectronic product through a multimodal image acquisition device.

3. The optoelectronic product detection method based on image processing according to claim 1, characterized in that, Extracting multi-scale feature vectors from the complex defect inspection samples specifically includes: Use a multi-scale convolutional network to extract features from the complex defect inspection samples to generate local feature maps at multiple different resolutions; Perform weighted fusion on all the local feature maps to obtain a global feature map, and then extract multi-scale feature vectors from the global feature map.

4. The optoelectronic product detection method based on image processing according to claim 1, characterized in that, Constructing the fused scale feature map of the optoelectronic product based on the multi-scale feature vector and the light source pattern specifically includes: Performing weighted fusion processing on the multi-scale feature vector and the light source pattern to obtain a weighted feature map; Using a multi-scale feature vector fusion network with bidirectional cross-scale connections to perform hierarchical fusion on the weighted feature map to generate a preliminary fused scale feature map; Adopting a deformable attention mechanism to perform key feature screening and optimization on the preliminary fused scale feature map to obtain a fused scale feature map.

5. The optoelectronic product detection method based on image processing according to claim 1, wherein, The defect segmentation model is a multi-scale feature vector fusion model based on a convolutional neural network.

6. The optoelectronic product detection method based on image processing according to claim 1, characterized in that, Extracting the defect probability map of the defect area specifically includes: Performing feature analysis on the defect area to obtain multiple defect feature values; Generating a defect probability map based on all the defect feature values through a pre-trained defect probability model.

7. An optoelectronic product detection system based on image processing, which is used to execute an optoelectronic product detection method based on image processing as described in any one of claims 1 to 6, characterized in that, The optoelectronic product detection system includes: An image acquisition module for acquiring multi-modal images of the surface of the optoelectronic product and determining the light source pattern during acquisition; A preprocessing module for adaptively dynamically scaling the multi-modal images to obtain complex defect inspection samples and extracting multi-scale feature vectors from the complex defect inspection samples; A defect detection module for constructing a fused scale feature map of the optoelectronic product based on the multi-scale feature vector and the light source pattern, and performing preliminary defect detection on the fused scale feature map through a boundary recognition box optimized by a preset coordinate attention mechanism to obtain a defect area; A defect segmentation module for constructing a defect segmentation model based on the feature entropy and local gradient information of the defect area, and then performing secondary segmentation and defect type discrimination on the defect area through the defect segmentation model to obtain a fine segmentation result; An optimization detection module for extracting the defect probability map of the defect area and performing pixel-level intensity adjustment on the defect area based on the defect probability map and the fine segmentation result to obtain the defect detection result of the optoelectronic product.

Images