Photoelectric product detection method and system based on image processing

Abstract

The invention provides a photoelectric product detection method and system based on image processing, and relates to the technical field of image recognition, a complex defect detection sample is obtained by performing adaptive dynamic scaling on a multi-modal image on the surface of a photoelectric product, and a multi-scale feature vector is extracted from the complex defect detection sample; a fusion scale feature map of the photoelectric product is constructed based on the multi-scale feature vector and a light source mode, and a defect area is obtained through preliminary detection in the fusion scale feature map; constructing a defect subdivision model according to the feature entropy and the local gradient information of the defect region, and performing secondary segmentation and defect type discrimination on the defect region through the defect subdivision model to obtain a fine segmentation result; and pixel-level intensity adjustment is carried out on the defect area based on the defect probability graph of the defect area and the fine segmentation result to obtain a defect detection result of the photoelectric product, and optimal detection of fine defects on the surface of the photoelectric product can be realized so as to improve the reliability of the detection result of the photoelectric product.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and more specifically, to an optoelectronic product detection method and system based on image processing. Background Art

[0002] Image recognition technology is an important branch of the field of computer vision. It aims to enable computers to understand and interpret the content of digital images. Image recognition technology processes and analyzes images to identify objects, scenes or features in images. It is widely used in face recognition, autonomous driving, medical image analysis and other fields.

[0003] In optoelectronic product inspection, image processing-based methods have become mainstream. By using machine vision systems to scan the product surface, defects in optoelectronic products, such as scratches, stains or structural abnormalities, are detected. However, existing optoelectronic product inspection methods often rely on feature extraction at a single scale, which makes it difficult to fully capture the diversity of complex defects, resulting in missed or false detections. In the face of different light source modes or changes in product materials, existing methods lack an adaptive adjustment mechanism, which affects the accuracy and robustness of inspection. In addition, in the segmentation of defective areas, traditional methods lack the accuracy of detecting precise defect boundaries, resulting in a decrease in the reliability of inspection results. Therefore, how to achieve optimized detection of fine defects on the surface of optoelectronic products to improve the reliability of optoelectronic product inspection results is a difficult problem currently faced by the industry. Summary of the invention

[0004] The present application provides an optoelectronic product inspection method and system based on image processing, which can realize optimized inspection of fine defects on the surface of optoelectronic products to improve the reliability of inspection results of optoelectronic products.

[0005] In a first aspect, the present application provides an optoelectronic product detection method and system based on image processing, wherein the optoelectronic product detection method comprises the following steps: Collect multimodal images of the surface of optoelectronic products and determine the light source mode during acquisition; Adaptively and dynamically scaling the multimodal image to obtain a complex defect inspection sample, and extracting a multi-scale feature vector from the complex defect inspection sample; Based on the multi-scale feature vector and the light source mode, a fused scale feature map of the optoelectronic product is constructed, and preliminary defect detection is performed on the fused scale feature map through a boundary recognition frame optimized by a preset coordinate attention mechanism to obtain a defect area; A defect segmentation model is constructed according to the characteristic entropy and local gradient information of the defect area, and then the defect area is secondary segmented and the defect type is distinguished by the defect segmentation model to obtain a fine segmentation result; A defect probability map of the defect area is extracted, and pixel-level intensity adjustment is performed on the defect area based on the defect probability map and the fine segmentation result to obtain a defect detection result of the optoelectronic product.

[0006] In this embodiment, a multimodal image of the surface of the optoelectronic product is acquired by a multimodal image acquisition device.

[0007] In this embodiment, adaptive dynamic scaling is performed on the multimodal image to obtain a complex defect inspection sample, specifically including: Performing preliminary scaling processing on the multimodal image by using a preset dynamic scaling strategy to obtain a preliminary scaled image sequence; generating a local scaling factor according to brightness, contrast and texture statistical characteristics of a local area in the preliminary scaled image sequence; The local scaling factor is used to dynamically scale the preliminary scaled image sequence to obtain a complex defect inspection sample.

[0008] In this embodiment, extracting a multi-scale feature vector from the complex defect inspection sample specifically includes: A multi-scale convolutional network is used to extract features of the complex defect inspection sample to generate a plurality of local feature maps at different resolutions; All local feature maps are weightedly fused to obtain a global feature map, and then a multi-scale feature vector is extracted from the global feature map.

[0009] In this embodiment, constructing a fused scale feature map of an optoelectronic product based on the multi-scale feature vector and the light source mode specifically includes: Performing weighted fusion processing on the multi-scale feature vector and the light source pattern information to obtain a weighted feature map; The weighted feature maps are hierarchically fused using a multi-scale feature vector fusion network with bidirectional cross-scale connections to generate a preliminary fused scale feature map. A deformable attention mechanism is used to screen and optimize key features of the preliminary fused scale feature map to obtain a fused scale feature map.

[0010] In this embodiment, the preset coordinate attention mechanism is a coordinate attention mechanism that integrates average pooling and maximum pooling.

[0011] In this embodiment, the defect segmentation model is a multi-scale feature vector fusion model based on a convolutional neural network.

[0012] In this embodiment, the defect area is segmented twice and the defect type is identified by the defect segmentation model, and the fine segmentation result obtained specifically includes: Using an improved semantic segmentation network to perform fine segmentation on the defect area to obtain defect boundary features and defect shape features; Linear discriminant analysis is performed on the defect boundary features and defect shape features to obtain a fine segmentation result.

[0013] In this embodiment, extracting the defect probability map of the defect area specifically includes: Performing feature analysis on the defect area to obtain multiple defect feature values; The defect probability map is generated based on all defect feature values ​​through the pre-trained defect probability model.

[0014] In a second aspect, the present application provides an optoelectronic product detection system based on image processing, which is used to perform an optoelectronic product detection method based on image processing, and the optoelectronic product detection system includes: An image acquisition module is used to acquire multimodal images of the surface of optoelectronic products and determine the light source mode during acquisition; A preprocessing module, used for adaptively and dynamically scaling the multimodal image to obtain a complex defect inspection sample, and extracting a multi-scale feature vector from the complex defect inspection sample; A defect detection module, used to construct a fused scale feature map of the optoelectronic product based on the multi-scale feature vector and the light source mode, and perform preliminary defect detection on the fused scale feature map through a boundary recognition frame optimized by a preset coordinate attention mechanism to obtain a defect area; A defect segmentation module is used to construct a defect segmentation model according to the characteristic entropy and local gradient information of the defect area, and then perform secondary segmentation and defect type discrimination on the defect area through the defect segmentation model to obtain a fine segmentation result; The optimization detection module is used to extract a defect probability map of the defect area, and adjust the pixel-level intensity of the defect area based on the defect probability map and the fine segmentation result to obtain a defect detection result of the optoelectronic product.

[0015] The technical solution provided by the embodiments disclosed in this application has the following beneficial effects: By collecting multimodal images of the surface of optoelectronic products, the light source mode during collection is determined; the multimodal images are adaptively and dynamically scaled to obtain complex defect inspection samples, and multiscale feature vectors are extracted from the complex defect inspection samples; a fused scale feature map of the optoelectronic product is constructed based on the multiscale feature vector and the light source mode, and a preliminary defect detection is performed on the fused scale feature map through a boundary recognition box optimized by a preset coordinate attention mechanism to obtain a defect area; a defect segmentation model is constructed based on the feature entropy and local gradient information of the defect area, and then the defect area is secondary segmented and the defect type is discriminated through the defect segmentation model to obtain a fine segmentation result; a defect probability map of the defect area is extracted, and pixel-level intensity adjustment is performed on the defect area based on the defect probability map and the fine segmentation result to obtain a defect detection result of the optoelectronic product.

[0016] It can be seen that in this application, optimized detection of fine defects on the surface of optoelectronic products can be achieved; first, through the synchronous acquisition of multimodal images, combined with the intelligent switching of dynamic light source modes, it is possible to solve the problem of feature loss caused by low defect contrast and reflective interference under a single light source. Among them, different light source modes can highlight the differentiated optical responses of defects such as cracks, bubbles, scratches, etc., and enhance the comprehensiveness of defect feature expression; secondly, through dynamic scaling strategy and local scaling factor generation, combined with the hierarchical feature fusion of multi-scale convolutional networks, the problems of insufficient resolution of tiny defects and lack of contextual information of large-size defects caused by traditional fixed scaling are solved. The scaling factor is generated based on local brightness, contrast and texture features, which can enhance the resolution of the defect area and avoid edge blurring caused by global scaling. The global feature map is generated through weighted fusion to ensure that micron-level cracks and millimeter-level The synchronous capture of scratch features is conducive to improving the detection rate of complex defects; then, the multi-scale feature vector is weightedly fused with the light source pattern information, and the coordinate attention mechanism is used to optimize the boundary recognition box to solve the problems of weak correlation between features and light sources and inaccurate boundary positioning in traditional methods. The weighted fusion of light source patterns and multi-scale feature vectors can enhance the distinction between defect areas and backgrounds, and suppress reflective artifacts; a defect segmentation model is constructed based on feature entropy and local gradient information, which can solve the problem of low segmentation accuracy of blurred defects and overlapping defects in traditional single detection, and improve the accuracy of defect classification; finally, a pixel-level probability map is generated through the defect probability model, and the defect area intensity is dynamically adjusted in combination with the fine segmentation results. Among them, pixel-level weights are generated through the probability map and the segmentation results to balance the brightness consistency of defects and backgrounds and avoid detail loss caused by over-correction.

[0017] In summary, the technical solution adopted in this application can achieve optimized detection of fine defects on the surface of optoelectronic products, so as to improve the reliability of the inspection results of optoelectronic products. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 is a flow chart of a photoelectric product detection method based on image processing provided in the present application; Figure 2 is an exemplary flow chart for determining a complex defect inspection sample provided by the present application; Figure 3 is an exemplary flow chart for determining defect areas provided by the present application; Figure 4 It is a module structure diagram of the optoelectronic product detection system provided in this application. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] The embodiment of the present application provides an optoelectronic product detection method and system based on image processing, the core of which is to determine the light source mode during the acquisition by collecting multimodal images of the surface of the optoelectronic product; adaptively and dynamically scale the multimodal images to obtain complex defect inspection samples, and extract multiscale feature vectors from the complex defect inspection samples; construct a fused scale feature map of the optoelectronic product based on the multiscale feature vector and the light source mode, perform preliminary defect detection on the fused scale feature map through a boundary recognition box optimized by a preset coordinate attention mechanism, and obtain a defect area; construct a defect segmentation model based on the feature entropy and local gradient information of the defect area, and then perform secondary segmentation and defect type discrimination on the defect area through the defect segmentation model to obtain a fine segmentation result; extract a defect probability map of the defect area, and perform pixel-level intensity adjustment on the defect area based on the defect probability map and the fine segmentation result to obtain a defect detection result of the optoelectronic product.

[0022] Embodiment 1: In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1As shown in FIG. 1 , this figure is an exemplary flow chart of a photoelectric product detection method based on image processing according to this embodiment of the present application, and the photoelectric product detection method includes the following steps: In step S1, a multimodal image of the surface of the optoelectronic product is collected, and a light source mode during the collection is determined.

[0023] In specific implementation, multimodal images of the surface of optoelectronic products are synchronously collected through a multimodal image acquisition device within a specified time period, and the multimodal images include visible light images, infrared images, and depth images, etc., wherein the multimodal image acquisition device continuously captures image frames obtained by different sensors within a collection window of 3-5 seconds, and each frame records detailed information of the surface of the optoelectronic products under their respective spectra, such as color texture, thermal distribution characteristics, and three-dimensional structure data, providing rich and multi-angle information input for subsequent image adaptive processing and defect detection.

[0024] It should be noted that the multimodal images in this application refer to different modal image data synchronously collected by multiple image sensors in the same time period, including: visible light images, which can provide color texture and detail information of the product surface, infrared images, which can reflect the temperature distribution and thermal radiation characteristics of the product surface, and depth images, which can record the three-dimensional structure and shape changes of the product surface. The multimodal images capture different physical properties of the surface of optoelectronic products, which complement each other and help improve the accuracy of overall detection. In addition, the light source mode refers to the lighting conditions and parameter settings used in the image acquisition process, including the color, brightness, and illumination angle of the light source. These parameters directly affect the acquisition effect and signal-to-noise ratio of different modal images. By analyzing the characteristics of light intensity distribution, edge information, etc. in the acquired image, the system can determine the current light source mode and provide accurate lighting reference for subsequent adaptive dynamic scaling and defect detection, thereby ensuring the effective fusion and utilization of image data of each modality.

[0025] In step S2, the multimodal image is adaptively and dynamically scaled to obtain a complex defect inspection sample, and a multi-scale feature vector is extracted from the complex defect inspection sample.

[0026] Preferably, in this embodiment, reference Figure 2 As shown in FIG. 1 , this figure is an exemplary flow chart of determining a complex defect inspection sample in an embodiment of the present application. In this embodiment, adaptive dynamic scaling of the multimodal image is performed to obtain a complex defect inspection sample, which can be specifically implemented by the following steps: First, in step S21, the multimodal image is subjected to preliminary scaling processing by a preset dynamic scaling strategy to obtain a preliminary scaled image sequence; Then, in step S22, a local scaling factor is generated according to the brightness, contrast and texture statistical characteristics of the local area in the preliminary scaled image sequence; Finally, in step S23, the local scaling factor is used to perform dynamic scaling processing on the preliminary scaled image sequence to obtain a complex defect inspection sample.

[0027] In the specific implementation, first, a preset dynamic scaling strategy is used to perform preliminary scaling processing on the multimodal image, that is, the preset dynamic scaling strategy refers to combining image normalization with automatic exposure correction technology to balance the overall brightness and color saturation of each modal image (such as visible light, infrared, and depth map). The cv2.normalize function in OpenCV can be used to normalize the multimodal image, and the brightness distribution of each image can be unified in combination with the automatic exposure adjustment algorithm. The results obtained after normalization and pre-scaling are a preliminary scaled image sequence, which ensures that the subsequent local feature extraction has a unified basis; then, each preliminary scaled image in the preliminary scaled image sequence is divided into regions using a 16×16 grid to obtain multiple local regions. For each local region, the grayscale histogram of the region is calculated using the calcHist function of OpenCV, and the Sobel operator (such as cv 2. Sobel) calculates the local edge gradient, and uses the standard deviation of the grayscale value in the calculated area as the regional contrast index. When the regional contrast index is ≥0.6, the local scaling factor is set to a lower value (e.g., 1.0 to 1.1), indicating that the details of the area are relatively sufficient; when the contrast index is between 0.3 and 0.6, a medium scaling factor is set (e.g., 1.2 to 1.4); when the contrast index is <0.3, a higher scaling factor is set (e.g., 1.5 to 1.8) to magnify the area with insufficient details. In other embodiments, the local scaling factor can also be set according to actual inspection requirements, which is not limited here; finally, for each local area, the cv2.resize function of OpenCV is used to adjust the size of the local area according to the corresponding local scaling factor, and then the local area is recombined into a complete image through the image stitching function of OpenCV, so that the combined image is used as a complex defect inspection sample.

[0028] It should be noted that the multimodal image includes visible light images, infrared images, depth images, etc., and each mode reflects the different physical properties of the product surface, such as color texture, heat distribution and three-dimensional structure; the generation of the local scaling factor is based on the brightness, contrast and texture statistical characteristics of the local area, which can ensure that the defect area can be magnified in detail during the image adaptive scaling process, thereby helping to reduce the risk of missed detection and improve detection accuracy; dynamic scaling processing adaptively adjusts the scaling ratio to achieve a more obvious magnification effect for high-detail areas (such as tiny scratches or cracks), while maintaining the overall visual continuity of low-detail areas, providing high-quality samples for subsequent multi-scale feature vector extraction and defect detection.

[0029] In this embodiment, the multi-scale feature vector can be extracted from the complex defect inspection sample in the following manner, namely: A multi-scale convolutional network is used to extract features of the complex defect inspection sample to generate a plurality of local feature maps at different resolutions; All local feature maps are weightedly fused to obtain a global feature map, and then a multi-scale feature vector is extracted from the global feature map.

[0030] In the specific implementation, first, by designing multiple convolutional layers, each layer uses convolution kernels of different sizes (such as 3×3, 5×5, 7×7, etc.) to capture the detailed information of local areas in the image at different resolutions, thereby generating multiple local feature maps. This process can be analyzed using the Pandas library and implemented based on the OpenCV library. The feature maps output by each convolutional layer can reflect the defect characteristics of different scales in the sample; secondly, all the generated local feature maps are weighted fused. The normalized features in the feature maps of each scale can be integrated by pixel-by-pixel weighted summation, and the mean square error and structural similarity are calculated and weighted fused to obtain a comprehensive global feature map; finally, the global feature map can be processed using principal component analysis (PCA) to extract multi-scale feature vectors that contain both fine-grained local information and global description capabilities, providing accurate feature basis for subsequent detection modules.

[0031] It should be noted that, in this embodiment, through the design of a multi-scale convolutional network and a weighted fusion strategy, the comprehensive utilization of multi-level and multi-resolution information in complex defect inspection samples is realized, and the environmental interference factors and real defect characteristics are effectively separated, thereby improving the accuracy and robustness of defect detection, and providing reliable data support for subsequent defect location and type discrimination.

[0032] In step S3, a fused scale feature map of the optoelectronic product is constructed based on the multi-scale feature vector and the light source mode, and preliminary defect detection is performed on the fused scale feature map through a boundary recognition box optimized by a preset coordinate attention mechanism to obtain a defect area.

[0033] In this embodiment, the fusion scale feature map of the optoelectronic product is constructed based on the multi-scale feature vector and the light source mode in the following manner, namely: Performing weighted fusion processing on the multi-scale feature vector and the light source pattern information to obtain a weighted feature map; The weighted feature maps are hierarchically fused using a multi-scale feature vector fusion network with bidirectional cross-scale connections to generate a preliminary fused scale feature map. A deformable attention mechanism is used to screen and optimize key features of the preliminary fused scale feature map to obtain a fused scale feature map.

[0034] In the specific implementation, firstly, the light source mode parameters (such as light source intensity, wavelength, irradiation angle, etc.) can be encoded as feature vectors and spliced ​​with multi-scale feature vectors. The learnable weight matrix is ​​introduced through machine learning technology, and the spliced ​​feature vectors are weighted and summed element by element to generate a weighted feature map. By fusing the ambient light information and defect features, the subsequent processing can automatically compensate for the feature differences caused by light changes, thereby improving the overall detection robustness. Secondly, according to the structure of the bidirectional feature pyramid network, a top-down and bottom-up bidirectional transmission path is constructed to hierarchically fuse feature maps of different resolutions. The features of each layer are normalized and weighted fused to obtain a preliminary fused scale feature map, which is conducive to enhancing the comprehensiveness and consistency of feature expression. Finally, the offset is determined by the autoregressive analysis algorithm, and the sampling points in the feature map are dynamically adjusted according to the offset, so that the attention module can adaptively focus on the key structure of the defect area, and perform element-by-element multiplication of the key structure of the defect area with the preliminary fused scale feature map (which can be achieved through the OpenCV library) to obtain a fused scale feature map, which can enhance the expression of important defect features, reduce environmental interference, and further improve the accuracy and stability of detection.

[0035] It should be noted that, in this embodiment, by weighted fusion of multi-scale feature vectors and light source pattern information, and using a bidirectional cross-scale connection module to achieve hierarchical feature fusion, and finally using a deformable attention mechanism to optimize key features, it effectively achieves a multi-level comprehensive expression of complex defect information, providing accurate and robust feature support for optoelectronic product inspection systems, thereby significantly improving detection accuracy.

[0036] Preferably, in this embodiment, reference Figure 3 As shown in FIG. 1 , this figure is an exemplary flow chart for determining a defect area in an embodiment of the present application. In this embodiment, a boundary recognition frame optimized by a preset coordinate attention mechanism is used to perform preliminary defect detection on the fused scale feature map, and the defect area can be obtained by the following steps: First, in step S31, the fused scale feature map is input into a preset boundary recognition frame optimized by the coordinate attention mechanism to obtain a fused scale feature map after attention optimization; Then, in step S32, the attention focus area in the attention optimized fusion scale feature map is segmented to obtain a potential defect area; Finally, in step S33, the fused image of the potential defect area is extracted, and then a morphological operation is performed on the fused image to obtain the defect area.

[0037] It should be noted that the preset coordinate attention mechanism is a coordinate attention mechanism that integrates average pooling and maximum pooling. By integrating global statistics and local significant information, it can more accurately focus on the key features of the defect area and provide efficient feature support for preliminary defect detection.

[0038] In the specific implementation, first, average pooling is performed on each channel of the fused scale feature map to extract global statistical information, and maximum pooling is used for the same feature map to capture local significant information. The results of the two poolings are then spliced, and feature fusion is achieved through 1×1 convolution mapping to generate an attention weight map. The attention weight map is then multiplied pixel by pixel with the original fused scale feature map to obtain the attention-optimized fused scale feature map, which can effectively suppress redundant background information and provide a more accurate data basis for subsequent region proposals. Then, the high response values ​​of the attention-optimized fused scale feature map are threshold segmented to obtain the attention-focused area, and the attention-focused area is integrated using a clustering algorithm to obtain the potential defect area. Finally, operations such as corrosion and dilation can be used to remove noise, smooth the boundaries of the candidate regions, and extract accurate defect boundaries through contour detection, thereby outputting the defect area after morphological optimization processing, so that the detection results have high boundary accuracy and regional coherence.

[0039] In step S4, a defect segmentation model is constructed according to the characteristic entropy and local gradient information of the defect area, and then the defect area is secondary segmented and the defect type is distinguished by the defect segmentation model to obtain a fine segmentation result.

[0040] It should be noted that the defect segmentation model in this embodiment is a multi-scale feature vector fusion model based on convolutional neural network. The improved semantic segmentation network can fully extract the detailed information of the defect area, and quantitatively distinguish the defect boundaries and shape features through linear discriminant analysis, which can effectively separate environmental interference from real defect features, thereby realizing refined segmentation of defect areas and defect type discrimination, providing reliable and accurate defect segmentation data for subsequent optoelectronic product inspection.

[0041] In the specific implementation, first, the Sobel operator is used to calculate the gradient of the defect area, and the local gradient map obtained can capture the edge changes and texture details in the defect area, providing significant features for subsequent segmentation. The feature entropy reflects the randomness and information complexity of the texture in the area, and can effectively distinguish environmental noise from real defect features. The information entropy formula can be used to calculate the histogram of the local gradient amplitude and output an entropy value map. The entropy value of each pixel in the entropy value map can represent the uncertainty of local information; then, the convolutional neural network based on the U-Net architecture is improved through channel splicing, so that the improved convolutional neural network is used as a defect segmentation model, wherein the fusion features are extracted through the multi-scale convolution layer of the improved convolutional neural network. Through the fused feature map, the model can more comprehensively capture the edge, texture and structural information of the defect area, thereby realizing the secondary segmentation of the defect area and the defect type discrimination.

[0042] It should be noted that this embodiment can effectively separate environmental interference factors from real defect features by inputting local gradient information and feature entropy as key auxiliary features into the defect segmentation model, thereby improving the accuracy and robustness of defect segmentation, and providing an accurate correction basis for subsequent defect location and type discrimination.

[0043] In this embodiment, the defect area is segmented twice and the defect type is identified by the defect segmentation model to obtain a fine segmentation result, which can be specifically obtained in the following manner, namely: Using an improved semantic segmentation network to perform fine segmentation on the defect area to obtain defect boundary features and defect shape features; Linear discriminant analysis is performed on the defect boundary features and defect shape features to obtain a fine segmentation result.

[0044] In the specific implementation, the defect area is input into an improved semantic segmentation network, which adopts a deep convolutional architecture such as U-Net and is optimized to improve the ability to capture local details. The boundary features and shape features in the defect area are extracted at multiple scales through the improved semantic segmentation network to generate a high-resolution defect boundary map and a defect shape map. Then, the defect boundary features are extracted from the defect boundary map through an existing feature extraction model, and the defect shape features are extracted from the defect shape map through the same existing feature extraction model. Then, the extracted defect boundary features and defect shape features are reduced in dimension using linear discriminant analysis, and then the discriminant indicators (mean square error and structural similarity) between different defect types are calculated to finely classify each defect area to obtain a fine segmentation result.

[0045] It should be noted that the improved semantic segmentation network in this embodiment is obtained through multi-scale feature extraction and fine-grained structure optimization, and can accurately capture the boundaries and shapes of defect areas; through linear discriminant analysis, it can utilize feature dimensionality reduction and classification capabilities to efficiently integrate defect boundary features and shape features, thereby providing an accurate basis for defect type discrimination.

[0046] In step S5, a defect probability map of the defect area is extracted, and pixel-level intensity adjustment is performed on the defect area based on the defect probability map and the fine segmentation result to obtain a defect detection result of the optoelectronic product.

[0047] In this embodiment, the defect probability map of the defect area may be extracted in the following manner: Performing feature analysis on the defect area to obtain multiple defect feature values; The defect probability map is generated based on all defect feature values ​​through the pre-trained defect probability model.

[0048] In a specific implementation, the defect area is preprocessed by an edge detection algorithm, and a convolutional neural network is used to perform statistical analysis on various features in the area, such as color, texture, shape, and brightness, so as to obtain a plurality of defect feature values; then, the defect feature values ​​are formed into a defect feature value set, and a comprehensive evaluation is performed on the defect feature value set through a Bayesian network, and then the defect probability value of each pixel is output to obtain a defect probability map.

[0049] It should be noted that pixel-level intensity adjustment based on the defect probability map and the fine segmentation result can not only accurately highlight the defect area and reduce background noise interference, but also effectively balance local details and overall brightness, thereby significantly improving the reliability and detection accuracy of optoelectronic product inspection results; in specific implementation, the defect probability map can be used as a weight map and multiplied pixel by pixel with the original pixel intensity of the area corresponding to the fine segmentation result to achieve local enhancement or suppression. At the same time, the processed image is further optimized by an adaptive smoothing filtering algorithm to eliminate noise in the inspection image, and finally the optoelectronic product defect detection result after pixel-level intensity adjustment is output, and the optoelectronic product defect detection result is visualized through the visualization library in OpenCV.

[0050] It can be seen that in this application, optimized detection of fine defects on the surface of optoelectronic products can be achieved; first, through the synchronous acquisition of multimodal images, combined with the intelligent switching of dynamic light source modes, it is possible to solve the problem of feature loss caused by low defect contrast and reflective interference under a single light source. Among them, different light source modes can highlight the differentiated optical responses of defects such as cracks, bubbles, scratches, etc., and enhance the comprehensiveness of defect feature expression; secondly, through dynamic scaling strategy and local scaling factor generation, combined with the hierarchical feature fusion of multi-scale convolutional networks, the problems of insufficient resolution of tiny defects and lack of contextual information of large-size defects caused by traditional fixed scaling are solved. The scaling factor is generated based on local brightness, contrast and texture features, which can enhance the resolution of the defect area and avoid edge blurring caused by global scaling. The global feature map is generated through weighted fusion to ensure that micron-level cracks and millimeter-level The synchronous capture of scratch features is conducive to improving the detection rate of complex defects; then, the multi-scale feature vector is weightedly fused with the light source pattern information, and the coordinate attention mechanism is used to optimize the boundary recognition box to solve the problems of weak correlation between features and light sources and inaccurate boundary positioning in traditional methods. The weighted fusion of light source patterns and multi-scale feature vectors can enhance the distinction between defect areas and backgrounds, and suppress reflective artifacts; a defect segmentation model is constructed based on feature entropy and local gradient information, which can solve the problem of low segmentation accuracy of blurred defects and overlapping defects in traditional single detection, and improve the accuracy of defect classification; finally, a pixel-level probability map is generated through the defect probability model, and the defect area intensity is dynamically adjusted in combination with the fine segmentation results. Among them, pixel-level weights are generated through the probability map and the segmentation results to balance the brightness consistency of defects and backgrounds and avoid detail loss caused by over-correction.

[0051] In summary, the technical solution adopted in this application can achieve optimized detection of fine defects on the surface of optoelectronic products, so as to improve the reliability of the inspection results of optoelectronic products.

[0052] Embodiment 2: This application provides a photoelectric product detection system based on image processing, referring to Figure 4 As shown, this figure is a module structure diagram of the optoelectronic product detection system shown in this embodiment of the present application, and the optoelectronic product detection system includes: The image acquisition module 100 is used to acquire multimodal images of the surface of the optoelectronic product and determine the light source mode during acquisition; A preprocessing module 200 is used to adaptively and dynamically scale the multimodal image to obtain a complex defect inspection sample, and extract a multi-scale feature vector from the complex defect inspection sample; A defect detection module 300 is used to construct a fused scale feature map of the optoelectronic product based on the multi-scale feature vector and the light source mode, and perform preliminary defect detection on the fused scale feature map through a boundary recognition frame optimized by a preset coordinate attention mechanism to obtain a defect area; The defect segmentation module 400 is used to construct a defect segmentation model according to the characteristic entropy and local gradient information of the defect area, and then perform secondary segmentation and defect type discrimination on the defect area through the defect segmentation model to obtain a fine segmentation result; The optimization detection module 500 is used to extract a defect probability map of the defect area, and perform pixel-level intensity adjustment on the defect area based on the defect probability map and the fine segmentation result to obtain a defect detection result of the optoelectronic product.

[0053] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0054] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, the storage medium including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically-erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0055] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. A photoelectric product detection method based on image processing, characterized in that: The photoelectric product detection method comprises the following steps: Collect multimodal images of the surface of optoelectronic products and determine the light source mode during acquisition; Adaptively and dynamically scaling the multimodal image to obtain a complex defect inspection sample, and extracting a multi-scale feature vector from the complex defect inspection sample; Based on the multi-scale feature vector and the light source mode, a fused scale feature map of the optoelectronic product is constructed, and preliminary defect detection is performed on the fused scale feature map through a boundary recognition frame optimized by a preset coordinate attention mechanism to obtain a defect area; A defect segmentation model is constructed according to the characteristic entropy and local gradient information of the defect area, and then the defect area is secondary segmented and the defect type is distinguished by the defect segmentation model to obtain a fine segmentation result; A defect probability map of the defect area is extracted, and pixel-level intensity adjustment is performed on the defect area based on the defect probability map and the fine segmentation result to obtain a defect detection result of the optoelectronic product.

2. The method for detecting optoelectronic products based on image processing according to claim 1, characterized in that: Multimodal images of the surface of optoelectronic products are collected by multimodal image acquisition equipment.

3. The method for detecting optoelectronic products based on image processing according to claim 1, characterized in that: Adaptively and dynamically scaling the multimodal image to obtain a complex defect inspection sample specifically includes: Performing preliminary scaling processing on the multimodal image by using a preset dynamic scaling strategy to obtain a preliminary scaled image sequence; generating a local scaling factor according to brightness, contrast and texture statistical characteristics of a local area in the preliminary scaled image sequence; The local scaling factor is used to dynamically scale the preliminary scaled image sequence to obtain a complex defect inspection sample.

4. The method for detecting optoelectronic products based on image processing according to claim 1, characterized in that: Extracting a multi-scale feature vector from the complex defect inspection sample specifically includes: A multi-scale convolutional network is used to extract features of the complex defect inspection sample to generate a plurality of local feature maps at different resolutions; All local feature maps are weightedly fused to obtain a global feature map, and then a multi-scale feature vector is extracted from the global feature map.

5. The method for detecting optoelectronic products based on image processing according to claim 1, characterized in that: Constructing a fusion scale feature map of a photoelectric product based on the multi-scale feature vector and the light source mode specifically includes: Performing weighted fusion processing on the multi-scale feature vector and the light source pattern information to obtain a weighted feature map; The weighted feature maps are hierarchically fused using a multi-scale feature vector fusion network with bidirectional cross-scale connections to generate a preliminary fused scale feature map. A deformable attention mechanism is used to screen and optimize key features of the preliminary fused scale feature map to obtain a fused scale feature map.

6. The method for detecting optoelectronic products based on image processing according to claim 1, characterized in that: The preset coordinate attention mechanism is a coordinate attention mechanism that integrates average pooling and maximum pooling.

7. The method for detecting optoelectronic products based on image processing according to claim 1, characterized in that: The defect segmentation model is a multi-scale feature vector fusion model based on convolutional neural network.

8. The method for detecting optoelectronic products based on image processing according to claim 1, characterized in that: The defect area is segmented twice and the defect type is identified by the defect segmentation model, and the fine segmentation result obtained specifically includes: Using an improved semantic segmentation network to perform fine segmentation on the defect area to obtain defect boundary features and defect shape features; Linear discriminant analysis is performed on the defect boundary features and defect shape features to obtain a fine segmentation result.

9. The method for detecting optoelectronic products based on image processing according to claim 1, characterized in that: Extracting the defect probability map of the defect area specifically includes: Performing feature analysis on the defect area to obtain multiple defect feature values; The defect probability map is generated based on all defect feature values ​​through the pre-trained defect probability model.

10. An optoelectronic product inspection system based on image processing, used to execute an optoelectronic product inspection method based on image processing as claimed in any one of claims 1 to 9, characterized in that: The optoelectronic product detection system comprises: An image acquisition module is used to acquire multimodal images of the surface of optoelectronic products and determine the light source mode during acquisition; A preprocessing module, used for adaptively and dynamically scaling the multimodal image to obtain a complex defect inspection sample, and extracting a multi-scale feature vector from the complex defect inspection sample; A defect detection module, used to construct a fused scale feature map of the optoelectronic product based on the multi-scale feature vector and the light source mode, and perform preliminary defect detection on the fused scale feature map through a boundary recognition frame optimized by a preset coordinate attention mechanism to obtain a defect area; A defect segmentation module is used to construct a defect segmentation model according to the characteristic entropy and local gradient information of the defect area, and then perform secondary segmentation and defect type discrimination on the defect area through the defect segmentation model to obtain a fine segmentation result; The optimization detection module is used to extract a defect probability map of the defect area, and adjust the pixel-level intensity of the defect area based on the defect probability map and the fine segmentation result to obtain a defect detection result of the optoelectronic product.

Images