Multi-mode DBC substrate surface defect detection method, device and system based on deep learning, medium and program product

Abstract

The invention provides a multi-mode DBC substrate surface defect detection method, device and system based on deep learning, a medium and a program product, and aims to perform comprehensive defect imaging by using diversified light source and spectrum technologies so as to highlight defect characteristics and effectively prevent missing of defect detection; by adopting a non-fixed multi-scale image cutting method, omission of small-size defects is prevented, so that the detection precision and accuracy of small target defects are improved. Through verification, the multi-mode DBC substrate surface defect detection technology based on deep learning provided by the invention can efficiently and accurately identify and detect various defect types.

Description

Technical Field

[0001] The present application relates to the technical field of DBC product defect detection, and in particular to a multimodal DBC substrate surface defect detection method, device, system, medium and program product based on deep learning. Background Art

[0002] DBC (Direct Bonding Copper), or direct copper-clad ceramic substrate, is a new composite material that covers a highly insulating aluminum oxide or aluminum nitride ceramic substrate with copper metal. This material is widely used in high-power semiconductor packaging modules, high-temperature gaskets, and refrigerators. However, during the manufacturing process, DBC ceramic substrates may produce some defects that affect product performance, such as bubbles, oxidation, discoloration, cracks, edge collapse, copper deficiency, etc. Therefore, it is very important to detect surface defects on DBC ceramic substrates.

[0003] At present, non-contact inspection of DBC ceramic substrates is mainly based on machine vision technology, using high-resolution cameras and high-magnification lenses to image DBC ceramic substrates, pre-processing images and extracting and locating features in defective areas through image processing algorithms, and classifying them according to the morphology of features. For some special defects, such as internal bubbles and voids, the interior can be scanned by ultrasonic scanning microscope detection technology or X-rays, and imaging can be performed through the sound wave reflection of internal defects. These technologies have their own characteristics, and it is usually necessary to select the most appropriate solution based on the actual application scenario.

[0004] Since the processing of DBC ceramic substrates is complex and delicate, it involves many key steps and process control links. Its surface will have problems such as complex background, inconsistent texture, brightness changes, etc. in machine vision imaging, and the types of defect features vary, which brings great challenges to the feature extraction of defects. Traditional image processing methods are currently unable to accurately identify the above defect features. In recent years, the powerful ability of deep learning technology in complex environments has attracted more and more attention, and its technology has developed very rapidly. Therefore, the method based on deep learning has gradually become the mainstream method for detecting such products, and its detection effect is inferior to that of traditional image processing algorithms.

[0005] At present, deep learning methods have been used for DBC substrate detection, but existing imaging solutions often choose a single type of monochromatic light source to image the DBC ceramic substrate, which leads to the inability to be well compatible with defects of various shapes and color differences. Summary of the invention

[0006] In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a technical solution for solving the defect that the existing detection scheme is not well compatible with various forms and color differences.

[0007] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides a multimodal DBC substrate surface defect detection method based on deep learning, comprising: collecting multimodal images of DBC ceramic substrates through a light source combination; stitching the collected multimodal images to form a whole image of DBC ceramic substrates based on image stitching technology; annotating the whole image of DBC ceramic substrates; performing multi-scale cropping on the whole image of DBC ceramic substrates; preprocessing the annotated and multi-scale cropped images of each modality to obtain a trainable image data set, wherein the number of channels of the image in the image data set is N times the number of channels of the original image, and N is the number of modal categories; modifying the convolution size of the input layer of the network model according to the number of channels of the image in the image data set, and using the image data set to train the modified network model to obtain a surface defect detection model suitable for DBC ceramic substrates; using the ONNX framework to deploy the trained surface defect detection model to a target device; and using the target device deployed with the surface defect detection model to perform real-time online detection on the DBC ceramic substrate to be detected.

[0008] In some embodiments of the first aspect of the present application, the acquiring of multimodal images of the DBC ceramic substrate by using a light source combination includes: acquiring the multimodal images of the DBC ceramic substrate by using a combined light source of a white coaxial light source and a multi-angle multi-spectral ring light source.

[0009] In some embodiments of the first aspect of the present application, the image stitching technology is used to stitch the collected multimodal images to form a complete image of the DBC ceramic substrate, which includes: performing image preprocessing on the multimodal images; using a feature algorithm to detect feature points in the multimodal images, and matching feature points in different images accordingly; calculating the transformation structure between images through matched feature points to perform posture estimation, and transforming the image according to the estimated posture to align it with a reference image; fusing the transformed image with the reference image to obtain a complete image of the DBC ceramic substrate.

[0010] In some embodiments of the first aspect of the present application, during the image fusion process, weighted averaging is performed on the image pixels in the overlapping area to eliminate the stitching seams generated in the overlapping area during the image stitching process.

[0011] In some embodiments of the first aspect of the present application, the multi-scale cropping of the entire image of the DBC ceramic substrate includes: cropping the original multimodal image according to the following multiple sizes: W / 2*H / 2, W*H, 2W*2H, 3W*3H, W represents the width of the original multimodal image, and H represents the height of the original multimodal image; wherein the cropped image of the W / 2*H / 2 scale is upsampled before being sent to the network model.

[0012] In some embodiments of the first aspect of the present application, the number of channels of the image in the image data set is N times the number of channels of the original image, where N is the number of modal categories, including: for the N multimodal images collected, each multimodal image contains three channels of red, green, and blue, and the N modal images are stitched on the channels in sequence to obtain a single image with a channel number of (3*N).

[0013] In some embodiments of the first aspect of the present application, the target device deployed with a surface defect detection model is used to perform real-time online detection of the DBC ceramic substrate to be inspected, and the detection results include any one or more of the following: ceramic oxidation, foreign matter, ceramic discoloration, ceramic cracks, scratches, edge collapse, bumps, copper defects, and bubbles.

[0014] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides a multimodal DBC substrate surface defect detection system based on deep learning, including: an image acquisition module, used to acquire multimodal images of DBC ceramic substrates through a light source combination; a defect detection module, which is communicatively connected to the image acquisition module; after receiving the multimodal image, the defect detection module executes the multimodal DBC substrate surface defect detection method based on deep learning.

[0015] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal DBC substrate surface defect detection method based on deep learning.

[0016] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer implements the deep learning-based multimodal DBC substrate surface defect detection method.

[0017] To achieve the above-mentioned purpose and other related purposes, the fifth aspect of the present application provides a computer device, including a memory, a processor and a computer program stored in the memory; the processor executes the computer program to implement the deep learning-based multimodal DBC substrate surface defect detection method.

[0018] As described above, the multimodal DBC substrate surface defect detection method, device, system, medium and program product based on deep learning of the present application have the following beneficial effects: the present application utilizes a variety of light sources and spectral technologies to perform comprehensive defect imaging to highlight defect characteristics and effectively prevent the omission of defect detection; by adopting a non-fixed multi-scale image cropping method, the omission of small-size defects is prevented, thereby improving the detection precision and accuracy of small target defects; it has been verified that the multimodal DBC substrate surface defect detection technology based on deep learning proposed in the present application can efficiently and accurately identify and detect the various defect types mentioned above. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Shown is a schematic diagram of an application scenario of multi-modal DBC substrate surface defect detection based on deep learning in one embodiment of the present application.

[0020] Figure 2 Shown is a flow chart of a multimodal DBC substrate surface defect detection method based on deep learning in one embodiment of the present application.

[0021] Figure 3 Shown is a schematic diagram of four different modality images captured using four different light source combinations in one embodiment of the present application.

[0022] Figure 4 Shown is a schematic diagram of the image stitching process in one embodiment of the present application.

[0023] Figure 5 Shown is a schematic diagram of the DBC substrate surface defect detection results in one embodiment of the present application.

[0024] Figure 6 Shown is a structural schematic diagram of a multimodal DBC substrate surface defect detection system based on deep learning in one embodiment of the present application.

[0025] Figure 7 Shown is a schematic block diagram of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0026] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0027] Before further describing the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are applicable to the following interpretations:

[0028] DBC (Direct Bonding Copper), a direct copper-clad ceramic substrate, is a new type of composite material that covers a highly insulating aluminum oxide or aluminum nitride ceramic substrate with copper metal. DBC substrates are widely used in high-power semiconductor packaging modules, high-temperature gaskets, and refrigerators due to their excellent thermal conductivity, insulation, and mechanical properties.

[0029] Image annotation is a task in the field of computer vision that involves identifying and marking specific objects, scenes, or areas in an image. The annotation process requires adding labels to the image manually or automatically so that the machine learning model can recognize and learn the features in the image.

[0030] Image cropping is the process of selecting an area from the original image and cutting it out to create a new, smaller image. The purpose of cropping is usually to remove unnecessary background, highlight a specific part of the image, or adjust the composition of the image. In computer vision and image processing, cropping can also be used to prepare image data for further analysis or as input to machine learning models. Cropping can be fixed in size and ratio, or dynamic, adjusting based on the image content and analysis requirements.

[0031] YOLOv8 (You Only Look Once version 8), a real-time object detection model based on deep learning, is the latest version of the YOLO series. It can quickly and accurately identify multiple objects in an image through a single-stage detection method. The features of YOLOv8 include efficient feature extraction and prediction mechanisms, support for multiple input sizes, and a good balance between speed and accuracy.

[0032] Faster R-CNN is a classic two-stage object detection model that combines the Region Proposal Network (RPN) and Fast R-CNN. It generates candidate regions through RPN, and then classifies and regresses these regions. Faster R-CNN has significantly improved speed and accuracy compared to previous R-CNN and Fast R-CNN.

[0033] DETR (Detection Transformer) is a Transformer-based end-to-end object detection model proposed by Facebook in 2020. It simplifies the traditional object detection process by converting the object detection task into a set prediction problem, avoiding the use of predefined anchor boxes and post-processing steps (such as non-maximum suppression NMS).

[0034] Figure 1 A schematic diagram of an application scenario of multimodal DBC substrate surface defect detection based on deep learning in an embodiment of the present invention is shown. This application scenario mainly includes a multimodal image acquisition device 1, a computer device 2 and a DBC ceramic substrate 3 to be tested.

[0035] The image acquisition device 1 can be a camera, such as a color camera with a high-magnification and high-precision lens, which can collect multiple images of different modes using a combination of multiple light sources for the DBC ceramic substrate 3 to be tested. Subsequently, the image acquisition device 1 sends the collected images to the computer device 1, which splices these images to form a complete image of the DBC ceramic substrate, and performs tasks such as labeling, cropping, and model training on the complete image of the DBC ceramic substrate. Finally, the trained model is deployed in an industrial computer for real-time online detection, thereby realizing multi-modal basic defect detection of DBC ceramic substrates based on deep learning, with high detection efficiency and high accuracy.

[0036] The above is a certain explanation of the application scenario of one embodiment of the present invention. Hereinafter, the implementation process and principle of the embodiment of the present invention will be further described in conjunction with the accompanying drawings.

[0037] Figure 2 A flowchart of a multi-modal DBC substrate surface defect detection method based on deep learning in an embodiment of the present invention is shown, including the following steps:

[0038] Step S21: collecting a multimodal image of the DBC ceramic substrate by combining light sources.

[0039] In this embodiment, a camera is used to collect multimodal images of the DBC ceramic substrate to be inspected. The camera is preferably a color camera integrated with a high-power and high-precision lens, and the light source is a combination of a white coaxial light source and a multi-angle multi-spectrum annular light source.

[0040] It should be understood that the color camera can capture rich color information, which is crucial for application scenarios such as substrate surface defect detection that require color recognition and analysis. High-power lenses can provide more detailed image details, which is very important for the detection and analysis of tiny defects on the surface of DBC boards. In order to be able to shoot a complete DBC ceramic substrate, the prior art selects a low-power lens. This solution will reduce the detection accuracy and cannot detect subtle defects. High-precision lenses ensure the clarity and resolution of the image, making image analysis more accurate. The white coaxial light source can provide uniform illumination, reduce shadows and reflections, make the object features in the image clearer, and reduce the interference caused by the reflection of the substrate surface, which is suitable for capturing the overall appearance and surface flatness of the substrate. The multi-angle multi-spectrum ring light source can provide illumination from different angles, which helps to highlight the three-dimensional features and surface defects of the substrate, such as cracks, depressions and protrusions. It should also be noted here that, unlike the combined light source used in the embodiment of the present application, although a single light source also has the function of providing illumination, it cannot provide sufficient illumination uniformity, which will cause shadows and reflection problems in the image. Similarly, although non-coaxial light sources can also provide light, they cannot provide light coaxial with the lens, which will affect the clarity and accuracy of the image.

[0041] For example, Figure 3 A schematic diagram showing four different modal images collected using four different light source combinations according to an embodiment of the present application is shown. The four different light source combinations are, for example, a white light coaxial light source and a multi-angle multi-spectrum ring light source. These light source combinations can be adjusted according to the actual detection requirements of the DBC ceramic substrate to obtain the required multimodal data. It should be understood that the four different light source combinations referred to here do not specifically refer to four different types of light sources, but rather different imaging effects on the same product produced by a combination of white coaxial light sources and multi-angle multi-spectrum ring light sources with different brightness, different colors, and different angles. In actual applications, the selection and combination of light sources should be based on the specific characteristics and requirements of the DBC ceramic substrate to be detected, as well as the required image modality type. By optimizing the light source combination, the accuracy and efficiency of detection can be improved, so Figure 3 This is only an example and is not intended to limit the scope of application of the technical solution of the present invention.

[0042] Step S22: stitching the collected multimodal images to form a complete image of the DBC ceramic substrate based on image stitching technology.

[0043] Image stitching is a technology that combines multiple images into a wider viewing angle or higher resolution image. In this embodiment, the collected multi-modal images are stitched together based on the image stitching technology to form a complete image of the DBC ceramic substrate.

[0044] Figure 4The following is a schematic diagram of the process of image stitching in an embodiment of the present application:

[0045] Step S22a: performing image preprocessing on the multimodal image.

[0046] The purpose of image preprocessing is to improve image quality and prepare for subsequent feature detection and matching. Image preprocessing methods include but are not limited to histogram matching, smoothing filtering, and enhancement transformation. Histogram matching refers to adjusting the brightness and contrast of an image to make the lighting conditions of different images more consistent. Smoothing filtering refers to reducing image noise, such as Gaussian filtering and median filtering, to clarify image details. Enhancement transformation refers to enhancing certain features of an image, such as edges and textures, to improve the accuracy of feature detection.

[0047] Step S22b: Use a feature algorithm to detect feature points in the multimodal image, and match feature points in different images accordingly.

[0048] The purpose of feature point detection and matching is to find unique points in an image, which can be identified and matched in different images. Feature point detection methods include but are not limited to using SIFT algorithm, SURF algorithm, ORB algorithm, etc. to detect feature points in an image. These algorithms can detect significant features such as corners and edges in an image. Feature matching methods include but are not limited to using FLANN matching algorithm or other algorithms to perform feature matching and select high-quality matching points.

[0049] It should be understood that SIFT (Scale-Invariant Feature Transform) is an algorithm for detecting and describing local features, which can extract salient features in images under different scales, orientations, and lighting conditions. It first identifies scale-invariant feature points in the image, and then generates a unique descriptor for each key point, which can be used to match feature points of the same scene in different images. SURF (Speeded Up Robust Features) is an accelerated version of the SIFT algorithm, which is optimized for computational efficiency while maintaining invariance to scale and rotation. SURF uses integral images and approximate Hessian matrices to quickly calculate feature points and descriptors, making it more suitable for real-time applications. ORB (Oriented FAST and Rotated BRIEF) is a combination of Oriented FAST (feature point detection) and Rotated BRIEF (feature description). It is a fast feature point detection and description algorithm. ORB not only detects key points, but also assigns directions to each key point and generates rotation-invariant descriptors, which makes it more robust in feature matching. FLANN (Fast Library for Approximate Nearest Neighbors) is a library for fast approximate nearest neighbor search, which is often used to find the nearest neighbors of feature descriptors in large-scale data sets, which is very useful in the feature matching stage. FLANN provides a variety of algorithms, such as KD tree and hierarchical clustering tree, to adapt to different search needs and performance requirements.

[0050] Step S22c: Calculate the transformation structure between the images through the matched feature points to perform pose estimation, and transform the image according to the estimated pose to align it with the reference image.

[0051] The purpose of pose estimation and image transformation is to determine the geometric relationship between images and transform the images so that they can be correctly aligned. The pose estimation method includes but is not limited to calculating the transformation matrix between images, such as the homography matrix, by matching feature points. The image transformation method includes but is not limited to applying the transformation matrix to rotate, scale, translate, etc. the image to align it with the reference image.

[0052] Step S22d: Fusing the transformed image with the reference image to obtain a complete image of the DBC ceramic substrate.

[0053] The purpose of image fusion is to merge the transformed image with the reference image to form a seamless panoramic image. Image fusion methods include but are not limited to multi-band blending, Poisson fusion and other technologies. Multi-band blending refers to the fusion of images in different frequency bands and then recombining them to reduce visual artifacts. Poisson fusion refers to the use of Poisson's equation to fuse images and naturally handle lighting changes and occluded areas.

[0054] Preferably, during the image fusion process, the image pixels in the overlapping area are weighted averaged to eliminate the stitching seams generated in the overlapping area during the image stitching process. It should be understood that in image stitching, when two or more images overlap geographically or visually, these overlapping parts are the overlapping area. In the overlapping area, pixels at the same position may have different intensity values ​​in different images. The weighted average fusion method assigns different weights to these pixels and then calculates the weighted average to generate the pixel value of the position in the final image.

[0055] Specifically, the weighted average fusion process includes the following: First, determine the overlapping area of ​​the two images to be stitched. For each pixel in the overlapping area, calculate its distance to the nearest overlapping area boundary and normalize it. Second, use the normalized distance value as the input of the Sigmoid function to calculate the weight of each pixel. Finally, according to the calculated weight, perform weighted average fusion on the pixel values ​​of the two images in the overlapping area.

[0056] It should be understood that for each pixel in the overlapping area, the distance to the nearest overlapping area boundary is calculated and normalized. This distance can be the vertical distance to the edge or the Euclidean distance. Normalization refers to mapping the calculated distance value to between 0 and 1. This is usually achieved by dividing the distance of each pixel by the maximum distance of the overlapping area. Normalization can ensure that the weight is within a reasonable range for subsequent processing. The Sigmoid function is an activation function that can be applied to machine learning and image processing, which can map the input value to between 0 and 1. The normalized distance value is used as the input of the Sigmoid function, and the calculated value is the weight of each pixel, which reflects the importance of the position of the pixel in the overlapping area.

[0057] The specific calculation formulas for weight calculation, normalization calculation, and weighted fusion are as follows:

[0058]

[0059] f(x,y)=w l *f l (x,y)+w r *f r (x,t); Formula (3)

[0060] Among them, w l 、w r Respectively represent the weights of the overlapping areas of the left and right images; x l 、x r Respectively represent the normalized distances of the left and right boundaries of the overlapping area; f(x,y) represents the pixel value of the fused image at the coordinate (x,y); f l (x,y) represents the pixel value of the left image at the coordinate (x,y); f r (x,y) indicates the coordinates of the image on the right (x ,y)’s pixel value.

[0061] Step S23: marking the entire image of the DBC ceramic substrate.

[0062] The purpose of annotating the entire image is to reduce the workload of manual annotation and improve annotation efficiency and accuracy. Annotation methods include but are not limited to using open source software such as labelImg for annotation. labelImg is a popular open source image annotation tool that supports multiple types of annotations, such as rectangular boxes (for object detection), polygons (for object segmentation), etc. The operational process of using the annotation tool labelImg for annotation includes: Importing the image to be annotated into the labelImg tool. Create corresponding annotations based on the image content, such as drawing rectangular boxes to annotate objects. Adjust the position and size of the annotations to ensure that they accurately cover the objects in the image. Save the annotation results in a specific format, such as XML or JSON, which are usually compatible with the data formats required for machine learning model training.

[0063] It should be understood that high-quality annotated data is the basis for training accurate and efficient machine learning models. Accurate annotations help the model better learn the features and locations of objects in the image. Annotated data is also used to evaluate the performance of the model by comparing the model's predictions with the actual annotations to calculate metrics such as accuracy and recall.

[0064] Step S24: performing multi-scale cropping on the entire image of the DBC ceramic substrate.

[0065] Multi-scale cropping is an image preprocessing technique that involves cropping images at different scales to provide different-sized inputs to deep learning models. This approach helps the model learn features at different levels, thereby improving its ability to understand the image content.

[0066] Exemplarily, the multi-scale cropping method can be: W / 2*H / 2, W*H, 2W*2H, 3W*3H, etc., which respectively indicate that the cropping size is half, original size, double, and triple of the width and height of the original image; wherein, the cropped image of the W / 2*H / 2 scale is upsampled before being sent to the network model. It should be understood that the reason for cropping the image to a smaller size such as W / 2*H / 2 is usually to improve the detection accuracy of small targets, which enables the model to observe the details in the image more carefully, because when the image size is reduced, the proportion of defects in the image relative to the image size becomes larger, making it easier to be captured by the model. Subsequently, these smaller-sized images are upsampled, that is, the resolution of the image is increased back to the original size or a larger size, which can further improve the detection accuracy and accuracy of small targets. Upsampling can be achieved through different algorithms, such as nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, or using deep learning techniques such as transposed convolution or PixelShuffle, etc., which are not limited in this embodiment.

[0067] If the deep network input image size used is 640*640, the cropping sizes are: 320*320, 640*640, 1280*1280, 1920*1920. Cropping of different sizes can help the model learn multi-level features from local details to global context. The model can improve its adaptability to scale changes by observing images of different sizes. By training at different scales, the model can better utilize its capacity and learn more complex feature representations. It should be understood that for DBC ceramic substrates, since they may have defects of different sizes, multi-scale cropping can ensure that these defects are captured in image blocks of different sizes. However, there is no need to set multi-scale cropping for ordinary images. For example, medical imaging usually needs to maintain the original size and proportion to ensure the accuracy of diagnosis, and multi-scale cropping may change important features in the image.

[0068] Preferably, a preset overlapping area is set when cropping an image to avoid defects that are missing at the cropping boundary. For example, to avoid partial defects being missing at the cropping boundary, a 20% overlapping area can be set for each cropping. By presetting the overlapping area, information loss can be avoided. There may be important features or defects at the edges or corners of the image. If there is no overlapping area, this key information may be lost during the cropping process.

[0069] Step S25: preprocessing the annotated and multi-scale cropped images of each modality to obtain a trainable image dataset, wherein the number of channels of the images in the image dataset is N times the number of channels of the original images, where N is the number of modality categories.

[0070] This involves preprocessing the multimodal image dataset into a format suitable for deep learning model training and inference. In deep learning, the model usually requires input with fixed size and number of channels. Therefore, for the N multimodal images collected, each multimodal image contains three channels: red, green, and blue. The N modal images are stitched in sequence on the channels to obtain a single image with (3*N) channels.

[0071] Take 4 multimodal images as an example. Each image contains three channels, RGB. It is not feasible to directly feed these images into the network for training because they do not meet the network's input requirements. Therefore, it is necessary to stitch the four images in sequence on the channels to obtain a single image with 12 channels. Specifically, the four multimodal images are stitched in sequence on the channel dimension, that is, the RGB channels of each image are arranged in sequence to form a single image with 12 channels, where each channel corresponds to a color channel in the original multimodal image.

[0072] Step S26: modifying the convolution size of the input layer of the network model according to the number of channels of the image in the image data set, and using the image data set to train the modified network model to obtain a surface defect detection model suitable for DBC ceramic substrates.

[0073] In the target detection task, the input image usually has three channels, RGB, but in the embodiment of the present application, since there are a total of 4 multimodal images, the number of channels of the image in the image dataset has increased to 12. In view of this, it is necessary to modify the conventional convolutional neural network (CNN) input layer so that it can accept 12 channel inputs. The modification method is, for example: the first convolution layer of the input layer needs to adjust the size of its weight matrix to match the new number of channels. For example, if the original convolution kernel is 3x3x3 (spatial size x number of channels), it now needs to be changed to 3x3x12. Once the input layer is appropriately modified, the rest of the network can remain unchanged because the convolution operation is channel independent.

[0074] Preferably, the network model used in the training is a YOLOv8 model. YOLO (You Only Look Once) is a popular target detection algorithm, and YOLOv8 is a version thereof. In this solution, YOLOv8 is used as a basic model and trained to process 12-channel input images. In addition, other target detection frameworks such as Faster-RCNN or DETR (Detection Transformer) may also be used, which is not limited in this application.

[0075] Step S27: Use the ONNX framework to deploy the trained surface defect detection model to the target device.

[0076] The ONNX (Open Neural Network Exchange) framework is an open model format standard for representing deep learning models. It allows models to be trained in one framework and seamlessly converted and used in different frameworks without retraining. ONNX defines a unified model representation that allows models to be ported between different deep learning frameworks while maintaining the integrity and accuracy of the model. ONNX allows models to be converted between different deep learning frameworks, such as from PyTorch to TensorFlow or ONNX runtime, which provides flexibility for model deployment. ONNX provides optimization tools that can optimize models, reduce model size, increase inference speed, and reduce memory consumption. ONNX simplifies the process of model deployment because it provides a standardized method to convert and run models.

[0077] Step S28: using the target device deployed with the surface defect detection model to perform real-time online detection on the DBC ceramic substrate to be detected.

[0078] During inference, multi-modal and multi-scale images are used for inference, and the detection results at different scales are summarized as the final detection result. Through verification, the multi-modal DBC substrate surface defect detection method based on deep learning proposed in this application can well detect the following types of defects, which include but are not limited to Figure 5 The following figures show the inspection effects of the corresponding defect types: ceramic oxidation, foreign matter, ceramic discoloration, ceramic cracks, scratches, edge collapse, bumps, copper defects, and bubbles. These figures are intended to provide auxiliary visual understanding for reference.

[0079] It should be understood that ceramic oxidation refers to the reaction of ceramic materials with oxygen under high temperature or electrochemical action, which usually leads to the formation of an oxide layer on the surface of ceramic materials, affecting their performance. Foreign matter refers to materials that are not ceramic materials or designed and are accidentally mixed into the product. Ceramic discoloration refers to the inconsistent color caused by uneven temperature control or differences in raw material composition in the ceramic substrate process, that is, the color of some areas does not match the overall design or expected color. Ceramic cracks refer to cracks on the surface or inside of ceramic materials. These cracks may be caused by factors such as internal stress, external impact or temperature changes in the material. The presence of cracks will reduce the strength and durability of ceramic materials. Scratches refer to linear damage to the surface of ceramic materials caused by hard objects, usually appearing as long and thin scratches. In industrial testing, scratches are one of the common defects that affect the appearance and performance of products. Edge collapse refers to the damage to the edge of the material during the processing of ceramic substrates, especially during cutting or breaking. Bumps refer to damage to ceramic substrates caused by collisions during transportation or handling. This damage may be surface depressions, scratches or other forms of deformation. Copper deficiency refers to the absence or insufficiency of copper material, which may be caused by defects in the processing process or uneven materials. Bubbles refer to bubble-like defects formed on the surface of ceramic substrate materials, usually caused by gas or liquid inside the material being trapped under the surface.

[0080] It is worth noting that: in terms of visual imaging, existing imaging schemes often select a single type of monochromatic light source to image the DBC ceramic substrate, which is not very compatible with defects of various forms and color differences. Based on this, the present application adopts a color camera plus a high-magnification lens and a combination of multiple types and multi-spectral light sources to perform multi-modal imaging of features, and strictly control the surface defects of the DBC ceramic substrate. At the same time, in order to be able to shoot a complete DBC ceramic substrate, the prior art selects a small-magnification lens. This solution will reduce the detection accuracy and cannot detect subtle defects. In response to this problem, the present invention adopts a high-magnification lens solution and uses image stitching technology to completely stitch it. Existing DBC ceramic substrate defect detection methods based on deep learning often only use single-mode, single-size images for network training and reasoning positioning and segmentation defects. Since the defects of DBC ceramic substrates have different characteristic forms (size, shape, etc.) and different brightness changes, single-mode single-size images can easily cause defects to be missed. In response to the problem of single-mode imaging, the present invention provides a multi-modal imaging solution for DBC ceramic substrates, which uses different light source types and different spectra to perform comprehensive imaging of defects and highlight the characteristics of defects. At the same time, the multimodal images are channel-integrated and sent to the deep learning network for training and reasoning.

[0081] In the process of industrial inspection by deep learning, in order to improve the detection accuracy and detect small targets well, the image will be cropped to a fixed size, such as 640*640, when it is sent to the deep network. This processing method will have the following problems: if the defect area is larger than the cropped image size, it is often not well detected; if the cropped size is large, the image will be downsampled before being sent to the network to keep it consistent with the input size of the deep network model, which will cause the loss of small target defects. Therefore, in view of the problem that single-scale images cannot be compatible with defects of different sizes, this application proposes a multi-scale image training and reasoning method to crop the same defective image with different resolutions. Assuming that the input size of the network is W*H, then the cropping method itself is W / 2*H / 2, W*H, 2W*2H, 3W*3H, etc., which can be adjusted according to the maximum and minimum size of the defect. The present invention only proposes an industrial defect detection training and reasoning method. Among them, W / 2*H / 2 will upsample the network before being sent to the network, which can improve the detection accuracy and accuracy of small targets. When the DBC ceramic substrate inspection results are finally output, the inspection results of different sizes are used as the final defect extraction results.

[0082] In the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. For example, the first XX and the second XX are only for distinguishing different XXs, and do not limit their sequence. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different. In addition, words such as "exemplary" or "for example" represent examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0083] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, ab, ac, bc or abc, where a, b, c can be single or multiple.

[0084] Figure 6 The schematic diagram of the structure of a multi-modal DBC substrate surface defect detection system based on deep learning in one embodiment of the present application is shown. The system 600 in the present application includes an image acquisition module 601 and a defect detection module 602.

[0085] The image acquisition module 601 is used to acquire multimodal images of the DBC ceramic substrate through a light source combination. The defect detection module 602 is connected to the image acquisition module 601 in communication; after receiving the multimodal image, the defect detection module 602 performs the following steps: Figure 2 The multimodal DBC substrate surface defect detection method based on deep learning is shown.

[0086] In the embodiment of the present application, the image acquisition module 601 can be a camera, preferably a color camera with a high-magnification lens and a combination of multiple types and multi-spectral light sources to perform multi-modal imaging of features. The defect detection module 602 can be a computer device, such as a personal computer (PC), a server, a workstation, a network device, an industrial computer, etc., which is not limited in this embodiment.

[0087] It should be understood that the specific process of each module executing the above corresponding steps has been described in detail in the above method embodiment, and for the sake of brevity, it will not be repeated here. The division of modules in the embodiment of the present application is schematic, which is only a logical function division, and there may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application can be integrated in a processor, or it can exist physically separately, or two or more modules can be integrated in one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0088] Figure 7 is a schematic block diagram of a computer device provided in an embodiment of the present application. Figure 7 As shown in FIG. 1 , the computer device includes: at least one processor 701, a memory 702, at least one network interface 703 and a user interface 705. The various components in the device are coupled together via a bus system 704. It can be understood that the bus system 704 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 704 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 704 is not described in detail. Figure 7 In the specification, various buses are labeled as bus systems.

[0089] The user interface 705 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0090] It is understood that the memory 702 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of exemplary but not limiting explanation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0091] The memory 702 in the embodiment of the present invention is used to store various categories of data to support the operation of the electronic terminal 700. Examples of these data include: any executable program for operating on the electronic terminal 700, such as an operating system 7021 and an application 7022; the operating system 7021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 7022 can include various applications, such as a media player (Media Player), a browser (Browser), etc., for implementing various application services. The XX method provided by the embodiment of the present invention can be included in the application 7022.

[0092] The method disclosed in the above embodiment of the present invention can be applied to the processor 701, or implemented by the processor 701. The processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 701 or the instruction in the form of software. The above processor 701 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 701 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 701 may be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0093] In an exemplary embodiment, the electronic terminal 700 may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the aforementioned method.

[0094] According to the method provided in the embodiment of the present application, the present application also provides a computer program product, which includes: a computer program code, when the computer program code is run on a computer, the computer executes Figure 2 or Figure 4 The method of the illustrated embodiment.

[0095] According to the method provided in the embodiment of the present application, the present application also provides a computer-readable storage medium, which stores a program code, and when the program code is run on a computer, the computer executes Figure 2 or Figure 4 The method of the illustrated embodiment.

[0096] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program and / or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components may reside in a process and / or an execution thread, and a component may be located on a computer and / or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may, for example, communicate through local and / or remote processes according to signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system and / or a network, such as the Internet interacting with other systems through signals).

[0097] Those of ordinary skill in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0098] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0099] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0100] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0101] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0102] In the above embodiments, the functions of each functional unit can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When loading and executing computer program instructions (programs) on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integrations. Available media may be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., high-density digital video discs (DVDs), or semiconductor media (e.g., solid state disks (SSDs), etc.).

[0103] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0104] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0105] In summary, the present application provides a multi-modal DBC substrate surface defect detection method, device, system, medium and program product based on deep learning. The present application uses a variety of light sources and spectral technologies to perform comprehensive defect imaging to highlight defect features and effectively prevent the omission of defect detection; by adopting a non-fixed multi-scale image cropping method, the omission of small-sized defects is prevented, thereby improving the detection precision and accuracy of small target defects; after verification, the multi-modal DBC substrate surface defect detection technology based on deep learning proposed in the present application can efficiently and accurately identify and detect the above-mentioned various defect types. Therefore, the present application effectively overcomes the various shortcomings in the prior art and has a high industrial utilization value.

[0106] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A multimodal DBC substrate surface defect detection method based on deep learning, characterized in that: include: Multimodal images of DBC ceramic substrates are collected through light source combination; Based on the image stitching technology, the collected multi-modal images are stitched together to form a complete image of the DBC ceramic substrate; Annotating the entire image of the DBC ceramic substrate; Performing multi-scale cropping on the entire image of the DBC ceramic substrate; Preprocessing the labeled and multi-scale cropped images of each modality to obtain a trainable image dataset, where the number of channels of the image in the image dataset is N times the number of channels of the original image, where N is the number of modality categories; Modifying the convolution size of the input layer of the network model according to the number of channels of the image in the image data set, and training the modified network model using the image data set to obtain a surface defect detection model suitable for DBC ceramic substrates; Use the ONNX framework to deploy the trained surface defect detection model to the target device; The target device deployed with the surface defect detection model is used to perform real-time online detection of the DBC ceramic substrate to be inspected.

2. The multimodal DBC substrate surface defect detection method based on deep learning according to claim 1 is characterized in that: The method of collecting the multimodal image of the DBC ceramic substrate by combining light sources includes: collecting the multimodal image of the DBC ceramic substrate by using a combined light source of a white coaxial light source and a multi-angle multi-spectrum annular light source.

3. The multimodal DBC substrate surface defect detection method based on deep learning according to claim 1 is characterized in that: The method of stitching the collected multi-modal images to form a complete image of the DBC ceramic substrate based on the image stitching technology includes: Performing image preprocessing on the multimodal image; Use feature algorithms to detect feature points in multimodal images and match feature points in different images accordingly; The transformed structure between the images is calculated by matching feature points to perform pose estimation, and the image is transformed according to the estimated pose to align it with the reference image; The transformed image is fused with the reference image to obtain the whole image of the DBC ceramic substrate.

4. The multi-modal DBC substrate surface defect detection method based on deep learning according to claim 3 is characterized in that: During the image fusion process, the image pixels in the overlapping area are weighted averaged to eliminate the stitching seams generated in the overlapping area during the image stitching process.

5. The multi-modal DBC substrate surface defect detection method based on deep learning according to claim 1 is characterized in that: The multi-scale cropping of the whole image of the DBC ceramic substrate includes: cropping the original multimodal image according to the following multiple sizes: W / 2*H / 2, W*H, 2W*2H, 3W*3H, W represents the width of the original multimodal image, and H represents the height of the original multimodal image; wherein the cropped image of the W / 2*H / 2 scale is upsampled before being sent to the network model.

6. The multi-modal DBC substrate surface defect detection method based on deep learning according to claim 1 is characterized in that: The number of channels of the image in the image dataset is N times the number of channels of the original image, where N is the number of modal categories, including: for the N multimodal images collected, each multimodal image contains three channels: red, green, and blue. The N modal images are stitched on the channels in sequence to obtain a single image with a channel number of (3*N).

7. The multi-modal DBC substrate surface defect detection method based on deep learning according to claim 1, characterized in that: The target device deployed with the surface defect detection model is used to perform real-time online detection on the DBC ceramic substrate to be detected, and the detection results include any one or more of the following: ceramic oxidation, foreign matter, ceramic discoloration, ceramic cracks, scratches, edge collapse, bumps, copper defects, and bubbles.

8. A multi-modal DBC substrate surface defect detection system based on deep learning, characterized in that: include: An image acquisition module, used for acquiring multimodal images of a DBC ceramic substrate through a light source combination; A defect detection module, which is communicatively connected with the image acquisition module; After receiving the multimodal image, the defect detection module executes the multimodal DBC substrate surface defect detection method based on deep learning as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal DBC substrate surface defect detection method based on deep learning as described in any one of claims 1 to 7 is implemented.

10. A computer program product, characterized in that The computer program product includes a computer program code, and when the computer program code is executed on a computer, the computer is enabled to implement a multimodal DBC substrate surface defect detection method based on deep learning as described in any one of claims 1 to 7.

11. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the multimodal DBC substrate surface defect detection method based on deep learning as described in any one of claims 1 to 7.

Images