Surface defect detection method, system and related products based on multi-light source fusion

Abstract

The present invention discloses a surface defect detection method, system, and related products based on multi-light source fusion, including: collecting multiple sets of images of a sample under test under different light source conditions; annotating image defects to obtain an annotated dataset, and merging the same set of multi-light source image masks; establishing a multi-light source fusion segmentation model with multiple image inputs, multi-feature fusion, and single image output; training the fusion segmentation model based on the multi-light source dataset and optimizing the training process; converting the trained model into an accelerated model for production line deployment, thereby achieving non-destructive, real-time, and automated detection of product surface defects. The present invention can comprehensively capture key information of multiple defects through varying light sources, solving the problems of low efficiency and insufficient accuracy of traditional industrial quality inspection methods.

Description

Technical Field

[0001] The present invention relates to the field of industrial quality inspection technology, and in particular to a surface defect detection method, system and related products based on multi-light source fusion. Background Art

[0002] Product defect detection is crucial for quality control in industrial manufacturing. In complex industrial manufacturing processes, mechanical products can develop various defects, such as scrapes, scratches, dents, and internal holes, due to improper design, equipment failure, or harsh operating environments. These defects not only compromise the product's appearance and quality but can also negatively impact its performance and service life. Automated defect detection technology can not only improve product quality but also reduce production costs.

[0003] Traditional methods use techniques such as feature extraction, template matching, and edge detection to extract and detect defects. However, these methods are not universally applicable to complex defects, large lighting variations, and large-scale data processing. Manual parameter adjustment is often required based on the situation, which is very time-consuming and labor-intensive. Advanced deep learning technologies can better adapt to complex defect scenarios and perform well on large-scale data. However, in actual production environments, lighting conditions and environmental changes can interfere with the visual appearance of defects. This requires defect detection systems to be able to adaptively adjust lighting to increase effective imaging and feature information of defects, and requires detection models to have sufficient generalization capabilities to adapt to different products, different defects, and different lighting conditions. Currently, multi-light source detection technology is widely used in industrial quality inspection due to its strong robustness and adaptability.

[0004] In existing multi-light source defect detection technologies, there are two main strategies for processing multi-light source images. One is to fuse images obtained under different light sources and then use the fused image for defect detection. This method is computationally complex and involves complex image alignment and fusion algorithms, requiring significant processing time and hardware resources. The other method is to perform defect detection on images obtained under each light source separately and then comprehensively evaluate the results of multiple inspections. This method requires independent inspection of each image, resulting in wasted computing resources and potentially conflicting results when integrating the results. Given the numerous limitations of existing methods in complex industrial environments, the market urgently needs an automated surface defect detection solution that is highly adaptable, highly real-time, and avoids repeated inspections.

[0005] Therefore, how to provide a surface defect detection method, system and related products based on multi-light source fusion with high efficiency and high adaptability is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0006] In response to the aforementioned research status and existing problems, this paper provides a surface defect detection method, system, and related products based on multi-light source fusion. These methods aim to address the low efficiency, poor real-time performance, and insufficient precision of traditional defect detection methods. Furthermore, by introducing multi-light source fusion technology, the model demonstrates excellent adaptability, generalization, and robustness in complex environments and with complex defects.

[0007] The present invention first provides a surface defect detection method based on multi-light source fusion, comprising the following steps:

[0008] Steps to build a multi-light source fusion segmentation model based on the encoder-spatial selector-decoder framework:

[0009] The encoder is used to extract multi-layer semantic information of the multi-light source image and obtain multi-light source intermediate features at multiple scales;

[0010] The spatial selector fuses the multiple light source intermediate features based on the information score mask of each of the multiple light source intermediate features to obtain light source fusion features corresponding to each scale;

[0011] The decoder is used to fuse the light source fusion features corresponding to each scale to obtain a detection result;

[0012] Model training steps:

[0013] Taking multiple sets of images of the sample at different positions, each set of images is an image of a multi-light source mode taken at the same position;

[0014] Perform defect location mask annotation for each set of images to obtain a multi-light source annotation dataset;

[0015] The multi-light source fusion segmentation model is trained based on the multi-light source labeled dataset, and the training process is iteratively optimized to obtain a trained multi-light source fusion segmentation model.

[0016] Defect detection steps:

[0017] Receive the real-time multi-light source image stream of the product, input it into the trained multi-light source fusion segmentation model, and output the real-time detection results of product surface defects.

[0018] Preferably, the multi-light source fusion segmentation model includes:

[0019] The encoder is an N-branch M-level encoder, wherein the N branches respectively receive images of N light source modes taken at the same position, and the N branches respectively extract semantic information of the images at different scales, and the N branches share parameters; the M levels correspond to M spatial selectors; M>1, N>1.

[0020] Preferably, the space selector includes an STS module; wherein,

[0021] The STS module receives the intermediate features of the N light sources extracted by the N branches as input and calculates the information score mask Q of each intermediate feature n , the information score mask represents the importance score of each spatial position in the intermediate feature:

[0022] Q n =Sigmoid(Conv(DWConv(f n )))

[0023] Among them, {f n |n∈[1,N]} is the intermediate feature of N light sources, DWConv(·) represents the deep convolutional network, and Conv(·) represents the standard convolutional network;

[0024] Multiply the information score masks of N intermediate features by the corresponding intermediate features element by element, and add the product results to the corresponding intermediate features element by element to obtain N new feature maps;

[0025] Compare the N new feature maps, and the fused light source fusion feature f q The patch of each image pixel space position is filled by the maximum value of the patch at the corresponding image pixel space position in the N new feature maps:

[0026] f q =φ({f n +Q n ·f n |n∈[1,N]})

[0027] Here, φ(·) represents the maximum value of each image pixel spatial location.

[0028] Preferably, each branch consists of M stages; each stage contains Overlap PatchEmbedding and Transformer block. The input patch of each branch passes through M stages in sequence, and the number of channels of each stage is different, so as to obtain M-level multi-light source intermediate features with different image resolutions.

[0029] Preferably, the iterative optimization training process includes the steps of setting an optimization strategy:

[0030] Use Lion optimization algorithm;

[0031] A weighted combination of Cross-Entropy loss and Lovász-Softmax Loss is used as the loss function to optimize the Acc and IoU indicators;

[0032] The Warm Up learning rate strategy is adopted. In the initial stage of training, the LinearLR strategy is used until the learning rate reaches the peak. Then, the learning rate is gradually reduced according to the PolyLR strategy until the learning rate reaches 1×e after the last iteration. -8 .

[0033] Preferably, the step of capturing multiple sets of images of the sample at different positions includes:

[0034] Each time the light source is switched, the sending of the shooting instruction is automatically triggered. A group of images with different light source modes are captured for the same shooting position to obtain a group of multi-light source images.

[0035] Preferably, the step of masking defect positions for each set of images includes:

[0036] Marking defect positions on the multi-light source image to generate a mask image with the same size as the multi-light source image;

[0037] The mask images of the same set of multi-light source images are merged to make the marking of the defect positions in the same set of multi-light source images consistent.

[0038] The present invention also provides a surface defect detection system based on multi-light source fusion according to the surface defect detection method based on multi-light source fusion, comprising:

[0039] A multi-light source image acquisition module is used to capture multiple sets of images of a sample at different locations, with each set of images being captured at the same location in a multi-light source mode, serving as sample data for constructing a multi-light source annotation dataset; and to control the camera to automatically capture the multi-light source image stream of the product as the light source array changes;

[0040] Image detection module, including:

[0041] The model construction unit includes the steps of constructing a multi-light source fusion segmentation model based on an encoder-spatial selector-decoder framework: the encoder is used to extract multi-layer semantic information of the multi-light source image to obtain multi-light source intermediate features at multiple scales; the spatial selector is used to fuse the multi-light source intermediate features based on the information score mask of each multi-light source intermediate feature to obtain light source fusion features corresponding to each scale; the decoder is used to fuse the light source fusion features corresponding to each scale to obtain a detection result;

[0042] The model training unit trains the multi-light source fusion segmentation model based on the multi-light source annotation dataset, and iteratively optimizes the training process to obtain a trained multi-light source fusion segmentation model;

[0043] A defect detection unit is configured to receive a real-time multi-light source image stream of a product, input the stream into the trained multi-light source fusion segmentation model, and output a real-time detection result of surface defects of the product;

[0044] The model deployment module is used to convert the multi-light source fusion segmentation model into an accelerated model that is easy to deploy on the production line, and generate a production line report based on the real-time detection results.

[0045] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored on the memory and runnable on the processor, wherein the memory is used to store a multi-light source image stream of a product, a multi-light source fusion segmentation model, and production line data; when the processor executes the program, the steps of the surface defect detection method based on multi-light source fusion are implemented.

[0046] The present invention also provides a non-transitory computer-readable storage medium on which a computer program and a multi-light source fusion segmentation model are stored. When the computer program is executed by a processor, the steps of the surface defect detection method based on multi-light source fusion are implemented.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. This invention utilizes an editable stereo light source specifically designed for defect detection, effectively reducing the need for frequent light source changes and repeated shooting in traditional multi-light source inspections, thereby significantly improving inspection efficiency. The stereo light source creates a variety of different lighting effects, effectively simulating lighting conditions in different scenarios and enabling the system to adapt to a variety of complex inspection environments.

[0049] 2. Utilizing an encoder-spatial selector-decoder architecture, a multi-light source feature fusion strategy is designed to comprehensively capture and deeply mine surface defect features from different light sources. This method simply feeds multi-light source images into a deep network and fuses the intermediate features of these multiple light sources to produce a comprehensive defect segmentation result. This eliminates the need for complex image alignment and fusion, effectively avoiding the resource waste associated with multiple inspections, and ultimately achieves efficient and rapid multi-light source defect detection.

[0050] 3. The use of deep learning methods for multi-light source feature fusion is more adaptive and intelligent than traditional multi-light source image fusion. First, traditional fusion strategies require manual adjustment based on on-site conditions, while deep learning networks can adaptively learn the optimal feature extraction and fusion strategies, performing better in complex industrial scenarios. Second, traditional fusion only considers the direct fusion of local features, while deep fusion can capture global contextual information, preserve local details, and promote the integrity of defect information. In addition, multi-level feature fusion not only enhances the system's sensitivity to subtle defects, but also ensures that even under complex lighting conditions, multiple types of defects can be accurately distinguished and identified.

[0051] 4. The system's built-in SegFormer backbone network, with its superior optimization algorithm, not only accelerates model training but also enables rapid and accurate defect identification in actual inspections, achieving both real-time performance and accuracy. The system's built-in multi-level feature extraction mechanism automatically adjusts to meet the needs of defect detection at varying scales, enabling accurate identification of both tiny cracks and large-scale damage, demonstrating excellent generalization capabilities.

[0052] 5. The present invention integrates functional modules such as image acquisition, model training, and model deployment into a system to solve the problem of long industrial deployment project cycles.

[0053] In summary, the present invention achieves the purpose of surface defect detection of industrial products through deep learning technology, multi-light source feature fusion method, image processing technology, model training optimization, automated detection and other technical means. It has strong practicality and promotion value, and can provide better solutions for the field of industrial quality inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only embodiments of the present invention. Those skilled in the art can also derive other drawings based on the provided drawings without inventive effort.

[0055] Figure 1 A flow chart of a surface defect detection method based on multi-light source fusion provided by an embodiment of the present invention;

[0056] Figure 2 A general model framework diagram provided for an embodiment of the present invention;

[0057] Figure 3 A schematic diagram of the STS module structure provided in an embodiment of the present invention;

[0058] Figure 4A schematic diagram of the structure of a surface defect detection system based on multi-light source fusion provided by an embodiment of the present invention;

[0059] Figure 5 Schematic diagram of the surface defect detection equipment based on multi-light source fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0061] The first aspect of the embodiment of the present invention discloses a surface defect detection method based on multi-light source fusion, such as Figure 1 As shown, the following steps are included:

[0062] Steps to build a multi-light source fusion segmentation model based on the encoder-spatial selector-decoder framework:

[0063] The encoder is used to extract multi-layer semantic information of multi-light source images and obtain multi-light source intermediate features at multiple scales; the spatial selector fuses the multi-light source intermediate features based on the information score mask of each multi-light source intermediate feature to obtain the light source fusion features corresponding to each scale; the decoder is used to fuse the light source fusion features corresponding to each scale to obtain the detection results.

[0064] Model training steps:

[0065] Multiple sets of images are taken for the sample at different locations, with each set of images being images of multiple light sources taken at the same location. Defect location masks are annotated for each set of images to obtain a multi-light source annotated dataset. A multi-light source fusion segmentation model is trained based on the multi-light source annotated dataset, and the training process is iteratively optimized to obtain a trained multi-light source fusion segmentation model.

[0066] Defect detection steps:

[0067] Receive the real-time multi-light source image stream of the product, input it into the trained multi-light source fusion segmentation model, and output the real-time detection results of product surface defects.

[0068] In one embodiment, a shared multi-light image feature extraction encoder, feature fuser, and a feature decoder fusion segmentation network (MFSNet) is constructed, including:

[0069] The encoder is an N-branch M-level encoder. The N branches receive images of N light source modes taken at the same location, and the N branches extract semantic information at different scales from the images. The N branches share parameters; the M levels correspond to M spatial selectors; M>1, N>1.

[0070] In this embodiment, Figure 2 As shown in the figure, based on the SegFormer framework, an N-branch four-stage encoder based on MiT is established as the backbone network of the multi-light source fusion segmentation model. The four-stage encoder generates feature maps at four different scales, each containing feature information at different scales. Due to the multi-branch design, each branch is responsible for extracting features at a specific scale under the corresponding illumination mode. As a result, the feature maps output at each stage are able to better capture the different details of the image.

[0071] A spatial token selection (STS) is used between the encoder and decoder. Each of the four stages corresponds to four spatial token selections. The extracted multi-illuminant intermediate features are fused to generate a single feature map. Specifically, N branches extract multiple layers of semantic information from the multi-illuminant image, and all branches share parameters.

[0072] It should be noted that the purpose of setting encoder shared parameters is to reduce model parameters and improve training efficiency. This is achieved by instantiating an identical module object. For example, define a shared module share_module and reuse the same module in N branches. This ensures that the weights of share_module are exactly the same in different branches and are updated together during training:

[0073]

[0074] SegFormer Head acts as a decoder to fuse the four feature maps obtained in the four stages to obtain the prediction result.

[0075] In this embodiment, the encoder and decoder in the MFSNet model can be replaced with any segmentation model with an encoder-decoder structure, such as SegFormer, SegNeXt, DeepLab, etc., according to the characteristics of the defects of the product to be inspected. This embodiment only shows one of the models. Taking into account the accuracy and speed issues, SegFormer was selected as the default model after multiple tests. It has both high accuracy and high detection speed. In addition, the number of spatial marker selectors depends on the number of input feature maps of the decoder and has nothing to do with the number of branches. For example, the decoder of DeepLabV3+ has two input features, so two spatial marker selectors are required to fuse the two levels of feature maps respectively to obtain the two feature maps required by the decoder.

[0076] In this embodiment, if Figure 3 As shown, the space selector includes an STS module; wherein,

[0077] First, the input of the STS module is a set of N feature maps {f n |n∈[1,N],f n ∈H×W×C}, each feature map has extracted specific spatial features through the processing of the previous layers of the network. These feature maps represent the representation of the same object under different lighting conditions and contain rich diversity information. For each input feature map f n Apply the STS module to calculate the information score mask Q of each intermediate feature n , the information score mask represents the importance score of each spatial position in the intermediate feature:

[0078] Q n =Sigmoid(Conv(DWConv(f n )))

[0079] Among them, {f n |n∈[1,N]} is the intermediate feature of N light sources, DWConv(·) represents the deep convolutional network, and Conv(·) represents the standard convolutional network;

[0080] Then, the feature map enhancement operation is performed. The information score mask of the N intermediate features is multiplied element-by-element with the corresponding intermediate features. This operation enhances important features while suppressing unimportant positions. The product result is added element-by-element with the corresponding intermediate features to obtain N new feature maps.

[0081] Finally, the feature map fusion operation is performed. N new feature maps are compared, and the fused light source fusion feature f q The patch at each image pixel space position is filled with the maximum value of the patch at the corresponding image pixel space position in the N new feature maps:

[0082] f q =φ({f n +Q n ·f n |n∈[1,N]})

[0083] Here, φ(·) represents the maximum value at each spatial location of each image pixel. This approach not only enhances the features of key areas but also selects the most effective patch at each spatial location among the N features, allowing the model to focus on the key areas of the object. The resulting feature map contains the best feature information under different light sources, resulting in higher expressiveness and robustness.

[0084] It should be noted that a "patch" refers to a small local area extracted from the original image or feature map. It is used to describe a small set of pixels in an image or feature map, usually square and of a fixed size. When fusing N feature maps, each feature map is composed of many such small blocks (patches). The fusion operation selects the maximum value among the N patches at each position to form a new fused feature map. This is done to retain the most representative information in each feature map while removing redundant or unimportant parts.

[0085] In one embodiment, each branch consists of M stages; each stage contains OverlapPatch Embedding and Transformer block. The input patch of each branch passes through M stages in sequence, and each stage has a different number of channels, obtaining M levels of multi-light source intermediate features with different image resolutions.

[0086] In this example, N input images of size H×W×3 enter N branches with shared parameters respectively. The input patch of each branch passes through four stages with channels of (64, 128, 256, 512) and corresponding stacking ratios of [2, 2, 2, 2] in sequence to obtain multi-level features with original image resolutions of {1 / 4, 1 / 8, 1 / 16, 1 / 32}. The multi-branch features obtained in each stage are filtered and selected by an STS module and finally fused to obtain four feature maps of different resolutions with multi-light source properties.

[0087] The specific implementation process is as follows:

[0088] Taking three 2448×2048×3 input images as an example, the horizontal and vertical ratios are locked and the input images are reduced to 612×512×3, normalized, and then the three images are fed into three encoder branches with shared parameters. The Overlap Patch Embedding of Stage-1 is a 7×7 convolution with a stride of 4, while Stages-2, 3, and 4 are all 3×3 convolutions with a stride of 2. By reducing the resolution and increasing the number of channels, information of different scales is extracted layer by layer. The Transformer block of each stage includes a multi-head self-attention mechanism (MHSA) and a mixed feed-forward network (Mix-FFN).

[0089] MHSA introduces a reduction rate R to process features in blocks, reducing the complexity of the self-attention mechanism. The formula is as follows:

[0090]

[0091] Where N = H × W is the length of the sequence, K is the sequence to be reduced, Reshape(N / R,C·R)(K) reshapes K into a sequence of shape N / R × (C·R), and Linear(C1,C2)(·) takes a C1-dimensional tensor as input and produces a C2-dimensional tensor as output. R is set to [64, 16, 4, 1] for Stage-1 to Stage-4.

[0092] The formula of the multi-head self-attention mechanism is as follows:

[0093]

[0094] Among them, each head Q, K, V has the same size N×C.

[0095] Mix-FFN consists of a 3×3 depthwise convolution and two multi-layer perceptrons (MLPs). The depthwise convolution is used to extract local features, and the multi-layer perceptron is applied to the channel dimension, and the features are linearly transformed through the fully connected layer. The formula is as follows:

[0096] x out =MLP(GELU(Conv 3×3 (MLP(x in ))))+x in

[0097] where x in is the feature from the self-attention module, and GELU represents the Gaussian Error Linear Unit activation function.

[0098] After the four stages, each of the three branches obtains feature maps of four different resolutions, totaling 12 feature maps. Next, four spatial tag selectors are connected to fuse the feature maps from the four stages, ultimately extracting four feature maps of different resolutions.

[0099] In one embodiment, the step of capturing multiple sets of images of the sample at different positions includes:

[0100] Each time the light source is switched, the sending of the shooting instruction is automatically triggered. A group of images with different light source modes are captured for the same shooting position to obtain a group of multi-light source images.

[0101] In this embodiment, N lighting methods that are most suitable for product defects are explored and determined; the light source controller controls the industrial camera to take pictures through the serial port, and sends a signal to the industrial camera to take a picture every time the light source is switched, and a group of N multi-light source images are taken for the same picture at one time.

[0102] When executing:

[0103] First, explore the optimal lighting method for defects. To ensure that all defects are clearly visible, the 3D light source's lighting angle and brightness need to be continuously adjusted based on the characteristics of the surface defects on the product being inspected. Through experimentation, we identify numerous optimal lighting methods and save these light source patterns.

[0104] Then, during the acquisition process, the system will automatically switch the corresponding light source mode, and the light source controller will control the camera to take pictures through the serial port. Every time the light source is switched, a signal will be sent to the camera to take a picture, and a group of N multi-light source images will be taken for the same picture at one time.

[0105] After completing a set of light source images, adjust the product's inspection position or replace the product under inspection and repeat the acquisition process until all images are captured. Images from the same light source are saved in the same folder for subsequent resolution and processing.

[0106] In one embodiment, the step of masking defect locations for each set of images includes:

[0107] The defect locations are marked on the multi-light source images to generate a mask image with the same size as the multi-light source images. The mask images of the same group of multi-light source images are merged to ensure that the defect locations in the same group of multi-light source images are marked consistently, thus avoiding missing any defects.

[0108] In this embodiment, the saved mask image is a grayscale image, the grayscale value represents the label value of the corresponding defect type, and the background portion is marked as 0 by default.

[0109] It should be noted that the operation of merging mask images of the same set of multi-light source images is intended to achieve the following result: if a pixel position is marked as a defect on any mask image in a set of masks, then the position will also be marked as a defect on the final merged mask image.

[0110] Under normal circumstances, specific defects appear either present or absent under different light sources and do not develop into other defects. Therefore, defects can be combined using an OR operation. For example, if there are three mask images, denoted as M1, M2, and M3, the combined mask M can be expressed using the logical operation: M = M1 OR M2 OR M3. In special cases, where there may be overlapping areas of different defects, image comparison and consistency annotation can be used to eliminate uncertainty and reduce misjudgments or omissions.

[0111] In this example, the Segment Anything Model (SAM) is introduced for automatic annotation. The SAM model automatically generates high-quality segmentation results, significantly improving annotation efficiency, especially when processing large amounts of image data. This annotation accuracy is higher than traditional manual methods, reducing the occurrence of annotation errors. Using this annotation software, users simply click a few points or boxes on the image as prompts, and the model automatically completes the remaining defect segmentation and annotation.

[0112] In this embodiment, the generated multi-light source dataset is divided into a training set and a validation set according to a certain ratio for model training. For example, the generated multi-light source dataset is divided into a training set and a validation set according to a certain ratio (e.g., 4:1). In cases where there is less defect data, data augmentation techniques can be used to expand the dataset and improve the robustness and generalization ability of the model. Specific data augmentation methods include random rotation, random flipping, color dithering, and noise superposition.

[0113] In one embodiment, the iterative optimization training process includes the steps of setting an optimization strategy:

[0114] Use Lion optimization algorithm;

[0115] A weighted combination of Cross-Entropy loss and Lovász-Softmax Loss is used as the loss function to optimize the Acc and IoU indicators;

[0116] The Warm Up learning rate strategy is adopted. In the initial stage of training, the LinearLR strategy is used until the learning rate reaches the peak. Then, the learning rate is gradually reduced according to the PolyLR strategy until the learning rate reaches 1×e after the last iteration. -8 :

[0117] Loss = βLoss lovász+(1-β)Loss ce

[0118] Wherein, β is the loss balance factor, and β is set to 0.1.

[0119] In this embodiment, a preprocessing process is also included before training: the constructed multi-light source dataset is preprocessed, including formatting and online data enhancement, including but not limited to cropping, random flipping, random resizing, and normalization.

[0120] The preprocessed multiple images are fed into the optimized MFSNet network for training.

[0121] In this embodiment, training also includes an evaluation process: running the trained model on the test set (and validation set) generates a detailed evaluation report including key indicators such as IoU (Intersection over Union), RAcc (Detection Rate), FPR (False Positive Rate), and FNR (Missed Detection Rate). Through indicator analysis, it is determined whether the model performance meets the expected requirements. If it does not meet the expectations, the user is prompted to readjust the dataset or retrain the model parameters. If it meets the expectations, the model deployment process is started.

[0122] In one embodiment, the multi-light source fusion segmentation model is converted into a format suitable for production line deployment, such as ONNX or TensorRT, to accelerate the inference process. The converted model is then integrated into the production line's inspection system. The integration process includes connecting to the camera, light source control system, data acquisition module, and industrial automation system.

[0123] The multi-light source images acquired in real time are fed into the acceleration model for inference. The model outputs defect detection results, enabling non-destructive, real-time, and automated detection of product surface defects. The industrial control system then executes corresponding actions (such as rejecting defective products).

[0124] Over time, product defect types or characteristics may change, leading to degraded model performance. Therefore, regular model updates and retraining are necessary. Building on the existing dataset and model, new datasets are added and the model is retrained using the defect location mask annotation process and an iterative optimization training process. The retrained model undergoes optimization through pruning, quantization, and other methods before being converted and deployed. Continuous model updates and maintenance ensure the long-term effectiveness and stability of the production line defect detection system. All operations are performed within this system, accelerating the model update and iteration process on the production line.

[0125] Based on the same inventive idea as the first aspect of the embodiment, Figure 4 As shown, the second aspect of the embodiment of the present invention provides a surface defect detection system based on multi-light source fusion, including:

[0126] The multi-light source image acquisition module is used to capture multiple sets of images of the sample at different locations. Each set of images is a multi-light source mode image captured at the same location, which serves as sample data for constructing a multi-light source annotation dataset. The module also controls the camera to automatically capture the product's multi-light source image stream as the light source array changes.

[0127] Image detection module, including:

[0128] The model construction unit constructs a multi-light source fusion segmentation model based on the encoder-spatial selector-decoder framework: the encoder is used to extract multi-layer semantic information of the multi-light source image and obtain multi-light source intermediate features at multiple scales; the spatial selector fuses the multi-light source intermediate features based on the information score mask of each multi-light source intermediate feature to obtain the light source fusion features corresponding to each scale; the decoder is used to fuse the light source fusion features corresponding to each scale to obtain the detection result;

[0129] The model training unit trains the multi-light source fusion segmentation model based on the multi-light source annotation dataset, and iteratively optimizes the training process to obtain a trained multi-light source fusion segmentation model;

[0130] The defect detection unit is used to receive the real-time multi-light source image stream of the product, input it into the trained multi-light source fusion segmentation model, and output the real-time detection results of product surface defects;

[0131] The model deployment module is used to convert the multi-light source fusion segmentation model into an accelerated model that is easy to deploy on the production line, and generate production line reports based on real-time detection results.

[0132] In one embodiment, the multi-light source image acquisition module includes visual light source software, a stereo light source, an industrial camera, a bracket, and a test bench. The stereo light source and industrial camera are used to capture multiple sets of images of the sample under test under different lighting conditions. The visual light source software is then used to adjust the light source and camera parameters, adjusting the lighting angle and brightness based on the defect characteristics, and saving the required light source array. Automatic light-cutting mode allows for easy capture of continuous multi-light source images. During model deployment and inference, the stereo light source and industrial camera must be coordinated via serial communication to ensure the camera can capture a complete set of multi-light source images as the light source array changes.

[0133] In one embodiment, the image detection module also includes an image preprocessing module for preprocessing the constructed multi-light source dataset. This includes formatting and online data augmentation, including but not limited to cropping, random flipping, random resizing, and normalization. Specifically, this module performs fixed cropping on the image to remove excess large background areas, and then performs online data augmentation and normalization on the image before each training iteration to enhance generalization.

[0134] In one embodiment, the image detection module also includes a result display and analysis module for plotting the results of the model inference test set and generating a table containing model evaluation indicators such as IoU, RAcc, FPR, FNR, etc. for user reference.

[0135] In one embodiment, the model deployment module includes:

[0136] The model conversion unit is used to optimize the trained model and convert it into other accelerated model formats such as TensorRT or ONNX.

[0137] The production line simulation unit is used to simulate the real-time inspection environment of the production line, receive multi-light source image streams and inference images from the camera in real time, and discover defects. It also generates production line reports for easy viewing and verification of the defect detection effect of the model.

[0138] The model deployment module receives multi-light source image streams from the production line and inputs them into the acceleration model for processing, thus achieving non-destructive, real-time and automated detection of product surface defects.

[0139] In this embodiment, the stereoscopic light source can adopt an editable stereoscopic light source for defect detection provided by Guangzhou Xuanshijia Electronic Technology Co., Ltd. The 48 light-emitting channels are attached to the inner wall of the dome-shaped structure, which can light up the LED arrays at different positions and adjust the brightness parameters to achieve multi-angle lighting of the stereoscopic light source, so that defects of different types and positions can be presented.

[0140] In this embodiment, the industrial camera can be a Hikvision (model) industrial camera, which has a Micro-B 3.0 interface and meets the system's USB 3.0 communication protocol. The camera has 5 million pixels, a resolution of 2448×2048, and a frame rate of 70fps. This camera can clearly capture industrial surface defects.

[0141] Based on the same inventive idea as the method described in the first aspect of the embodiment, the third aspect of the embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and run on the processor, the memory is used to store the product multi-light source image stream, the multi-light source fusion segmentation model and the production line data; the steps of the surface defect detection method based on multi-light source fusion are implemented when the processor executes the program.

[0142] The solution of the present invention can be implemented and executed on a computer device with data query capabilities, wherein the computer device can be a terminal or a server. The terminal can be a smartphone, tablet computer, laptop computer, touch screen, game console, personal computer (PC), personal digital assistant (PDA), etc., but is not limited to these.

[0143] In this embodiment, Figure 5 As shown in the figure, the communication connection between the stereo light source, industrial camera, electronic equipment and transmission equipment. The communication process between the stereo light source, industrial camera, computing equipment and transmission equipment includes:

[0144] First, the conveying equipment sends the product to be inspected to the designated inspection position. After the product is in place, it sends a signal to the light source controller that it is in place.

[0145] Then, the light source that receives the signal will automatically switch the light source mode and send a signal to the camera to take pictures every time the light is cut off, ensuring that each light source switch can effectively take pictures.

[0146] Then, the obtained image stream is acquired by the computing device and sent to the inference model for defect detection to obtain the detection results.

[0147] Finally, the computing device transmits the sorting information of whether the product is qualified to the conveying device, instructing it on the next step of operation.

[0148] In this embodiment, the electronic device can be composed of a host computer equipped with an Intel Core(TM) I9-13900x CPU @ 5.8GHz (with 32GB of memory) and an NVIDIA GeForce RTX-4090 graphics processing unit (GPU) (with 24GB of memory). The processor can be a CPU or GPU processor. The input device includes a display, a keyboard and a mouse. The display is used to display the test results in real time, and the keyboard and mouse are used to perform various human-computer interaction operations.

[0149] In this embodiment, the conveying equipment can be auxiliary equipment such as a conveyor belt and a robotic arm, which is used to accurately convey the products to be inspected to the inspection position, receive sorting signals from the host, and take corresponding actions.

[0150] Based on the same inventive idea as the method described in the first aspect of the embodiment, the fourth aspect of the embodiment of the present invention provides a non-transitory computer-readable storage medium, on which a computer program and a multi-light source fusion segmentation model are stored. When the computer program is executed by the processor, the steps of the surface defect detection method based on multi-light source fusion are implemented.

[0151] The electronic device and non-transitory computer-readable storage medium provided in the embodiments of the present invention correspond one-to-one to the method. Therefore, the electronic device and the non-transitory computer-readable storage medium also have similar beneficial technical effects to their corresponding methods. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the device and the medium will not be repeated here.

[0152] The following is a specific example to illustrate the implementation process of the above surface defect detection method and system. Taking the mobile phone case defect detection as an example, the specific subdivision is into the following steps:

[0153] Step 1: Sample Preparation and Lighting Adjustment. A robotic arm picks up defective phone case samples from a conveyor belt and moves them under the camera and stereo light source. The phone case's posture and camera focus are adjusted to ensure clear visibility of surface details. The visual lighting software adjusts the stereo light source's illumination angle and brightness to clearly display surface defects. The software also saves the three optimal light source array configurations to the light source controller for subsequent acquisition and testing.

[0154] Step 2: Multi-light source image acquisition. Set the stereo light source to automatic light-cutting mode, gradually switching the light source at the preset angle and brightness. Control the robotic arm to rotate the phone case in sequence, capturing multi-light source images of the four sides and four corners of each sample and saving them to the host computer.

[0155] Step 3: Dataset Construction. The robotic arm picks up the next phone case sample and repeats step 2 until sufficient multi-light source image data has been collected. Defects such as scratches, dirt, and white spots are annotated using annotation software. Image masks from the same image under different lighting conditions are then merged. The dataset is then divided into training, validation, and test sets using a specific ratio (e.g., 8:1:1).

[0156] Step 4: Model training. Enter the number of multi-light source branches, activate the system's built-in multi-light source fusion segmentation model, and train it on a multi-light source image dataset of phone cases. The model gradually optimizes parameters, ultimately generating the optimal model file.

[0157] Step 5: Model Evaluation. The system runs the trained model on the test set (and validation set) and generates a detailed evaluation report including key metrics such as IoU, RAcc, FPR, and FNR. This metric analysis determines whether the model's performance meets expectations. If not, the user is prompted to readjust the dataset or retrain the model parameters. If it does, the system proceeds to the next step.

[0158] Step 6: Model conversion and deployment: Convert the best trained model file to TensorRT format and load it into the system to achieve accelerated model inference.

[0159] Step 7: Real-time Inspection. A conveyor belt delivers the phone case to the inspection area, where a robotic arm precisely positions the case for inspection. Once in position, the robotic arm sends a signal to the light source controller, which then switches light sources and captures images from multiple sources. The captured images are transmitted in real time to the host computer, where an acceleration model is used for defect detection. Inspection results are displayed in real time, highlighting any defects.

[0160] Step 8: Comprehensive Inspection and Decision-Making. The robotic arm rotates the phone case. By rotating the arm and switching light sources, the inspection process repeats, ensuring that all four sides and corners of the phone case are inspected using multiple light source images. After inspection, the phone case is returned to the conveyor. The host computer comprehensively determines whether the phone case has defects based on the inspection results and issues corresponding instructions to control the conveyor's direction, determining whether the phone case should proceed to the next processing step or be marked as defective.

[0161] Step 9: Batch Inspection and Report Generation. Repeat steps 6 and 7 until all phone cases are inspected. The system generates a production line report for the user to review.

[0162] Compared with traditional industrial quality inspection technologies, this invention can effectively reduce the consumption of manpower and material resources, and through multi-light source fusion technology, utilizes complementary information under different lighting conditions, greatly improving the adaptability, generalization and robustness of detection, showing great practical application potential.

[0163] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] The above is a detailed introduction to the surface defect detection method, system and related products based on multi-light source fusion provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0165] In this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

Claims

1. A surface defect detection method based on multi-light source fusion, characterized in that: The steps include: Steps to build a multi-light source fusion segmentation model based on the encoder-spatial selector-decoder framework: The encoder is used to extract multi-layer semantic information of the multi-light source image and obtain multi-light source intermediate features at multiple scales; The spatial selector fuses the multiple light source intermediate features based on the information score mask of each of the multiple light source intermediate features to obtain light source fusion features corresponding to each scale; The decoder is used to fuse the light source fusion features corresponding to each scale to obtain a detection result; The encoder receives N images of light source modes taken at the same position, and the N branches extract semantic information of the images at different scales. The spatial selector includes an STS module; wherein the STS module receives the intermediate features of the N light sources extracted by the N branches as input, and calculates the information score mask Q of each intermediate feature n , the information score mask represents the importance score of each spatial position in the intermediate feature: Q n =Sigmoid(Conv(DWConv(f n ))) Among them, {f n |n∈[1,N]} is the intermediate feature of N light sources, DWConv(·) represents the deep convolutional network, and Conv(·) represents the standard convolutional network; Multiply the information score masks of N intermediate features by the corresponding intermediate features element by element, and add the product results to the corresponding intermediate features element by element to obtain N new feature maps; Compare the N new feature maps, and the fused light source fusion feature f q The patch of each image pixel space position is filled by the maximum value of the patch at the corresponding image pixel space position in the N new feature maps: f q =φ({f n +Q n ·f n |n∈[1,N]}) Where φ(·) represents the maximum value of each image pixel spatial position; Model training steps: Taking multiple sets of images of the sample at different positions, each set of images is an image of a multi-light source mode taken at the same position; Perform defect location mask annotation for each set of images to obtain a multi-light source annotation dataset; The multi-light source fusion segmentation model is trained based on the multi-light source labeled dataset, and the training process is iteratively optimized to obtain a trained multi-light source fusion segmentation model. Defect detection steps: Receive the real-time multi-light source image stream of the product, input it into the trained multi-light source fusion segmentation model, and output the real-time detection results of product surface defects.

2. The surface defect detection method based on multi-light source fusion according to claim 1, characterized in that: The multi-light source fusion segmentation model includes: The encoder is an N-branch M-level encoder, wherein the N branches respectively receive images of N light source modes taken at the same position, and the N branches respectively extract semantic information of the images at different scales, and the N branches share parameters; the M levels correspond to M spatial selectors; M>1, N>1.

3. The surface defect detection method based on multi-light source fusion according to claim 2, characterized in that: Each branch consists of M stages; each stage contains Overlap Patch Embedding and Transformer block. The input patch of each branch passes through M stages in sequence. The number of channels in each stage is different, and M-level multi-light source intermediate features with different image resolutions are obtained.

4. The surface defect detection method based on multi-light source fusion according to claim 1, characterized in that: The iterative optimization training process includes the steps of setting the optimization strategy: Use Lion optimization algorithm; A weighted combination of Cross-Entropy loss and Lovász-Softmax Loss is used as the loss function to optimize the Acc and IoU indicators; The Warm Up learning rate strategy is adopted. In the initial stage of training, the LinearLR strategy is used until the learning rate reaches the peak. Then, the learning rate is gradually reduced according to the PolyLR strategy until the learning rate reaches 1×e after the last iteration. -8 .

5. The surface defect detection method based on multi-light source fusion according to claim 1, characterized in that: The step of taking multiple sets of images of the sample at different positions includes: Each time the light source is switched, the sending of the shooting instruction is automatically triggered. A group of images with different light source modes are captured for the same shooting position to obtain a group of multi-light source images.

6. The surface defect detection method based on multi-light source fusion according to claim 1, characterized in that: The step of masking defect positions for each set of images includes: Marking defect positions on the multi-light source image to generate a mask image with the same size as the multi-light source image; The mask images of the same set of multi-light source images are merged to make the marking of the defect positions in the same set of multi-light source images consistent.

7. A surface defect detection system based on multi-light source fusion and a surface defect detection method based on multi-light source fusion according to any one of claims 1 to 6, characterized in that: include: The multi-light source image acquisition module is used to capture multiple sets of images of the sample at different locations. Each set of images is a multi-light source mode image captured at the same location, which serves as sample data for constructing a multi-light source annotation dataset. and controlling the camera to automatically capture the multi-light source image stream of the product as the light source array changes; Image detection module, including: The model construction unit includes the steps of constructing a multi-light source fusion segmentation model based on an encoder-spatial selector-decoder framework: the encoder is used to extract multi-layer semantic information of the multi-light source image to obtain multi-light source intermediate features at multiple scales; the spatial selector is used to fuse the multi-light source intermediate features based on the information score mask of each multi-light source intermediate feature to obtain light source fusion features corresponding to each scale; the decoder is used to fuse the light source fusion features corresponding to each scale to obtain a detection result; The model training unit trains the multi-light source fusion segmentation model based on the multi-light source annotation dataset, and iteratively optimizes the training process to obtain a trained multi-light source fusion segmentation model; A defect detection unit is configured to receive a real-time multi-light source image stream of a product, input the stream into the trained multi-light source fusion segmentation model, and output a real-time detection result of surface defects of the product; The model deployment module is used to convert the multi-light source fusion segmentation model into an accelerated model that is easy to deploy on the production line, and generate a production line report based on the real-time detection results.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The memory is used to store product multi-light source image streams, multi-light source fusion segmentation models, and production line data; when the processor executes the program, the steps of the surface defect detection method based on multi-light source fusion as described in any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium, characterized in that A computer program and a multi-light source fusion segmentation model are stored thereon, and when the computer program is executed by a processor, the steps of the surface defect detection method based on multi-light source fusion as described in any one of claims 1 to 6 are implemented.

Images