Surface defect detection method and system based on multi-light-source fusion and related products

Abstract

The invention discloses a surface defect detection method and system based on multi-light source fusion and a related product. The method comprises the following steps: collecting multiple groups of images of a sample to be detected under different light source conditions; marking image defects to obtain a marked data set, and combining the same group of multi-light-source image masks; establishing a multi-light-source fusion segmentation model with multi-image input, multi-feature fusion and single-image output; training the fusion segmentation model based on a multi-light-source data set, and optimizing a training process; the trained model is converted into an acceleration model for production line deployment, and nondestructive, real-time and automatic detection of product surface defects is achieved. According to the method, key information of various defects can be comprehensively captured through the changing light source, and the problems that a traditional industrial quality inspection method is low in efficiency and insufficient in precision are solved.

Description

Technical Field

[0001] The present invention relates to the field of industrial quality inspection technology, and in particular to a surface defect detection method, system and related products based on multi-light source fusion. Background Art

[0002] Product defect detection is crucial in quality control in industrial manufacturing. In the complex industrial manufacturing process, mechanical products may have various defects such as abrasions, scratches, pits and inner holes due to improper design, equipment failure or harsh working environment. These defects not only damage the appearance and quality of the product, but also may have a negative impact on the performance and service life of the product. The use of automatic defect detection technology can not only improve the quality of manufactured products, but also reduce production costs.

[0003] Traditional methods use feature extraction, template matching, edge detection and other technologies to extract and detect defects, but these methods are not universal in the processing of complex defects, large lighting changes and large-scale data. Usually, manual parameter adjustment is required according to the situation, which is very time-consuming and labor-intensive. Advanced deep learning technology can better adapt to complex defect scenarios and perform well in large-scale data. However, in actual production environments, lighting conditions and environmental changes can interfere with the visual appearance of defects. This requires the defect detection system to be able to adaptively adjust the lighting, increase the effective imaging and feature information of the defects, and require the detection model to have sufficient generalization ability to adapt to different products, different defects and different lighting conditions. At present, multi-light source detection technology is widely used in the field of industrial quality inspection due to its strong robustness and adaptability.

[0004] In the existing multi-light source defect detection technology, there are two main strategies for processing multi-light source images. One is to fuse images under different light sources first, and then perform defect detection on the fused image. This method has high computational complexity, involves complex image alignment and fusion algorithms, and requires high processing time and hardware resources. The other is to perform defect detection on images under each light source separately, and then conduct a comprehensive evaluation of multiple detection results. This method requires independent detection of each image, resulting in a waste of computing resources, and the integration of results may produce contradictions. In view of the many limitations of existing methods in dealing with complex industrial environments, the market urgently needs an automated surface defect detection solution that is highly adaptable, real-time, and avoids multiple repeated detections.

[0005] Therefore, how to provide a surface defect detection method, system and related products based on multi-light source fusion with high efficiency and high adaptability is a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0006] In view of the above research status and existing problems, the present invention provides a surface defect detection method, system and related products based on multi-light source fusion, aiming to solve the shortcomings of traditional defect detection methods such as low efficiency, poor real-time performance and insufficient precision. In addition, by introducing multi-light source fusion technology, the model can show excellent adaptability, generalization and robustness in complex environments and complex defects.

[0007] The present invention first provides a surface defect detection method based on multi-light source fusion, comprising the following steps:

[0008] Steps to build a multi-light source fusion segmentation model based on the encoder-spatial selector-decoder framework:

[0009] The encoder is used to extract multi-layer semantic information of the multi-light source image and obtain multi-light source intermediate features of multiple scales;

[0010] The spatial selector fuses the multiple light source intermediate features based on the information score mask of each of the multiple light source intermediate features to obtain light source fusion features corresponding to each scale;

[0011] The decoder is used to fuse the light source fusion features corresponding to each scale to obtain a detection result;

[0012] Model training steps:

[0013] Taking multiple sets of images of the sample at different positions, each set of images is an image in a multi-light source mode taken at the same position;

[0014] Perform defect position mask annotation for each group of images to obtain a multi-light source annotation dataset;

[0015] The multi-light source fusion segmentation model is trained based on the multi-light source annotation dataset, and the training process is iteratively optimized to obtain a trained multi-light source fusion segmentation model;

[0016] Defect detection steps:

[0017] The real-time multi-light source image stream of the product is received and input into the trained multi-light source fusion segmentation model, and the real-time detection result of the surface defects of the product is output.

[0018] Preferably, the multi-light source fusion segmentation model includes:

[0019] The encoder is an N-branch M-level encoder, wherein the N branches respectively receive images of N light source modes taken at the same position, the N branches respectively extract semantic information of the images at different scales, and the N branches share parameters; the M levels correspond to M spatial selectors; M>1, N>1.

[0020] Preferably, the space selector comprises an STS module; wherein,

[0021] The STS module receives the intermediate features of the N light sources extracted by the N branches as input and calculates the information score mask Q of each intermediate feature. n , the information score mask represents the importance score of each spatial position in the intermediate feature:

[0022] Q n =Sigmoid(Conv(DWConv(f n )))

[0023] Among them, {f n |n∈[1,N]} is the intermediate feature of N light sources, DWConv(·) represents the deep convolutional network, and Conv(·) represents the standard convolutional network;

[0024] Multiply the information score masks of N intermediate features by the corresponding intermediate features element by element, and add the product results to the corresponding intermediate features element by element to obtain N new feature maps;

[0025] The N new feature maps are compared, and the fused light source fusion feature f q The patch at each image pixel spatial position is filled with the maximum value of the patch at the corresponding image pixel spatial position in the N new feature maps:

[0026] f q =φ({f n +Q n ·f n |n∈[1,N]})

[0027] Here, φ(·) represents the maximum value of each image pixel spatial position.

[0028] Preferably, each branch is composed of M stages; each stage includes Overlap Patch Embedding and Transformer block, and the input patch of each branch passes through M stages in sequence, and the number of channels of each stage is different, so as to obtain M-level multi-light source intermediate features of different image resolutions.

[0029] Preferably, the iterative optimization training process includes the steps of setting an optimization strategy:

[0030] Use Lion optimization algorithm;

[0031] A weighted combination of Cross-Entropy loss and Lovász-Softmax Loss is used as the loss function to optimize the Acc and IoU indicators;

[0032] The Warm Up learning rate strategy is adopted. The LinearLR strategy is used in the initial stage of training until the learning rate reaches the peak. Then the PolyLR strategy is used to gradually reduce the learning rate until the learning rate reaches 1×e after the last iteration. -8 .

[0033] Preferably, the step of taking multiple sets of images of the sample at different positions comprises:

[0034] Each time the light source is switched, the sending of the shooting command is automatically triggered, and a group of images with different light source modes are shot for the same shooting position screen to obtain a group of multi-light source images.

[0035] Preferably, the step of masking defect positions for each group of images comprises:

[0036] Marking defect positions on the multi-light source image to generate a mask image with the same size as the multi-light source image;

[0037] The mask images of the same group of multi-light source images are merged to make the marking of the defect positions in the same group of multi-light source images consistent.

[0038] The present invention also provides a surface defect detection system based on multi-light source fusion according to the surface defect detection method based on multi-light source fusion, comprising:

[0039] A multi-light source image acquisition module is used to take multiple sets of images of a sample at different positions, each set of images being images of a multi-light source mode taken at the same position, as sample data for constructing a multi-light source annotation data set; and to control the camera to automatically acquire the multi-light source image stream of the product as the light source array changes;

[0040] Image detection module, including:

[0041] The model building unit includes the steps of building a multi-light source fusion segmentation model based on an encoder-spatial selector-decoder framework: the encoder is used to extract multi-layer semantic information of a multi-light source image to obtain multi-light source intermediate features at multiple scales; the spatial selector fuses the multi-light source intermediate features based on the information score mask of each of the multi-light source intermediate features to obtain light source fusion features corresponding to each scale; the decoder is used to fuse the light source fusion features corresponding to each scale to obtain a detection result;

[0042] A model training unit trains a multi-light source fusion segmentation model based on a multi-light source annotation dataset, and iteratively optimizes the training process to obtain a trained multi-light source fusion segmentation model;

[0043] A defect detection unit, used to receive a real-time multi-light source image stream of a product, input it into the trained multi-light source fusion segmentation model, and output a real-time detection result of a product surface defect;

[0044] The model deployment module is used to convert the multi-light source fusion segmentation model into an acceleration model that is easy to deploy on the production line, and generate a production line report based on the real-time detection results.

[0045] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein the memory is used to store a product multi-light source image stream, a multi-light source fusion segmentation model, and production line data; when the processor executes the program, the steps of the surface defect detection method based on multi-light source fusion are implemented.

[0046] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program and a multi-light source fusion segmentation model are stored. When the computer program is executed by a processor, the steps of the surface defect detection method based on multi-light source fusion are implemented.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. The present invention uses an editable stereo light source dedicated to defect detection, which effectively reduces the need for frequent light source replacement and repeated shooting in traditional multi-light source detection, thereby greatly improving detection efficiency. The stereo light source is used to create a variety of different light source effects, effectively simulating the lighting conditions in different scenes, so that the system can adapt to various complex detection environments.

[0049] 2. Adopting the encoder-spatial selector-decoder structure, a multi-light source feature fusion strategy is designed to achieve comprehensive capture and deep mining of surface defect features from different light sources. This method only needs to feed the multi-light source images into the deep network, and through the fusion of multi-light source intermediate features, the overall defect segmentation result can be obtained. There is no need for complex image alignment and image fusion, and it effectively avoids the waste of resources caused by multiple detections, thus achieving efficient and fast multi-light source defect detection.

[0050] 3. The deep learning method is used for multi-light source feature fusion, which is more adaptive and intelligent than the traditional multi-light source image fusion. First, the traditional fusion strategy needs to be manually adjusted according to the on-site conditions, while the deep learning network can adaptively learn the best feature extraction and fusion strategy, which performs better for complex industrial scenes. Secondly, traditional fusion only considers the direct fusion of local features, while deep fusion can capture global context information, retain local details and facilitate the integrity of defect information. In addition, multi-level feature fusion not only enhances the system's sensitivity to subtle defects, but also ensures that even under complex lighting conditions, it can accurately distinguish and identify various types of defects.

[0051] 4. The system's built-in SegFormer backbone network, with its excellent optimization algorithm, not only accelerates the model training process, but also can quickly and accurately complete defect identification in actual detection, achieving a dual improvement in real-time performance and accuracy. The system's built-in multi-level feature extraction mechanism can automatically adjust to meet the needs of defect detection of different scales, whether it is a tiny crack or a large area of ​​damage, it can achieve accurate identification and show good generalization ability.

[0052] 5. The present invention integrates functional modules such as image acquisition, model training, and model deployment into a system to solve the problem of long industrial deployment project cycle.

[0053] To sum up, the present invention achieves the purpose of surface defect detection of industrial products through deep learning technology, multi-light source feature fusion method, image processing technology, model training optimization, automated detection and other technical means. It has strong practicality and promotion value and can provide better solutions for the field of industrial quality inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work.

[0055] Figure 1 A flow chart of a surface defect detection method based on multi-light source fusion provided by an embodiment of the present invention;

[0056] Figure 2 A general model framework diagram provided for an embodiment of the present invention;

[0057] Figure 3 A schematic diagram of the STS module structure provided in an embodiment of the present invention;

[0058] Figure 4A schematic diagram of the structure of a surface defect detection system based on multi-light source fusion provided by an embodiment of the present invention;

[0059] Figure 5 A schematic diagram of the association of a surface defect detection device based on multi-light source fusion provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] The first aspect of the embodiment of the present invention discloses a surface defect detection method based on multi-light source fusion, such as Figure 1 As shown, the following steps are included:

[0062] Steps to build a multi-light source fusion segmentation model based on the encoder-spatial selector-decoder framework:

[0063] The encoder is used to extract multi-layer semantic information of multi-light source images and obtain multi-light source intermediate features at multiple scales; the spatial selector fuses the multi-light source intermediate features based on the information score mask of each multi-light source intermediate feature to obtain the light source fusion features corresponding to each scale; the decoder is used to fuse the light source fusion features corresponding to each scale to obtain the detection result.

[0064] Model training steps:

[0065] Take multiple groups of images for the sample at different positions, each group of images is an image of a multi-light source mode taken at the same position; mark the defect position mask for each group of images to obtain a multi-light source annotation data set; train the multi-light source fusion segmentation model based on the multi-light source annotation data set, and iteratively optimize the training process to obtain a trained multi-light source fusion segmentation model;

[0066] Defect detection steps:

[0067] Receive the real-time multi-light source image stream of the product, input it into the trained multi-light source fusion segmentation model, and output the real-time detection results of product surface defects.

[0068] In one embodiment, a fusion segmentation network (MFSNet, Multi-light Fusion Segmentation network) of a shared multi-light image feature extraction encoder, a feature fuser, and a feature decoder is constructed, including:

[0069] The encoder is an N-branch M-level encoder. The N branches receive images of N light source modes taken at the same position respectively. The N branches extract semantic information from the images at different scales respectively, and the N branches share parameters; the M levels correspond to M spatial selectors; M>1, N>1.

[0070] In this embodiment, Figure 2 As shown in the figure, based on the SegFormer framework, an N-branch four-stage encoder based on MiT is established as the backbone network of the multi-light source fusion segmentation model. In the four-stage encoder, four feature maps of different scales are obtained, which contain feature information of different scales. Due to the multi-branch design, each branch is responsible for extracting features of a specific scale under the corresponding illumination mode, so the feature map output at each stage can better capture the different details of the image.

[0071] The encoder and decoder are connected by a spatial token selector (STS). The four stages correspond to four spatial token selectors, which fuse the extracted multi-light source intermediate features to generate a single feature map. In particular, N branches extract multi-layer semantic information from multi-light source images, and multiple branches share parameters.

[0072] It should be noted that the purpose of setting encoder shared parameters is to reduce model parameters and improve training efficiency. This is achieved by instantiating an identical module object. For example, define a shared module share_module, and N branches reuse the same module. This ensures that the weights of share_module are exactly the same in different branches and are updated together during training:

[0073]

[0074] SegFormer Head acts as a decoder to fuse the four feature maps obtained in the four stages to obtain the prediction result.

[0075] In this embodiment, the encoder and decoder in the MFSNet model can be replaced with any segmentation model with an encoder-decoder structure, such as SegFormer, SegNeXt, DeepLab, etc., according to the characteristics of the defects of the product to be inspected. This embodiment only shows one of the models. Considering the accuracy and speed issues comprehensively, SegFormer is selected as the default model after multiple tests. It has both high accuracy and high detection speed. In addition, the number of spatial label selectors depends on the number of input feature maps of the decoder and has nothing to do with the number of branches. For example, the decoder input features of DeepLabV3+ are two, then two corresponding spatial label selectors are needed to fuse the two levels of feature maps respectively to obtain the two feature maps required by the decoder.

[0076] In this embodiment, if Figure 3 As shown, the space selector includes an STS module; wherein,

[0077] First, the input of the STS module is a set of N feature maps f from different light sources. n |n∈[1,N],f n ∈H×W×C}, each feature map has extracted specific spatial features through the processing of the previous layers of the network. These feature maps represent the representation of the same object under different lighting conditions and contain rich diversity information. For each input feature map f n Apply the STS module to calculate the information score mask Q of each intermediate feature n , the information score mask represents the importance score of each spatial position in the intermediate feature:

[0078] Q n =Sigmoid(Conv(DWConv(f n )))

[0079] Among them, {f n |n∈[1,N]} is the intermediate feature of N light sources, DWConv(·) represents the deep convolutional network, and Conv(·) represents the standard convolutional network;

[0080] Then, the feature map enhancement operation is performed. The information score mask of the N intermediate features is multiplied element by element with the corresponding intermediate features. This operation enhances important features while suppressing unimportant positions. The product result is added element by element with the corresponding intermediate feature to obtain N new feature maps;

[0081] Finally, the feature map fusion operation is performed. N new feature maps are compared, and the fused light source fusion feature f q The patch at each image pixel spatial position is filled with the maximum value of the patch at the corresponding image pixel spatial position in the N new feature maps:

[0082] f q =φ({f n +Q n ·f n |n∈[1,N]})

[0083] Among them, φ(·) represents the maximum value of each image pixel spatial position. In this way, the model not only strengthens the features of the key areas, but also selects the most effective patch in each spatial position of the N features, so that the model focuses on the key areas of the object. The final output feature map contains the best feature information under different light sources, with higher expressiveness and robustness.

[0084] It should be noted that a "patch" refers to a smaller local area extracted from the original image or feature map. It is used to describe a small set of pixels in an image or feature map, usually square and of a fixed size. When fusing N feature maps, each feature map is composed of many such small blocks (patches). The fusion operation selects the maximum value of the N patches at each position to form a new fused feature map. This is done to retain the most representative information in each feature map while removing redundant or unimportant parts.

[0085] In one embodiment, each branch is composed of M stages; each stage includes OverlapPatch Embedding and Transformerblock, and the input patch of each branch passes through M stages in sequence, each stage has a different number of channels, and M-level multi-light source intermediate features of different image resolutions are obtained.

[0086] In this embodiment, N input images of size H×W×3 enter N branches with shared parameters respectively; the input patch of each branch passes through 4 stages with channels of (64, 128, 256, 512) and corresponding stacking ratios of [2, 2, 2, 2] in turn to obtain multi-level features of the original image resolution {1 / 4, 1 / 8, 1 / 16, 1 / 32}; the multi-branch features obtained in each stage are filtered and selected through an STS module, and finally fused to obtain 4 feature maps of different resolutions with multi-light source properties.

[0087] The specific implementation process is as follows:

[0088] Taking three input images of 2448×2048×3 as an example, the horizontal and vertical ratios are locked, the input images are reduced to 612×512×3, and normalized. Then the three images enter three encoder branches with shared parameters respectively. The Overlap Patch Embedding of Stage-1 is a 7×7 convolution with a stride of 4. Stage-2, Stage-3, and Stage-4 are all 3×3 convolutions with a stride of 2. By reducing the resolution and increasing the number of channels, information of different scales is extracted layer by layer. The Transformerblock of each stage includes a multi-head self-attention mechanism (MHSA) and a mixed feedforward network (Mix-FFN).

[0089] MHSA introduces a reduction rate R to process features in blocks, reducing the complexity of the self-attention mechanism. The formula is as follows:

[0090]

[0091] Where N = H × W is the length of the sequence, K is the sequence to be reduced; Reshape(N / R,C·R)(K) means reshaping K into a sequence of shape N / R×(C·R); Linear(C1,C2)(·) means that the linear layer takes a C1-dimensional tensor as input and generates a C2-dimensional tensor as output. R is set to [64,16,4,1] for Stage-1 to Stage-4.

[0092] The formula of the multi-head self-attention mechanism is as follows:

[0093]

[0094] Among them, each head Q, K, V has the same size N×C.

[0095] Mix-FFN consists of a 3×3 deep convolution and two multi-layer perceptrons (MLP). The deep convolution is used for local feature extraction, and the multi-layer perceptron is applied to the channel dimension, and the features are linearly transformed through the fully connected layer. The formula is as follows:

[0096] x out =MLP(GELU(Conv 3×3 (MLP(x in ))))+x in

[0097] where x in is the feature from the self-attention module, and GELU represents the Gaussian Error Linear Unit activation function.

[0098] After the above four stages, the three branches all obtain 12 feature maps with four different resolutions. Then, four spatial tag selectors are connected to fuse the feature maps from the four stages respectively, and finally four feature maps with different resolutions are extracted.

[0099] In one embodiment, the step of taking multiple sets of images of the sample at different positions includes:

[0100] Each time the light source is switched, the sending of the shooting command is automatically triggered, and a group of images with different light source modes are shot for the same shooting position screen to obtain a group of multi-light source images.

[0101] In this embodiment, N lighting methods that are most suitable for product defects are explored and determined; the light source controller controls the industrial camera to take pictures through the serial port, and sends a signal to the industrial camera to take a picture every time the light source is switched, and a group of N multi-light source images are taken for the same picture at one time.

[0102] When executing:

[0103] First, explore the best lighting method for defects. In order to ensure that all defects can be clearly presented, it is necessary to continuously adjust the luminous angle and brightness of the stereo light source according to the characteristics of the surface defects of the product to be inspected. Through experiments, N optimal lighting methods are determined and these light source modes are saved.

[0104] Then, during the acquisition process, the system will automatically switch the corresponding light source mode, and the light source controller will control the camera to take pictures through the serial port. Each time the light source is switched, a signal will be sent to the camera to take a picture, and a group of N multi-light source images will be taken for the same picture at one time.

[0105] After completing a set of light source photos, adjust the product's inspection position or change the inspection product, and repeat the acquisition operation until all images are acquired. The same set of light source images are saved in the same folder for subsequent identification and processing.

[0106] In one embodiment, the step of marking defect position masks for each set of images includes:

[0107] The defect positions of the multi-light source images are marked to generate a mask image with the same size as the multi-light source images; the mask images of the same group of multi-light source images are merged to make the marking of the defect positions in the same group of multi-light source images consistent, thereby avoiding missing any defects.

[0108] In this embodiment, the saved mask image is a grayscale image, the grayscale value represents the label value of the corresponding defect type, and the background part is marked as 0 by default.

[0109] It should be noted that the operation of merging mask images of the same set of multi-light source images is intended to achieve the following result: if a pixel position is marked as a defect on any mask image in a set of masks, then the position will also be marked as a defect on the final merged mask image.

[0110] Under normal circumstances, a specific defect will appear in a state of presence or absence under different light sources, and will not develop into other defects, so defects can be merged in an OR manner. For example, if there are three mask images, recorded as M1, M2, and M3, then the merged mask M can be expressed by a logical operation: M = M1OR M2 OR M3. In special cases, for possible overlapping areas of different defects, image comparison and consistency annotation can be used to eliminate uncertainty and reduce misjudgments or omissions.

[0111] In this embodiment, the SAM (Segment Anything Model) model is introduced for automatic annotation. The SAM model can automatically generate high-quality segmentation results, which can greatly improve the annotation efficiency, especially when processing a large amount of image data. The annotation accuracy will be higher than the traditional manual method, reducing the occurrence of annotation errors. Using this annotation software, the user only needs to click a small number of points or boxes on the image as prompts, and the model can automatically complete the remaining defect segmentation annotations.

[0112] In this embodiment, the generated multi-light source data set is divided into a training set and a validation set according to a certain ratio for model training. For example, the generated multi-light source data set is divided into a training set and a validation set according to a certain ratio (for example, 4:1). In the case of less defect data, data enhancement technology can be used to expand the data set to improve the robustness and generalization ability of the model. Specific data enhancement methods include random rotation, random flipping, color jitter, noise superposition, etc.

[0113] In one embodiment, the iterative optimization training process includes the steps of setting the optimization strategy:

[0114] Use Lion optimization algorithm;

[0115] A weighted combination of Cross-Entropy loss and Lovász-Softmax Loss is used as the loss function to optimize the Acc and IoU indicators;

[0116] The Warm Up learning rate strategy is adopted. The LinearLR strategy is used in the initial stage of training until the learning rate reaches the peak. Then the PolyLR strategy is used to gradually reduce the learning rate until the learning rate reaches 1×e after the last iteration. -8 :

[0117] Loss = βLoss lovász+(1-β)Loss ce

[0118] Wherein, β is the loss balance factor, and β is set to 0.1.

[0119] In this embodiment, a preprocessing process is also included before training: the constructed multi-light source dataset is preprocessed, including formatting and online data enhancement, including but not limited to cropping, random flipping, random resizing, and normalization.

[0120] The preprocessed multiple images are sent to the optimized MFSNet network for training.

[0121] In this embodiment, the training also includes an evaluation process: running the trained model on the test set (and validation set) to generate a detailed evaluation report including key indicators such as IoU (intersection over union ratio), RAcc (detection rate), FPR (false positive rate), FNR (missed positive rate), etc. Through indicator analysis, it is judged whether the performance of the model meets the expected requirements. If it does not meet the expectations, the user is prompted to readjust the data set or retrain the model parameters. If it meets the expectations, the model deployment operation is entered.

[0122] In one embodiment, the multi-light source fusion segmentation model is converted into a format suitable for production line deployment, such as ONNX, TensorRT, etc., to accelerate the reasoning process. The converted model is integrated into the detection system of the production line, and the integration process includes docking with the camera, light source control system, data acquisition module, and industrial automation system.

[0123] The multi-light source images collected in real time are input into the acceleration model for inference. The model will output defect detection results, realize non-destructive, real-time and automated detection of product surface defects, and perform corresponding operations (such as removing unqualified products) through the industrial control system.

[0124] Over time, the defect type or characteristics of the product may change, resulting in a decrease in model performance. Therefore, the model needs to be updated and retrained regularly. On the basis of the original data set and model, a new data set is added to retrain the model according to the steps of defect location mask annotation and iterative optimization of the training process. The retrained new model is converted and deployed after pruning, quantization and other optimization methods. Through continuous model updates and maintenance, the long-term effectiveness and stability of the production line defect detection system are ensured. All operations are carried out in this set of methods and systems, which speeds up the process of model update and iteration on the production line.

[0125] Based on the same inventive idea as the first aspect of the embodiment, Figure 4 As shown, the second aspect of the embodiment of the present invention provides a surface defect detection system based on multi-light source fusion, including:

[0126] The multi-light source image acquisition module is used to take multiple sets of images of the sample at different positions, each set of images is an image of a multi-light source mode taken at the same position, which is used as sample data for constructing a multi-light source annotation data set; and control the camera to automatically collect the multi-light source image stream of the product as the light source array changes;

[0127] Image detection module, including:

[0128] The model building unit includes steps of building a multi-light source fusion segmentation model based on an encoder-spatial selector-decoder framework: the encoder is used to extract multi-layer semantic information of the multi-light source image and obtain multi-light source intermediate features at multiple scales; the spatial selector fuses the multi-light source intermediate features based on the information score mask of each multi-light source intermediate feature to obtain light source fusion features corresponding to each scale; the decoder is used to fuse the light source fusion features corresponding to each scale to obtain the detection result;

[0129] A model training unit trains the multi-light source fusion segmentation model based on the multi-light source annotation dataset, and iteratively optimizes the training process to obtain a trained multi-light source fusion segmentation model;

[0130] The defect detection unit is used to receive the real-time multi-light source image stream of the product, input it into the trained multi-light source fusion segmentation model, and output the real-time detection results of the product surface defects;

[0131] The model deployment module is used to convert the multi-light source fusion segmentation model into an accelerated model that is easy to deploy on the production line, and generate production line reports based on real-time detection results.

[0132] In one embodiment, the multi-light source image acquisition module includes visual light source software, stereo light source, industrial camera, bracket, and test bench; stereo light source and industrial camera are used to collect multiple sets of images of the sample to be tested under different light source conditions, and the parameters of the light source and camera are adjusted by using visual light source software, and the lighting angle and brightness are adjusted according to the defect characteristics, and the required light source array is saved; continuous multi-light source images can be easily collected in the automatic light cutting mode. During model deployment and inference, the stereo light source and industrial camera need to be used in conjunction with serial communication to ensure that the camera can collect a complete set of multi-light source images as the light source array changes.

[0133] In one embodiment, the image detection module also includes an image preprocessing module for preprocessing the constructed multi-light source data set, including formatting and online data enhancement, including but not limited to cropping, random flipping, random resizing, and normalization. Specifically, it includes: performing fixed cropping on the image, removing redundant large-area background, and then performing online data enhancement and normalization on the image before each training iter to enhance generalization.

[0134] In one embodiment, the image detection module also includes a result display and analysis module, which is used to plot the results of the model reasoning test set and generate a table containing model evaluation indicators such as IoU, RAcc, FPR, FNR, etc. for user reference.

[0135] In one embodiment, the model deployment module includes:

[0136] The model conversion unit is used to optimize the trained model and convert it into other accelerated model formats such as TensorRT or ONNX.

[0137] The production line simulation unit is used to simulate the real-time detection environment of the production line, receive multi-light source image streams, inference images and discover defects from the camera in real time, and generate production line reports to facilitate viewing and verifying the defect detection effect of the model.

[0138] The model deployment module receives the multi-light source image stream on the production line and inputs it into the acceleration model for processing, thus achieving non-destructive, real-time and automated detection of product surface defects.

[0139] In this embodiment, the three-dimensional light source can adopt an editable three-dimensional light source for defect detection provided by Guangzhou Xuanshijia Electronic Technology Co., Ltd. The 48 light-emitting channels are attached to the inner wall of the dome-shaped structure, which can light up the LED arrays at different positions and adjust the brightness parameters, so as to realize multi-angle lighting of the three-dimensional light source, so that defects of different types and positions can be presented.

[0140] In this embodiment, the industrial camera can be a Hikvision (model) industrial camera, which has a Micro-B3.0 interface, meets the USB3.0 communication protocol of the system, has a camera with 5 million pixels, a resolution of 2448×2048, and a frame rate of 70fps. The camera can clearly capture industrial surface defects.

[0141] Based on the same inventive idea as the method described in the first aspect of the embodiment, the third aspect of the embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and run on the processor, the memory being used to store a product multi-light source image stream, a multi-light source fusion segmentation model, and production line data; the steps of a surface defect detection method based on multi-light source fusion are implemented when the processor executes the program.

[0142] The solution of the present invention can be implemented on a computer device with data query capability, and executed by the computer device, wherein the computer device can be a terminal or a server. The terminal can be a smart phone, a tablet computer, a laptop, a touch screen, a game console, a personal computer (PC), a personal digital assistant (PDA) and other terminal devices, but is not limited thereto.

[0143] In this embodiment, Figure 5 As shown, the stereo light source, industrial camera, electronic device and transmission device are connected in communication. The communication process between the stereo light source, industrial camera, computing device and transmission device includes:

[0144] First, the conveying equipment delivers the product to be inspected to the designated inspection position. After the product is in place, it sends a signal to the light source controller that it is in place.

[0145] Then, the light source that receives the signal will automatically switch the light source mode and send a signal to the camera to take pictures every time the light is cut off, ensuring that each light source change can effectively take pictures.

[0146] Then, the obtained image stream is acquired by the computing device and sent to the inference model for defect detection to obtain the detection results.

[0147] Finally, the computing device transmits the sorting information of whether the product is qualified to the conveying device, instructing it on the next step of operation.

[0148] In this embodiment, the electronic device may be a computer equipped with Intel Core(TM)

[0149] The system consists of a host computer with an I9-13900xCPU@5.8GHz (32GB memory) and an NVIDIA GeForce RTX-4090 graphics processing unit (GPU) (24GB memory). The processor can be a CPU or GPU processor. The input devices include a display, a keyboard and a mouse. The display is used to display the test results in real time, and the keyboard and mouse are used to perform various human-computer interaction operations.

[0150] In this embodiment, the conveying device may be an auxiliary device such as a conveyor belt or a robotic arm, which is used to accurately convey the product to be inspected to the inspection position, receive the sorting signal from the host, and perform corresponding actions.

[0151] Based on the same inventive idea as the method described in the first aspect of the embodiment, the fourth aspect of the embodiment of the present invention provides a non-transitory computer-readable storage medium, on which a computer program and a multi-light source fusion segmentation model are stored. When the computer program is executed by a processor, the steps of the surface defect detection method based on multi-light source fusion are implemented.

[0152] The electronic device and non-transitory computer-readable storage medium and method provided in the embodiments of the present invention correspond one to one. Therefore, the electronic device and the non-transitory computer-readable storage medium also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the device and the medium will not be repeated here.

[0153] The following is a specific example to illustrate the implementation process of the above surface defect detection method and system. Taking the mobile phone shell defect detection as an example, it is specifically divided into the following steps:

[0154] Step 1: Sample preparation and light source adjustment. The robot arm picks up the defective samples of the mobile phone case from the conveyor belt, moves the samples under the camera and stereo light source, adjusts the posture of the mobile phone case and the focal length of the camera, and ensures that the details on the surface of the mobile phone case are clearly visible. The visual light source software adjusts the luminous angle and brightness of the stereo light source to clearly display the surface defects, and saves the three best light source array configurations to the light source controller for subsequent collection and detection.

[0155] Step 2: Multi-light source image acquisition. Set the stereo light source to automatic light-cutting mode, so that it gradually switches the light source at the preset angle and brightness. Control the robotic arm to rotate the phone case in sequence, capture the multi-light source images of the four sides and four corners of each sample, and save them to the host computer.

[0156] Step 3: Dataset construction. The robot arm takes the next mobile phone case sample and repeats the second step until enough multi-light source image data of the mobile phone case is collected. Defects such as scratches, dirt, and white spots are annotated using annotation software, and then image masks of the same picture under different lighting conditions are merged. The data set is then divided into training set, validation set, and test set according to the ratio (e.g. 8:1:1).

[0157] Step 4: Model training. Enter the number of multi-light source branches, start the system's built-in multi-light source fusion segmentation model, and train the multi-light source image dataset of the mobile phone case. The model gradually optimizes the parameters and finally obtains the best model file.

[0158] Step 5: Model evaluation. The system runs the trained model on the test set (and validation set) and generates a detailed evaluation report including key indicators such as IoU, RAcc, FPR, FNR, etc. Through indicator analysis, it is determined whether the performance of the model meets the expected requirements. If it does not meet the expectations, the user is prompted to readjust the data set or retrain the model parameters. If it meets the expectations, proceed to the next step.

[0159] Step 6: Model conversion and deployment. Convert the best model file obtained through training into TensorRT format and load it into the system to achieve accelerated reasoning of the model.

[0160] Step 7: Real-time detection. The conveyor belt delivers the phone case to be tested to the test area, and the robotic arm accurately positions the phone case at the test position. After the robotic arm is in place, it sends a signal to the light source controller, which starts switching the light source and captures multi-light source images after receiving the command. The captured images are transmitted to the host computer in real time, and the acceleration model is used for defect detection. The test results are displayed in real time, and the defective areas are marked.

[0161] Step 8: All-round inspection and decision-making. The robot arm rotates the phone case, and repeats the inspection operation by rotating the robot arm and switching the light source to ensure that the multi-light source images of the four sides and four corners of the phone case are inspected. After the inspection is completed, the phone case is put back on the conveyor belt. The upper computer comprehensively judges whether the phone case has defects based on the inspection results, and issues corresponding instructions to control the direction of the conveyor belt to decide whether the phone case will enter the next step of processing or be marked as a defective product.

[0162] Step 9: Batch testing and report generation. Repeat steps 6 and 7 until all mobile phone cases are tested. The system generates a production line report for users to view.

[0163] Compared with traditional industrial quality inspection technology, the present invention can effectively reduce the consumption of manpower and material resources, and through multi-light source fusion technology, utilizes complementary information under different lighting conditions, greatly improving the adaptability, generalization and robustness of detection, showing great practical application potential.

[0164] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] The above is a detailed introduction to a surface defect detection method, system and related products based on multi-light source fusion provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0166] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

Claims

1. A surface defect detection method based on multi-light source fusion, characterized in that: The steps include: Steps to build a multi-light source fusion segmentation model based on the encoder-spatial selector-decoder framework: The encoder is used to extract multi-layer semantic information of the multi-light source image and obtain multi-light source intermediate features of multiple scales; The spatial selector fuses the multiple light source intermediate features based on the information score mask of each of the multiple light source intermediate features to obtain light source fusion features corresponding to each scale; The decoder is used to fuse the light source fusion features corresponding to each scale to obtain a detection result; Model training steps: Taking multiple sets of images of the sample at different positions, each set of images is an image in a multi-light source mode taken at the same position; Perform defect position mask annotation for each group of images to obtain a multi-light source annotation dataset; The multi-light source fusion segmentation model is trained based on the multi-light source annotation dataset, and the training process is iteratively optimized to obtain a trained multi-light source fusion segmentation model; Defect detection steps: The real-time multi-light source image stream of the product is received and input into the trained multi-light source fusion segmentation model, and the real-time detection result of the surface defects of the product is output.

2. The surface defect detection method based on multi-light source fusion according to claim 1 is characterized in that: The multi-light source fusion segmentation model includes: The encoder is an N-branch M-level encoder, wherein the N branches respectively receive images of N light source modes taken at the same position, the N branches respectively extract semantic information of the images at different scales, and the N branches share parameters; the M levels correspond to M spatial selectors; M>1, N>1.

3. The surface defect detection method based on multi-light source fusion according to claim 2 is characterized in that: The space selector includes an STS module; wherein, The STS module receives the intermediate features of the N light sources extracted by the N branches as input and calculates the information score mask Q of each intermediate feature. n , the information score mask represents the importance score of each spatial position in the intermediate feature: Q n =Sigmoid(Conv(DWConv(f n ))) Among them, {f n |n∈[1,N]} is the intermediate feature of N light sources, DWConv(·) represents the deep convolutional network, and Conv(·) represents the standard convolutional network; Multiply the information score masks of N intermediate features by the corresponding intermediate features element by element, and add the product results to the corresponding intermediate features element by element to obtain N new feature maps; The N new feature maps are compared, and the fused light source fusion feature f q The patch at each image pixel spatial position is filled with the maximum value of the patch at the corresponding image pixel spatial position in the N new feature maps: f q =φ({f n +Q n ·f n |n∈[1,N]}) Here, φ(·) represents the maximum value of each image pixel spatial position.

4. The surface defect detection method based on multi-light source fusion according to claim 2 is characterized in that: Each branch consists of M stages; each stage contains Overlap Patch Embedding and Transformer block. The input patch of each branch passes through M stages in sequence. The number of channels in each stage is different, and M-level multi-light source intermediate features with different image resolutions are obtained.

5. The surface defect detection method based on multi-light source fusion according to claim 1 is characterized in that: The iterative optimization training process includes the steps of setting the optimization strategy: Use Lion optimization algorithm; A weighted combination of Cross-Entropy loss and Lovász-Softmax Loss is used as the loss function to optimize the Acc and IoU indicators; The Warm Up learning rate strategy is adopted. The LinearLR strategy is used in the initial stage of training until the learning rate reaches the peak. Then the PolyLR strategy is used to gradually reduce the learning rate until the learning rate reaches 1×e after the last iteration. -8 .

6. The surface defect detection method based on multi-light source fusion according to claim 1 is characterized in that: The step of taking multiple sets of images for the sample at different positions comprises: Each time the light source is switched, the sending of the shooting command is automatically triggered, and a group of images with different light source modes are shot for the same shooting position screen to obtain a group of multi-light source images.

7. The surface defect detection method based on multi-light source fusion according to claim 1 is characterized in that: The step of masking defect positions for each group of images comprises: Marking defect positions on the multi-light source image to generate a mask image with the same size as the multi-light source image; The mask images of the same group of multi-light source images are merged to make the marking of the defect positions in the same group of multi-light source images consistent.

8. A surface defect detection system based on multi-light source fusion according to any one of claims 1 to 7, characterized in that: include: The multi-light source image acquisition module is used to take multiple sets of images for samples at different positions, each set of images is an image of a multi-light source mode taken at the same position, and serves as sample data for constructing a multi-light source annotation dataset; and controlling the camera to automatically collect the multi-light source image stream of the product as the light source array changes; Image detection module, including: The model building unit includes the steps of building a multi-light source fusion segmentation model based on an encoder-spatial selector-decoder framework: the encoder is used to extract multi-layer semantic information of a multi-light source image to obtain multi-light source intermediate features at multiple scales; the spatial selector fuses the multi-light source intermediate features based on the information score mask of each of the multi-light source intermediate features to obtain light source fusion features corresponding to each scale; the decoder is used to fuse the light source fusion features corresponding to each scale to obtain a detection result; A model training unit trains the multi-light source fusion segmentation model based on the multi-light source annotation dataset, and iteratively optimizes the training process to obtain a trained multi-light source fusion segmentation model; A defect detection unit, used to receive a real-time multi-light source image stream of a product, input it into the trained multi-light source fusion segmentation model, and output a real-time detection result of a product surface defect; The model deployment module is used to convert the multi-light source fusion segmentation model into an acceleration model that is easy to deploy on the production line, and generate a production line report based on the real-time detection results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The memory is used to store product multi-light source image streams, multi-light source fusion segmentation models and production line data; when the processor executes the program, the steps of the surface defect detection method based on multi-light source fusion as described in any one of claims 1-7 are implemented.

10. A non-transitory computer-readable storage medium, characterized in that: A computer program and a multi-light source fusion segmentation model are stored thereon, and when the computer program is executed by a processor, the steps of the surface defect detection method based on multi-light source fusion as described in any one of claims 1 to 7 are implemented.

Images