Mini-LED surface defect detection method and related device

Abstract

The invention belongs to the technical field of target detection in computer vision, and discloses a Mini-LED surface defect detection method and a related device. The Mini-LED surface defect detection method comprises the following steps: acquiring a Mini-LED image of which the surface defect is to be detected; based on the obtained Mini-LED image to be subjected to surface defect detection, performing defect detection by using the trained defect detection model to obtain a surface defect detection result; wherein the defect detection model comprises a data input module, a feature extraction module, a feature fusion module, a relation modeling module and a detection module which are connected in sequence; the relation modeling module comprises a position embedding module and a relation reasoning module. According to the technical scheme, the problem of redundancy detection can be effectively solved, the detection precision and the recall rate can be improved, and missed detection and false detection are reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection in computer vision, and in particular relates to a Mini-LED surface defect detection method and related devices. Background Art

[0002] Submillimeter light-emitting diode (Mini-LED) is a new display technology. Compared with traditional LED, it has smaller size, higher integration, high brightness, fast response and high resolution, and can produce more delicate display effects. Currently, Mini-LED is widely used in advanced fields such as wearable devices, smart medical displays, high-definition televisions and car displays.

[0003] As the physical size of Mini-LEDs shrinks, the difficulty of manufacturing increases, and the yield rate decreases. The rapid and accurate detection of Mini-LED chips has become a key link in industrial production. Explanatory, due to the small size and dense distribution of chips, the feature information of a single target is insufficient, and industrial detection requires algorithms with fast speed and easy deployment. Therefore, the defect detection of Mini-LED chips still faces huge challenges.

[0004] At present, defect detection technology is mainly divided into two directions: traditional machine vision detection and deep learning detection. Traditional manual detection methods have defects such as low efficiency, high cost, and easy human error. With the rapid development of deep learning technology, automatic detection technology has effectively overcome the limitations of traditional manual detection and has gradually become an important part of the production process.

[0005] In the detection field, FasterR-CNN is the representative of two-stage detectors, and SSD and YOLO are typical representatives of single-stage detectors. The two-stage detection model FasterR-CNN divides the detection task into two steps. First, the candidate regions are generated using the region proposal network (RPN), and then classification and bounding box regression are performed on these candidate regions. Although FasterR-CNN performs well in accuracy, its reasoning speed is relatively slow. In contrast, the single-stage detection model aims to achieve target detection through a single forward propagation to significantly improve the speed. Among them, SSD performs detection on different feature layers and can handle targets of different sizes while maintaining a fast reasoning speed. YOLO divides the input image into grids and directly performs bounding box regression and category prediction on the feature map, which has efficient real-time performance. In addition, Transformer-based detection models have gradually attracted attention in recent years. DETR (DEtection TRansformer) regards target detection as a set prediction problem, adopts an end-to-end framework, and effectively captures the long-distance dependencies between objects in the image through the self-attention mechanism. Its innovation is that it simplifies the traditional detection process and eliminates the need for a region generation network, thus providing a new detection idea. Among the above existing models, the YOLO series has the fastest detection speed while ensuring the detection effect, becoming the preferred solution for real-time target detection and providing strong support for industrial defect detection.

[0006] At present, the application of automatic detection technology in industrial defect detection still faces many challenges. Compared with ordinary objects, industrial defects usually show more diverse and complex morphologies, which hinders the performance of general detection models. For example, scratch defects in the Mini-LED surface defect dataset have a large aspect ratio, which makes it more complicated to completely contain the defects in a bounding box, resulting in split detection results; in addition, when encountering irregularly shaped and densely configured defects, the model may generate too many bounding boxes, resulting in redundant detection results. In addition, unlike general objects, industrial defects often show unique relationship patterns that are closely related to the machine operation status or environmental conditions. For example, defects caused by equipment wear usually form a predictable distribution on the product surface and tend to gather in specific areas; the spatial and semantic interaction relationships between different defects also show certain patterns. For example, foreign body defects in the Mini-LED dataset usually appear on lamp beads and show a tendency to appear in pieces. Most existing defect detection methods usually locate and classify instances in isolation, and fail to make full use of these potential patterns and relationships, resulting in low detection accuracy. Summary of the invention

[0007] The purpose of the present invention is to provide a Mini-LED surface defect detection method and related devices to solve one or more of the above-mentioned technical problems. The technical solution disclosed in the present invention can effectively solve the problem of redundant detection, improve the accuracy and recall rate of detection, reduce missed detection and false detection, and has broad application prospects.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect of the present invention, a Mini-LED surface defect detection method is provided, comprising the following steps: Acquire a Mini-LED image to be inspected for surface defects; Based on the acquired Mini-LED image to be inspected for surface defects, defect detection is performed using a trained defect detection model to obtain a surface defect detection result; Wherein, the defect detection model includes: A data input module, used for inputting Mini-LED image data; The feature extraction module is used to perform deep convolution operations on the input Mini-LED image to generate multi-scale feature maps of different scales; The feature fusion module is used to perform convolution fusion operations on the input multi-scale feature maps to obtain the fused feature maps; A relation modeling module is used to input the fused feature map, and to enhance the representation capability of defect features by integrating spatial and semantic information to obtain the modeled relation feature representation; wherein the relation modeling module includes a position embedding module and a relation reasoning module; the position embedding module is used to input the fused feature map, model the relative position and scale relationship between two groups of objects, and generate a position matrix; the relation reasoning module is used to input the position matrix and generate a global relation feature representation; The detection module is used to input the relationship feature representation after modeling and perform defect detection, and output the surface defect detection results.

[0009] A further improvement of the present invention is that The surface defect detection result includes a boundary box and category information of the defect.

[0010] A further improvement of the present invention is that In the position embedding module, the step of executing the input fused feature map, modeling the relative position and scale relationship between the two groups of objects, and generating the position matrix includes: The relative position and scaling relationship between the bounding boxes of the two objects is calculated to generate a position matrix that is used to represent the geometric relationship between the bounding boxes. The expression is: ; In the formula, For the bounding box The center coordinates of For the bounding box Width and height of For the bounding box The center coordinates of For the bounding box Width and height of .

[0011] A further improvement of the present invention is that In the position embedding module, a sparsity constraint is imposed when generating a position matrix; In the position embedding module, sine and cosine functions are used to map geometric features when generating a position matrix, and converted into an embedding of a higher dimension.

[0012] A further improvement of the present invention is that In the relationship reasoning module, the step of inputting the position matrix to generate a global relationship feature representation includes: Generate geometric relationship weights and semantic relationship weights and merge them to obtain comprehensive relationship weights of the two groups of objects; wherein the semantic relationship weights are generated by performing a dot product operation on the semantic feature vectors of the two groups of objects; Based on the comprehensive relationship weight, the semantic features of all objects are weighted and aggregated to generate a global relationship feature representation; Among them, in the process of generating geometric relationship weights and semantic relationship weights and fusing the two, the fusion calculation expression is: ; In the formula, represents the comprehensive relationship weight; Represents the weight of geometric relationship; Represents the semantic relationship weight.

[0013] A further improvement of the present invention is that The relationship modeling module is also provided with a relationship feature selection mechanism; the relationship feature selection mechanism divides the bounding box into two categories of high confidence instances and low confidence instances according to the classification score of each bounding box and a preset threshold; the features of the high confidence instances are kept unchanged; and the low confidence instances are optimized and enhanced by introducing additional relationship features; Among them, the calculation formula for optimization enhancement is: ; In the formula, Represents the optimized and enhanced feature map; represents the original feature map; Represents relationship characteristics.

[0014] A further improvement of the present invention is that The defect detection model is trained using a supervised training method, and the loss function used in the training process is expressed as: ; In the formula, It is the rectangular box loss, which is used to measure the difference between the predicted bounding box and the real box in position and shape; It is the classification loss, which is used to calculate the gap between the category probability distribution output by the model and the true category; is the cross entropy loss, which is used to optimize the distribution around the target position; They are The weight coefficient of .

[0015] A second aspect of the present invention provides a Mini-LED surface defect detection system, comprising: An image acquisition module, used to acquire a Mini-LED image to be inspected for surface defects; A defect detection module, used to perform defect detection based on the acquired Mini-LED image to be inspected for surface defects using a trained defect detection model to obtain a surface defect detection result; Wherein, the defect detection model includes: A data input module, used for inputting Mini-LED image data; The feature extraction module is used to perform deep convolution operations on the input Mini-LED image to generate multi-scale feature maps of different scales; The feature fusion module is used to perform convolution fusion operations on the input multi-scale feature maps to obtain the fused feature maps; A relation modeling module is used to input the fused feature map, and to enhance the representation capability of defect features by integrating spatial and semantic information to obtain the modeled relation feature representation; wherein the relation modeling module includes a position embedding module and a relation reasoning module; the position embedding module is used to input the fused feature map, model the relative position and scale relationship between two groups of objects, and generate a position matrix; the relation reasoning module is used to input the position matrix and generate a global relation feature representation; The detection module is used to input the relationship feature representation after modeling and perform defect detection, and output the surface defect detection results.

[0016] According to a third aspect of the present invention, there is provided an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the Mini-LED surface defect detection method as described in any one of the first aspects of the present invention is implemented.

[0017] In a fourth aspect of the present invention, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the Mini-LED surface defect detection method as described in any one of the first aspects of the present invention is implemented.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The present invention discloses a new Mini-LED surface defect detection solution, which aims to fundamentally solve the redundant detection problem existing in traditional detection methods, and significantly improve the detection accuracy and recall rate, while effectively reducing the incidence of missed detection and false detection; the technical solution of the present invention not only proposes an efficient solution for quality control in the Mini-LED production process, but also its technical principles and application potential also indicate that it will have a far-reaching impact in a wider range of industrial detection fields. Specifically explanatory, the core of the technical solution of the present invention lies in the newly proposed relationship modeling module, which realizes the accurate capture of complex spatial and semantic relationships between defect instances by fusing position embedding and relationship reasoning. The position embedding module enables the model to understand the relative position of defects in the image, while the relationship reasoning module further analyzes the intrinsic connection between these defects. Through this meticulous relationship modeling, the model can more accurately perceive the location information and feature details of the target, providing richer and more accurate clues for subsequent defect identification. In the technical solution of the present invention, the enhanced features output by the relationship modeling module are integrated into the target detection network. This fusion process not only improves the network's sensitivity to defect features, but also enhances its ability to recognize defects with special morphologies. In particular, for defects with high aspect ratios or dense distributions, traditional methods are often difficult to accurately identify. The present invention significantly improves the distinguishability of these defects in the feature space through feature enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below; obviously, the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 is a schematic flow chart of a Mini-LED surface defect detection method in an embodiment of the present invention; Figure 2 It is a flowchart of a Mini-LED surface defect detection method based on YOLOv8 and relationship modeling in a specific embodiment of the present invention; Figure 3 is a structural schematic diagram of a defect detection model in an embodiment of the present invention; Figure 4 is a structural diagram of a relationship modeling module in an embodiment of the present invention; Figure 5 is a schematic diagram of the detection results of the Mini-LED surface defect detection method provided by an embodiment of the present invention; Figure 6 It is a schematic diagram of a Mini-LED surface defect detection system in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention; it is obvious that the described embodiments and technical solutions are only part of the embodiments of the present invention, not all of the embodiments.

[0022] All other embodiments obtained by those of ordinary skill in the art without creative work based on the technical solutions disclosed in the embodiments of the present invention belong to the scope of protection of the present invention. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] See also Figure 1 , a Mini-LED surface defect detection method provided by an embodiment of the present invention comprises the following steps: Step 1, obtaining a Mini-LED image to be inspected for surface defects; Step 2: Based on the Mini-LED image to be inspected for surface defects obtained in step 1, defect detection is performed using the trained defect detection model to obtain a surface defect detection result; in a specific exemplary technical solution, the output surface defect detection result includes a boundary box and a category of the defect; The defect detection model includes: a data input module (Input), a feature extraction module (Backbone), a feature fusion module (Neck), a relationship modeling module and a detection module (Prediction); The data input module is used to input Mini-LED image data; further, the input image data can also be pre-processed such as data enhancement to ensure that the image data is suitable for subsequent feature extraction and processing; The feature extraction module is used to perform a deep convolution operation on the Mini-LED image, extract the basic features of the image, and generate multi-scale feature maps of different scales; The feature fusion module is used to input a multi-scale feature map and perform a convolution fusion operation to obtain a fused feature map, so as to ensure that the defect detection model network can detect defects of different sizes at different feature scales; The relationship modeling module is used to input the fused feature map, and obtain the modeled relationship feature map by integrating spatial and semantic information to enhance the representation ability of defect features; wherein, the relationship modeling module includes a position embedding module and a relationship reasoning module, wherein the position embedding module is used to input the fused feature map, and obtain the position embedding feature map after integrating the feature representation of the position information; the relationship reasoning module is used to input the position embedding feature map and integrate the feature representation of the relationship between the targets to obtain the relationship feature map; The detection module is used to input the modeled relationship feature graph and perform detection, and output the defect detection result; illustratively, the defect detection result may include information such as the location, size and category of the defect; in addition, the detection module can generate a final prediction box based on the defect detection model, and calculate the target loss to guide the training and optimization of the model.

[0024] The embodiment of the present invention provides a new Mini-LED surface defect detection solution, which can effectively solve the problem of redundant detection, improve the detection accuracy and recall rate, reduce missed detection and false detection, and has broad application prospects. The improved technical means of the embodiment of the present invention aims to construct the position relationship and feature relationship between defect instances, perform feature enhancement, and introduce them into the target detection network; use the supervised learning method to train the network, and use the spatial and semantic information between target instances to improve the distinction of defects with special morphology in the feature space, thereby improving the detection accuracy of defect targets and realizing the calibration of defect targets in Mini-LED images. Further explanatory, a new relationship modeling module is proposed in the defect detection model of the embodiment of the present invention. By introducing position embedding and relationship reasoning mechanisms, it can more effectively capture the spatial and semantic relationships between defect instances, perceive the position and feature information of the target, and construct a defect detection model for the detection of complex morphological defects. In summary, the Mini-LED surface defect detection method disclosed in the embodiment of the present invention shows stronger adaptability when dealing with complex morphological defects, especially defects with high aspect ratios or densely distributed defects, effectively avoiding the problem of redundant detection, improving the accuracy and recall rate of detection, and reducing missed detections and false detections. The relational modeling module proposed in the technical solution of the present invention is based on basic convolution and matrix operations, and the input and output data formats are consistent, without excessive reliance, and can be easily applied to various deep defect detection models, with broad application prospects.

[0025] In one embodiment of the present invention, the step of training the defect detection model includes: Collect Mini-LED image sample data and finely annotate the defective areas in the image samples. The annotation content includes information such as the defect bounding box and defect category, so as to provide accurate supervision signals for the model in the subsequent training process. Based on the collected Mini-LED image sample data, the constructed defect detection model is trained to obtain a trained defect detection model.

[0026] In the specific exemplary technical solution, the defect detection model is based on the architecture of the YOLOv8 model, using the superior feature extraction and multi-scale feature processing capabilities of the YOLOv8 model, combined with the relationship modeling module to quantify the interaction between the spatial and semantic features of defects, so as to achieve high-precision detection and positioning of defect instances in complex industrial scenarios. Specifically, the defect detection model first extracts features from the input image through deep convolution operations, and generates multi-scale feature maps of different scales layer by layer; the generation process of multi-scale feature maps enables the model to effectively capture defect information of different sizes and shapes; the relative position and scaling relationship between the bounding boxes are encoded by calculating position embedding, and the spatial relationship between objects is introduced; the relationship reasoning module combines geometric and semantic features to further model and quantify the relationship between objects. Through the comprehensive use of geometric and semantic features, the relationship reasoning module can generate more accurate relationship weights, thereby helping the model to better understand the interaction between defect instances. Furthermore, the application of the relationship feature selection mechanism to dynamically adjust features can improve the model's detection ability for low-confidence instances, and can dynamically optimize the model's detection effect without affecting the features of high-confidence instances. Furthermore, a prediction box is generated at the output end and a non-maximum suppression (NMS) operation is set. The prediction box is retained for target loss function calculation, and the final loss value is transmitted back to the defect detection model for parameter update.

[0027] In one embodiment of the present invention, the position embedding module in the relationship modeling module encodes information related to the spatial position of the object by creating a high-dimensional visual embedding to map the geometric features of the object and facilitate the capture of the relationship between defect instances; the model evaluates the relative position and size relationship between each two defect instances to determine their spatial relationship.

[0028] Specifically, the position embedding module calculates the relative position and scale relationship between the bounding boxes of two objects and generates a position matrix to represent the geometric relationship between the bounding boxes. The expression is: ; In the formula, the bounding box By its center coordinates and its width and height Definition, Bounding Box Likewise; Through this calculation method, the model can obtain the relative position information (such as horizontal and vertical offsets) and scale relationship (such as the ratio of width to height) between the bounding boxes of two objects, thereby constructing a geometric feature representation with spatial invariance, ensuring that the model's embedded features remain robust in the face of translation and scaling transformations.

[0029] In addition, in order to further improve computational efficiency and reduce memory consumption, the position embedding module imposes sparsity constraints when generating the position matrix, retaining only defect instance pairs with significant geometric relationships, reducing unnecessary computational overhead. In addition, the position embedding module further converts it into a high-dimensional representation. Specifically, the four geometric features are mapped using sine and cosine functions and converted into a higher-dimensional embedding (such as a 64-dimensional high-dimensional embedding), so that it can better adapt to the high-dimensional feature space of the deep learning model.

[0030] In one embodiment of the present invention, the relational reasoning module in the relational modeling module analyzes the semantic features of each defect instance, uses an attention mechanism to quantify the interactions between these features, and captures the relationship between defect instances at the semantic feature level, so that the model can not only perceive the features of each defect instance, but also better understand the intrinsic connection between instance features.

[0031] Specifically, after the relative position and scale relationship between the two groups of objects is modeled through the position embedding module, the relational reasoning module performs a dot product operation on the semantic feature vectors of the two groups of objects to generate a semantic relationship weight. This weight is used to measure the semantic similarity or degree of association between defect instances, and is a measure of the connection at the object content level. Through the dot product operation of this semantic feature, the model can identify defect instances with similar attributes or categories, and further optimize the ability to distinguish defects based on this similarity.

[0032] In the embodiment of the present invention, the relational reasoning module combines geometric and semantic features to calculate the relational features between two groups of objects; after generating the geometric relational weights and the semantic relational weights, the relational reasoning module combines the two to obtain the comprehensive relational weights of each pair of objects. This weight fusion takes into account both geometric and semantic factors, providing the model with a more comprehensive relational modeling result. Specifically, the fusion formula of geometric and semantic relational weights is as follows: ; In the formula, represents the relationship weight, represents the geometric relationship weight, Represents the weight of semantic relationship; In the embodiment of the present invention, through this fusion operation, the relational reasoning module can perform weighted aggregation on the semantic features of all objects to generate a global relational feature representation that reflects the spatial and semantic relational interaction information between defect instances.

[0033] In one embodiment of the present invention, the relationship feature selection mechanism in the relationship modeling module dynamically adjusts the application of relationship features to reduce the impact of irrelevant information. The mechanism effectively controls the relationship weight through the classification score threshold p (exemplarily, such as p=0.6), thereby reasonably filtering the output of the model so that only object features with high confidence are retained, while low confidence instances are further optimized or adjusted in the process.

[0034] In the embodiment of the present invention, the relational feature selection mechanism divides the bounding boxes into two categories: high-confidence and low-confidence instances according to the classification score of each bounding box. For high-confidence bounding boxes, the model assumes that the corresponding features have fully represented the core information of the object, so its features are kept unchanged to avoid potential noise interference caused by further operations. For low-confidence instances, additional relational features are introduced for optimization, thereby enhancing the representation ability of such instances, so that the model can more accurately identify these potential defect targets.

[0035] In this mechanism, the enhanced feature map The calculation formula is: ; In the formula, represents the original feature map, Represents the characteristics of the relationship, Represents the enhanced feature map.

[0036] In one embodiment of the present invention, the target loss calculation module is used to measure the deviation between the prediction result and the target label, and improve the detection performance of the model by optimizing the loss function of YOLOv8. The loss function contains three main loss terms, each of which undertakes a different optimization task. The calculation formula is: ; In the formula, It is the rectangular box loss (bounding box regression loss), which is used to measure the difference between the predicted bounding box and the real box in position and shape; It is the classification loss (confidence regression loss), which is used to calculate the gap between the category probability distribution output by the model and the true category; In the form of cross entropy, it focuses on optimizing the distribution around the target position; this loss will select the two positions closest to the label y for probability optimization, thereby guiding the model to focus more on the specific location of the target and the distribution information of its neighboring area. This loss can help the model converge to the target position more quickly and accurately, and reduce the interference of uncertain areas. Rectangular box loss weight Set to 7.5, confidence regression loss weight is 1.5, Loss Weight Both are 0.5.

[0037] Referring to FIG. 2 , a Mini-LED surface defect detection method based on YOLOv8 and relational modeling provided by a specific embodiment of the present invention may specifically include the following steps: Collect Mini-LED images and finely annotate the defective areas in the images. The annotations include the bounding box of the defect and the defect category information, so as to provide accurate supervision signals for the model in the subsequent training process. Build a defect detection model for identifying and locating Mini-LED defect images; The constructed defect detection model is trained using the Mini-LED surface defect detection dataset to obtain a trained defect detection model; Use the trained defect detection model to perform defect detection on Mini-LED images and output the defect bounding box and category.

[0038] Referring to FIG. 3 , in the embodiment of the present invention, the defect detection model used is based on the YOLOv8 network architecture, specifically including: Data input module, used for data input and preprocessing, including data enhancement of input images; The feature extraction module uses a series of convolution and deconvolution layers to perform deep convolution operations on the Mini-LED surface defect images to extract the basic features of the image and generate a multi-scale feature map; The feature fusion module performs convolution fusion on multi-scale features. By fusing feature maps from different stages of Backbone, it enhances the feature representation capability and ensures that the network can detect defects of different sizes at different feature scales. The relation modeling module, including the position embedding module and the relation reasoning module, enhances the representation capability of defect features by integrating spatial and semantic information; The detection module is responsible for the final target detection and classification tasks. It generates the final prediction box and prediction category based on the network model, calculates the target loss, and guides the training and optimization of the model.

[0039] Specifically and exemplarily, the detection process of the method of the embodiment of the present invention may include the following steps: Step 1: Construct a dataset of Mini-LED surface defect images in an industrial environment. The collected images satisfy the three RGB color channels and contain the corresponding manual annotation results. The pixel size of the dataset is 360*240. The dataset is randomly divided into training and test sets. The collected images are transmitted to the computer that executes the algorithm.

[0040] Step 2: Construct a defect detection model, including a sequentially connected feature extraction module, a multi-scale feature fusion module, a relationship modeling module, and a detection module; See also Figure 4 ,The relation reasoning module includes the position embedding module, the relation reasoning module and the relation feature selection mechanism; Among them, the position embedding module encodes the relative position and scaling relationship between bounding boxes and introduces the spatial relationship between objects; the relationship reasoning module combines geometric and semantic features to further model and quantify the relationship between objects. Through the comprehensive use of geometric and semantic features, the relationship reasoning module can generate more accurate relationship weights, thereby helping the model to better understand the interaction between defect instances; the relationship feature selection mechanism dynamically adjusts the application of features to improve the model's detection ability for low-confidence instances, and can dynamically optimize the model's detection effect without affecting the features of high-confidence instances; a prediction box is generated at the output end and a non-maximum suppression (NMS) operation is set, the prediction box is retained for target loss function calculation, and the final loss value is transmitted back to the defect detection model for parameter update.

[0041] Step 3: Data preprocessing. Before inputting into the network training, each Mini-LED surface defect image is first scaled to 320 pixels on the long side while maintaining the aspect ratio. Then, random left-right, top-bottom scaling and image stitching are performed for data enhancement. The manual labeling results are also transformed accordingly. Finally, all images are padded with peripheral pixels to keep the image size consistent.

[0042] Step 4: During the training process, after the image is input into the network, it passes through the feature extraction module and the feature fusion module to obtain three layers of effective feature maps. Then, each layer is input into the relationship modeling module for feature enhancement, and the three layers of enhanced feature maps are output. Finally, the defect location and classification results are output through the detection module. See also Figure 5In the embodiment of the present invention, mAP, precision, and recall are used to evaluate the defect detection results. It can be seen that the new method provided by the embodiment of the present invention has a higher mAP value, indicating that the method of the present invention can effectively detect Mini-LED surface defects and has obvious advantages over the conventional YOLOv8 method.

[0043] In the following specific embodiments of the present invention: the Mini-LED surface defect dataset used contains a total of 2545 RGB color images of the surface of Mini-LED lamp beads taken in real industrial scenes, with an image size of 360*240 pixels; the images are randomly shuffled and divided into a training set consisting of 2292 images and a test set consisting of 253 images, respectively. The images will first be scaled to 320*320 pixels in the preprocessing stage; the initialization method of the defect detection model parameters in the embodiment of the present invention is that the parameters of the feature extraction module are initialized by the pre-trained YOLOv8 network, and the remaining modules are randomly initialized; the operating environment is a computer with a framework such as PyTorch, which can read a given image and complete the construction and training of the model of this method. The training time of the embodiment of the present invention on a Gold 6626R@2.90GHz CPU, 8G memory and NVIDIA GeForce RTX3090 GPU is about 5 hours.

[0044] The specific implementation steps include: first setting relevant training parameters, setting the optimizer used for network update in the present invention to a stochastic gradient descent optimizer, setting its momentum value to 0.9, and setting its learning rate to 0.01. The deep target detection network consists of a feature extraction module containing 5 stages, a multi-scale feature fusion module, a relationship modeling module and a detection module. The input of the network is an RGB image. The feature extraction module first extracts the texture and abstract semantic information of the detection defect. The 5-level feature map output by the multi-scale feature fusion module is subjected to information fusion to obtain a 3-level feature map, which is passed to the relationship modeling module for feature enhancement, and finally input into the detection module to obtain a detection result image containing a rectangular anchor box mark and classification confidence. When using the divided data set for network training, 8 pictures are randomly selected from the training set each time to enter the network, and the selected stochastic gradient descent optimizer is used to update the parameters. The training is completed after 300 rounds of iteration on the data set. Finally, the pictures in the test set are input into the trained network for detection to obtain the results of the embodiment of the invention.

[0045] In summary, the embodiment of the present invention discloses a Mini-LED surface defect detection method based on YOLOv8 and relational modeling, including the following steps: first, collect Mini-LED images and annotate the boundary boxes and category information of defects; second, build a defect detection model, the network is based on the YOLOv8 architecture, and is mainly composed of data input, feature extraction, feature fusion, relational modeling and detection modules. The feature extraction module extracts basic image features through deep convolution operations, and the feature fusion module performs convolution fusion on multi-scale features to ensure that the network can effectively detect defects of different sizes. The relational modeling module uses the attention mechanism to analyze the semantic features of defect instances by encoding the spatial position relationship between objects, captures the intrinsic spatial and semantic relationships between defects, and dynamically adjusts the application of features through the relational feature selection mechanism to filter irrelevant information; finally, the parameters are updated by calculating the loss function. The technical solution of the embodiment of the present invention shows stronger adaptability when processing complex morphological defects, effectively avoids redundant detection, and improves the detection accuracy and recall rate.

[0046] The following are device embodiments of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0047] See also Figure 6 In an embodiment of the present invention, a Mini-LED surface defect detection system is provided, comprising: An image acquisition module, used to acquire a Mini-LED image to be inspected for surface defects; A defect detection module, used to perform defect detection based on the acquired Mini-LED image to be inspected for surface defects using a trained defect detection model to obtain a surface defect detection result; Wherein, the defect detection model includes: A data input module, used for inputting Mini-LED image data; The feature extraction module is used to perform deep convolution operations on the input Mini-LED image to generate multi-scale feature maps of different scales; The feature fusion module is used to perform convolution fusion operations on the input multi-scale feature maps to obtain the fused feature maps; A relation modeling module is used to input the fused feature map, and to enhance the representation capability of defect features by integrating spatial and semantic information to obtain the modeled relation feature representation; wherein the relation modeling module includes a position embedding module and a relation reasoning module; the position embedding module is used to input the fused feature map, model the relative position and scale relationship between two groups of objects, and generate a position matrix; the relation reasoning module is used to input the position matrix and generate a global relation feature representation; The detection module is used to input the relationship feature representation after modeling and perform defect detection, and output the surface defect detection results.

[0048] In one embodiment of the present invention, a computer device is provided, the computer device includes a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used to perform the operation of the Mini-LED surface defect detection method.

[0049] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM (Random Access Memory) memory, or a non-volatile memory (non-volatile memory), such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the Mini-LED surface defect detection method in the above embodiment.

[0050] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) containing computer-usable program codes.

[0051] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0052] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0053] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A Mini-LED surface defect detection method, characterized in that: The following steps are involved: Acquire a Mini-LED image to be inspected for surface defects; Based on the acquired Mini-LED image to be inspected for surface defects, defect detection is performed using a trained defect detection model to obtain a surface defect detection result; Wherein, the defect detection model includes: A data input module, used for inputting Mini-LED image data; The feature extraction module is used to perform deep convolution operations on the input Mini-LED image to generate multi-scale feature maps of different scales; The feature fusion module is used to perform convolution fusion operations on the input multi-scale feature maps to obtain the fused feature maps; A relation modeling module is used to input the fused feature map, and to enhance the representation capability of defect features by integrating spatial and semantic information to obtain the modeled relation feature representation; wherein the relation modeling module includes a position embedding module and a relation reasoning module; the position embedding module is used to input the fused feature map, model the relative position and scale relationship between two groups of objects, and generate a position matrix; the relation reasoning module is used to input the position matrix and generate a global relation feature representation; The detection module is used to input the relationship feature representation after modeling and perform defect detection, and output the surface defect detection results.

2. A Mini-LED surface defect detection method according to claim 1, characterized in that: The surface defect detection result includes a boundary box and category information of the defect.

3. A Mini-LED surface defect detection method according to claim 1, characterized in that: In the position embedding module, the step of executing the input fused feature map, modeling the relative position and scale relationship between the two groups of objects, and generating the position matrix includes: The relative position and scaling relationship between the bounding boxes of the two objects is calculated to generate a position matrix that is used to represent the geometric relationship between the bounding boxes. The expression is: ； In the formula, For the bounding box The center coordinates of For the bounding box Width and height of For the bounding box The center coordinates of For the bounding box Width and height of .

4. A Mini-LED surface defect detection method according to claim 3, characterized in that: In the position embedding module, a sparsity constraint is imposed when generating a position matrix; In the position embedding module, sine and cosine functions are used to map geometric features when generating a position matrix, and converted into an embedding of a higher dimension.

5. The Mini-LED surface defect detection method according to claim 1, characterized in that: In the relationship reasoning module, the step of inputting the position matrix to generate a global relationship feature representation includes: Generate geometric relationship weights and semantic relationship weights and merge them to obtain comprehensive relationship weights of the two groups of objects; wherein the semantic relationship weights are generated by performing a dot product operation on the semantic feature vectors of the two groups of objects; Based on the comprehensive relationship weight, the semantic features of all objects are weighted and aggregated to generate a global relationship feature representation; Among them, in the process of generating geometric relationship weights and semantic relationship weights and fusing the two, the fusion calculation expression is: ； In the formula, represents the comprehensive relationship weight; Represents the weight of geometric relationship; Represents the semantic relationship weight.

6. The Mini-LED surface defect detection method according to claim 1, characterized in that: The relationship modeling module is also provided with a relationship feature selection mechanism; the relationship feature selection mechanism divides the bounding box into two categories of high confidence instances and low confidence instances according to the classification score of each bounding box and a preset threshold; the features of the high confidence instances are kept unchanged; and the low confidence instances are optimized and enhanced by introducing additional relationship features; Among them, the calculation formula for optimization enhancement is: ； In the formula, Represents the optimized and enhanced feature map; represents the original feature map; Represents relationship characteristics.

7. The Mini-LED surface defect detection method according to claim 1, characterized in that: The defect detection model is trained using a supervised training method, and the loss function used in the training process is expressed as: ； In the formula, It is the rectangular box loss, which is used to measure the difference between the predicted bounding box and the real box in position and shape; It is the classification loss, which is used to calculate the gap between the category probability distribution output by the model and the true category; is the cross entropy loss, which is used to optimize the distribution around the target position; They are The weight coefficient of .

8. A Mini-LED surface defect detection system, characterized in that: include: An image acquisition module, used to acquire a Mini-LED image to be inspected for surface defects; A defect detection module, used to perform defect detection based on the acquired Mini-LED image to be inspected for surface defects using a trained defect detection model to obtain a surface defect detection result; Wherein, the defect detection model includes: A data input module, used for inputting Mini-LED image data; The feature extraction module is used to perform deep convolution operations on the input Mini-LED image to generate multi-scale feature maps of different scales; The feature fusion module is used to perform convolution fusion operations on the input multi-scale feature maps to obtain the fused feature maps; A relation modeling module is used to input the fused feature map, and to enhance the representation capability of defect features by integrating spatial and semantic information to obtain the modeled relation feature representation; wherein the relation modeling module includes a position embedding module and a relation reasoning module; the position embedding module is used to input the fused feature map, model the relative position and scale relationship between two groups of objects, and generate a position matrix; the relation reasoning module is used to input the position matrix and generate a global relation feature representation; The detection module is used to input the relationship feature representation after modeling and perform defect detection, and output the surface defect detection results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the Mini-LED surface defect detection method as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the Mini-LED surface defect detection method according to any one of claims 1 to 7 is implemented.

Images