A method and related device for detecting surface defects in Mini-LEDs

Abstract

This invention belongs to the field of target detection technology in computer vision, and discloses a method and related apparatus for detecting surface defects in Mini-LEDs. The Mini-LED surface defect detection method includes: acquiring a Mini-LED image to be detected for surface defects; and performing defect detection using a trained defect detection model based on the acquired Mini-LED image to obtain surface defect detection results. The defect detection model includes a sequentially connected data input module, feature extraction module, feature fusion module, relationship modeling module, and detection module. The relationship modeling module includes a location embedding module and a relationship inference module. This invention effectively solves the problem of redundant detection, improves detection accuracy and recall, and reduces missed detections and false detections.

Description

Technical Field

[0001] This invention belongs to the field of target detection technology in computer vision, and specifically relates to a method and related device for detecting surface defects in Mini-LEDs. Background Technology

[0002] Mini-LEDs are a new display technology that, compared to traditional LEDs, features smaller size, higher integration, high brightness, fast response, and high resolution, enabling more refined display effects. Currently, Mini-LEDs are widely used in advanced fields such as wearable devices, smart medical displays, high-definition televisions, and automotive displays.

[0003] As the physical size of Mini-LEDs shrinks, their manufacturing difficulty increases, leading to a decrease in yield. Rapid and accurate inspection of Mini-LED chips has become a critical aspect of industrial production. Explained, due to the small size and dense distribution of the chips, the feature information of a single target is insufficient, and industrial inspection requires algorithms that are fast and easy to deploy. Therefore, defect detection of Mini-LED chips still faces significant challenges.

[0004] Currently, defect detection technology is mainly divided into two major directions: traditional machine vision-based inspection and deep learning-based inspection. Traditional manual inspection methods suffer from low efficiency, high cost, and susceptibility to human error. With the rapid development of deep learning technology, automated inspection technology has effectively overcome the limitations of traditional manual inspection and has gradually become an important part of the production process.

[0005] In the field of object detection, Faster R-CNN is a representative of two-stage detectors, while SSD and YOLO are typical examples of single-stage detectors. The two-stage detection model, Faster R-CNN, divides the detection task into two steps: first, it uses a Region Proposal Network (RPN) to generate candidate regions, and then performs classification and bounding box regression on these candidate regions. Although Faster R-CNN performs well in terms of accuracy, its inference speed is relatively slow. In contrast, single-stage detection models aim to achieve object detection with a single forward propagation, significantly improving speed. SSD performs detection on different feature layers, enabling it to handle targets of different sizes while maintaining fast inference speed. YOLO divides the input image into a grid and directly performs bounding box regression and class prediction on the feature map, achieving efficient real-time performance. Furthermore, Transformer-based detection models have gradually gained attention in recent years. DETR (DEtection Transformer) treats object detection as an ensemble prediction problem, employing an end-to-end framework. It effectively captures long-distance dependencies between objects in an image through a self-attention mechanism. Its innovation lies in simplifying the traditional detection process and eliminating the need for a region generation network, thus providing a new approach to object detection. Among these existing models, the YOLO series offers the fastest detection speed while ensuring detection effectiveness, making it the preferred solution for real-time target detection and providing strong support for industrial defect detection.

[0006] Currently, the application of automated inspection technology in industrial defect detection still faces many challenges. Compared to ordinary objects, industrial defects typically exhibit more diverse and complex morphologies, hindering the performance of general detection models. For example, scratches in the Mini-LED surface defect dataset have a large aspect ratio, making it more complex to completely include the defect within a bounding box, leading to fragmented detection results. Furthermore, when encountering irregularly shaped and densely packed defects, the model may generate too many bounding boxes, resulting in redundant detection results. Moreover, unlike ordinary objects, industrial defects often exhibit unique relationship patterns closely related to machine operating conditions or environmental conditions. For instance, defects caused by equipment wear usually form a predictable distribution on the product surface, tending to cluster in specific areas. The spatial and semantic interactions between different defects also exhibit certain patterns; for example, foreign object defects in the Mini-LED dataset typically appear on LED beads and tend to appear in patches. Most existing defect detection methods typically locate and classify instances in isolation, failing to fully utilize these potential patterns and relationships, resulting in low detection accuracy. Summary of the Invention

[0007] The purpose of this invention is to provide a method and related apparatus for detecting surface defects in Mini-LEDs, thereby solving one or more of the aforementioned technical problems. The technical solution disclosed in this invention can effectively solve the problem of redundant detection, improve detection accuracy and recall, reduce missed detections and false detections, and has broad application prospects.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a method for detecting surface defects in Mini-LEDs, comprising the following steps:

[0010] Acquire Mini-LED images of the surfaces to be inspected;

[0011] Based on the acquired Mini-LED image of the surface defect to be detected, the trained defect detection model is used to perform defect detection and obtain the surface defect detection results.

[0012] The defect detection model includes:

[0013] The data input module is used to input Mini-LED image data;

[0014] The feature extraction module is used to perform depth convolution operations on the input Mini-LED image to generate multi-scale feature maps of different scales.

[0015] The feature fusion module is used to perform convolutional fusion operations on the input multi-scale feature maps to obtain the fused feature maps.

[0016] A relationship modeling module is used to input the fused feature map and enhance the representation capability of defect features by integrating spatial and semantic information to obtain a modeled relationship feature representation. This module includes a position embedding module and a relationship inference module. The position embedding module is used to input the fused feature map and model the relative position and scale relationship between two sets of objects to generate a position matrix. The relationship inference module is used to input the position matrix and generate a global relationship feature representation.

[0017] The detection module is used to input the modeled relational feature representation and perform defect detection, and output the surface defect detection results.

[0018] A further improvement of the present invention is that,

[0019] The surface defect detection results include the defect's bounding box and category information.

[0020] A further improvement of the present invention is that,

[0021] In the location embedding module, the steps of executing the input fused feature map, modeling the relative position and scale relationship between the two sets of objects, and generating the location matrix include:

[0022] The relative position and scaling relationship between the bounding boxes of two objects are calculated to generate a position matrix, which represents the geometric relationship between the bounding boxes. The expression is as follows:

[0023] ;

[0024] In the formula, bounding box The center coordinates, bounding box Width and height; bounding box The center coordinates, bounding box Width and height.

[0025] A further improvement of the present invention is that,

[0026] In the location embedding module, sparsity constraints are applied when generating the location matrix;

[0027] In the location embedding module, when generating the location matrix, sine and cosine functions are used to map the geometric features, transforming them into a higher-dimensional embedding.

[0028] A further improvement of the present invention is that,

[0029] In the relation reasoning module, the step of generating a global relation feature representation by inputting the position matrix includes:

[0030] Geometric relation weights and semantic relation weights are generated and fused to obtain the comprehensive relation weights of the two sets of objects; among them, semantic relation weights are generated by performing a dot product operation on the semantic feature vectors of the two sets of objects.

[0031] Based on the comprehensive relation weight, the semantic features of all objects are weighted and aggregated to generate a global relation feature representation;

[0032] In the process of generating geometric relation weights and semantic relation weights and fusing them, the fusion calculation expression is as follows:

[0033] ;

[0034] In the formula, Indicates the weight of the overall relationship; Indicates the weight of geometric relationships; Represents the semantic relation weight.

[0035] A further improvement of the present invention is that,

[0036] The relationship modeling module also includes a relationship feature selection mechanism. Based on the classification score and preset threshold of each bounding box, the relationship feature selection mechanism divides the bounding boxes into two categories: high-confidence instances and low-confidence instances. The features of high-confidence instances remain unchanged, while the features of low-confidence instances are optimized and enhanced by introducing additional relationship features.

[0037] The calculation formula for optimization and enhancement is as follows:

[0038] ;

[0039] In the formula, This represents the optimized and enhanced feature map; Represents the original feature map; Indicates relational characteristics.

[0040] A further improvement of the present invention is that,

[0041] The defect detection model is trained using a supervised training method, and the loss function used during training is expressed as follows:

[0042] ;

[0043] In the formula, The rectangular box loss measures the difference in position and shape between the predicted bounding box and the ground truth box. The classification loss is used to calculate the difference between the class probability distribution output by the model and the true class. Cross-entropy loss is used to optimize the distribution around the target location; They are respectively The weighting coefficients.

[0044] In a second aspect, the present invention provides a Mini-LED surface defect detection system, comprising:

[0045] The image acquisition module is used to acquire Mini-LED images of the surface defects to be detected;

[0046] The defect detection module is used to perform defect detection based on the acquired Mini-LED image of the surface defect to be detected, using a trained defect detection model to obtain the surface defect detection results.

[0047] The defect detection model includes:

[0048] The data input module is used to input Mini-LED image data;

[0049] The feature extraction module is used to perform depth convolution operations on the input Mini-LED image to generate multi-scale feature maps of different scales.

[0050] The feature fusion module is used to perform convolutional fusion operations on the input multi-scale feature maps to obtain the fused feature maps.

[0051] A relationship modeling module is used to input the fused feature map and enhance the representation capability of defect features by integrating spatial and semantic information to obtain a modeled relationship feature representation. This module includes a position embedding module and a relationship inference module. The position embedding module is used to input the fused feature map and model the relative position and scale relationship between two sets of objects to generate a position matrix. The relationship inference module is used to input the position matrix and generate a global relationship feature representation.

[0052] The detection module is used to input the modeled relational feature representation and perform defect detection, and output the surface defect detection results.

[0053] In a third aspect, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the Mini-LED surface defect detection method as described in any one of the first aspects of the present invention.

[0054] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the Mini-LED surface defect detection method as described in any one of the first aspects of the present invention.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] This invention discloses a novel surface defect detection scheme for Mini-LEDs, aiming to fundamentally solve the problem of redundant detection in traditional methods and significantly improve detection accuracy and recall, while effectively reducing the incidence of missed and false detections. This invention not only provides an efficient solution for quality control in the Mini-LED production process, but its technical principles and application potential also indicate a profound impact on a wider range of industrial inspection fields. Specifically, the core of this invention lies in the newly proposed relation modeling module. This module, by integrating position embedding and relation reasoning, achieves accurate capture of the complex spatial and semantic relationships between defect instances. The position embedding module enables the model to understand the relative position of defects in the image, while the relation reasoning module further analyzes the intrinsic connections between these defects. Through this meticulous relation modeling, the model can more accurately perceive the target's location information and feature details, providing richer and more accurate clues for subsequent defect identification. In the technical solution of this invention, the enhanced features output by the relation modeling module are integrated into the target detection network. This fusion process not only improves the network's sensitivity to defect features, but also enhances its ability to identify defects with special shapes. In particular, traditional methods often have difficulty accurately identifying defects with high aspect ratios or dense distributions. However, this invention significantly improves the distinguishability of these defects in the feature space through feature enhancement. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0058] Figure 1 This is a flowchart illustrating a method for detecting surface defects in Mini-LEDs, as described in an embodiment of the present invention.

[0059] Figure 2 This is a flowchart illustrating a method for detecting surface defects in Mini-LEDs based on YOLOv8 and relational modeling, as described in a specific embodiment of the present invention.

[0060] Figure 3 This is a schematic diagram of the defect detection model in an embodiment of the present invention;

[0061] Figure 4 This is a schematic diagram of the structure of the relationship modeling module in an embodiment of the present invention;

[0062] Figure 5This is a schematic diagram of the detection results of the Mini-LED surface defect detection method provided in the embodiment of the present invention;

[0063] Figure 6 This is a schematic diagram of a Mini-LED surface defect detection system in an embodiment of the present invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention; obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0065] Based on the technical solutions disclosed in the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0066] Please see Figure 1 The present invention provides a method for detecting surface defects in Mini-LEDs, comprising the following steps:

[0067] Step 1: Obtain the Mini-LED image of the surface defect to be detected;

[0068] Step 2: Based on the Mini-LED image of the surface defect to be detected obtained in Step 1, perform defect detection using the trained defect detection model to obtain the surface defect detection result; in a specific exemplary technical solution, the output surface defect detection result includes the defect bounding box and category;

[0069] The defect detection model includes: a data input module (Input), a feature extraction module (Backbone), a feature fusion module (Neck), a relationship modeling module, and a detection module (Prediction).

[0070] The data input module is used to input Mini-LED image data; furthermore, it can also perform preprocessing such as data enhancement on the input image data to ensure that the image data is suitable for subsequent feature extraction and processing.

[0071] The feature extraction module is used to perform deep convolution operation on Mini-LED images to extract the basic features of the images and generate multi-scale feature maps of different scales.

[0072] The feature fusion module is used to input multi-scale feature maps and perform convolutional fusion operations to obtain fused feature maps, so as to ensure that the defect detection model network can detect defects of different sizes at different feature scales.

[0073] The relationship modeling module is used to input the fused feature map, and to obtain the modeled relationship feature map by integrating spatial and semantic information to enhance the representation capability of defect features. The relationship modeling module includes a location embedding module and a relationship reasoning module. The location embedding module is used to input the fused feature map, and after incorporating feature representations of location information, to obtain a location embedding feature map. The relationship reasoning module is used to input the location embedding feature map and incorporate feature representations of the relationships between targets to obtain a relationship feature map.

[0074] The detection module is used to input the modeled relational feature map and perform detection, and output defect detection results. For example, the defect detection results may include information such as the location, size and category of the defect. In addition, the detection module can generate the final prediction box according to the defect detection model and calculate the target loss to guide the training and optimization of the model.

[0075] This invention provides a novel surface defect detection scheme for Mini-LEDs, effectively solving the problem of redundant detection, improving detection accuracy and recall, reducing false negatives and missed detections, and possessing broad application prospects. The improved technical means of this invention aim to construct positional and feature relationships between defect instances, enhance features, and introduce them into a target detection network. Supervised learning is used to train the network, utilizing spatial and semantic information between target instances to improve the discriminative power of defects with special morphologies in the feature space, thereby improving defect target detection accuracy and achieving the labeling of defect targets in Mini-LED images. Further, this invention proposes a new relationship modeling module in its defect detection model. By introducing position embedding and relationship reasoning mechanisms, it can more effectively capture the spatial and semantic relationships between defect instances, perceive the position and feature information of targets, and construct a defect detection model for detecting complex-shaped defects. In summary, the Mini-LED surface defect detection method disclosed in this invention exhibits stronger adaptability when handling defects with complex shapes, especially high aspect ratio defects or densely distributed defects. It effectively avoids redundant detection, improves detection accuracy and recall, and reduces missed detections and false detections. The relational modeling module proposed in this invention, based on basic convolution and matrix operations, has consistent input and output data formats, requires minimal reliance, and can be easily applied to various depth defect detection models, demonstrating broad application prospects.

[0076] In one embodiment of the present invention, the training steps of the defect detection model include:

[0077] Mini-LED image sample data is collected, and the defective regions in the image samples are finely annotated. The annotation includes information such as the bounding box of the defect and the defect category, so as to provide accurate supervision signals for the model in subsequent training.

[0078] Based on the collected Mini-LED image sample data, the constructed defect detection model is trained to obtain a trained defect detection model.

[0079] In a specific exemplary technical solution, the defect detection model is based on the YOLOv8 model architecture. Leveraging the superior feature extraction and multi-scale feature processing capabilities of the YOLOv8 model, and combined with a relationship modeling module, it quantifies the interaction of spatial and semantic features between defects, thereby achieving high-precision detection and localization of defect instances in complex industrial scenarios. Specifically, the defect detection model first extracts features from the input image through deep convolution operations, generating multi-scale feature maps of different scales layer by layer. The generation process of multi-scale feature maps allows the model to effectively capture defect information of different sizes and shapes. By calculating position embeddings, the relative positions and scaling relationships between bounding boxes are encoded, introducing spatial relationships between objects. The relationship reasoning module combines geometric and semantic features to further model and quantify the relationships between objects. Through the comprehensive utilization of geometric and semantic features, the relationship reasoning module can generate more accurate relationship weights, thereby helping the model better understand the interactions between defect instances. Furthermore, the relationship feature selection mechanism dynamically adjusts the application of features, improving the model's detection capability for low-confidence instances. Without affecting the features of high-confidence instances, it can dynamically optimize the model's detection performance. Furthermore, a predicted bounding box is generated at the output end and non-maximum suppression (NMS) is set. The predicted bounding box is retained for the target loss function calculation, and the final loss value is fed back to the defect detection model for parameter update.

[0080] In one embodiment of the invention, the position embedding module in the relationship modeling module encodes spatial position-related information of objects by creating a high-dimensional visual embedding to map the geometric features of the objects and facilitate the capture of relationships between defect instances; the model evaluates the relative position and size relationship between every two defect instances to determine their spatial relationships.

[0081] Specifically, the position embedding module calculates the relative position and scaling relationship between the bounding boxes of two objects, generating a position matrix to represent the geometric relationship between the bounding boxes, expressed as:

[0082] ;

[0083] In the formula, the bounding box From its central coordinates and its width and height Definition, bounding box That is also true;

[0084] Through this calculation method, the model can obtain the relative position information (such as horizontal and vertical offsets) and scale relationship (such as the ratio of width to height) between two object bounding boxes, thereby constructing a spatially invariant geometric feature representation, ensuring that the model's embedded features remain robust when facing translation and scaling transformations.

[0085] Furthermore, to further improve computational efficiency and reduce memory consumption, the location embedding module imposes sparsity constraints when generating the location matrix, retaining only defect instance pairs with significant geometric relationships, thus reducing unnecessary computational overhead. In addition, the location embedding module performs further high-dimensional representation transformation. Specifically, it uses sine and cosine functions to map the four geometric features, transforming them into a higher-dimensional embedding (such as a 64-dimensional high-dimensional embedding), enabling it to better adapt to the high-dimensional feature space of deep learning models.

[0086] In one embodiment of the present invention, the relation reasoning module in the relation modeling module analyzes the semantic features of each defect instance, uses an attention mechanism to quantify the interaction between these features, captures the relationship between defect instances at the semantic feature level, so that the model can not only perceive the features of each defect instance, but also better understand the intrinsic connection between instance features.

[0087] Specifically, after modeling the relative position and scale relationship between two sets of objects through the position embedding module, the relationship reasoning module performs a dot product operation on the semantic feature vectors of the two sets of objects to generate semantic relationship weights. These weights are used to measure the semantic similarity or correlation between defect instances, which is a measure of the connection at the content level of the objects. Through this dot product operation of semantic features, the model can identify defect instances with similar attributes or categories, and further optimize the ability to distinguish defects based on this similarity.

[0088] In this embodiment of the invention, the relation reasoning module combines geometric and semantic features to calculate the relationship features between two sets of objects. After generating geometric relation weights and semantic relation weights, the relation reasoning module combines the two to obtain the comprehensive relation weight for each pair of objects. This weight fusion considers both geometric and semantic factors, providing the model with a more comprehensive relation modeling result. Specifically, the fusion formula for geometric and semantic relation weights is as follows:

[0089] ;

[0090] In the formula, Represents relation weights. Represents the geometric relationship weights. Indicates the semantic relation weight;

[0091] In this embodiment of the invention, through this fusion operation, the relational reasoning module can perform weighted aggregation of the semantic features of all objects to generate a global relational feature representation that reflects the spatial and semantic relational interaction information between defective instances.

[0092] In one embodiment of the present invention, the relation feature selection mechanism in the relation modeling module dynamically adjusts the application of relation features to reduce the influence of irrelevant information. This mechanism effectively controls the relation weights through a classification score threshold p (exemplarily, p=0.6), thereby reasonably filtering the model's output so that only object features with high confidence are retained, while low-confidence instances are further optimized or adjusted in the process.

[0093] In this embodiment of the invention, the relation feature selection mechanism divides bounding boxes into two categories—high-confidence and low-confidence instances—based on their classification scores. For high-confidence bounding boxes, the model assumes that their corresponding features already sufficiently represent the core information of the object, thus keeping their features unchanged to avoid potential noise interference from further operations. For low-confidence instances, additional relation features are introduced for optimization, thereby enhancing the representational ability of this type of instance and enabling the model to more accurately identify these potential defective targets.

[0094] Under this mechanism, the enhanced feature map The calculation formula is:

[0095] ;

[0096] In the formula, Represents the original feature map. Indicates relational characteristics, This represents the enhanced feature map.

[0097] In one embodiment of the present invention, the target loss calculation module is used to measure the deviation between the predicted result and the target label, and to improve the detection performance of the model by optimizing the YOLOv8 loss function. This loss function contains three main loss terms, each undertaking a different optimization task, and its calculation formula is as follows:

[0098] ;

[0099] In the formula, The bounding box loss (bounding box regression loss) is used to measure the difference in position and shape between the predicted bounding box and the ground truth box. This is the classification loss (confidence regression loss), used to calculate the difference between the class probability distribution output by the model and the true class. The cross-entropy loss method focuses on optimizing the distribution around the target location. This loss selects the two positions closest to the label y for probability optimization, guiding the model to focus more on the specific location of the target and the distribution information of its neighboring region. This loss helps the model converge to the target location more quickly and accurately, reducing interference from uncertain regions. (Rectangular box loss weights) Set to 7.5, confidence level regression loss weight It is 1.5. Loss weights Both are 0.5.

[0100] Please refer to Figure 2. A specific embodiment of the present invention provides a method for detecting surface defects of Mini-LEDs based on YOLOv8 and relational modeling, which may include the following steps:

[0101] Mini-LED images are acquired, and regions containing defects in the images are finely annotated. The annotations include the bounding boxes of the defects and the defect category information, so as to provide accurate supervision signals for the model during subsequent training.

[0102] Construct a defect detection model for identifying and locating defects in Mini-LED images;

[0103] The constructed defect detection model was trained using the Mini-LED surface defect detection dataset to obtain the trained defect detection model.

[0104] The trained defect detection model is used to detect defects in Mini-LED images, and the bounding boxes and categories of defects are output.

[0105] Please refer to Figure 3. In this embodiment of the invention, the defect detection model is based on the YOLOv8 network architecture, specifically including:

[0106] The data input module is used for data input and preprocessing, including data augmentation of the input image;

[0107] The feature extraction module uses a series of convolutional and deconvolutional layers to perform depth convolution operations on Mini-LED surface defect images, extracting the basic features of the images and generating multi-scale feature maps.

[0108] The feature fusion module performs convolutional fusion on multi-scale features. By fusing feature maps from different stages of the backbone, it enhances feature representation capabilities and ensures that the network can detect defects of different sizes at different feature scales.

[0109] The relation modeling module, including the location embedding module and the relation reasoning module, enhances the representation capability of defect features by integrating spatial and semantic information;

[0110] The detection module is responsible for the final object detection and classification tasks. It generates the final predicted bounding boxes and predicted categories based on the network model, calculates the object loss, and guides the training and optimization of the model.

[0111] Specifically, as an example, the detection process of the method in the embodiments of the present invention may include the following steps:

[0112] Step 1: Construct a dataset of Mini-LED surface defects in an industrial environment. The collected images satisfy the three color channels of RGB and include corresponding manual annotation results. The dataset pixel size is 360*240. The dataset is randomly divided into training and test sets. The collected images are sent to the computer that executes the algorithm.

[0113] Step 2: Construct a defect detection model, including a sequential connection feature extraction module, a multi-scale feature fusion module, a relationship modeling module, and a detection module;

[0114] Please see Figure 4 The relation reasoning module includes a location embedding module, a relation reasoning module, and a relation feature selection mechanism;

[0115] The system comprises several modules: a location embedding module that encodes the relative positions and scaling relationships between bounding boxes, introducing spatial relationships between objects; a relationship inference module that combines geometric and semantic features to further model and quantify the relationships between objects, generating more accurate relationship weights through the comprehensive use of geometric and semantic features, thus helping the model better understand the interactions between defect instances; a relationship feature selection mechanism that dynamically adjusts the application of features to improve the model's ability to detect low-confidence instances, dynamically optimizing the model's detection performance without affecting the features of high-confidence instances; and a prediction box that generates a prediction box at the output and sets non-maximum suppression (NMS) operation, retaining the prediction box for calculating the target loss function, and feeding the final loss value back to the defect detection model for parameter updates.

[0116] Step 3: Data preprocessing. Before inputting the data into the network for training, each Mini-LED surface defect image is first scaled to a length of 320 pixels while maintaining the aspect ratio. Then, random left-right and up-down scaling and image stitching are performed for data augmentation. The manually labeled results are also transformed accordingly. Finally, all images are filled with outer pixels to maintain consistent image size.

[0117] Step 4: Training process. After the image is input into the network, it passes through the feature extraction module and the feature fusion module to obtain 3 effective feature maps. Then, each layer is input into the relationship modeling module for feature enhancement. The enhanced feature maps of the 3 layers are output. Finally, the detection module outputs the location and classification results of defects.

[0118] Please see Figure 5 In this embodiment of the invention, mAP, precision, and recall are used to evaluate the defect detection results. It can be seen that the new method provided by this embodiment of the invention has a high mAP value, indicating that the method of the present invention can effectively detect surface defects of Mini-LEDs and has obvious advantages over the conventional YOLOv8 method.

[0119] In the following specific embodiments of the present invention: the Mini-LED surface defect dataset used contains 2545 RGB color images of Mini-LED bead surfaces taken in real industrial scenes, with an image size of 360*240 pixels; these images are randomly shuffled and divided into a training set consisting of 2292 images and a test set consisting of 253 images. During the preprocessing stage, the images are first scaled to 320*320 pixels. In this embodiment, the defect detection model parameters are initialized as follows: the parameters of the feature extraction module are initialized using a pre-trained YOLOv8 network, while the parameters of the remaining modules are randomly initialized. The operating environment is a computer with frameworks such as PyTorch, capable of reading given images and completing the construction and training of the model. The training time for this embodiment on a Gold 6626R@2.90GHz CPU, 8GB of RAM, and an NVIDIA GeForce RTX3090 GPU is approximately 5 hours.

[0120] The specific implementation steps include: First, setting relevant training parameters, setting the optimizer used for network updates in this invention to a stochastic gradient descent optimizer, setting its momentum value to 0.9, and its learning rate to 0.01. The deep object detection network consists of a feature extraction module with five stages, a multi-scale feature fusion module, a relation modeling module, and a detection module. The network input is an RGB image. First, the feature extraction module extracts the texture and abstract semantic information of the detected defects. Its output, a five-level feature map, is fused by the multi-scale feature fusion module to obtain a three-level feature map, which is then passed to the relation modeling module for feature enhancement. Finally, it is input to the detection module to obtain a detection result image containing rectangular anchor box labels and classification confidence. When training the network using a partitioned dataset, eight images are randomly selected from the training set and input into the network each time. The selected stochastic gradient descent optimizer is used for parameter updates. Training is completed after 300 iterations on the dataset. Finally, images from the test set are input into the trained network for detection to obtain the results of the embodiment of the invention.

[0121] In summary, this invention discloses a method for detecting surface defects in Mini-LEDs based on YOLOv8 and relational modeling, comprising the following steps: First, acquiring Mini-LED images and annotating the bounding boxes and category information of defects; second, constructing a defect detection model, which is based on the YOLOv8 architecture and mainly consists of data input, feature extraction, feature fusion, relational modeling, and detection modules. The feature extraction module extracts basic image features through deep convolution operations, while the feature fusion module performs convolutional fusion on multi-scale features to ensure that the network can effectively detect defects of different sizes. The relational modeling module encodes the spatial positional relationships between objects, analyzes the semantic features of defect instances using an attention mechanism, captures the inherent spatial and semantic relationships between defects, and dynamically adjusts the application of features through a relational feature selection mechanism to filter irrelevant information; finally, updating parameters by calculating a loss function. The technical solution of this invention demonstrates stronger adaptability in handling complex-shaped defects, effectively avoids redundant detection, and improves detection accuracy and recall.

[0122] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0123] Please see Figure 6 In this embodiment of the invention, a Mini-LED surface defect detection system is provided, comprising:

[0124] The image acquisition module is used to acquire Mini-LED images of the surface defects to be detected;

[0125] The defect detection module is used to perform defect detection based on the acquired Mini-LED image of the surface defect to be detected, using a trained defect detection model to obtain the surface defect detection results.

[0126] The defect detection model includes:

[0127] The data input module is used to input Mini-LED image data;

[0128] The feature extraction module is used to perform depth convolution operations on the input Mini-LED image to generate multi-scale feature maps of different scales.

[0129] The feature fusion module is used to perform convolutional fusion operations on the input multi-scale feature maps to obtain the fused feature maps.

[0130] A relationship modeling module is used to input the fused feature map and enhance the representation capability of defect features by integrating spatial and semantic information to obtain a modeled relationship feature representation. This module includes a position embedding module and a relationship inference module. The position embedding module is used to input the fused feature map and model the relative position and scale relationship between two sets of objects to generate a position matrix. The relationship inference module is used to input the position matrix and generate a global relationship feature representation.

[0131] The detection module is used to input the modeled relational feature representation and perform defect detection, and output the surface defect detection results.

[0132] In one embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used to perform the operation of a Mini-LED surface defect detection method.

[0133] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the Mini-LED surface defect detection method in the above embodiments.

[0134] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0135] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for detecting surface defects in Mini-LEDs, characterized in that, Includes the following steps: Acquire Mini-LED images of the surfaces to be inspected; Based on the acquired Mini-LED image of the surface defect to be detected, the trained defect detection model is used to perform defect detection and obtain the surface defect detection results. The defect detection model includes: The data input module is used to input Mini-LED image data; The feature extraction module is used to perform depth convolution operations on the input Mini-LED image to generate multi-scale feature maps of different scales. The feature fusion module is used to perform convolutional fusion operations on the input multi-scale feature maps to obtain the fused feature maps. A relationship modeling module is used to input the fused feature map and enhance the representation capability of defect features by integrating spatial and semantic information to obtain a modeled relationship feature representation. This module includes a position embedding module and a relationship inference module. The position embedding module is used to input the fused feature map and model the relative position and scale relationship between two sets of objects to generate a position matrix. The relationship inference module is used to input the position matrix and generate a global relationship feature representation. The detection module is used to input the modeled relational feature representation and perform defect detection, and output the surface defect detection results; In the location embedding module, the steps of executing the input fused feature map, modeling the relative position and scale relationship between the two sets of objects, and generating the location matrix include: The relative position and scaling relationship between the bounding boxes of two objects are calculated to generate a position matrix, which represents the geometric relationship between the bounding boxes. The expression is as follows: ； In the formula, bounding box The center coordinates, bounding box Width and height; bounding box The center coordinates, bounding box Width and height; In the location embedding module, sparsity constraints are applied when generating the location matrix; In the location embedding module, when generating the location matrix, sine and cosine functions are used to map the geometric features, transforming them into a higher-dimensional embedding.

2. The method for detecting surface defects in Mini-LEDs according to claim 1, characterized in that, The surface defect detection results include the defect's bounding box and category information.

3. The method for detecting surface defects in Mini-LEDs according to claim 1, characterized in that, In the relation reasoning module, the step of generating a global relation feature representation by inputting the position matrix includes: Geometric relation weights and semantic relation weights are generated and fused to obtain the comprehensive relation weights of the two sets of objects; among them, semantic relation weights are generated by performing a dot product operation on the semantic feature vectors of the two sets of objects. Based on the comprehensive relation weight, the semantic features of all objects are weighted and aggregated to generate a global relation feature representation; In the process of generating geometric relation weights and semantic relation weights and fusing them, the fusion calculation expression is as follows: ； In the formula, Indicates the weight of the overall relationship; Indicates the weight of geometric relationships; Represents the semantic relation weight.

4. The method for detecting surface defects in Mini-LEDs according to claim 1, characterized in that, The relationship modeling module also includes a relationship feature selection mechanism. Based on the classification score and preset threshold of each bounding box, the relationship feature selection mechanism divides the bounding boxes into two categories: high-confidence instances and low-confidence instances. The features of high-confidence instances remain unchanged, while the features of low-confidence instances are optimized and enhanced by introducing additional relationship features. The calculation formula for optimization and enhancement is as follows: ； In the formula, This represents the optimized and enhanced feature map; Represents the original feature map; Indicates relational characteristics.

5. The method for detecting surface defects in Mini-LEDs according to claim 1, characterized in that, The defect detection model is trained using a supervised training method, and the loss function used during training is expressed as follows: ； In the formula, The rectangular box loss measures the difference in position and shape between the predicted bounding box and the ground truth box. The classification loss is used to calculate the difference between the class probability distribution output by the model and the true class. Cross-entropy loss is used to optimize the distribution around the target location; They are respectively The weighting coefficients.

6. A Mini-LED surface defect detection system, characterized in that, include: The image acquisition module is used to acquire Mini-LED images of the surface defects to be detected; The defect detection module is used to perform defect detection based on the acquired Mini-LED image of the surface defect to be detected, using a trained defect detection model to obtain the surface defect detection results. The defect detection model includes: The data input module is used to input Mini-LED image data; The feature extraction module is used to perform depth convolution operations on the input Mini-LED image to generate multi-scale feature maps of different scales. The feature fusion module is used to perform convolutional fusion operations on the input multi-scale feature maps to obtain the fused feature maps. A relationship modeling module is used to input the fused feature map and enhance the representation capability of defect features by integrating spatial and semantic information to obtain a modeled relationship feature representation. This module includes a position embedding module and a relationship inference module. The position embedding module is used to input the fused feature map and model the relative position and scale relationship between two sets of objects to generate a position matrix. The relationship inference module is used to input the position matrix and generate a global relationship feature representation. The detection module is used to input the modeled relational feature representation and perform defect detection, and output the surface defect detection results; In the location embedding module, the steps of executing the input fused feature map, modeling the relative position and scale relationship between the two sets of objects, and generating the location matrix include: The relative position and scaling relationship between the bounding boxes of two objects are calculated to generate a position matrix, which represents the geometric relationship between the bounding boxes. The expression is as follows: ； In the formula, bounding box The center coordinates, bounding box Width and height; bounding box The center coordinates, bounding box Width and height; In the location embedding module, sparsity constraints are applied when generating the location matrix; In the location embedding module, when generating the location matrix, sine and cosine functions are used to map the geometric features, transforming them into a higher-dimensional embedding.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the Mini-LED surface defect detection method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the Mini-LED surface defect detection method as described in any one of claims 1 to 5.

Images