Image description method and device for Mini LED defect detection
By combining a Transformer-based multimodal model with an image processing library, the interpretability and intuitiveness issues of Mini LED defect detection are solved, enabling accurate defect detection and intuitive display, thus meeting the production line's need for rapid and accurate quality assessment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-30
- Publication Date
- 2026-04-07
AI Technical Summary
Existing Mini LED defect detection methods are insufficient in terms of interpretability and intuitiveness, making it difficult to meet the production line's need for rapid and accurate quality assessment.
By employing a Transformer-based multimodal model, combined with a visual encoder and a language decoder, defect description text is generated through image acquisition, annotation, data augmentation, and model training. The OpenCV and PIL libraries are then used for visualization, enabling accurate defect detection and intuitive display.
It enables intuitive presentation of Mini LED defect detection results, meets the production line's need for rapid and accurate quality assessment, and provides necessary technical support.
Smart Images
Figure CN121810573A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to an image description method and apparatus for Mini LED defect detection. Background Technology
[0002] Mini LED is a next-generation display technology with chip sizes between 100 and 300 micrometers, falling between traditional LEDs and Micro LEDs. Due to its advantages such as low power consumption, fast response, and high resolution, Mini LED has been widely used in various products, including televisions, mobile phones, and smartwatches. However, despite relatively mature manufacturing processes and skilled workers, Mini LED production remains susceptible to various defects. Because this technology requires extremely high precision, even minute defects can cause the entire Mini LED panel to malfunction.
[0003] In actual production, traditional manual point-by-point inspection methods are not only inefficient but also lack accuracy. Machine vision-based inspection methods can solve these problems. In recent years, with the development of deep learning, related algorithms have gradually replaced traditional image processing methods and demonstrated strong performance in the field of defect detection. Among them, commonly used algorithms in defect detection are classified into three categories: classification, object detection, and semantic segmentation. Classification algorithms can distinguish whether a defect exists, but cannot determine the location of the defect. Object detection methods can provide the location of the defect, but it is difficult to accurately describe the shape and extent of the defect. Semantic segmentation methods can simultaneously obtain the type, location, and shape of the defect, and have higher detection accuracy. However, the results of semantic segmentation are usually presented in the form of a mask image, which is not intuitive enough for production line quality inspectors and makes it difficult to quickly understand the defect type and its impact on product performance.
[0004] In summary, although semantic segmentation technology has shown good adaptability in Mini LED defect detection, existing detection methods still have shortcomings in terms of interpretability and intuitiveness, and cannot meet the production line's need for rapid and accurate quality assessment. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of the prior art. This invention provides an image description method and device for Mini LED defect detection, which realizes intuitive presentation of defect detection results and can meet the needs of production lines for rapid and accurate quality assessment.
[0006] To address the aforementioned technical problems, embodiments of the present invention provide an image description method for Mini LED defect detection, the method comprising: Obtain an image of a Mini LED light group located on a Mini LED panel; generate corresponding defect description text based on the annotation results in the Mini LED light group image; and construct a dataset using the Mini LED light group image and the corresponding defect description text, the dataset including a training dataset and a validation dataset. A Transformer-based multimodal model is constructed, and the dataset is input into the multimodal model for training to form a trained multimodal model, which includes a visual encoder and a language decoder. Obtain an image of the Mini LED light group to be processed, and input the image of the Mini LED light group to be processed into a trained multimodal model for prediction processing to obtain the segmentation result and description text corresponding to the image of the Mini LED light group to be processed. The defects in the Mini LED light group image to be processed are visualized using the OpenCV library and the segmentation results, and the descriptive text is drawn on the visualized Mini LED light group image to be processed for display using the PIL library.
[0007] Optionally, obtaining the image of the Mini LED light group located on the Mini LED panel includes: Based on the high-definition image acquisition device, the Mini LED on the Mini LED panel is image acquired and processed according to the preset exposure frequency to obtain several Mini LED images. Several Mini LED images are stitched together to form a complete Mini LED image corresponding to the Mini LED panel; The coordinates of the entire Mini LED image are matched using a template matching algorithm. The coordinate matching results are then used to cut the Mini LED light group images in the entire Mini LED image to obtain the Mini LED light group images located on the Mini LED panel.
[0008] Optionally, generating corresponding defect description text based on the annotation results in the Mini LED light group image includes: The Mini LED light group image is filled into a square image, and the resolution of the square image is scaled down to 256*256; A 256*256 forward-facing image is input into the SAM model, and the input image is semi-automatically annotated based on the prompting mechanism of the SAM model to obtain the annotation results in the Mini LED light group image. The annotated defects include foreign object category, LED bead defect category, and LED bead abnormality category. The corresponding segmentation mask is generated based on the annotation results in the Mini LED light group image, and the generated segmentation mask is input into a Python script. The Python script generates corresponding defect description text based on the pixel values contained in the segmentation mask.
[0009] Optionally, constructing a dataset using the Mini LED light group image and the corresponding defect description text includes: The Mini LED light group image is enhanced by random rotation and horizontal flipping to form an enhanced Mini LED light group image. Random rotation is achieved by generating a rotation angle by random sampling within a preset angle range and rotating the image using affine transformation. Horizontal flipping is achieved by mirroring the image pixel matrix in the horizontal direction. A dataset was constructed using enhanced images of Mini LED light clusters and corresponding defect description texts.
[0010] Optionally, the visual encoder is a Transformer-based semantic segmentation model used to segment defects in the MiniLED light group image and output the defect features extracted from the MiniLED light group image. The language decoder uses a pre-trained open-source large language model to generate a corresponding description of the defect based on the defect features output by the visual encoder.
[0011] Optionally, the step of inputting the dataset into the multimodal model for training to form a trained multimodal model includes: The training dataset from the dataset is input into the multimodal model for training. After training is completed, the validation dataset from the dataset is input into the trained multimodal model to obtain the output test segmentation results and test description results. The pixel-level cross-entropy loss and Dice loss are calculated using the test segmentation results and the real mask of the corresponding Mini LED light group image in the verification dataset to obtain the pixel-level cross-entropy loss results and Dice loss results. The pixel-level cross-entropy loss result and the Dice loss result are added together according to a preset ratio to obtain the first weighted loss result; The token-level cross-entropy loss is calculated by using the test description results and the corresponding defect description text in the verification dataset to obtain the token-level cross-entropy loss calculation result. The first weighted loss result is weighted and the token-level cross-entropy loss calculation result is weighted to obtain the second weighted loss result; Determine whether the second weighted loss result has reached a stationary state. If not, update the parameters of the trained multimodal model and input the training dataset from the dataset into the multimodal model for training. If a stable state is reached, the multimodal model training is complete.
[0012] Optionally, updating the parameters of the trained multimodal model includes: The visual encoder in the multimodal model is fully updated using backpropagation and gradient descent, and the language decoder in the multimodal model is fine-tuned using LoRA technology to form a multimodal model with updated parameters.
[0013] Optionally, the step of visualizing defects in the Mini LED light group image using the OpenCV library and the segmentation results includes: The "findContours" function in the OpenCV library is used to obtain the segmentation results of each defect contour in the Mini LED light group image to be processed; The "drawContours" function in the OpenCV library is used to draw the contours of each defect in the Mini LED light group image to be processed, and the drawing contours of each defect are obtained. Based on the drawn contours of each defect, the defects in the segmentation results of the Mini LED light group image to be processed are visualized.
[0014] Optionally, the step of using the PIL library to draw and display the descriptive text on the visualized Mini LED light group image includes: Use the fromarray function in the PIL.Image library to generate the corresponding PIL object; Use the Draw function in the PIL.ImageDraw library to create a drawing object that draws text on a visualized Mini LED light group image; The text function of the drawing object is used to draw the descriptive text onto the segmentation result of the Mini LED light group image that has been visualized, according to the PIL object, and then display it.
[0015] In addition, embodiments of the present invention also provide an image description device for Mini LED defect detection, the device comprising: Data construction module: used to obtain Mini LED light group images located on Mini LED panels, generate corresponding defect description text based on the annotation results in the Mini LED light group images, and construct a dataset using the Mini LED light group images and the corresponding defect description texts, the dataset including training dataset and validation dataset; Model training module: used to build a Transformer-based multimodal model, and input the dataset into the multimodal model for training to form a trained multimodal model, which includes a visual encoder and a language decoder; Prediction module: used to obtain the Mini LED light group image to be processed, and input the Mini LED light group image to be processed into the trained multimodal model for prediction processing to obtain the segmentation result and description text corresponding to the Mini LED light group image to be processed; Description drawing and display module: It is used to visualize the defects in the Mini LED light group image to be processed using the OpenCV library and the segmentation results, and to draw the description text on the visualized Mini LED light group image to be processed using the PIL library for display.
[0016] In this embodiment of the invention, a trained multimodal model is used to predict defects in the Mini LED lamp group image to be processed. Then, the prediction results are visualized using the OpenCV library, and the generated descriptive text is displayed on the image using the PIL library. This achieves accurate detection and description of defects in Mini LED images and provides visualization results, providing necessary technical support for quality inspectors to quickly assess product quality. Furthermore, it enables intuitive presentation of defect detection results, meeting the needs of the production line for rapid and accurate quality assessment. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 This is a flowchart illustrating the image description method for Mini LED defect detection in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structural composition of the image description device for Mini LED defect detection in an embodiment of the present invention; Figure 3This is a schematic diagram of the Mini LED image acquisition device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of the multimodal model provided in the embodiments of the present invention; Detailed Implementation
[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] Example 1, please refer to Figure 1 , Figure 1 This is a flowchart illustrating the image description method for Mini LED defect detection in an embodiment of the present invention.
[0021] like Figure 1 As shown, an image description method for Mini LED defect detection includes: S101: Obtain an image of the Mini LED light group located on the Mini LED panel, generate corresponding defect description text based on the annotation results in the Mini LED light group image, and construct a dataset using the Mini LED light group image and the corresponding defect description text. The dataset includes a training dataset and a validation dataset. In this embodiment of the invention, obtaining the Mini LED light group image located on the Mini LED panel includes: performing image acquisition processing on the Mini LEDs on the Mini LED panel according to a preset exposure frequency using a high-definition image acquisition device to obtain several Mini LED images; stitching the several Mini LED images together to form a whole Mini LED image corresponding to the Mini LED panel; performing coordinate matching processing on the coordinates of the whole Mini LED image based on a template matching algorithm, obtaining the coordinate matching result, and then performing segmentation processing on the Mini LED light group image in the whole Mini LED image to obtain the Mini LED light group image located on the Mini LED panel.
[0022] Furthermore, the step of generating corresponding defect description text based on the annotation results in the Mini LED light group image includes: filling the Mini LED light group image into a square image and scaling the resolution of the square image to 256*256; inputting the 256*256 scaled positive image into the SAM model, and performing semi-automatic annotation processing on the input image based on the prompting mechanism of the SAM model to obtain the annotation results in the Mini LED light group image, wherein the annotated defects include foreign object categories, LED bead defect categories, and LED bead anomaly categories; generating a corresponding segmentation mask based on the annotation results in the Mini LED light group image, and inputting the generated segmentation mask into a Python script, wherein the Python script generates corresponding defect description text based on the pixel values contained in the segmentation mask.
[0023] Furthermore, the step of constructing a dataset using the Mini LED light group image and the corresponding defect description text includes: performing image enhancement processing on the Mini LED light group image using random rotation and horizontal flipping to form an enhanced Mini LED light group image, wherein random rotation is achieved by generating a rotation angle through random sampling within a preset angle range and performing rotation processing on the image using affine transformation, and horizontal flipping is achieved by performing a mirror transformation on the image pixel matrix in the horizontal direction; and constructing a dataset using the enhanced Mini LED light group image and the corresponding defect description text.
[0024] Specifically, high-definition images of the Mini LED panel are acquired using a high-definition image acquisition device, with a preset exposure frequency of 10 images per second; images of the incoming Mini LED material are captured below, and an algorithm is used to stitch together the images corresponding to a whole material board to obtain an image of the entire material board. A template matching algorithm is used to match the coordinates of the material board, and then images of each Mini LED light group are cut out to construct a dataset; such as Figure 3 and Figure 4 As shown, several Mini LED images are stitched together to form a complete Mini LED image corresponding to the Mini LED panel. Then, the coordinates of the complete Mini LED image are matched using a template matching algorithm. The coordinate matching results are used to cut the Mini LED light group images in the complete Mini LED image to obtain the Mini LED light group images located on the Mini LED panel.
[0025] Then, the Mini LED light group image is filled into a square image, and the resolution of the square image is scaled down to 256. 256, resolution at 256 At a scale of 256, subsequent training efficiency can be improved; then, based on the SAM (Segment Anything Model) prompting mechanism, semi-automatic annotation is implemented to construct semantic segmentation labels for Mini LED images, which will be scaled down to 256. A 256-fold positive-direction image is input into the SAM model. The SAM model's prompting mechanism performs semi-automatic annotation processing on the input image to obtain the annotated defects in the Mini LED light group image. The annotated defects include foreign object defect categories, LED bead defect categories, and LED bead abnormality defect categories. A corresponding segmentation mask is generated based on the defect categories of the annotated defects in the Mini LED light group image. The generated segmentation mask is then input into a Python script, which generates corresponding defect description text based on the pixel values contained in the segmentation mask. Specifically, the image is annotated using the SAM-based prompting mechanism to generate a segmentation mask. The mask is then input into the Python script, which generates corresponding description text based on the pixel values contained in the mask.
[0026] Then, in the dataset construction, image enhancement processing of Mini LED light group images is performed using random rotation and horizontal flipping to form enhanced Mini LED light group images. Random rotation is achieved by generating rotation angles through random sampling within a preset angle range and then rotating the image using affine transformation. Horizontal flipping is achieved by mirroring the image pixel matrix in the horizontal direction, that is, swapping the pixels at the left and right positions of the image. The enhanced Mini LED light group images are then used to construct the dataset with the corresponding defect description text.
[0027] The image data processing method is not limited to the specific implementations mentioned above. The method and proportion of dataset division can be flexibly set, the image preprocessing method is also unrestricted, and the annotation tools and data augmentation methods used can also be diversified. The general principle is that the selection of tools should prioritize convenience and efficiency, and the preprocessing methods and data augmentation methods should be selected in combination with the characteristics of the dataset. The purpose is to improve the prediction accuracy of the model.
[0028] S102: Construct a Transformer-based multimodal model, and input the dataset into the multimodal model for training to form a trained multimodal model, wherein the multimodal model includes a visual encoder and a language decoder; In the specific implementation of this invention, the visual encoder is a Transformer-based semantic segmentation model used to segment defects in the Mini LED light group image and output the defect features extracted from the Mini LED light group image; the language decoder uses a pre-trained open-source large language model to generate a corresponding defect description based on the defect features output by the visual encoder.
[0029] Furthermore, the step of inputting the dataset into the multimodal model for training to form a trained multimodal model includes: inputting the training dataset from the dataset into the multimodal model for training; after training is completed, inputting the validation dataset from the dataset into the trained multimodal model to obtain the output test segmentation results and test description results; and using the test segmentation results and the corresponding Mini... The pixel-level cross-entropy loss and Dice loss are calculated using the true mask of the LED light group image to obtain pixel-level cross-entropy loss results and Dice loss results. The pixel-level cross-entropy loss results and Dice loss results are added according to a preset ratio to obtain a first weighted loss result. The token-level cross-entropy loss is calculated using the test description result and the corresponding defect description text in the verification dataset to obtain a token-level cross-entropy loss calculation result. The first weighted loss result and the token-level cross-entropy loss calculation result are weighted to obtain a second weighted loss result. It is determined whether the second weighted loss result has reached a stationary state. If it has not, the parameters of the trained multimodal model are updated, and the training dataset in the dataset is input into the multimodal model for training. If it has reached a stationary state, the multimodal model training is complete.
[0030] Furthermore, the parameter update of the trained multimodal model includes: performing a full update of the weights of the visual encoder in the multimodal model using backpropagation and gradient descent, and fine-tuning the parameters of the language decoder in the multimodal model using LoRA technology, to form a multimodal model with updated parameters.
[0031] Specifically, the structure of the constructed Transformer-based multimodal model is as follows: Figure 4As shown; the visual encoder is a Transformer-based semantic segmentation model used to segment defects in Mini LED light group images and output the defect features extracted from the Mini LED light group images; the language decoder uses a pre-trained open-source large language model to generate corresponding defect descriptions based on the defect features output by the visual encoder; the language decoder uses a pre-trained open-source large language model, such as BERT, LLaMA, GPT, Qianwen, and BLIP, to generate corresponding defect descriptions based on the extracted image features. During training, LoRA (Low-Rank Adaptation) technology is used to fine-tune the large language model. The steps include: introducing a low-rank adaptation module into a specified linear layer of the pre-trained language model and keeping the original model parameters frozen; aligning and mapping the image features output by the visual encoder with the input embedding of the language model to form a feature sequence that can be received by the language model; during training, only the parameters of the low-rank adaptation module are updated, and the language model is effectively fine-tuned through the learning of this low-rank structure.
[0032] During the training of the multimodal model, the training dataset from the dataset is input into the multimodal model for training. After training, the validation dataset from the dataset is input into the trained multimodal model to obtain the output test segmentation results and test description results. Then, the test segmentation results are used with the ground truth mask of the corresponding Mini LED light group image in the validation dataset to calculate pixel-wise cross-entropy loss and Dice loss, obtaining pixel-wise cross-entropy loss results and Dice loss results. The pixel-wise cross-entropy loss results and Dice loss results are added according to a preset ratio to obtain the first weighted loss result. Finally, the test description results are used with the corresponding defect description text in the validation dataset to perform token-level cross-entropy loss. The multimodal model is processed by calculating the cross-entropy loss (LOS) to obtain the token-level cross-entropy loss result. Finally, it is determined whether the second weighted loss result has reached a stationary state. If it has not reached a stationary state, the parameters of the trained multimodal model are updated, and the training dataset in the dataset is input into the multimodal model for training. If it has reached a stationary state, the multimodal model training is complete.
[0033] During parameter updates, backpropagation and gradient descent are used to update all parameters of the visual encoder in the multimodal model, and LoRA technology is used to fine-tune the parameters of the language decoder in the multimodal model. That is, without performing an overall parameter update, the multimodal model with updated parameters can be formed.
[0034] S103: Obtain the Mini LED light group image to be processed, and input the Mini LED light group image to be processed into the trained multimodal model for prediction processing to obtain the segmentation result and description text corresponding to the Mini LED light group image to be processed. In the specific implementation of this invention, it is first necessary to obtain the Mini LED light group image to be processed, and then input the Mini LED light group image to be processed into the trained multimodal model for prediction processing to obtain the segmentation result and description text corresponding to the Mini LED light group image to be processed.
[0035] S104: Using the OpenCV library and the segmentation results, the defects in the Mini LED light group image to be processed are visualized, and the descriptive text is drawn on the visualized Mini LED light group image to be processed and displayed using the PIL library.
[0036] In the specific implementation of this invention, the visualization processing of defects in the Mini LED light group image to be processed using the OpenCV library and the segmentation results includes: using the "findContours" function in the OpenCV library to obtain the contours of each defect in the segmentation results of the Mini LED light group image to be processed; using the "drawContours" function in the OpenCV library to draw the contours of each defect in the Mini LED light group image to be processed, obtaining the drawn contours of each defect; and visualizing the defects in the segmentation results of the Mini LED light group image to be processed based on the drawn contours of each defect.
[0037] Furthermore, the step of using the PIL library to draw the descriptive text on the visualized Mini LED light group image for display includes: using the fromarray function in the PIL.Image library to generate a corresponding PIL object; using the Draw function in the PIL.ImageDraw library to create a drawing object for drawing text on the visualized Mini LED light group image; and using the text function of the drawing object to draw the descriptive text on the visualized Mini LED light group image according to the PIL object for display.
[0038] Specifically, after obtaining the segmentation results and descriptive text corresponding to the Mini LED light group image to be processed, the first step is to use the "findContours" function in the OpenCV library to obtain the contours of each defect in the segmentation results of the Mini LED light group image to be processed; then, the "drawContours" function in the OpenCV library is used to draw the contours of each defect in the Mini LED light group image to be processed, obtaining the drawn contours of each defect; finally, the defects in the segmentation results of the Mini LED light group image to be processed are visualized based on the drawn contours of each defect.
[0039] During the demonstration, the fromarray function in the PIL.Image library is used to generate the corresponding PIL object; the Draw function in the PIL.ImageDraw library is used to create a drawing object to draw text on the visualized Mini LED light group image; and the text function of the drawing object is used to draw the descriptive text on the segmentation result of the visualized Mini LED light group image according to the PIL object for display.
[0040] In this embodiment of the invention, a trained multimodal model is used to predict defects in the Mini LED lamp group image to be processed. Then, the prediction results are visualized using the OpenCV library, and the generated descriptive text is displayed on the image using the PIL library. This achieves accurate detection and description of defects in Mini LED images and provides visualization results, providing necessary technical support for quality inspectors to quickly assess product quality. Furthermore, it enables intuitive presentation of defect detection results, meeting the needs of the production line for rapid and accurate quality assessment.
[0041] Example 2, please refer to Figure 2 , Figure 2 This is a schematic diagram of the structural composition of an image description device for Mini LED defect detection in an embodiment of the present invention.
[0042] like Figure 2 As shown, an image description device for Mini LED defect detection includes: Data construction module 201: used to obtain Mini LED light group images located on Mini LED panels, generate corresponding defect description text based on the annotation results in the Mini LED light group images, and construct a dataset using the Mini LED light group images and the corresponding defect description text, the dataset including a training dataset and a validation dataset; In this embodiment of the invention, obtaining the Mini LED light group image located on the Mini LED panel includes: performing image acquisition processing on the Mini LEDs on the Mini LED panel according to a preset exposure frequency using a high-definition image acquisition device to obtain several Mini LED images; stitching the several Mini LED images together to form a whole Mini LED image corresponding to the Mini LED panel; performing coordinate matching processing on the coordinates of the whole Mini LED image based on a template matching algorithm, obtaining the coordinate matching result, and then performing segmentation processing on the Mini LED light group image in the whole Mini LED image to obtain the Mini LED light group image located on the Mini LED panel.
[0043] Furthermore, the step of generating corresponding defect description text based on the annotation results in the Mini LED light group image includes: filling the Mini LED light group image into a square image and scaling the resolution of the square image to 256*256; inputting the 256*256 scaled positive image into the SAM model, and performing semi-automatic annotation processing on the input image based on the prompting mechanism of the SAM model to obtain the annotation results in the Mini LED light group image, wherein the annotated defects include foreign object categories, LED bead defect categories, and LED bead anomaly categories; generating a corresponding segmentation mask based on the annotation results in the Mini LED light group image, and inputting the generated segmentation mask into a Python script, wherein the Python script generates corresponding defect description text based on the pixel values contained in the segmentation mask.
[0044] Furthermore, the step of constructing a dataset using the Mini LED light group image and the corresponding defect description text includes: performing image enhancement processing on the Mini LED light group image using random rotation and horizontal flipping to form an enhanced Mini LED light group image, wherein random rotation is achieved by generating a rotation angle through random sampling within a preset angle range and performing rotation processing on the image using affine transformation, and horizontal flipping is achieved by performing a mirror transformation on the image pixel matrix in the horizontal direction; and constructing a dataset using the enhanced Mini LED light group image and the corresponding defect description text.
[0045] Specifically, high-definition images of the Mini LED panel are acquired using a high-definition image acquisition device, with a preset exposure frequency of 10 images per second; images of the incoming Mini LED material are captured below, and an algorithm is used to stitch together the images corresponding to a whole material board to obtain an image of the entire material board. A template matching algorithm is used to match the coordinates of the material board, and then images of each Mini LED light group are cut out to construct a dataset; such as Figure 3 and Figure 4As shown, several Mini LED images are stitched together to form a complete Mini LED image corresponding to the Mini LED panel. Then, the coordinates of the complete Mini LED image are matched using a template matching algorithm. The coordinate matching results are used to cut the Mini LED light group images in the complete Mini LED image to obtain the Mini LED light group images located on the Mini LED panel.
[0046] Then, the Mini LED light group image is filled into a square image, and the resolution of the square image is scaled down to 256. 256, resolution at 256 At a scale of 256, subsequent training efficiency can be improved; then, based on the SAM (Segment Anything Model) prompting mechanism, semi-automatic annotation is implemented to construct semantic segmentation labels for Mini LED images, which will be scaled down to 256. A 256-fold positive-direction image is input into the SAM model. The SAM model's prompting mechanism performs semi-automatic annotation processing on the input image to obtain the annotated defects in the Mini LED light group image. The annotated defects include foreign object defect categories, LED bead defect categories, and LED bead abnormality defect categories. A corresponding segmentation mask is generated based on the defect categories of the annotated defects in the Mini LED light group image. The generated segmentation mask is then input into a Python script, which generates corresponding defect description text based on the pixel values contained in the segmentation mask. Specifically, the image is annotated using the SAM-based prompting mechanism to generate a segmentation mask. The mask is then input into the Python script, which generates corresponding description text based on the pixel values contained in the mask.
[0047] Then, in the dataset construction, image enhancement processing of Mini LED light group images is performed using random rotation and horizontal flipping to form enhanced Mini LED light group images. Random rotation is achieved by generating rotation angles through random sampling within a preset angle range and then rotating the image using affine transformation. Horizontal flipping is achieved by mirroring the image pixel matrix in the horizontal direction, that is, swapping the pixels at the left and right positions of the image. The enhanced Mini LED light group images are then used to construct the dataset with the corresponding defect description text.
[0048] The image data processing method is not limited to the specific implementations mentioned above. The method and proportion of dataset division can be flexibly set, the image preprocessing method is also unrestricted, and the annotation tools and data augmentation methods used can also be diversified. The general principle is that the selection of tools should prioritize convenience and efficiency, and the preprocessing methods and data augmentation methods should be selected in combination with the characteristics of the dataset. The purpose is to improve the prediction accuracy of the model.
[0049] Model training module 202: used to construct a Transformer-based multimodal model, and input the dataset into the multimodal model for training to form a trained multimodal model, wherein the multimodal model includes a visual encoder and a language decoder; In the specific implementation of this invention, the visual encoder is a Transformer-based semantic segmentation model used to segment defects in the Mini LED light group image and output the defect features extracted from the Mini LED light group image; the language decoder uses a pre-trained open-source large language model to generate a corresponding defect description based on the defect features output by the visual encoder.
[0050] Furthermore, the step of inputting the dataset into the multimodal model for training to form a trained multimodal model includes: inputting the training dataset from the dataset into the multimodal model for training; after training is completed, inputting the validation dataset from the dataset into the trained multimodal model to obtain the output test segmentation results and test description results; and using the test segmentation results and the corresponding Mini... The pixel-level cross-entropy loss and Dice loss are calculated using the true mask of the LED light group image to obtain pixel-level cross-entropy loss results and Dice loss results. The pixel-level cross-entropy loss results and Dice loss results are added according to a preset ratio to obtain a first weighted loss result. The token-level cross-entropy loss is calculated using the test description result and the corresponding defect description text in the verification dataset to obtain a token-level cross-entropy loss calculation result. The first weighted loss result and the token-level cross-entropy loss calculation result are weighted to obtain a second weighted loss result. It is determined whether the second weighted loss result has reached a stationary state. If it has not, the parameters of the trained multimodal model are updated, and the training dataset in the dataset is input into the multimodal model for training. If it has reached a stationary state, the multimodal model training is complete.
[0051] Furthermore, the parameter update of the trained multimodal model includes: performing a full parameter update on the visual encoder in the multimodal model using backpropagation and gradient descent, and fine-tuning the parameters of the language decoder in the multimodal model using LoRA technology, to form a parameter-updated multimodal model.
[0052] Specifically, the structure of the constructed Transformer-based multimodal model is as follows: Figure 4 As shown; the visual encoder is a Transformer-based semantic segmentation model used to segment defects in Mini LED light group images and output the defect features extracted from the Mini LED light group images; the language decoder uses a pre-trained open-source large language model to generate corresponding defect descriptions based on the defect features output by the visual encoder; the language decoder uses a pre-trained open-source large language model, such as BERT, LLaMA, GPT, Qianwen, and BLIP, to generate corresponding defect descriptions based on the extracted image features. During training, LoRA (Low-Rank Adaptation) technology is used to fine-tune the large language model. The steps include: introducing a low-rank adaptation module into a specified linear layer of the pre-trained language model and keeping the original model parameters frozen; aligning and mapping the image features output by the visual encoder with the input embedding of the language model to form a feature sequence that can be received by the language model; during training, only the parameters of the low-rank adaptation module are updated, and the language model is effectively fine-tuned through the learning of this low-rank structure.
[0053] During the training of the multimodal model, the training dataset from the dataset is input into the multimodal model for training. After training, the validation dataset from the dataset is input into the trained multimodal model to obtain the output test segmentation results and test description results. Then, the test segmentation results are used with the ground truth mask of the corresponding Mini LED light group image in the validation dataset to calculate pixel-wise cross-entropy loss and Dice loss, obtaining pixel-wise cross-entropy loss results and Dice loss results. The pixel-wise cross-entropy loss results and Dice loss results are added according to a preset ratio to obtain the first weighted loss result. Finally, the test description results are used with the corresponding defect description text in the validation dataset to perform token-level cross-entropy loss. The multimodal model is processed by calculating the cross-entropy loss (LOS) to obtain the token-level cross-entropy loss result. Finally, it is determined whether the second weighted loss result has reached a stationary state. If it has not reached a stationary state, the parameters of the trained multimodal model are updated, and the training dataset in the dataset is input into the multimodal model for training. If it has reached a stationary state, the multimodal model training is complete.
[0054] During parameter updates, backpropagation and gradient descent are used to update all parameters of the visual encoder in the multimodal model, and LoRA technology is used to fine-tune the parameters of the language decoder in the multimodal model. That is, without performing an overall parameter update, the multimodal model with updated parameters can be formed.
[0055] Prediction module 203: used to obtain the Mini LED light group image to be processed, and input the Mini LED light group image to be processed into the trained multimodal model for prediction processing to obtain the segmentation result and description text corresponding to the Mini LED light group image to be processed; In the specific implementation of this invention, it is first necessary to obtain the Mini LED light group image to be processed, and then input the Mini LED light group image to be processed into the trained multimodal model for prediction processing to obtain the segmentation result and description text corresponding to the Mini LED light group image to be processed.
[0056] Description drawing and display module 204: Used to visualize the defects in the Mini LED light group image to be processed using the OpenCV library and the segmentation results, and to draw the description text on the visualized Mini LED light group image to be processed using the PIL library for display.
[0057] In the specific implementation of this invention, the visualization processing of defects in the Mini LED light group image to be processed using the OpenCV library and the segmentation results includes: using the "findContours" function in the OpenCV library to obtain the contours of each defect in the segmentation results of the Mini LED light group image to be processed; using the "drawContours" function in the OpenCV library to draw the contours of each defect in the Mini LED light group image to be processed, obtaining the drawn contours of each defect; and visualizing the defects in the segmentation results of the Mini LED light group image to be processed based on the drawn contours of each defect.
[0058] Furthermore, the step of using the PIL library to draw the descriptive text on the visualized Mini LED light group image for display includes: using the fromarray function in the PIL.Image library to generate a corresponding PIL object; using the Draw function in the PIL.ImageDraw library to create a drawing object for drawing text on the visualized Mini LED light group image; and using the text function of the drawing object to draw the descriptive text on the visualized Mini LED light group image according to the PIL object for display.
[0059] Specifically, after obtaining the segmentation results and descriptive text corresponding to the Mini LED light group image to be processed, the first step is to use the "findContours" function in the OpenCV library to obtain the contours of each defect in the segmentation results of the Mini LED light group image to be processed; then, the "drawContours" function in the OpenCV library is used to draw the contours of each defect in the Mini LED light group image to be processed, obtaining the drawn contours of each defect; finally, the defects in the segmentation results of the Mini LED light group image to be processed are visualized based on the drawn contours of each defect.
[0060] During the demonstration, the fromarray function in the PIL.Image library is used to generate the corresponding PIL object; the Draw function in the PIL.ImageDraw library is used to create a drawing object to draw text on the visualized Mini LED light group image; and the text function of the drawing object is used to draw the descriptive text on the segmentation result of the visualized Mini LED light group image according to the PIL object for display.
[0061] In this embodiment of the invention, a trained multimodal model is used to predict defects in the Mini LED lamp group image to be processed. Then, the prediction results are visualized using the OpenCV library, and the generated descriptive text is displayed on the image using the PIL library. This achieves accurate detection and description of defects in Mini LED images and provides visualization results, providing necessary technical support for quality inspectors to quickly assess product quality. Furthermore, it enables intuitive presentation of defect detection results, meeting the needs of the production line for rapid and accurate quality assessment.
[0062] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.
[0063] Furthermore, the above provides a detailed description of the image description method and apparatus for Mini LED defect detection provided by the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. An image description method for Mini LED defect detection, characterized in that, The method includes: Obtain an image of a Mini LED light group located on a Mini LED panel; generate corresponding defect description text based on the annotation results in the Mini LED light group image; and construct a dataset using the Mini LED light group image and the corresponding defect description text, the dataset including a training dataset and a validation dataset. A Transformer-based multimodal model is constructed, and the dataset is input into the multimodal model for training to form a trained multimodal model, which includes a visual encoder and a language decoder. Obtain an image of the Mini LED light group to be processed, and input the image of the Mini LED light group to be processed into a trained multimodal model for prediction processing to obtain the segmentation result and description text corresponding to the image of the Mini LED light group to be processed. The defects in the Mini LED light group image to be processed are visualized using the OpenCV library and the segmentation results, and the descriptive text is drawn on the visualized Mini LED light group image to be processed for display using the PIL library.
2. The image description method according to claim 1, characterized in that, The process of obtaining an image of the Mini LED light group located on the Mini LED panel includes: Based on the high-definition image acquisition device, the Mini LED on the Mini LED panel is image acquired and processed according to the preset exposure frequency to obtain several Mini LED images. Several Mini LED images are stitched together to form a complete Mini LED image corresponding to the Mini LED panel; The coordinates of the entire Mini LED image are matched using a template matching algorithm. The coordinate matching results are then used to cut the Mini LED light group images in the entire Mini LED image to obtain the Mini LED light group images located on the Mini LED panel.
3. The image description method according to claim 1, characterized in that, The step of generating corresponding defect description text based on the annotation results in the Mini LED light group image includes: The Mini LED light group image is filled into a square image, and the resolution of the square image is scaled down to 256*256; A 256*256 forward-facing image is input into the SAM model, and the input image is semi-automatically annotated based on the prompting mechanism of the SAM model to obtain the annotation results in the Mini LED light group image. The annotated defects include foreign object category, LED bead defect category, and LED bead abnormality category. The corresponding segmentation mask is generated based on the annotation results in the Mini LED light group image, and the generated segmentation mask is input into a Python script. The Python script generates corresponding defect description text based on the pixel values contained in the segmentation mask.
4. The image description method according to claim 1, characterized in that, The process of constructing a dataset using the Mini LED light group images and the corresponding defect description text includes: The Mini LED light group image is enhanced by random rotation and horizontal flipping to form an enhanced Mini LED light group image. Random rotation is achieved by generating a rotation angle by random sampling within a preset angle range and rotating the image using affine transformation. Horizontal flipping is achieved by mirroring the image pixel matrix in the horizontal direction. A dataset was constructed using enhanced images of Mini LED light clusters and corresponding defect description texts.
5. The image description method according to claim 1, characterized in that, The visual encoder is a Transformer-based semantic segmentation model used to segment defects in Mini LED light group images and output the defect features extracted from the Mini LED light group images. The language decoder uses a pre-trained open-source large language model to generate a corresponding description of the defect based on the defect features output by the visual encoder.
6. The image description method according to claim 5, characterized in that, The step of inputting the dataset into the multimodal model for training to form a trained multimodal model includes: The training dataset from the dataset is input into the multimodal model for training. After training is completed, the validation dataset from the dataset is input into the trained multimodal model to obtain the output test segmentation results and test description results. The pixel-level cross-entropy loss and Dice loss are calculated using the test segmentation results and the real mask of the corresponding Mini LED light group image in the verification dataset to obtain the pixel-level cross-entropy loss results and Dice loss results. The pixel-level cross-entropy loss result and the Dice loss result are added together according to a preset ratio to obtain the first weighted loss result; The token-level cross-entropy loss is calculated by using the test description results and the corresponding defect description text in the verification dataset to obtain the token-level cross-entropy loss calculation result. The first weighted loss result is weighted and the token-level cross-entropy loss calculation result is weighted to obtain the second weighted loss result; Determine whether the second weighted loss result has reached a stationary state. If not, update the parameters of the trained multimodal model and input the training dataset from the dataset into the multimodal model for training. If a stable state is reached, the multimodal model training is complete.
7. The image description method according to claim 6, characterized in that, The parameter update of the trained multimodal model includes: The visual encoder in the multimodal model is fully updated using backpropagation and gradient descent, and the language decoder in the multimodal model is fine-tuned using LoRA technology to form a multimodal model with updated parameters.
8. The image description method according to claim 1, characterized in that, The process of visualizing defects in the Mini LED light group image using the OpenCV library and the segmentation results includes: The "findContours" function in the OpenCV library is used to obtain the segmentation results of each defect contour in the Mini LED light group image to be processed; The "drawContours" function in the OpenCV library is used to draw the contours of each defect in the Mini LED light group image to be processed, and the drawing contours of each defect are obtained. Based on the drawn contours of each defect, the defects in the segmentation results of the Mini LED light group image to be processed are visualized.
9. The image description method according to claim 1, characterized in that, The step of using the PIL library to draw and display the descriptive text on the visualized image of the Mini LED light group includes: Use the fromarray function in the PIL.Image library to generate the corresponding PIL object; Use the Draw function in the PIL.ImageDraw library to create a drawing object that draws text on a visualized Mini LED light group image; The text function of the drawing object is used to draw the descriptive text onto the segmentation result of the Mini LED light group image that has been visualized, according to the PIL object, and then display it.
10. An image description device for Mini LED defect detection, characterized in that, The device includes: Data construction module: used to obtain Mini LED light group images located on Mini LED panels, generate corresponding defect description text based on the annotation results in the Mini LED light group images, and construct a dataset using the Mini LED light group images and the corresponding defect description texts, the dataset including training dataset and validation dataset; Model training module: used to build a Transformer-based multimodal model, and input the dataset into the multimodal model for training to form a trained multimodal model, which includes a visual encoder and a language decoder; Prediction module: used to obtain the Mini LED light group image to be processed, and input the Mini LED light group image to be processed into the trained multimodal model for prediction processing to obtain the segmentation result and description text corresponding to the Mini LED light group image to be processed; Description drawing and display module: It is used to visualize the defects in the Mini LED light group image to be processed using the OpenCV library and the segmentation results, and to draw the description text on the visualized Mini LED light group image to be processed using the PIL library for display.