Glass defect ai-assisted labeling method and system, electronic device, and medium

Abstract

The application discloses an AI-assisted glass defect labeling method and system, electronic equipment and medium, and relates to the technical field of visual inspection of industrial vision intelligence. The method specifically comprises the following steps: determining a plurality of first prompt points at the defect positions of a to-be-labeled image; inputting the to-be-labeled image containing the first prompt points into a pre-trained defect labeling model, performing model reasoning by the defect labeling model to output a first prediction mask, wherein the defect labeling model is an encoder-decoder model based on a neural network; and generating a target defect contour on the to-be-labeled image based on the first prediction mask. Compared with the traditional method, the method only needs manual labeling of a plurality of prompt points near the defect image, and does not need accurate labeling of the contour edge of the defect, thereby greatly improving the labeling efficiency of the defect edge samples required by the recognition model.

Description

Technical Field

[0001] This invention relates to the field of visual inspection technology, and in particular to an AI-assisted annotation method, system, electronic device, and medium for glass defects. Background Technology

[0002] In the field of visual inspection for industrial intelligent vision, glass products, due to their high light transmittance, heat resistance, and chemical stability, are widely used in industries such as consumer electronics, automobile manufacturing, photovoltaic modules, and building curtain walls. Their surface quality directly determines the optical performance and lifespan of the final product. Common glass defects include bubbles, scratches, cracks, and inclusions. These defects are small in size and complex in shape, making them inefficient and prone to error when identified by the human eye. Therefore, existing technologies often use semantic segmentation models as recognition models to identify glass defects. Training high-precision segmentation models relies on a large number of precisely labeled defect edges as training samples. Current technologies obtain these precisely labeled defect edge samples by manually tracing the glass defect boundaries pixel by pixel. However, existing technologies require operators to trace the defect contour pixel by pixel, which can take several minutes for a single sample. For thousands or tens of thousands of samples, manual annotation is inefficient and costly. Summary of the Invention

[0003] This invention aims to address at least one of the technical problems existing in the prior art. To this end, this invention proposes an AI-assisted annotation method, system, electronic device, and medium for glass defects, which can solve the problem of low efficiency in manual annotation.

[0004] The AI-assisted annotation method for glass defects according to a first aspect of the present invention includes: Determine several initial cue points at the defect locations in the image to be annotated; The image to be labeled, containing the first cue point, is input into a pre-trained defect labeling model. The defect labeling model performs model inference to output a first prediction mask. The defect labeling model is a neural network-based encoder-decoder model. The encoder of the defect labeling model is used to extract image features of the image to be labeled. The defect labeling model is also used to extract cue point features of the first cue point. The image features and cue point features are interactively fused based on a multi-head self-attention mechanism to obtain fused features. The decoder of the defect labeling model is used to output the first prediction mask based on the fused features. Based on the first prediction mask, a target defect contour is generated on the image to be labeled.

[0005] The embodiments of the present invention have at least the following beneficial effects: The method provided by the embodiments of the present invention first clicks several cue points in the defect area of ​​the image to be labeled; then, the image and cue points are input into a pre-trained encoder-decoder model, the model extracts image features and cue point features, and fuses them through a multi-head self-attention mechanism, decoding and outputting a prediction mask; finally, the target defect contour is generated based on the prediction mask. Compared with traditional methods, this method only requires manual labeling of several cue points near the defect image, without the need for precise labeling of the defect contour edges, greatly improving the efficiency of labeling defect edge samples required by the recognition model.

[0006] According to some embodiments of the present invention, before the step of inputting the image to be labeled containing the first cue point into a pre-trained defect labeling model, the method includes: Multiple sample images are acquired, and a corresponding training mask is obtained based on each sample image, wherein the training mask includes a first defect region and a background region; Multiple sample cue points are determined on the first defect region of each sample image, wherein the sample cue points are uniformly distributed within the first defect region; A training sample set is obtained based on the sample image, the sample cue points, and the training mask.

[0007] According to some embodiments of the present invention, before the step of inputting the image to be labeled containing the first cue point into the pre-trained defect labeling model, a step of training the defect labeling model is further included, the training step comprising: Each sample image in the training sample set is converted into multiple initial feature vectors, and a position embedding is added to each initial feature vector; Feature extraction is performed on the initial feature vector after adding position embedding based on a multi-layer coding network to obtain sample image features. Each layer of the coding network includes a multi-head self-attention mechanism and a feedforward neural network. Perform a position encoding mapping operation on the sample prompt points to obtain sample prompt features; Perform concatenation and cross-attention calculation operations on the sample image features and the sample cue features to obtain sample fusion features; Perform upsampling and pixel-level probability prediction operations on the sample fusion features to obtain a sample prediction mask; The model parameters of the defect labeling model are updated based on the difference between the sample prediction mask and the training mask.

[0008] According to some embodiments of the present invention, the step of updating the model parameters of the defect labeling model based on the difference between the sample prediction mask and the training mask includes: The difference between the sample prediction mask and the training mask corresponding to the sample image is obtained based on the combined loss function, and the loss function value is obtained. The combined loss function is composed of the binary cross-entropy loss function and the Dice loss function. Based on the loss function value, the gradient of the model parameters of the defect labeling model is obtained through the backpropagation algorithm, and the model parameters are updated through the optimizer; the optimizer uses the adaptive moment estimation algorithm to adjust the learning rate, and the learning rate gradually decreases with each training round according to the cosine decay law.

[0009] According to some embodiments of the present invention, the step of generating a target defect contour on the image to be labeled based on the first prediction mask includes: The first prediction mask is binarized and connected component analysis is performed to remove the second defect region on the first prediction mask whose area is smaller than a preset threshold. The outer contour of the second defect region is extracted based on the boundary tracking algorithm to obtain the initial defect contour; The initial defect profile is simplified based on the Douglas-Puk algorithm to obtain the target defect profile.

[0010] According to some embodiments of the present invention, after the step of generating the target defect contour on the image to be labeled based on the first prediction mask, the method further includes: Obtain labeled images based on the defect labeling model, and incorporate the labeled images as new sample sets into the training sample set; The defect labeling model is retrained based on the expanded training sample set to update the model parameters.

[0011] According to some embodiments of the present invention, after the step of marking the first cue point on the image to be annotated, the method further includes: Obtain the image size of the image to be labeled, and adjust the image size of the image to be labeled according to the input size of the defect labeling model; the step of adjusting the image size of the image to be labeled includes: If the image size of the image to be labeled is smaller than the input size of the defect labeling model, then the image size of the image to be labeled is made equal to the input size by expanding the image. If the image size of the image to be labeled is equal to the input size of the defect labeling model, then the image to be labeled is directly input into the defect labeling model; If the image size of the image to be labeled is smaller than the input size of the defect labeling model, the image to be labeled is cropped based on the position of the first prompt point as the center, so that the image size of the image to be labeled is equal to the input size.

[0012] An AI-assisted labeling and warning system for glass defects according to a second aspect of the present invention includes: The pre-annotation module is used to determine several first cue points at the defect locations in the image to be annotated; A model recognition module is used to input the image to be labeled containing the first cue point into a pre-trained defect labeling model, and to perform model inference through the defect labeling model to output a first prediction mask. The defect labeling model is a neural network-based encoder-decoder model. The encoder of the defect labeling model is used to extract image features of the image to be labeled, and the defect labeling model is also used to extract cue point features of the first cue point. The image features and cue point features are interactively fused based on a multi-head self-attention mechanism to obtain fused features. The decoder of the defect labeling model is used to output the first prediction mask based on the fused features. A contour generation module is used to generate a target defect contour on the image to be labeled based on the first prediction mask.

[0013] An electronic device according to a third aspect of the present invention includes a memory and a processor, the memory being used to store at least one program, and the processor being used to load the at least one program to execute the AI-assisted annotation method for glass defects described in the above-described aspects.

[0014] A computer-readable storage medium according to a fourth aspect of the present invention includes a memory and a processor, the memory being configured to store at least one program, and the processor being configured to load the at least one program to perform the AI-assisted annotation method for glass defects as described in the above-described aspects.

[0015] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0016] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart of an AI-assisted annotation method for glass defects according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the training process of the defect annotation model according to an embodiment of the present invention. Figure 3This is a structural diagram of the defect annotation model according to an embodiment of the present invention; Figure 4 This is a structural diagram of an electronic device provided in another embodiment of the present invention. Detailed Implementation

[0017] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0018] In the description of this invention, "several" means one or more, "multiple" means two or more, "greater than," "less than," "exceeding," etc. are understood to exclude the stated number, and "above," "below," "within," etc. are understood to include the stated number. If "first," "second," etc. are used in the description, they are only for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the order of the indicated technical features.

[0019] In the field of visual inspection for industrial intelligent vision, glass products, due to their high light transmittance, heat resistance, and chemical stability, are widely used in industries such as consumer electronics, automobile manufacturing, photovoltaic modules, and building curtain walls. Their surface quality directly determines the optical performance and lifespan of the final product. Common glass defects include bubbles, scratches, cracks, and stones. These defects are small in size and complex in shape, making them inefficient and prone to error when recognized by the human eye. Therefore, existing technologies often use semantic segmentation models as recognition models to identify glass defects. Training high-precision segmentation models relies on a large number of precisely labeled defect edges as training samples. Current technologies obtain these precisely labeled defect edges by manually tracing each pixel along the glass defect boundary. However, current technologies require operators to trace each pixel along the defect contour, which can take several minutes for a single sample. For thousands or tens of thousands of samples, manual annotation is inefficient and costly. Furthermore, if the same batch of samples is annotated by different annotators, differences in their ability to distinguish defect locations and background areas can lead to inconsistent contour standards in the training samples, thus affecting the training effect of subsequent recognition models.

[0020] The concepts involved in this invention are explained below: Transformer: A deep learning model that is entirely based on self-attention mechanism and abandons the recurrent structure. It efficiently captures long-distance dependencies in sequences through parallel computing. Mask: A mask is a binary image of the same size as the original image, where each pixel has only two possible values ​​(usually 0 and 1, or 0 and 255). Its function is to identify which pixels in the image belong to the "target region" and which belong to the "background region". MHSA: Multi-Head Self-Attention, a multi-head self-attention mechanism that uses multiple parallel attention heads to simultaneously capture various long-distance dependencies between elements in the input sequence from different representation subspaces; FFN: Feed-Forward Network, is a neural network structure with unidirectional information flow and no feedback connections. It is often used as a key component in models such as Transformer to perform independent nonlinear transformations on the representation at each location. LN: Layer Normalization, which normalizes all features of a single sample to make model training more stable and converge faster. It is often used in architectures such as Transformer. Binary cross-loss: used to measure the difference between the predicted probability distribution and the true label in a binary classification problem. The prediction error for each sample is calculated independently and averaged. Dice loss: Based on set similarity metrics, it calculates the degree of overlap between the predicted result and the true label, and is often used to evaluate the overall consistency of regions in segmentation tasks.

[0021] Please refer to Figures 1-3 , Figure 1 This is a flowchart of an AI-assisted annotation method for glass defects according to an embodiment of the present invention. This embodiment discloses an AI-assisted annotation method for glass defects, which includes, but is not limited to, the following steps: Step S100: Determine several first prompting points at the defect locations in the image to be annotated; It should be noted that, in the preprocessing stage of the method provided by this invention, the image to be labeled needs to be pre-labeled by the labeler. The preprocessing steps specifically include: 1. Loading the image to be labeled I∈ Record the original size parameters ( ), where the original width of the image to be labeled is The height is 2. The user obtains the coordinates of the prompt by clicking on the defect area in the image with the mouse, and records the position of each click. , ), k≥1 and are positive integers, forming a set of prompt points P_click={( , ), ..., ( , To balance annotation efficiency and accuracy, it is recommended that the number of prompt points n∈[1,6], and the click positions should roughly evenly cover the defect area. The purpose of annotating the first prompt point in this invention is completely different from the traditional method of manually annotating pixel by pixel along the edge contour of the defect. The purpose of the traditional method is to directly annotate the complete contour of the glass defect, and the annotated image is directly used as a training sample for the model; while in this embodiment of the invention, annotating several first prompt points at the defect position of the image to be annotated is to serve as prompt words input for the defect annotation model. The time consumed by the traditional method to annotate a sample depends on the length of the defect contour. Since it is a pixel-by-pixel annotation method, even samples with short contours require several minutes to annotate. However, the annotation in this embodiment of the invention only requires the annotator to click a very small number of times within the defect position. For example, in some embodiments, a standard person can click 1 to 6 times at the defect position to annotate the first prompt point on the sample image, which is much faster and more efficient than the traditional method. It can be understood that the defect position in step S100 of this embodiment of the invention is not a region with strict boundaries, but rather an approximate location of the defect on the glass image determined by the annotator.

[0022] Step S200: The image to be labeled, containing the first cue point, is input into a pre-trained defect labeling model. The defect labeling model performs model inference to output a first prediction mask. The defect labeling model is a neural network-based encoder-decoder model. The encoder of the defect labeling model is used to extract image features of the image to be labeled. The defect labeling model is also used to extract cue point features of the first cue point. The image features and cue point features are interactively fused based on a multi-head self-attention mechanism to obtain fused features. The decoder of the defect labeling model is used to output the first prediction mask based on the fused features. It should be noted that the model structure of the defect annotation model provided in this embodiment of the invention is as follows: Figure 3As shown, the defect annotation model can fuse and correlate the features corresponding to the defect location in the image to be annotated with the features of the first cue point, and can identify the specific contour of the defect location based on the correlation between the two types of features. Specifically, the encoder extracts the global image features of the image to be annotated and the spatial location features of the first cue point, enabling the model to understand both the semantic content of the image and the sparse prior information provided by the user. Secondly, by introducing a multi-head self-attention mechanism to interactively fuse the image features and the cue point features, the cue point features can be used as query vectors to actively retrieve semantically related regions in the image features, thereby effectively enhancing the model's spatial perception ability of the defect location. Even with only a very small number of cue points, the model can accurately focus on the defect boundary region. Finally, the decoder recovers the spatial resolution pixel by pixel based on the fused features and outputs the first prediction mask, which can then be used to annotate the defect location. Compared to the traditional annotation method that requires manual point-by-point drawing of the defect contour, this step can generate a defect mask with near-manual annotation accuracy through a single forward inference of the pre-trained model, significantly improving the efficiency and consistency of glass defect annotation.

[0023] Step S300: Based on the first prediction mask, generate the target defect contour on the image to be labeled.

[0024] It should be noted that the mask is a binary image of the same size as the original image, where each pixel has only two possible values ​​(usually 0 and 1, or 0 and 255). The first predictive mask in this embodiment divides the original image to be labeled into a mask defect region and a mask background region. Binarization and other methods are then used to make the mask defect region and mask background region easier to distinguish, allowing for clear identification of the boundary between them. Finally, based on this, a target defect contour is generated on the image to be labeled, completing the labeling of the image.

[0025] It should be noted that, in steps S100 to S300, the AI-assisted annotation method for glass defects provided in this embodiment of the invention, compared with the traditional method, only requires manual annotation of a small number of prompt points (1 to 6 times) near the defect image, without the need for precise annotation of the contour edge of the defect, which greatly improves the efficiency of annotation of defect edge samples required by the recognition model.

[0026] In addition, prior to step S200 in the above-described embodiments, the following steps are included, but are not limited to: Step S201: Acquire multiple sample images and obtain a corresponding training mask based on each sample image, wherein the training mask includes a first defect region and a background region; Step S202: Determine multiple sample cue points on the first defect region of each sample image, wherein the sample cue points are evenly distributed within the first defect region; Step S203: Based on the sample images, sample cue points, and training mask, a training sample set is obtained.

[0027] It should be noted that the defect annotation model provided in this embodiment of the invention is an encoder-decoder model based on a neural network. Before performing actual auxiliary annotation, this model needs to learn parameters using training data. Steps S201 to S203 are the training data preparation steps for this model. During training data preparation, a certain number (approximately 100) of glass defect images are first collected. For each image, a pixel-by-pixel annotation method is used to accurately delineate the defect edge, generating a training mask (the pixel value of the first defect region is 1, and the background region is 0). The sequence of polygon vertices along the defect edge is then stored in JSON format. Next, to enhance the model's robustness to different numbers of cue points, a progressive sampling strategy is adopted: within the defect mask region of each image, 32 sets of cue points are randomly generated, where the nth set contains n sample cue points (n=1, 2, ..., 32). The coordinates of each sample cue point are uniformly and randomly sampled within the horizontal and vertical range of the defect region, and all sample points are guaranteed to fall within the defect region (i.e., the corresponding mask value at the location of the sample cue point is 1). This yields 32 sets of cue points of varying numbers for each image, which, together with the original image and the training mask, constitute the training samples.

[0028] Additionally, refer to Figure 2 Prior to step S200 in the above-described embodiment, a step of training the defect annotation model is further included, the training step including but not limited to the following steps: Step S210: Convert each sample image in the training sample set into multiple initial feature vectors, and add position embeddings to each initial feature vector; Step S220: Based on the multi-layer coding network, feature extraction is performed on the initial feature vector after adding position embedding to obtain sample image features. Each layer of the coding network contains a multi-head self-attention mechanism and a feedforward neural network. Step S230: Perform a position encoding and mapping operation on the sample prompt points to obtain sample prompt features; Step S240: Perform a concatenation operation and a cross-attention calculation operation on the sample image features and sample cue features to obtain the sample fusion features; Step S250: Perform upsampling and pixel-level probability prediction operations on the sample fusion features to obtain the sample prediction mask; Step S260: Update the model parameters of the defect annotation model based on the difference between the sample prediction mask and the training mask.

[0029] It should be noted that steps S210 to S260 together constitute the training process of the defect annotation model, enabling the model to annotate the input image to be annotated. The model iteratively optimizes the model parameters of each layer in a sample data-driven manner, gradually enabling the model to generate high-precision defect masks from sparse cue points. Specifically, the defect annotation model provided in this embodiment of the invention adopts a Transformer-based encoder-decoder architecture and is implemented using the TensorFlow deep learning framework.

[0030] Referring to steps S210-S220, the model architecture includes an image encoder. The image encoder converts an entire glass image into a series of feature vectors for subsequent model processing, enabling the defect annotation model to extract image coding features with global context awareness from the original image. Specifically, the image encoder in this embodiment adopts the Vision Transformer (ViT) architecture. The training process based on the image encoder is as follows: First, the input image I (size is height H × width W × 3 channels) is divided into image blocks of fixed size P × P, resulting in N_p = (H × W) / P² image blocks. Each image block is mapped to a feature vector of length D through a linear projection matrix E. The i-th image block... The projection result is E. Subsequently, since the model does not know the location of each image patch in the original image, a learnable vector representing the location needs to be added to each patch. Finally, the initial feature sequence is obtained. : ...Form ①, In formula ①, It is the pixel value of the i-th image patch after flattening (length is...). E is the projection matrix, with the shape ( ) D; It is a position embedding matrix with shape [formula missing]. Its values ​​are automatically learned during training. The image encoder segments the image into small blocks, converts each block into a feature vector, and adds a learnable location label to each vector, thus obtaining an image feature sequence containing spatial information. Further, the initial feature sequence obtained after initial embedding... The L layers are identical coding layers. Each layer contains two core sub-layers: Multi-Head Self-Attention (MHSA) and Feedforward Neural Network (FFN), and each sub-layer employs residual connections and layer normalization (LN). For the l-th layer (l=1, 2, ..., L), the input is the output of the previous layer. The calculation process is as follows: First, the multi-head self-attention sublayer performs layer normalization on the input, then feeds it into the multi-head self-attention module, and finally adds the result to the original input (residual connection): ...Formula ②, in To represent the attention output, specifically, the multi-head self-attention mechanism first processes the initial feature sequence of the input. Each feature vector at each position is mapped to a query, key, and value vector using three sets of learnable weight matrices. These vectors are then uniformly divided into multiple heads along the feature dimension, with each head independently processing a portion of the feature dimension. Within each head, an attention score is obtained by calculating the dot product of the query and all keys at all positions. After scaling and softmax normalization to obtain probability weights, the value vectors are weighted and summed to obtain the output of that head. Finally, the outputs of all heads are concatenated and linearly transformed using a learnable output weight matrix to obtain the final multi-head self-attention result. Next, the output of Equation ② is... After performing layer normalization, and then feeding the data into the feedforward neural network with residual connections, we can obtain: ·····Form ③, in This represents the output of FFN, the final output of the Lth layer. This refers to the sample image features obtained by the image encoder. In summary, the multi-layer encoder allows for full "communication" between image patches through multi-head self-attention, enhances nonlinear expressive power through the feedforward network, and ensures training stability through residual connections, thereby extracting high-quality sample image features suitable for defect segmentation.

[0031] Referring to step S230, step S230 constructs the prompt word encoder for the defect annotation model. The prompt word encoder is used to convert the coordinates of the prompt points clicked by the user (e.g., (x, y)) into a set of high-dimensional feature vectors for subsequent fusion with sample image features. The training process of the prompt word encoder is as follows: Input a set of prompt point coordinates, which includes n points, and the coordinates of each point are (x, y)... Normalize the above coordinates, that is, scale each coordinate value to [0,1] or [...]. Within the range of [1,1] (depending on the implementation), normalized coordinates p are obtained. For each normalized coordinate value p, a high-dimensional vector composed of sine and cosine functions is generated. This encoding contains L layers (L is the number of encoding layers), with each layer using an increasingly higher frequency, expressed by the formula: Formula ④ In formula ④, The frequency base is usually taken as ( =2), where L is the number of encoding layers. Then, the above Fourier encoding is performed on the x and y coordinates respectively, resulting in two vectors of length 2L. These vectors are then concatenated to form a vector of length 4L. To match the dimension D of the sample image features, the concatenated vector needs to be transformed by a linear transformation (multiplied by the weight matrix and biased) to map it to the final cue point feature vector, i.e., the sample cue features of dimension D. Finally, the feature vectors corresponding to all n cue points form a matrix. In summary, the prompt word encoder maps the two-dimensional coordinates of user clicks to a high-dimensional space using multi-frequency sine / cosine functions, and then performs a linear transformation to obtain consistent sample prompt features. This allows the model to fully utilize the location information provided by the set of prompt points in the samples to assist in defect segmentation.

[0032] Referring to step S240, step S240 constructs a feature fusion module for the defect annotation model. The function of the feature fusion module is to effectively combine sample image features and sample prompt features. The specific steps are as follows: First, the sequence representation of the sample image features obtained in the above embodiments can be expressed as... ,in is the number of image patches, and D is the dimension of each feature vector. This involves analyzing the features of the sample image. With sample hint features The two matrices are concatenated along their row dimensions without changing the dimension D of each eigenvector. The concatenated result is: =Concat( , )∈ ...Form ⑤, In formula ⑤, The former The first n positions correspond to sample image features, and the last n positions correspond to sample cue features. Further, the concatenated feature sequence is input into a cross-attention layer, which generates query, key, and value matrices based on the sample cue features and sample image features respectively: the query matrix is ​​generated using the portion corresponding to the sample cue features, and the key and value matrices are generated using the portion corresponding to the sample image features. Attention weights are obtained by calculating the similarity between the query and key matrices, and then the value matrices are weighted and summed, thus integrating the feature vector of each cue point with the image semantic information related to its location. Simultaneously, the image features also acquire spatial location constraint information through interaction with the cue point features. After this processing layer, the cue point features are updated to vectors incorporating image context information, and the image features are also enhanced in their response to the target defect region due to the guidance of the cue points.

[0033] Referring to step S250, step S250 constructs a mask decoder for the defect annotation model. The mask decoder's role is to progressively restore the fused features to the original image resolution and output the probability of each pixel belonging to a defect category. This decoder adopts a multi-level upsampling structure, consisting of three core components: a bilinear upsampling module, a multilayer perceptron (MLP), and a Transformer decoder layer. Specifically, the bilinear upsampling module progressively amplifies the resolution of the feature map by a factor of 2; the MLP performs a nonlinear transformation on the upsampled features; and the Transformer decoder layer refines the details of the mask edges through self-attention and cross-attention mechanisms. The final output layer uses the sigmoid activation function to predict the probability of each pixel belonging to a defect category. After the upsampling operation and pixel-level probability prediction operation, the sample image is obtained. Overall, the mask decoder completes the transformation from low-resolution, high-semantic fused features to high-resolution, pixel-level probability maps, which is a key step in generating accurate defect boundaries.

[0034] Referring to step S260, the purpose of step S260 is to adjust the internal parameters of the defect labeling model based on the difference between the sample prediction mask and the training mask, so that the model's prediction results for the current sample gradually approach the real labeling, thereby improving the model's segmentation accuracy and generalization ability for glass defects.

[0035] Furthermore, step S260 in the above-described embodiments includes, but is not limited to, the following steps: Step S261: Obtain the difference between the sample prediction mask and the training mask corresponding to the sample image based on the combined loss function, and obtain the loss function value. The combined loss function is composed of the binary cross-entropy loss function and the Dice loss function. Step S262: Based on the loss function value, the gradient of the model parameters of the defect labeling model is obtained through the backpropagation algorithm, and the model parameters are updated through the optimizer; the optimizer uses the adaptive moment estimation algorithm to adjust the learning rate, and the learning rate gradually decreases with each training round according to the cosine decay law.

[0036] It should be noted that steps S261-S262 utilize the difference value calculated by the loss function (i.e., the error between the predicted mask and the real mask) to adjust all trainable parameters of the defect labeling model through the backpropagation algorithm, so that the predicted mask output by the model in subsequent forward calculations gradually approaches the real mask. Specifically, this embodiment of the invention uses binary crossover loss. and Dice loss The weighted sum is used as the portfolio loss function: ·····Form 6, In formula ⑥, The weights for the binary crossover loss, The weights are those for the Dice loss. Binary cross-entropy loss independently judges each pixel, finely optimizing edges, but when the defect region is small (with extreme imbalance between positive and negative samples), the model tends to favor predicting the background. Dice loss, on the other hand, focuses on the similarity of the overall region, is insensitive to small targets, and effectively alleviates class imbalance. Therefore, combining the two (e.g., with a 1:1 weight ratio) balances pixel-level accuracy and region integrity, making it more suitable for scenarios where glass defects have small areas and diverse shapes. Furthermore, in deep learning model training, parameter updating refers to the process of fine-tuning the trainable parameters (such as weights and biases) within the model based on the loss function value, aiming to make the predicted results output by the model in the next round of forward computation closer to the true values. Specifically, firstly, the sample prediction mask is obtained through forward propagation, and the loss value under the current parameters is calculated using the combined loss function; then, the partial derivative of the loss value with respect to each parameter (i.e., the gradient) is calculated through the backpropagation algorithm. The gradient indicates the direction and magnitude of parameter adjustment. In some embodiments of this invention, the Adam optimizer is used for parameter updating, with the initial learning rate set to 1× The learning rate is gradually reduced to near zero with each training epoch, following a cosine annealing strategy. The Adam optimizer's role in parameter updates is to dynamically adjust the learning rate for each parameter based on the first-moment estimate (mean) and the second-moment estimate (uncentered variance) of the gradient. Unlike traditional gradient descent which uses a fixed learning rate, Adam maintains two momentum variables: a first-momentum (exponential moving average of the gradient) to smooth the update direction, and a second-momentum (exponential moving average of the squared gradient) to adaptively scale the update step size for each parameter. In each iteration, Adam first calculates the current gradient, then updates the two momentum variables, corrects for their biases, and finally uses the corrected momentum to calculate the parameter update amount, i.e.: ...Form ⑦, In formula ⑦, Indicates the model parameters after iteration. Indicates the model parameters before iteration. It is the first-order momentum (exponential moving average of the gradient). It is the second-order momentum (exponential moving average of the squared gradient). The smoothing factor (to avoid division by zero) is used, and the model parameters are all trainable parameters of the model, including the weights and biases in the image encoder, cue word encoder, feature fusion module, and mask decoder. This way, the update step size of each parameter can adapt to its gradient history fluctuations and decrease globally as training progresses, thus achieving stable convergence.

[0037] Furthermore, step S300 in the above-described embodiments includes, but is not limited to, the following steps: Step S310: Binarize the first prediction mask and perform connected component analysis to remove the second defect region on the first prediction mask whose area is smaller than a preset threshold. Step S320: Extract the outer contour of the second defect region based on the boundary tracking algorithm to obtain the initial defect contour; Step S330: The initial defect profile is simplified based on the Douglas-Puk algorithm to obtain the target defect profile.

[0038] It should be noted that steps S310 to S330 are the post-processing stages after the defect annotation model is generated, mainly including morphological processing, contour extraction, and contour simplification of the first prediction mask. Morphological processing involves binarizing the output first prediction mask and thresholding... =0.5, resulting in a binary mask. Performing connected component analysis on this binary mask yields a set of connected regions { }. Calculate each region The area (i.e., the number of complete pixels in each region) is calculated, and regions with an area smaller than a preset threshold are removed to obtain the main second defect region. In one embodiment of the invention, noise regions with an area smaller than 9 pixels are removed, retaining the main defect region and the second defect region. Further, a boundary tracking algorithm is used to extract the outer contour of the binary mask. If multiple connected regions exist, the contour of each region is extracted separately. The perimeter of each contour is calculated, and the contour with the longest perimeter is selected to obtain the initial defect contour. Furthermore, this embodiment of the invention also simplifies the initial defect contour based on the Douglas-Puk algorithm. Specifically, it performs polygon approximation on the target contour, reducing the number of vertices while maintaining shape features. The maximum vertical distance from all midpoints on the curve segment to the line connecting the beginning and end is considered. Less than simplified tolerance In this case, the straight line segment is used to replace the original curve segment to simplify the contour. In summary, this embodiment of the invention uses morphological processing to predict probabilities. Figure 2 Value-mapping and removal of small noise areas yield a clean binary defect mask; contour extraction extracts the boundary pixel sequence, and the most significant initial defect contour is selected by perimeter comparison; contour simplification uses the Douglas-Peucker algorithm to remove redundant vertices, making the contour storage compact while preserving the main geometric shape, resulting in the target defect contour. In some embodiments, users can also manually adjust the final generated target defect contour; finally, the target defect contour is bound to a defect category label and stored as a structured annotation record in JSON format.

[0039] In addition, after step S300 in the above-described embodiments, the following steps are included, but are not limited to: Step S340: Obtain the labeled images based on the defect labeling model, and incorporate the labeled images as new sample sets into the training sample set; Step S350: Retrain the defect annotation model based on the expanded training sample set to update the model parameters.

[0040] It should be noted that the model trained with only a small number of precisely labeled samples (e.g., 100 images) in the initial stage has limited accuracy; as users use the current model to assist in labeling and generate new high-quality samples, these new samples are added to the training set (i.e., ...). , Let t be the training sample set for the t-th iteration. For the newly added sample set annotated by AI, the specific parameter update process is as follows: =argmin ( ; )······Form ⑧, In formula ⑧, These are the parameters after the iteration; the parameters from the previous round were... In this embodiment of the invention, the model is retrained on the augmented dataset to minimize the total loss (typically from the parameters of the previous round). (Starting with fine-tuning) further improves the model, thus forming a positive feedback loop of "annotation → training → better annotation → retraining". The final defect annotation model is a general pre-trained model with good generalization ability, which can be directly used for auxiliary annotation of new images without the need for separate training for each defect, significantly reducing the subsequent annotation cost.

[0041] In addition, after step S100 in the above-described embodiments, the following steps are included, but are not limited to: Step S110: Obtain the image size of the image to be annotated, and adjust the image size of the image to be annotated according to the input size of the defect annotation model; the steps for adjusting the image size of the image to be annotated include: If the image size of the image to be labeled is smaller than the input size of the defect labeling model, the image size of the image to be labeled will be made equal to the input size by expanding the image. If the image size of the image to be labeled is equal to the input size of the defect labeling model, then the image to be labeled is directly input into the defect labeling model; If the image size of the image to be labeled is smaller than the input size of the defect labeling model, the image to be labeled is cropped based on the position of the first prompt point, so that the image size of the image to be labeled is equal to the input size.

[0042] It should be noted that the embodiments of the present invention also include a preprocessing stage for the image to be labeled. The purpose of step S100 is to ensure that the size of the image to be labeled meets the preset input size requirements of the defect labeling model, avoiding model inability to process or inference errors due to size mismatch. Specifically, the original size of the image to be labeled is ( The adaptive pruning strategy in this embodiment of the invention is as follows: If max( )≤1024: Directly input the original image into the model; If max( >1024: Calculate the minimum bounding rectangle formed by the first cue point on the image to be labeled. Let its width and height be ( ), the center is ( The dimensions of the cutting area are calculated as follows: = max( +margin, 1024), = max( +margin, 1024), where margin is the margin extension amount (default 128 pixels), ensuring that defects are completely contained within the cropping area.

[0043] Secondly, referring to Figure 2 This invention provides an AI-assisted labeling system for glass defects, the system specifically comprising: The pre-annotation module is used to determine several first cue points at the defect locations in the image to be annotated; The model recognition module is used to input the image to be labeled containing the first cue point into a pre-trained defect labeling model, and to perform model inference through the defect labeling model to output a first prediction mask. The defect labeling model is a neural network-based encoder-decoder model. The encoder of the defect labeling model is used to extract image features of the image to be labeled, and the defect labeling model is also used to extract cue point features of the first cue point. The image features and cue point features are interactively fused based on a multi-head self-attention mechanism to obtain fused features. The decoder of the defect labeling model is used to output the first prediction mask based on the fused features. The contour generation module is used to generate target defect contours on the image to be labeled based on a first prediction mask.

[0044] It should be noted that the system provided in this embodiment of the invention receives a small number of prompts clicked by the user through a pre-annotation module; the model recognition module automatically generates pixel-level defect masks using a pre-trained encoder-decoder network, the structure of which is as follows: Figure 2As shown, the contour generation module converts the mask into an editable vector contour, simplifying manual pixel-by-pixel annotation to a few clicks, significantly reducing annotation time and subjective bias. The system supports interactive correction, incremental learning, and general model iteration, continuously improving segmentation accuracy. This system is primarily applied in industrial visual inspection, particularly suitable for auxiliary annotation of minute defects such as bubbles, scratches, and cracks on the surface of glass products (e.g., mobile phone cover plates, automotive display panels, photovoltaic glass), providing efficient, consistent, and high-quality annotation data for deep learning model training, and accelerating the deployment and optimization of industrial quality inspection systems.

[0045] like Figure 4 As shown, Figure 4 This is a structural diagram of an electronic device provided in one embodiment of the present invention. The present invention also provides an electronic device, comprising: The processor 801 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 802 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and called by the processor 801 to execute the AI-assisted annotation method for glass defects according to the embodiments of this application. The 803 input / output interface is used to implement information input and output. The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804); The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0046] It should be noted that the electronic device that performs the AI-assisted labeling method for glass defects in the embodiments of the present invention refers in particular to a device with a battery management unit, such as a new energy vehicle, a consumer electronics product, etc., as well as any other electronic device that can be used to perform the method. The present invention does not impose any specific limitations on this.

[0047] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate, and may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0048] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0049] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. An AI-assisted annotation method for glass defects, characterized in that, include: Determine several initial cue points at the defect locations in the image to be annotated; The image to be labeled, containing the first cue point, is input into a pre-trained defect labeling model. The defect labeling model performs model inference to output a first prediction mask. The defect labeling model is a neural network-based encoder-decoder model. The encoder of the defect labeling model is used to extract image features of the image to be labeled. The defect labeling model is also used to extract cue point features of the first cue point. The image features and cue point features are interactively fused based on a multi-head self-attention mechanism to obtain fused features. The decoder of the defect annotation model is used to output a first prediction mask based on the fusion features; Based on the first prediction mask, a target defect contour is generated on the image to be labeled.

2. The AI-assisted annotation method for glass defects according to claim 1, characterized in that, Before the step of inputting the image to be labeled containing the first cue point into the pre-trained defect labeling model, the following steps are included: Multiple sample images are acquired, and a corresponding training mask is obtained based on each sample image, wherein the training mask includes a first defect region and a background region; Multiple sample cue points are determined on the first defect region of each sample image, wherein the sample cue points are uniformly distributed within the first defect region; A training sample set is obtained based on the sample image, the sample cue points, and the training mask.

3. The AI-assisted annotation method for glass defects according to claim 2, characterized in that, Before the step of inputting the image to be labeled containing the first cue point into the pre-trained defect labeling model, the method further includes a step of training the defect labeling model, the training step including: Each sample image in the training sample set is converted into multiple initial feature vectors, and a position embedding is added to each initial feature vector; Feature extraction is performed on the initial feature vector after adding position embedding based on a multi-layer coding network to obtain sample image features. Each layer of the coding network includes a multi-head self-attention mechanism and a feedforward neural network. Perform a position encoding mapping operation on the sample prompt points to obtain sample prompt features; Perform concatenation and cross-attention calculation operations on the sample image features and the sample cue features to obtain sample fusion features; Perform upsampling and pixel-level probability prediction operations on the sample fusion features to obtain a sample prediction mask; The model parameters of the defect labeling model are updated based on the difference between the sample prediction mask and the training mask.

4. The AI-assisted annotation method for glass defects according to claim 3, characterized in that, The step of updating the model parameters of the defect labeling model based on the difference between the sample prediction mask and the training mask includes: The difference between the sample prediction mask and the training mask corresponding to the sample image is obtained based on the combined loss function, and the loss function value is obtained. The combined loss function is composed of the binary cross-entropy loss function and the Dice loss function. Based on the loss function value, the gradient of the model parameters of the defect labeling model is obtained through the backpropagation algorithm, and the model parameters are updated through the optimizer; the optimizer uses the adaptive moment estimation algorithm to adjust the learning rate, and the learning rate gradually decreases with each training round according to the cosine decay law.

5. The AI-assisted annotation method for glass defects according to any one of claims 1 to 4, characterized in that, The step of generating the target defect contour on the image to be labeled based on the first prediction mask includes: The first prediction mask is binarized and connected component analysis is performed to remove the second defect region on the first prediction mask whose area is smaller than a preset threshold. The outer contour of the second defect region is extracted based on the boundary tracking algorithm to obtain the initial defect contour; The initial defect profile is simplified based on the Douglas-Puk algorithm to obtain the target defect profile.

6. The AI-assisted annotation method for glass defects according to any one of claims 1 to 4, characterized in that, After the step of generating the target defect contour on the image to be labeled based on the first prediction mask, the method further includes: Obtain labeled images based on the defect labeling model, and incorporate the labeled images as new sample sets into the training sample set; The defect labeling model is retrained based on the expanded training sample set to update the model parameters.

7. The AI-assisted annotation method for glass defects according to any one of claims 1 to 4, characterized in that, After the step of marking the first cue point on the image to be labeled, the method further includes: Obtain the image size of the image to be labeled, and adjust the image size of the image to be labeled according to the input size of the defect labeling model; the step of adjusting the image size of the image to be labeled includes: If the image size of the image to be labeled is smaller than the input size of the defect labeling model, then the image size of the image to be labeled is made equal to the input size by expanding the image. If the image size of the image to be labeled is equal to the input size of the defect labeling model, then the image to be labeled is directly input into the defect labeling model; If the image size of the image to be labeled is smaller than the input size of the defect labeling model, the image to be labeled is cropped based on the position of the first prompt point as the center, so that the image size of the image to be labeled is equal to the input size.

8. An AI-assisted labeling system for glass defects, characterized in that, include: The pre-annotation module is used to determine several first cue points at the defect locations in the image to be annotated; A model recognition module is used to input the image to be labeled containing the first cue point into a pre-trained defect labeling model, and to perform model inference through the defect labeling model to output a first prediction mask. The defect labeling model is a neural network-based encoder-decoder model. The encoder of the defect labeling model is used to extract image features of the image to be labeled, and the defect labeling model is also used to extract cue point features of the first cue point. The image features and cue point features are interactively fused based on a multi-head self-attention mechanism to obtain fused features. The decoder of the defect labeling model is used to output the first prediction mask based on the fused features. A contour generation module is used to generate a target defect contour on the image to be labeled based on the first prediction mask.

9. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store at least one program, and the processor being used to load the at least one program to execute the AI-assisted annotation method for glass defects according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the AI-assisted annotation method for glass defects as described in any one of claims 1 to 7.

Images