A weld defect color image detection method and system based on YOLO-World

Abstract

The present application relates to a kind of YOLO-World-based weld defect color image detection method and system, realize guarantee known weld defect detection performance simultaneously by constructing natural language text prompt real-time detection unknown weld defect.The method comprises the following steps: collecting the RGB color image of weld surface, input the RGB color image and known defect category prompt word into pre-trained YOLO-World model, and detect to obtain first defect detection area and no detection result area;First defect detection area is judged based on confidence threshold determination criterion, and known defect detection area and suspicious area are obtained;Unknown defect category prompt word is constructed according to no detection result area and suspicious area, and unknown defect category prompt word is input into pre-trained YOLO-World model, and second defect detection area is detected, and second defect detection area is judged based on confidence threshold determination criterion, and unknown defect detection area is obtained.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, image processing and defect detection technology, and in particular to a method and system for detecting color images of weld defects based on YOLO-World. Background Technology

[0002] Welding, as the most fundamental and critical joining process in modern manufacturing, is widely used in aerospace, shipbuilding, pressure vessels, rail transportation, and bridge construction. The quality of welds directly affects the structural integrity, safety, reliability, and service life of products; therefore, rapid, stable, and accurate detection of weld defects has significant industrial value.

[0003] Currently, weld defect detection technologies mainly include manual visual inspection, X-ray inspection, and traditional machine vision methods. Manual visual inspection relies on the experience of inspectors, resulting in low efficiency and high subjectivity. X-ray inspection equipment has high investment costs and long inspection cycles, while also posing radiation safety risks, making it difficult to promote and widely adopt in some manufacturing scenarios. Regarding traditional machine vision methods, deep learning has driven the rapid development of target detection technology in recent years, with the YOLO series algorithms being widely adopted in industrial quality inspection due to their high accuracy and real-time performance. However, the traditional YOLO model is a closed-set detection method, which can only identify a limited number of categories predefined during the training phase. When new defect types appear on the production site, it is necessary to re-collect and label data and retrain the model, leading to long adaptation cycles and high upgrade costs.

[0004] A search of existing technical literature revealed a patent application (CN202510249561.3) entitled "A Precise Detection Method for Complex and Diverse Welding Defects in Steel Pipes Based on YOLOv5." This patent first divides the acquired X-ray image data into training and testing sets and performs preprocessing. Next, it constructs a novel detection and classification algorithm model, XCM-YOLOv5, based on YOLOv5, including a Backbone module, a Neck module, and a Head module. The Backbone module includes a Focus network, a CBAM module, and a CSP module. The CBAM attention mechanism enhances the model's ability to focus on the most relevant features to address the complexity and diversity of welding defects. Then, the Neck module, including an FPN+PAN module and a small target detection layer, detects various small target defects present in steel pipe welds. This patent features low computational cost, high network recognition accuracy, and the ability to detect various weld defects, with advantages of fast detection speed and high accuracy. However, the patent has problems such as the inability of the YOLOv5-based model to effectively identify unknown weld defects in practical applications. Summary of the Invention

[0005] Therefore, it is necessary to provide a YOLO-World-based color image detection method and system for weld defects to address the aforementioned technical problems. This system can ensure the detection performance of known weld defects while simultaneously detecting unknown weld defects in real time through the construction of natural language text prompts.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] On one hand, the present invention provides a method for detecting weld defects using color images based on YOLO-World, comprising:

[0008] S1: Acquire RGB color images of the weld surface, input the RGB color images and fixed prompts corresponding to known defect categories into the pre-trained YOLO-World model for the first stage of detection, and output the first defect detection area and the area with no detection result. The defect detection area includes the detection box, the defect category and the confidence level corresponding to the defect category.

[0009] S2: If the confidence level in the first defect detection area is greater than the high confidence threshold, the first defect detection area is a known defect detection area; if the confidence level in the first defect detection area is less than the high confidence threshold but greater than the first stage detection confidence threshold, the first defect detection area is a suspicious area.

[0010] S3: Based on the regions with no detection results or the suspicious regions, a three-step prompt word engineering method is used to construct unknown defect category prompt words. The unknown defect category prompt words are then input into the pre-trained YOLO-World model for the second stage of detection, and the second defect detection region is output.

[0011] S4: If the confidence level in the second defect detection area is greater than the second-stage detection confidence threshold, the second defect detection area is an unknown defect detection area.

[0012] Preferably, if the confidence level in the first defect detection area is less than the first-stage detection confidence threshold, the first defect detection area is a background noise area unrelated to the defect; if the confidence level in the second defect detection area is less than the second-stage detection confidence threshold, the second defect detection area is subject to manual review.

[0013] Preferably, before step S1, the method further includes:

[0014] RGB color images of the weld surface were acquired and a dataset was constructed.

[0015] Based on the dataset, the YOLOv8l-Worldv2 model is trained using a multi-task loss function to obtain a pre-trained YOLO-World model. The YOLOv8l-Worldv2 model is a model obtained by training the YOLO-World model on a large-scale dataset.

[0016] Preferably, the model training based on transfer learning includes:

[0017] The dataset is divided according to a preset ratio to obtain a training set, a validation set, and a test set;

[0018] The hyperparameters for model training are preset, the training set and the validation set are input into the YOLOv8l-Worldv2 model and trained using a multi-task loss function until the loss function converges, thus obtaining the pre-trained YOLO-World model.

[0019] The performance of the pre-trained YOLO-World model is evaluated using the test set.

[0020] Preferably, the calculation expression for the multi-task loss function is as follows:

[0021]

[0022] in, Indicates the total loss. This represents the bounding box regression loss. Represents classification loss, Represents distributed focus loss. Indicates the bounding box regression loss weights. Represents the classification loss weights. This represents the weight of the distributed focus loss.

[0023] Preferably, during training, data augmentation methods are used to enhance the training set, including: Mosaic data augmentation method, Mixup data augmentation method, HSV color space enhancement method, and horizontal flipping method;

[0024] The calculation expression for the Mosaic data augmentation method is as follows:

[0025]

[0026] in, This represents a new image formed by randomly stitching together four RGB color images from the dataset. This indicates a splicing operation. This represents the randomly cropped region of the first image. This represents the randomly cropped region of the second image. This represents the randomly cropped region of the third image. This represents the randomly cropped region of the fourth image;

[0027] The calculation expression for the Mixup data augmentation method is as follows:

[0028]

[0029] in, This represents a new image formed by blending two RGB color images in the dataset. express distributed, Indicates the mixing ratio, from Distributed sampling express Hyperparameters of the distribution This represents the first image. This refers to the second image;

[0030] The calculation expression for the HSV color space enhancement method is as follows:

[0031]

[0032] in, This represents the original hue of the RGB color images in the dataset. This represents an additive perturbation of hue. Indicates the enhanced hue. This represents the original saturation of the RGB color images in the dataset. The multiplicative perturbation representing saturation. Indicates the enhanced saturation. This represents the original brightness of the RGB color images in the dataset. Represents the multiplicative perturbation of brightness. This indicates the enhanced brightness.

[0033] Preferably, the pre-trained YOLO-World model predicts the weld defect category based on the input RGB color image and defect category prompts, and the calculation expression is as follows:

[0034]

[0035] in, Indicates the confidence level. This indicates the image region where the defect is located in the RGB color image. This indicates the defect category corresponding to the defect category prompt. Represents a set of defect categories. Represents a single defect category in a set of defect categories. The visual feature vector representing the image region is obtained by encoding the RGB color image in the pre-trained YOLO-World model's image encoder. The text feature vector representing the defect category is obtained by encoding the defect category prompt word in the text encoder of the pre-trained YOLO-World model. Indicates temperature parameter, Represents the cosine similarity function. This represents the natural exponential function.

[0036] Preferably, the three-step prompt word engineering includes:

[0037] The regions with no detection results or the suspicious regions are systematically observed from different dimensions to obtain observation features;

[0038] The observed features are integrated at the natural language level to obtain a representational description;

[0039] Based on the representational description and the pre-trained YOLO-World model, unknown defect category prompts of different granularities are generated.

[0040] Preferably, after step S4, the method further includes:

[0041] The RGB color image containing the unknown defect detection region is added to the dataset, and the pre-trained YOLO-World model is incrementally trained. The calculation expression is as follows:

[0042]

[0043] in, This represents the YOLO-World model after incremental training. This represents the current YOLO-World model. This indicates fine-tuning of the pre-trained model. This represents the training data comprised of the existing dataset. This represents the training data consisting of newly added RGB color images.

[0044] On the other hand, the present invention provides a YOLO-World-based color image detection system for weld defects, comprising: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the steps of the YOLO-World-based color image detection method for weld defects as described above.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] This invention provides a method and system for color image detection of weld defects based on YOLO-World. The method employs a two-stage cascaded detection process based on the YOLO-World model to detect weld defects. The first stage automatically detects known defects without manual intervention. For areas that cannot be identified in the first stage or whose identification results are questionable, a three-step prompt word engineering method is used to construct prompt words for unknown defect categories applicable to different scenarios, which are then input into the YOLO-World model. The visual-language joint encoding mechanism of the YOLO-World model enables cross-modal alignment of visual features in RGB color images with textual features in text descriptions. This allows the model to detect weld defects without relying on fixed category indices, instead using semantic similarity for target matching, thus possessing open-vocabulary detection capabilities and enabling the detection of unknown defects. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the process for color image detection of weld defects based on YOLO-World in an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of the two-stage cascaded detection process of the YOLO-World-based color image detection method for weld defects in an embodiment of the present invention.

[0049] Figure 3 This is a schematic diagram of the dataset label distribution for the YOLO-World-based color image detection method for weld defects in an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram illustrating the Mosaic data augmentation effect of the YOLO-World-based color image detection method for weld defects in an embodiment of the present invention.

[0051] Figure 5 This is a schematic diagram of the training process curve of the YOLO-World-based color image detection method for weld defects in an embodiment of the present invention;

[0052] Figure 6 This is a schematic diagram of the normalized confusion matrix used by the YOLO-World-based color image detection method for weld defects in an embodiment of the present invention to classify defects.

[0053] Figure 7 This is a schematic diagram of the Precision-Recall curve of the YOLO-World-based color image detection method for weld defects in an embodiment of the present invention;

[0054] Figure 8 This is a schematic diagram illustrating the detection effect of the YOLO-World-based color image detection method for weld defects in an embodiment of the present invention.

[0055] Figure 9 This is a schematic diagram showing the distribution and statistics of the two-stage detection process of the YOLO-World-based color image detection method for weld defects in this embodiment of the invention.

[0056] Figure 10 This is a schematic diagram of the two-stage cascaded detection effect of the YOLO-World-based color image detection method for weld defects in an embodiment of the present invention.

[0057] Figure 11 This is a schematic diagram showing the performance comparison of various models of the YOLO-World-based color image detection method for weld defects on the test set in this embodiment of the invention.

[0058] Figure 12 This is a schematic diagram of the three-step prompting word process of the YOLO-World-based color image detection method for weld defects in an embodiment of the present invention.

[0059] Figure 13 This is a schematic diagram of the incremental feature learning process of the YOLO-World-based color image detection method for weld defects in an embodiment of the present invention.

[0060] Figure 14 This is a schematic diagram of the architecture of a YOLO-World-based color image detection system for weld defects in an embodiment of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0062] Example 1

[0063] like Figure 1 As shown in the figure, this embodiment proposes a color image detection method for weld defects based on YOLO-World, the method including:

[0064] S1: Acquire RGB color images of the weld surface, input the RGB color images and fixed prompts corresponding to known defect categories into the pre-trained YOLO-World model for the first stage of detection, and output the first defect detection area and the area with no detection result. The defect detection area includes the detection box, the defect category and the confidence level corresponding to the defect category.

[0065] The specific implementation of this step is as follows, such as Figure 2 As shown, this invention proposes a two-stage cascaded detection mechanism, dividing the detection process into two stages. The cascaded detection process can be described as follows:

[0066]

[0067] in, Given a set of known defect warning words. For the set of new defect description prompts, For the set of confidence scores, The confidence threshold is used. The first stage (automatic detection of known defects) involves using preset known defect prompts to perform detection, obtain detection results and confidence scores. This stage is completed automatically without manual intervention. The specific detection in the first stage is as follows: First, RGB color images of the weld surface are acquired using an industrial camera, and a dataset containing multiple defect categories is constructed. Known defect types include high splatter, irregular beadshape, poor start and stop, porosity, surface crack, and undercut. For statistical purposes, this embodiment also classifies good weld profiles as known defect types. After obtaining fixed prompts for the above known defect types, the prompts and RGB color images are input into the pre-trained YOLO-World model. The YOLO-World model receives two types of input: image input. and text prompt input The testing process can be described as follows:

[0068]

[0069] in, To detect bounding box sets, For category prediction, The confidence score is... This is the model mapping function. The final output yields the first defect detection area and the area with no detection result.

[0070] S2: If the confidence level in the first defect detection area is greater than the high confidence threshold, the first defect detection area is a known defect detection area; if the confidence level in the first defect detection area is less than the high confidence threshold but greater than the first stage detection confidence threshold, the first defect detection area is a suspicious area.

[0071] The specific implementation of this step is as follows: the present invention uses a confidence threshold judgment criterion to determine the recognition result of the first stage, as follows:

[0072]

[0073] The selection of the three confidence thresholds in this invention follows the design principle of "layered progression, erring on the side of too many rather than too few." =0.5 is the high confidence threshold, used to distinguish between high-confidence detections and suspicious regions. In the field of object detection, 0.5 is a widely adopted standard confidence cutoff. Detection results above this value generally have high reliability and can be regarded as a deterministic judgment of the model on known defects; detection results below this value indicate that the model's judgment on the region is uncertain, which may be a weak feature manifestation of a known defect or a new defect type not covered in the training set. Choosing 0.5 as the cutoff point ensures the detection accuracy of known defects without missing noteworthy suspicious regions.

[0074] =0.15 is the confidence threshold for the first-stage detection, i.e., the lowest confidence threshold for the first-stage detection. The goal of the first stage is to capture as comprehensively as possible all defect-related regions in the image, including defects with weak features or small sizes. Setting a lower detection threshold is to improve the recall rate of the first stage, ensuring that no potential defect signals are missed. Higher than... The detections were directly classified as high-confidence results, ranging between 0.15 and... The detection between these values ​​is marked as a suspicious area, triggering the second stage. Detections below 0.15 are classified as background noise areas unrelated to defects. That is, the value should not be too low, otherwise it will introduce a lot of background noise, causing the second stage to be frequently triggered and reducing the overall efficiency of the system; it should also not be too high, otherwise some weak feature defects may be missed.

[0075] S3: Based on the regions with no detection results or the suspicious regions, a three-step prompt word engineering method is used to construct unknown defect category prompt words. The unknown defect category prompt words are then input into the pre-trained YOLO-World model for the second stage of detection, and the second defect detection region is output.

[0076] The specific implementation of this step is as follows: The core of the YOLO-World model's vision-language joint encoding mechanism lies in encoding the defect category described in natural language into a semantic vector through the CLIP text encoder. Simultaneously, the backbone network extracts multi-scale visual features from the image, achieving cross-modal alignment of visual and text features in the RepVL-PAN (Re-parameterizable Vision-Language Path Aggregation Network). This allows the model to no longer rely on fixed category indices, but instead perform target matching through semantic similarity, thus enabling it to detect any text description category—that is, open-vocabulary detection. The second stage (unknown defect open-vocabulary detection) triggers open-vocabulary detection on the images of the regions with no detection results obtained in step S1 and the images of the suspicious regions obtained in step S2. After constructing unknown defect category prompts of different granularities applicable to the scenario using a three-step prompt word engineering method, the unknown defect category prompts are input into the pre-trained YOLO-World model, ultimately outputting the second defect detection region.

[0077] S4: If the confidence level in the second defect detection area is greater than the second-stage detection confidence threshold, the second defect detection area is an unknown defect detection area.

[0078] The specific implementation of this step is as follows. The second-stage detection confidence threshold is set to determine the target area. This second stage performs open-vocabulary detection, aiming to discover novel defect types that the model has never seen in the training set in areas with no detection results and suspicious regions. Since the model's confidence in responses to unseen categories is naturally low, an extremely low detection threshold is necessary to maximize the capture of potential new defect signals. Even if the detection confidence is low, the system records it and provides it to the manual review stage for final confirmation, thus achieving a balance between sensitivity and false positive rate to maximize the capture of potential new defects. When the second-stage detection also fails to detect new defects (i.e., no effective results in either stage, or suspicious regions that are not confirmed in the second stage), the system automatically marks the image as "requiring manual review" to ensure that no potential new defect types are missed.

[0079] Example 2

[0080] This embodiment further illustrates the YOLO-World-based color image detection method for weld defects proposed in the above embodiments.

[0081] In an optional embodiment, prior to step S1, the method further includes:

[0082] RGB color images of the weld surface were acquired and a dataset was constructed.

[0083] Based on the dataset, the YOLOv8l-Worldv2 model is trained using a multi-task loss function to obtain a pre-trained YOLO-World model. The YOLOv8l-Worldv2 model is a model obtained by training the YOLO-World model on a large-scale dataset.

[0084] In the specific implementation process, RGB color images of the weld surface were acquired using industrial cameras to form the dataset required for training. A total of 1039 images were collected, covering 7 categories of weld defects. The different label distributions for each category are as follows: Figure 3 As shown in Table 1, the dataset is divided into training, validation, and test sets, providing sufficient sample diversity for model training.

[0085] Table 1

[0086]

[0087] After the dataset is partitioned, the hyperparameters for model training are set (100 training epochs, input image size 640×640, batch size 16, SGD optimizer (momentum 0.937), initial learning rate 0.01, and cosine annealing strategy). Once configured, the training set is input into the YOLOv8l-Worldv2 model and trained using a multi-task loss function. During training, a validation set is used for hyperparameter tuning, and data augmentation methods are employed to enhance the training set, significantly improving the model's generalization ability. This embodiment uses Mosaic augmentation to enhance the training set, such as... Figure 4 As shown, four images are stitched together for training, effectively increasing the diversity of the training samples. Figure 5 As shown, the changing trends of various indicators during training reveal that the loss functions (box localization loss, category classification loss, and distribution focus loss) steadily decrease and converge during training. The mean accuracy (mAP@50) at an intersection-union ratio (IU) threshold of 0.5 and the mean accuracy (mAP@50-95) at IU thresholds of 0.5 to 0.95 continuously increase, eventually reaching 0.890 and 0.675 respectively. This indicates that the model successfully converged and achieved high detection performance, ultimately yielding the pre-trained YOLO-World model.

[0088] After training, the model's performance metrics were evaluated on the test set. The model demonstrated excellent detection performance across all seven types of weld defects. As shown in Table 2, the evaluation metrics included precision, recall, mean precision at an intersection-with-union (IU) threshold of 0.5, and mean precision for IU thresholds ranging from 0.5 to 0.95. The mean precision for surface cracks at an IU threshold of 0.5 reached 99.5%, while the recall rates for good weld examples and undercuts both reached 100%. Overall, the mean precision at an IU threshold of 0.5 reached 89.0%, and the mean precision for IU thresholds ranging from 0.5 to 0.95 reached 67.5%, indicating that the model can not only accurately detect the presence of defects but also precisely locate their boundaries.

[0089] Table 2

[0090]

[0091] The model's accuracy in classifying various defects is as follows: Figure 6 As shown, the values ​​on the diagonal represent the correct classification rate for each category. Among them, the categories with high accuracy (≥95%) include irregular weld shapes (100%), surface cracks (100%), undercut (100%), and good weld examples (92%); the categories with medium accuracy (67%-92%) include dense porosity (80%), premature start and end (75%), and a lot of spatter (71%). It can be seen that the model exhibits high classification accuracy in most categories. Confusion mainly occurs between defect categories with similar visual features (such as a lot of spatter and dense porosity), which is related to the similarity of these defects in appearance.

[0092] The PR curves formed by precision and recall for each category, and their corresponding mean precision with an intersection-over-union (IoU) threshold of 0.5, are shown below. Figure 7 As shown, the average accuracy of surface cracks and undercut reaches the highest crossover ratio threshold of 0.995 with an average accuracy of 0.5. Good weld examples reach 0.972, and premature start and end reach 0.921, all of which perform well. The average accuracy of all categories with a crossover ratio threshold of 0.5 is 0.890.

[0093] The specific detection results of the model are as follows: Figure 8 As shown, the model can accurately detect and locate various weld defects, with precise detection frame positions and reasonable confidence scores.

[0094] The results of the two-stage cascaded detection experiment using the test set are shown in Table 3. Among them, 14 images had suspicious regions or no detection, which triggered the second stage. The reasons for the triggering are analyzed in Table 4.

[0095] Table 3

[0096]

[0097] Table 4

[0098]

[0099] like Figure 9 As shown, 87.2% of the images can be detected in the first stage without needing to call the open vocabulary detection function in the second stage, significantly improving detection efficiency. This means that in practical industrial applications, most weld images can be quickly detected for known defects. 12.9% of the images triggered the second stage of detection due to the presence of suspicious areas or no detection results (11 images triggered due to suspicious areas, and 3 images triggered due to no detection results at all, achieving a reasonable allocation of computing resources). Among these images, 2.8% showed new defects with pre-described text, and 10.1% required manual review. The second stage successfully detected new defects in 3 images (confidence levels 0.055~0.132), verifying the effectiveness of open vocabulary detection in real-world scenarios. For the 11 images that did not yield valid results in either stage, the system automatically marked them as "requiring manual review," supporting a human-machine collaborative detection mode and reducing the risk of missed detections.

[0100] The representative detection performance of the two-stage cascaded detection method on the test set is as follows: Figure 10 As shown in the figure. Images starting with the label "S1:" indicate that the detection was completed in the first stage only (high confidence detection of known defects). Images starting with the label "S2 NEW:" indicate that new defects were successfully detected in the second stage. Images with red labels indicate samples that were not detected in either stage and require manual review. Detection boxes starting with the label "[?]" indicate suspicious areas, detection boxes starting with the label "[NEW]" indicate areas where unknown defects were detected, and the remaining unlabeled areas are areas where known defects were detected.

[0101] The two-stage cascaded detection method proposed in this invention not only ensures high efficiency in detecting known defects (87.2% completed in the first stage), but also achieves rapid response capability to new defects through open vocabulary detection in the second stage, while ensuring that no potential defects are missed through a manual review mechanism.

[0102] Furthermore, as shown in Table 5, the YOLOv8l-Worldv2 model used in this invention is trained and tested on the same weld defect dataset as models such as YOLOv5l (a classic YOLO series model), YOLOv8l, YOLO11l (the latest YOLO model released in 2024), and RT-DETR-l (a real-time detection model based on Transformer), using the same data augmentation strategies and training configurations.

[0103] Table 5

[0104]

[0105] like Figure 11 As shown, this invention performs comparably to other models in known defect detection, achieving an average precision of 89.0% with an intersection-union ratio (IU) threshold of 0.5. This is on par with other mainstream models (YOLOv8l's 88.5%, YOLOv11l's 89.8%, etc.), with a precision difference of only 0.5%-0.8%, a negligible difference, demonstrating the excellent performance of this invention in known defect detection tasks. The recall rate of this invention reaches 93.1%, the highest among all models, indicating that it can detect various defects more comprehensively with the lowest false negative rate. This invention achieves open vocabulary detection capabilities that other models lack, at a negligible difference of only 0.8% in average precision with an IU threshold of 0.5. When new defects appear on the production line, the detection rate of closed-set models (YOLOv5l, YOLOv8l, YOLO11l, RT-DETR-l) is zero. Data must be collected, labeled, and trained again, which takes several weeks. However, this invention detects new defects by means of two-stage cascaded detection, which only requires modification of text prompts, with a response time of less than 1 minute.

[0106] In one optional embodiment, the calculation expression for the multi-task loss function is as follows:

[0107]

[0108] in, Indicates the total loss. The bounding box regression loss uses CIoU (Complete Intersection over Union) Loss, which considers the overlapping area of ​​the bounding boxes, the distance between their center points, and their aspect ratio. This results in faster convergence and more accurate localization compared to traditional IoU Loss. This represents the classification loss, used to optimize the prediction accuracy of defect categories. This represents the Distributed Focal Loss, which models bounding box regression as a discrete probability distribution, improving localization accuracy, especially for regression of targets with ambiguous boundaries. Indicates the bounding box regression loss weights. Represents the classification loss weights. The distributed focus loss weights are used to jointly optimize the three loss terms, enabling the model to simultaneously learn accurate target localization and correct category classification.

[0109] In one optional embodiment, the data augmentation method includes: a Mosaic data augmentation method, a Mixup data augmentation method, an HSV color space augmentation method, and a horizontal flipping method;

[0110] Mosaic enhancement randomly concatenates four training images into a new image. The concatenation method is as follows:

[0111]

[0112] in, This represents a new image formed by randomly stitching together four RGB color images from the dataset. This indicates a splicing operation. This represents the randomly cropped region of the first image. This represents the randomly cropped region of the second image. This represents the randomly cropped region of the third image. This represents a randomly cropped region from the fourth image. In weld defect detection scenarios, Mosaic enhancement is particularly helpful in improving the model's ability to detect small defects such as porosity and micro-splashes. The advantages of this method are: it significantly increases the diversity of training samples, allowing the model to learn features from four images simultaneously in a single forward propagation; it forces the model to learn to recognize targets at a smaller scale, which is beneficial for detecting small target defects; and it reduces reliance on large batch training, thus lowering GPU memory requirements.

[0113] Mixup enhancement proportionally blends two images to generate new training samples. The calculation expression is as follows:

[0114]

[0115] in, This represents a new image formed by blending two RGB color images in the dataset. express distributed, Indicates the mixing ratio, from Distributed sampling express Hyperparameters of the distribution (in this embodiment) ), This represents the first image. The second image represents a situation where, in weld defect detection tasks, there may be blurred boundaries between different defect types (such as slight porosity and normal welds). Mixup enhancement helps the model better handle these boundary situations. This method generates intermediate state training samples through linear interpolation of the images, prompting the model to learn smoother decision boundaries and effectively preventing overfitting.

[0116] HSV color space enhancement randomly dithers the hue, saturation, and value to simulate different lighting and shooting conditions. The calculation expression is as follows:

[0117]

[0118] in, This represents the original hue of the RGB color images in the dataset. , indicating an additive perturbation of hue. Indicates the enhanced hue. This represents the original saturation of the RGB color images in the dataset. , representing the multiplicative perturbation of saturation. Indicates the enhanced saturation. This represents the original brightness of the RGB color images in the dataset. This represents the multiplicative perturbation of brightness. This indicates the enhanced brightness. The design principle of this method is that the lighting conditions in industrial sites are often unstable, and images from different times and workstations may have significant color differences. Through HSV enhancement, the model can learn to be robust to color changes, ensuring stable detection performance even when lighting conditions change during actual deployment. In particular, for defect types that depend on color features, such as weld slag (yellowish brown) and oxide areas (blue-purple), HSV enhancement helps the model learn a more robust color representation.

[0119] In one optional embodiment, the pre-trained YOLO-World model predicts the weld defect category based on the input RGB color image and defect category cue words. The core of this prediction is achieved through visual-text similarity matching, which in turn enables open-vocabulary detection. The calculation expression is as follows:

[0120]

[0121] in, Indicates the confidence level. This indicates the image region where the defect is located in the RGB color image. This indicates the defect category corresponding to the defect category prompt. Represents a set of defect categories. Represents a single defect category in a set of defect categories. The visual feature vector representing the image region is obtained by encoding the RGB color image in the pre-trained YOLO-World model's image encoder. The text feature vector representing the defect category is obtained by encoding the defect category prompt word in the text encoder of the pre-trained YOLO-World model. Indicates temperature parameter, Represents the cosine similarity function. This represents the natural exponential function. The innovation of this mechanism lies in the fact that traditional object detection models use fixed-dimensional classification heads, where the number of categories is determined during training and cannot be changed. In contrast, the open-vocabulary detection mechanism employed in this invention transforms category prediction into a similarity matching problem between visual and textual features. Since textual features can be dynamically generated during inference, the model can detect any new category without retraining. In the scenario of weld defect detection, when a new type of defect is discovered on the production line, the model can immediately acquire detection capabilities simply by describing the defect features in natural language, reducing the response time from several weeks in traditional methods to within one minute.

[0122] In an alternative embodiment, such as Figure 12 As shown, the three-step prompt word engineering includes:

[0123] The visual characteristics of defects are observed systematically from four dimensions: color, shape, location, and texture, forming a structured feature description and obtaining the observed features.

[0124] The observed features are integrated into a natural language representation, focusing on describing the appearance of the defects while avoiding the use of technical terms.

[0125] Based on the integrated representational descriptions, three levels of English cue words are generated. Coarse-grained cue words use general terminology to directly describe the defect type, medium-grained cue words use appearance to describe defect features, and fine-grained cue words use detailed visual features, including color, shape, texture, etc.

[0126] As shown in Tables 6 and 7, a three-step method was used to construct different granularity prompt words for the collected unknown defects of slag inclusion and slag not removed. It can be seen intuitively that the subjective defect description is transformed into a structured and reproducible prompt word generation process. The prompt words of different granularities are suitable for different scenarios (such as coarse granularity is suitable for rapid screening, and fine granularity is suitable for accurate detection). Moreover, the entire three-step process can be completed in a few minutes, which is much faster than the traditional data annotation-model retraining process.

[0127] Table 6

[0128]

[0129] Table 7

[0130]

[0131] In an optional embodiment, after the two-stage cascaded detection system successfully detects new defects in the second stage during actual deployment, these newly detected defect samples can be included in the training data to incrementally train the model and gradually expand its detection capabilities. After step S4, the system further includes:

[0132] The RGB color image containing the unknown defect detection region is added to the dataset, and the pre-trained YOLO-World model is incrementally trained. The calculation expression is as follows:

[0133]

[0134] in, This represents the YOLO-World model after incremental training. This represents the current YOLO-World model. This indicates fine-tuning of the pre-trained model. This represents the training data comprised of the existing dataset. This represents the training data consisting of newly added RGB color images.

[0135] In the specific implementation process, such as Figure 13 As shown, the two-stage cascaded detection system includes the following steps during operation:

[0136] Step 1: In the second stage, new defect types are discovered through open vocabulary detection, or defects that cannot be detected by the model are confirmed by manual review.

[0137] Step 2: Save the image and detection box of the newly detected defect, and have the technicians confirm and correct the annotation (since the model has provided preliminary detection results, the annotation workload is significantly reduced).

[0138] Step 3: Merge the newly labeled defect samples with the original training data and fine-tune the model. The training strategies include: using a lower learning rate (1 / 10 of the original learning rate) to avoid catastrophic forgetting; maintaining the sampling ratio of the original category data to ensure that the performance of known defect detection does not decrease; and adding new category prompt words to the model's text prompt set.

[0139] Step 4: After verifying that the incrementally trained model achieves the expected performance on both the old and new defects, replace the deployed model.

[0140] Example 3

[0141] This embodiment proposes a YOLO-World-based color image detection system for weld defects, comprising: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the steps of the YOLO-World-based color image detection method for weld defects as described above.

[0142] Specifically, such as Figure 14 As shown, this embodiment of the invention also provides a computer system, including a bus, a transceiver, an antenna, a bus interface, a processor, and a memory.

[0143] The transceiver is used to acquire input data, which includes RGB color images, fixed prompt words corresponding to known defect categories, and prompt words for unknown defect categories.

[0144] The processor is configured to perform a first-stage detection process on the RGB color image and the fixed prompt words corresponding to the known defect categories based on a pre-trained YOLO-World model to obtain a first defect detection region and a region with no detection results; and to perform a second-stage detection process on the RGB color image and the constructed prompt words for unknown defect categories to obtain a second defect detection region.

[0145] If the confidence level in the first defect detection area is greater than the high confidence threshold, the first defect detection area is a known defect detection area; if the confidence level in the first defect detection area is less than the high confidence threshold but greater than the first stage detection confidence threshold, the first defect detection area is a suspicious area.

[0146] If the confidence level in the second defect detection region is greater than the second-stage detection confidence threshold, then the second defect detection region is an unknown defect detection region.

[0147] A bus can include any number of interconnected buses and bridges, linking together various circuits including one or more processors represented by a processor and memories represented by memory. The bus can also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0148] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0149] Optionally, the processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPD).

Claims

1. A method for detecting weld defects using color images based on YOLO-World, characterized in that, include: S1: Acquire RGB color images of the weld surface, input the RGB color images and fixed prompts corresponding to known defect categories into the pre-trained YOLO-World model for the first stage of detection, and output the first defect detection area and the area with no detection result. The defect detection area includes the detection box, the defect category and the confidence level corresponding to the defect category. S2: If the confidence level in the first defect detection area is greater than the high confidence threshold, the first defect detection area is a known defect detection area; if the confidence level in the first defect detection area is less than the high confidence threshold but greater than the first stage detection confidence threshold, the first defect detection area is a suspicious area. S3: Based on the regions with no detection results or the suspicious regions, a three-step prompt word engineering method is used to construct unknown defect category prompt words. These unknown defect category prompt words are then input into the pre-trained YOLO-World model for the second stage of detection, outputting the second defect detection region. The three-step prompt word engineering method includes: The regions with no detection results or the suspicious regions are systematically observed from different dimensions to obtain observation features; The observed features are integrated at the natural language level to obtain a representational description; Based on the representational description and the pre-trained YOLO-World model, unknown defect category prompts of different granularities are generated; S4: If the confidence level in the second defect detection area is greater than the second-stage detection confidence threshold, the second defect detection area is an unknown defect detection area.

2. The method for detecting weld defects using color images based on YOLO-World according to claim 1, characterized in that, If the confidence level in the first defect detection region is less than the first-stage detection confidence threshold, the first defect detection region is a background noise region unrelated to the defect. If the confidence level in the second defect detection area is less than the confidence level threshold for the second stage detection, the second defect detection area will be subject to manual review.

3. The method for detecting weld defects using color images based on YOLO-World according to claim 1, characterized in that, Before step S1, the following are also included: RGB color images of the weld surface were acquired and a dataset was constructed. Based on the dataset, the YOLOv8l-Worldv2 model is trained using a multi-task loss function to obtain a pre-trained YOLO-World model. The YOLOv8l-Worldv2 model is a model obtained by training the YOLO-World model on a large-scale dataset.

4. The method for detecting weld defects using color images based on YOLO-World according to claim 3, characterized in that, The training of the model based on transfer learning includes: The dataset is divided according to a preset ratio to obtain a training set, a validation set, and a test set; The hyperparameters for model training are preset, the training set and the validation set are input into the YOLOv8l-Worldv2 model and trained using a multi-task loss function until the loss function converges, thus obtaining the pre-trained YOLO-World model. The performance of the pre-trained YOLO-World model is evaluated using the test set.

5. The method for detecting weld defects using color images based on YOLO-World according to claim 4, characterized in that, The calculation expression for the multi-task loss function is as follows: in, Indicates the total loss. This represents the bounding box regression loss. Represents classification loss, Represents distributed focus loss. Indicates the bounding box regression loss weights. Represents the classification loss weights. This represents the weight of the distributed focus loss.

6. The method for detecting weld defects using color images based on YOLO-World according to claim 4, characterized in that, During training, data augmentation methods are used to enhance the training set. These data augmentation methods include: Mosaic data augmentation, Mixup data augmentation, HSV color space enhancement, and horizontal flipping. The calculation expression for the Mosaic data augmentation method is as follows: in, This represents a new image formed by randomly stitching together four RGB color images from the dataset. This indicates a splicing operation. This represents the randomly cropped region of the first image. This represents the randomly cropped region of the second image. This represents the randomly cropped region of the third image. This represents the randomly cropped region of the fourth image; The calculation expression for the Mixup data augmentation method is as follows: in, This represents a new image formed by blending two RGB color images in the dataset. express distributed, Indicates the mixing ratio, from Distributed sampling express Hyperparameters of the distribution This represents the first image. This refers to the second image; The calculation expression for the HSV color space enhancement method is as follows: in, This represents the original hue of the RGB color images in the dataset. This represents an additive perturbation of hue. Indicates the enhanced hue. This represents the original saturation of the RGB color images in the dataset. The multiplicative perturbation representing saturation. Indicates the enhanced saturation. This represents the original brightness of the RGB color images in the dataset. Represents the multiplicative perturbation of brightness. This indicates the enhanced brightness.

7. The method for detecting weld defects using color images based on YOLO-World according to claim 1, characterized in that, The pre-trained YOLO-World model predicts weld defect categories based on the input RGB color image and defect category hints, and the calculation expression is as follows: in, Indicates the confidence level. This indicates the image region where the defect is located in the RGB color image. This indicates the defect category corresponding to the defect category prompt. Represents a set of defect categories. Represents a single defect category in a set of defect categories. The visual feature vector representing the image region is obtained by encoding the RGB color image in the pre-trained YOLO-World model's image encoder. The text feature vector representing the defect category is obtained by encoding the defect category prompt word in the text encoder of the pre-trained YOLO-World model. Indicates temperature parameter, Represents the cosine similarity function. This represents the natural exponential function.

8. The method for detecting weld defects using color images based on YOLO-World according to claim 3, characterized in that, After step S4, the following also includes: The RGB color image containing the unknown defect detection region is added to the dataset, and the pre-trained YOLO-World model is incrementally trained. The calculation expression is as follows: in, This represents the YOLO-World model after incremental training. This represents the current YOLO-World model. This indicates fine-tuning of the pre-trained model. This represents the training data comprised of the existing dataset. This represents the training data consisting of newly added RGB color images.

9. A color image detection system for weld defects based on YOLO-World, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the YOLO-World-based color image detection method for weld defects as described in any one of claims 1 to 8.

Images