Welding defect detection method and system based on image and ultrasonic wave and medium

Abstract

The embodiment of the invention provides a welding defect detection method and system based on an image and ultrasonic waves and a medium, and the method comprises the steps: obtaining an ultrasonic image, carrying out the noise reduction processing of the ultrasonic image, and carrying out the enhancement processing of the ultrasonic image after the noise reduction processing, and obtaining an enhanced image; performing region segmentation on the enhanced image to obtain a plurality of sub-region images, and extracting image features of each sub-region image; analyzing the image features based on a set feature interval to obtain welding features, and screening out a welding image based on the welding features; constructing a defect detection model, inputting the welding image into the defect detection model, and outputting welding defect data; transmitting the welding defect data to a terminal in real time according to a preset mode; and the welding image is detected and analyzed through the defect detection model, so that the welding defect characteristics are accurately analyzed, automatic detection of the welding defects is realized, and the welding defect detection precision is improved.

Description

Technical Field

[0001] The present application relates to the technical field of welding defect detection, and more particularly, to a welding defect detection method, system and medium based on images and ultrasonic waves. Background Art

[0002] Welding technology is an important industrial production method that establishes a permanent connection between components and is widely used in various departments. However, welding operations are susceptible to factors such as setting adjustments, environmental conditions, and whether operators comply with standards, resulting in defects such as cracks, incomplete fusion, and porosity, seriously affecting production safety. Metals, as important materials in modern industrial production and manufacturing, are welded into a cohesive whole using various metal workpieces. Metal welding has a wide range of applications in electrical equipment in daily life, as well as in the aerospace and defense industries. However, due to factors such as parameter settings, environmental conditions, and employee operation standards, welded parts will inevitably have defects such as cracks, incomplete welds, and bubbles. Traditional defect detection methods include magneto-optical imaging, x-ray inspection, ultrasonic testing, etc., and the images or the workpiece body are inspected and analyzed manually. However, in such a way, due to the subjectivity of manual detection, problems such as misdetection and missed detection are likely to occur during long-term evaluation. Summary of the Invention

[0003] The purpose of the embodiments of the present application is to provide a welding defect detection method, system and medium based on images and ultrasonic waves, which detect and analyze welding images through a defect detection model, so as to accurately analyze the characteristics of welding defects, realize the automatic detection of welding defects, and improve the accuracy of welding defect detection.

[0004] The embodiments of the present application also provide a welding defect detection method based on images and ultrasonic waves, including:

[0005] Obtain ultrasonic images, perform noise reduction processing on the ultrasonic images, and perform enhancement processing on the noise-reduced ultrasonic images to obtain enhanced images;

[0006] Perform region segmentation on the enhanced images to obtain several sub-region images, and extract the image features of each sub-region image;

[0007] Analyze the image features based on a set feature interval to obtain welding features, and screen out welding images based on the welding features;

[0008] Construct a defect detection model, input the welding images into the defect detection model, and output welding defect data;

[0009] Transmit the welding defect data to the terminal in real time according to a predetermined method.

[0010] Optionally, in the welding defect detection method based on image and ultrasound described in the embodiment of the present application, obtaining an ultrasonic image and performing noise reduction processing on the ultrasonic image specifically includes:

[0011] Acquire ultrasonic images and extract ultrasonic image features;

[0012] Compare the ultrasonic image features with the set features to obtain the feature deviation rate;

[0013] Determine whether the characteristic deviation rate is greater than or equal to a set characteristic deviation rate threshold;

[0014] If it is greater than or equal to the set feature deviation rate threshold, a noise feature is generated and the noise feature is removed;

[0015] If it is less than the set feature deviation rate threshold, the ultrasonic image features are judged to be normal.

[0016] Optionally, in the welding defect detection method based on image and ultrasound described in the embodiment of the present application, the ultrasonic image after noise reduction is enhanced to obtain an enhanced image, specifically including:

[0017] Acquire the ultrasonic image after noise reduction processing, and perform grayscale processing on the ultrasonic image after noise reduction processing to obtain a grayscale image;

[0018] Analyze the grayscale value of the image based on the grayscale image, compare the image grayscale value with the set grayscale threshold, and obtain the grayscale difference;

[0019] Screen out background features and artifact features based on grayscale difference;

[0020] The grayscale values of the workpiece features are enhanced and the grayscale values of the background features are blurred to obtain an enhanced image.

[0021] Optionally, in the welding defect detection method based on image and ultrasound described in the embodiment of the present application, the enhanced image is segmented into regions to obtain a plurality of sub-region images, and image features of each sub-region image are extracted, specifically including:

[0022] Set multiple grayscale thresholds to obtain the grayscale value of each pixel of the enhanced image;

[0023] Compare the gray value of each pixel with multiple gray thresholds to obtain a comparison result;

[0024] Based on the comparison results, the pixels are classified to obtain multiple categories, and based on the multiple categories, the pixels in the same category are divided into a region to obtain a plurality of sub-region images corresponding to different categories;

[0025] Analyze the resolution of the sub-region image and determine whether the resolution reaches the set resolution;

[0026] If the set resolution is reached, extract the image features of each sub-region image;

[0027] If the set resolution is not reached, adjust the magnitudes of multiple grayscale thresholds.

[0028] Optionally, in the welding defect detection method based on images and ultrasonic waves described in the embodiments of the present application, construct a defect detection model, input the welding image into the defect detection model, and output welding defect data, specifically including:

[0029] Construct a YOLOv7 model and a loss function, obtain ultrasonic images based on big data, and crop the regions with welding defects in the ultrasonic images to obtain a number of cropped images;

[0030] Based on the number of cropped images, set an allocation ratio, and divide the cropped images into a training set, a test set, and a validation set based on the allocation ratio;

[0031] Train the YOLOv7 model based on the training set and calculate the training loss value according to the loss function;

[0032] Compare the training loss value with the set loss threshold to obtain a loss difference value;

[0033] Adjust the hyperparameters of the YOLOv7 model based on the loss difference value, and test and validate the adjusted YOLOv7 model based on the test set and the validation set to obtain a defect detection model, and analyze the welding defect data based on the defect detection model.

[0034] Optionally, in the welding defect detection method based on images and ultrasonic waves described in the embodiments of the present application, the loss function includes a bounding box regression loss function, a classification loss function, and an objectness loss function, and a total loss function is obtained based on the weighted sum of the bounding box regression loss function, the classification loss function, and the objectness loss function.

[0035] In a second aspect, the embodiments of the present application provide a welding defect detection system based on images and ultrasonic waves, the system includes: a memory and a processor, the memory includes a program of the welding defect detection method based on images and ultrasonic waves, and when the program of the welding defect detection method based on images and ultrasonic waves is executed by the processor, the following steps are implemented:

[0036] Obtain an ultrasonic image, perform noise reduction processing on the ultrasonic image, and perform enhancement processing on the noise-reduced ultrasonic image to obtain an enhanced image;

[0037] Performing region segmentation on the enhanced image to obtain a number of sub-region images, and extracting image features of each sub-region image;

[0038] Analyze the image features based on the set feature interval to obtain welding features, and select welding images based on the welding features;

[0039] Construct a defect detection model, input the welding image into the defect detection model, and output the welding defect data;

[0040] The welding defect data is transmitted to the terminal in real time according to the predetermined method.

[0041] Optionally, in the welding defect detection system based on image and ultrasound described in the embodiment of the present application, obtaining an ultrasonic image and performing noise reduction processing on the ultrasonic image specifically includes:

[0042] Acquire ultrasonic images and extract ultrasonic image features;

[0043] Compare the ultrasonic image features with the set features to obtain the feature deviation rate;

[0044] Determine whether the characteristic deviation rate is greater than or equal to a set characteristic deviation rate threshold;

[0045] If it is greater than or equal to the set feature deviation rate threshold, a noise feature is generated and the noise feature is removed;

[0046] If it is less than the set feature deviation rate threshold, the ultrasonic image features are judged to be normal.

[0047] Optionally, in the welding defect detection system based on image and ultrasound described in the embodiment of the present application, the ultrasonic image after noise reduction is enhanced to obtain an enhanced image, specifically including:

[0048] Acquire the ultrasonic image after noise reduction processing, and perform grayscale processing on the ultrasonic image after noise reduction processing to obtain a grayscale image;

[0049] Analyze the grayscale value of the image based on the grayscale image, compare the image grayscale value with the set grayscale threshold, and obtain the grayscale difference;

[0050] Screen out background features and artifact features based on grayscale difference;

[0051] The grayscale values of the workpiece features are enhanced and the grayscale values of the background features are blurred to obtain an enhanced image.

[0052] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, which includes a welding defect detection method program based on images and ultrasonic waves. When the welding defect detection method program based on images and ultrasonic waves is executed by a processor, the steps of the welding defect detection method based on images and ultrasonic waves as described in any one of the above are implemented.

[0053] As can be seen from the above, an embodiment of the present application provides a welding defect detection method, system and medium based on images and ultrasonic waves. By acquiring ultrasonic images, performing noise reduction processing on the ultrasonic images, and performing enhancement processing on the noise-reduced ultrasonic images to obtain enhanced images; performing region segmentation on the enhanced images to obtain a number of sub-region images, and extracting the image features of each sub-region image; analyzing the image features based on a set feature interval to obtain welding features, screening out welding images based on the welding features; constructing a defect detection model, inputting the welding images into the defect detection model, and outputting welding defect data; transmitting the welding defect data to the terminal in a predetermined manner in real time; detecting and analyzing the welding images through the defect detection model, so as to accurately analyze the welding defect features, realize the automatic detection of welding defects, and improve the accuracy of welding defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0055] Figure 1 It is a flowchart of the welding defect detection method based on images and ultrasonic waves provided by an embodiment of the present application;

[0056] Figure 2 It is a flowchart of the noise reduction processing of ultrasonic images in the welding defect detection method based on images and ultrasonic waves provided by an embodiment of the present application;

[0057] Figure 3 It is a flowchart of the enhanced image acquisition method in the welding defect detection method based on images and ultrasonic waves provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0058] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Usually, the components of the embodiments of the present application described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents the selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts belong to the scope of protection of the present application.

[0059] It should be noted that similar reference numerals and letters indicate similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present application, terms such as "first" and "second" are only used for differential description and cannot be understood as indicating or implying relative importance.

[0060] Please refer to Figure 1 , Figure 1 which is a flowchart of a welding defect detection method based on image and ultrasonic in some embodiments of the present application. This welding defect detection method based on image and ultrasonic is used in a terminal device. This welding defect detection method based on image and ultrasonic includes the following steps:

[0061] S101, acquire an ultrasonic image, perform noise reduction processing on the ultrasonic image, and perform enhancement processing on the noise-reduced ultrasonic image to obtain an enhanced image;

[0062] S102, perform region segmentation on the enhanced image to obtain several sub-region images, and extract the image features of each sub-region image;

[0063] S103, analyze the image features based on a set feature interval to obtain welding features, and screen out welding images based on the welding features;

[0064] S104, construct a defect detection model, input the welding image into the defect detection model, and output welding defect data;

[0065] S105, transmit the welding defect data to the terminal in a predetermined manner in real time.

[0066] It should be noted that the present application is based on the YOLOv7 network model as the basic framework, establishes a welding defect detection model for ultrasonic images based on machine vision, adjusts the model parameters and adds an SE attention mechanism module, and locates and identifies the welding defects of the screened welding images, so as to accurately analyze the position, shape and contour of the welding defects according to the welding defect data.

[0067] Please refer to Figure 2 , Figure 2 which is a flowchart of ultrasonic image noise reduction processing for a welding defect detection method based on images and ultrasonic waves in some embodiments of the present application. According to the embodiments of the present invention, an ultrasonic image is obtained and the ultrasonic image is subjected to noise reduction processing, which specifically includes:

[0068] S201, obtain the ultrasonic image and extract the ultrasonic image features;

[0069] S202, compare the ultrasonic image features with the set features to obtain a feature deviation rate;

[0070] S203, determine whether the feature deviation rate is greater than or equal to the set feature deviation rate threshold;

[0071] S204, if it is greater than or equal to the set feature deviation rate threshold, generate noise features and eliminate the noise features;

[0072] S205, if it is less than the set feature deviation rate threshold, determine that the ultrasonic image features are normal.

[0073] It should be noted that image noise reduction refers to removing the noise introduced during the capture, transmission, or storage process without damaging the important information of the image, improving the reflection ability and reflection accuracy of ultrasonic image features, and preventing misjudgment of welding defects.

[0074] Please refer to Figure 3 , Figure 3 which is a flowchart of an enhanced image acquisition method for a welding defect detection method based on images and ultrasonic waves in some embodiments of the present application. According to the embodiments of the present invention, the noise-reduced ultrasonic image is subjected to enhancement processing to obtain an enhanced image, which specifically includes:

[0075] S301, obtain the noise-reduced ultrasonic image, perform grayscale processing on the noise-reduced ultrasonic image to obtain a grayscale image;

[0076] S302, analyze the image grayscale value based on the grayscale image, compare the image grayscale value with the set grayscale threshold to obtain a grayscale difference;

[0077] S303, screen out background features and workpiece features based on the grayscale difference;

[0078] S304, perform enhancement processing on the grayscale value of the workpiece features and perform blurring processing on the grayscale value of the background features to obtain an enhanced image.

[0079] It should be noted that when analyzing welding defects, only information such as the contour and position of the solder joints needs to be analyzed, and the color of the workpiece does not need to be concerned about. Therefore, processing the color image into grayscale by processing the three RGB components respectively greatly increases the amount of computation and reduces the processing efficiency.

[0080] According to an embodiment of the present invention, the enhanced image is subjected to region segmentation to obtain a plurality of sub-region images, and the image features of each sub-region image are extracted, specifically including:

[0081] Set multiple grayscale thresholds and obtain the grayscale value of each pixel point of the enhanced image;

[0082] Compare the grayscale value of each pixel point with the multiple grayscale thresholds to obtain a comparison result;

[0083] Classify the pixel points based on the comparison result to obtain multiple categories, and divide the pixel points within the same category into one region based on the multiple categories to obtain several sub-region images corresponding to different categories;

[0084] Analyze the resolution of the sub-region image and determine whether the resolution reaches the set resolution;

[0085] If the set resolution is reached, extract the image features of each sub-region image;

[0086] If the set resolution is not reached, adjust the magnitudes of the multiple grayscale thresholds.

[0087] It should be noted that image segmentation is a technology and process of dividing an image into several specific regions with unique properties and extracting the target of interest, so as to facilitate more in-depth analysis and processing.

[0088] According to an embodiment of the present invention, a defect detection model is constructed, and the welding image is input into the defect detection model to output welding defect data, specifically including:

[0089] Construct a YOLOv7 model and a loss function, obtain ultrasonic images based on big data, and crop the regions with welding defects in the ultrasonic images to obtain several cropped images;

[0090] Based on the number of cropped images, set an allocation ratio, and divide the cropped images into a training set, a test set, and a validation set based on the allocation ratio;

[0091] Train the YOLOv7 model based on the training set and calculate the training loss value according to the loss function;

[0092] Compare the training loss value with the set loss threshold to obtain a loss difference value;

[0093] Adjust the hyperparameters of the YOLOv7 model based on the loss difference value, and test and validate the adjusted YOLOv7 model based on the test set and the validation set to obtain a defect detection model, and analyze the welding defect data based on the defect detection model.

[0094] It should be noted that the C-scan image of the Doppler ultrasound detector, that is, the ultrasonic image, is obtained. The areas with welding defects in a total of 68 ultrasonic images are cropped, and finally 147 images are obtained and divided into 3 groups, namely 92 for the training set, 40 for the validation set, and 15 for the test set. The training iteration times are set to 3000, and 16 samples are set to participate in the training each time, and other parameters of the YOLOv7 model remain the default values.

[0095] The box loss, objective loss, accuracy, and recall rate curves of the training set and the test set during the training process. During the training process, the Box loss (bounding box regression loss value) and Objectness loss (objectness loss value) of the training set and the test set both gradually decrease. When the training iteration times are about 2000, the loss value tends to be stable, but due to overfitting, the loss value on the training set still tends to decrease. At the same time, the improvement of the accuracy and recall rate indicates that the detection accuracy is improving.

[0096] According to the embodiments of the present invention, the loss function includes a bounding box regression loss function, a classification loss function, and an objectness loss function, and the total loss function is obtained based on the weighted sum of the bounding box regression loss function, the classification loss function, and the objectness loss function.

[0097] It should be noted that the formula of the bounding box regression loss function is as follows:

[0098]

[0099] IoU represents the intersection over union of the predicted bounding box and the ground truth bounding box, ρ(b,b gt ) represents the Euclidean distance between the center points of the predicted bounding box b and the ground truth bounding box b gt C is the diagonal length of the smallest closed region covering the two bounding boxes, v is a term to measure the consistency of the aspect ratio of the bounding boxes, used to handle the problem of inconsistent aspect ratios, and α is a scaling parameter used to adjust the influence of the aspect ratio penalty term.

[0100] The formula of the classification loss function is as follows:

[0101]

[0102] C is the total number of classes; y o,c is a label indicating whether the predicted bounding box o belongs to class C; p o,c is the probability that the model predicts the bounding box o belongs to class C.

[0103] The formula for the object loss function is as follows:

[0104] L obj = y o log(p o ) + (1 - y o )log(1 - p o );

[0105] y o is a label indicating whether there is an object in the prediction box (1 means there is an object, 0 means there is no object); p o is the probability that there is an object in the model prediction box.

[0106] The overall loss function is the weighted sum of these individual losses:

[0107]

[0108] where λ box , λ cls and λ obj are hyperparameters used to adjust the relative importance of the losses of each part.

[0109] In a second aspect, an embodiment of the present application provides a welding defect detection system based on images and ultrasonic waves. The system includes: a memory and a processor. The memory includes a program for the welding defect detection method based on images and ultrasonic waves. When the program for the welding defect detection method based on images and ultrasonic waves is executed by the processor, the following steps are implemented:

[0110] Obtain an ultrasonic image, perform noise reduction processing on the ultrasonic image, and perform enhancement processing on the noise-reduced ultrasonic image to obtain an enhanced image;

[0111] Perform region segmentation on the enhanced image to obtain a number of sub-region images, and extract the image features of each sub-region image;

[0112] Analyze the image features based on a set feature interval to obtain welding features, and screen out welding images based on the welding features;

[0113] Construct a defect detection model, input the welding image into the defect detection model, and output welding defect data;

[0114] Transmit the welding defect data to the terminal in a predetermined manner in real time.

[0115] It should be noted that based on the YOLOv7 network model as the basic framework, this application establishes a welding defect detection model for ultrasonic images based on machine vision. By adjusting the model parameters and adding the SE attention mechanism module, and localizing and identifying welding defects in the selected welding images, the position, shape, and contour of welding defects can be accurately analyzed according to the welding defect data.

[0116] According to an embodiment of the present invention, an ultrasonic image is obtained, and the ultrasonic image is subjected to noise reduction processing, specifically including:

[0117] Obtain the ultrasonic image and extract the ultrasonic image features;

[0118] Compare the ultrasonic image features with the set features to obtain a feature deviation rate;

[0119] Determine whether the feature deviation rate is greater than or equal to the set feature deviation rate threshold;

[0120] If it is greater than or equal to the set feature deviation rate threshold, generate noise features and eliminate the noise features;

[0121] If it is less than the set feature deviation rate threshold, determine that the ultrasonic image features are normal.

[0122] It should be noted that image noise reduction refers to removing the noise introduced during the capture, transmission, or storage process without damaging the important information of the image, improving the reflection ability and reflection accuracy of ultrasonic image features, and preventing misjudgment of welding defects.

[0123] According to an embodiment of the present invention, the noise-reduced ultrasonic image is subjected to enhancement processing to obtain an enhanced image, specifically including:

[0124] Obtain the noise-reduced ultrasonic image, perform gray-scale processing on the noise-reduced ultrasonic image to obtain a gray-scale image;

[0125] Analyze the image gray-scale value based on the gray-scale image, compare the image gray-scale value with the set gray-scale threshold to obtain a gray-scale difference;

[0126] Based on the gray-scale difference, screen out the background features and workpiece features;

[0127] Strengthen the gray-scale value of the workpiece features and blur the gray-scale value of the background features to obtain an enhanced image.

[0128] It should be noted that when analyzing welding defects, only the contour, position, etc. of the solder joints need to be analyzed, and the color of the workpiece does not need to be concerned. Therefore, performing gray-scale processing on the color image and processing the three RGB components separately greatly increases the amount of computation and reduces the processing efficiency.

[0129] According to an embodiment of the present invention, the enhanced image is segmented into regions to obtain a number of sub-region images, and the image features of each sub-region image are extracted, specifically including:

[0130] Set multiple gray-scale thresholds, and obtain the gray-scale value of each pixel point of the enhanced image;

[0131] Compare the gray-scale value of each pixel point with the multiple gray-scale thresholds to obtain a comparison result;

[0132] Classify the pixel points based on the comparison result to obtain multiple categories, and divide the pixel points within the same category into one region based on the multiple categories to obtain a number of sub-region images corresponding to different categories;

[0133] Analyze the resolution of the sub-region image, and determine whether the resolution reaches the set resolution;

[0134] If the set resolution is reached, extract the image features of each sub-region image;

[0135] If the set resolution is not reached, adjust the magnitudes of the multiple gray-scale thresholds.

[0136] It should be noted that image segmentation is a technology and process of dividing an image into several specific regions with unique properties and extracting the target of interest, so as to facilitate further analysis and processing.

[0137] According to an embodiment of the present invention, a defect detection model is constructed, and the welding image is input into the defect detection model to output welding defect data, specifically including:

[0138] Construct a YOLOv7 model and a loss function, obtain ultrasonic images based on big data, and crop the regions with welding defects in the ultrasonic images to obtain a number of cropped images;

[0139] Based on the number of cropped images, set an allocation ratio, and divide the cropped images into a training set, a test set, and a validation set based on the allocation ratio;

[0140] Train the YOLOv7 model based on the training set, and calculate the training loss value according to the loss function;

[0141] Compare the training loss value with the set loss threshold to obtain a loss difference value;

[0142] Adjust the hyperparameters of the YOLOv7 model based on the loss difference value, and test and validate the adjusted YOLOv7 model based on the test set and the validation set to obtain a defect detection model, and analyze the welding defect data based on the defect detection model.

[0143] It should be noted that the C-scan image of the Doppler ultrasound detector, that is, the ultrasonic image, is obtained. The areas with welding defects in a total of 68 ultrasonic images are cropped, and finally 147 images are obtained and divided into 3 groups, namely 92 for the training set, 40 for the validation set, and 15 for the test set. The number of training iterations is set to 3000, and 16 samples are set to participate in the training each time. Other parameters of the YOLOv7 model remain the default values.

[0144] Box loss, objective loss, accuracy, and recall rate curves of the training set and the test set during the training process. During the training process, the Box loss (bounding box regression loss value) and Objectness loss (objectness loss value) of the training set and the test set both gradually decrease. When the number of training iterations is about 2000, the loss value tends to be stable, but due to overfitting, the loss value on the training set still tends to decrease. At the same time, the improvement of the accuracy and recall rate indicates that the detection accuracy is improving.

[0145] According to the embodiment of the present invention, the loss function includes a bounding box regression loss function, a classification loss function, and an objectness loss function, and the total loss function is obtained based on the weighted sum of the bounding box regression loss function, the classification loss function, and the objectness loss function.

[0146] It should be noted that the formula of the bounding box regression loss function is as follows:

[0147]

[0148] IoU represents the intersection over union of the predicted bounding box and the ground truth bounding box, ρ(b, b gt ) represents the Euclidean distance between the center points of the predicted bounding box b and the ground truth bounding box b gt , C is the diagonal length of the smallest closed region covering the two bounding boxes, v is a term for measuring the consistency of the aspect ratio of the bounding boxes, used to handle the problem of inconsistent aspect ratios, and α is a scaling parameter used to adjust the influence of the aspect ratio penalty term.

[0149] The formula of the classification loss function is as follows:

[0150]

[0151] C is the total number of classes; y o,c is a label indicating whether the predicted bounding box o belongs to class C; p o,c is the probability that the model predicts the bounding box o belongs to class C.

[0152] The formula of the objectness loss function is as follows:

[0153] L obj = y o log(p o )+(1 - y o) log(1 - p o );

[0154] y o is a label indicating whether there is an object in the prediction box (1 means there is an object, 0 means there is no object); p o is the probability that an object exists in the model prediction box.

[0155] The overall loss function is the weighted sum of these individual losses:

[0156]

[0157] where λ box , λ cls and λ obj are hyperparameters used to adjust the relative importance of the losses of each part.

[0158] The third aspect of the present invention provides a computer-readable storage medium, which includes a program for a welding defect detection method based on images and ultrasonic waves. When the program for the welding defect detection method based on images and ultrasonic waves is executed by a processor, the steps of the welding defect detection method based on images and ultrasonic waves as described in any one of the above are implemented.

[0159] A welding defect detection method, system and medium based on images and ultrasonic waves disclosed by the present invention, by acquiring ultrasonic images, performing noise reduction processing on the ultrasonic images, and performing enhancement processing on the noise-reduced ultrasonic images to obtain enhanced images; performing region segmentation on the enhanced images to obtain a plurality of sub-region images, and extracting the image features of each sub-region image; analyzing the image features based on a set feature interval to obtain welding features, screening welding images based on the welding features; constructing a defect detection model, inputting the welding images into the defect detection model, and outputting welding defect data; transmitting the welding defect data to a terminal in a predetermined manner in real time; detecting and analyzing the welding images through the defect detection model, so as to accurately analyze the welding defect features, realize the automatic detection of welding defects, and improve the welding defect detection accuracy.

[0160] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the couplings, direct couplings, or communication connections between the various components shown or discussed can be through some interfaces, and the indirect couplings or communication connections of devices or units can be electrical, mechanical, or other forms.

[0161] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] In addition, each functional unit in the embodiments of the present invention may all be integrated in a processing unit, or each unit may be separately regarded as a unit alone, or two or more units may be integrated in one unit; the above-mentioned integrated unit may be implemented in the form of hardware, or in the form of a combination of hardware and software functional units.

[0163] Those of ordinary skill in the art can understand that all or part of the steps to implement the above method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the aforementioned storage medium includes: removable storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks or optical disks, and other various media that can store program codes.

[0164] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of the present invention. And the aforementioned storage medium includes: removable storage devices, ROM, RAM, magnetic disks or optical disks, and other various media that can store program codes.

Claims

1. A welding defect detection method based on images and ultrasonic waves, characterized in that, Including: Obtain an ultrasonic image, perform noise reduction processing on the ultrasonic image, and perform enhancement processing on the noise-reduced ultrasonic image to obtain an enhanced image; Perform region segmentation on the enhanced image to obtain a number of sub-region images, and extract the image features of each sub-region image; Analyze the image features based on a set feature interval to obtain welding features, and screen out welding images based on the welding features; Construct a defect detection model, input the welding image into the defect detection model, and output welding defect data; Transmit the welding defect data to the terminal in a predetermined manner in real time.

2. The method for detecting welding defects based on images and ultrasonic waves according to claim 1, wherein Obtain an ultrasonic image, and perform noise reduction processing on the ultrasonic image, specifically including: Obtain an ultrasonic image and extract ultrasonic image features; Compare the ultrasonic image features with the set features to obtain a feature deviation rate; Determine whether the feature deviation rate is greater than or equal to a set feature deviation rate threshold; If it is greater than or equal to the set feature deviation rate threshold, generate noise features and eliminate the noise features; If it is less than the set feature deviation rate threshold, determine that the ultrasonic image features are normal.

3. The method for detecting welding defects based on images and ultrasonic waves according to claim 2, wherein Perform enhancement processing on the noise-reduced ultrasonic image to obtain an enhanced image, specifically including: Obtain the noise-reduced ultrasonic image, perform grayscale processing on the noise-reduced ultrasonic image to obtain a grayscale image; Analyze the image grayscale value based on the grayscale image, compare the image grayscale value with the set grayscale threshold to obtain a grayscale difference value; Screen out background features and workpiece features based on the grayscale difference value; Strengthen the grayscale value of the workpiece features and blur the grayscale value of the background features to obtain an enhanced image.

4. The method for detecting welding defects based on images and ultrasonic waves according to claim 3, characterized in that Perform region segmentation on the enhanced image to obtain a number of sub-region images, and extract the image features of each sub-region image, specifically including: Set multiple grayscale thresholds and obtain the grayscale value of each pixel point of the enhanced image; Compare the grayscale value of each pixel point with multiple grayscale thresholds to obtain a comparison result; Classify the pixel points based on the comparison result to obtain multiple categories, and divide the pixel points within the same category into one region based on the multiple categories to obtain several sub-region images corresponding to different categories; Analyze the resolution of the sub-region image and determine whether the resolution reaches the set resolution; If the set resolution is reached, extract the image features of each sub-region image; If the set resolution is not reached, adjust the size of the multiple grayscale thresholds.

5. The method for detecting welding defects based on images and ultrasonic waves according to claim 4, wherein Construct a defect detection model, input the welding image into the defect detection model, and output welding defect data, specifically including: Construct a YOLOv7 model and a loss function, obtain ultrasonic images based on big data, and crop the areas with welding defects in the ultrasonic images to obtain a number of cropped images; Based on the number of cropped images, set an allocation ratio, and divide the cropped images into a training set, a test set, and a validation set based on the allocation ratio; Train the YOLOv7 model based on the training set and calculate the training loss value according to the loss function; Compare the training loss value with the set loss threshold to obtain a loss difference value; The hyperparameters of the YOLOv7 model are adjusted based on the loss difference value, and the adjusted YOLOv7 model is tested and verified based on the test set and the validation set to obtain the defect detection model, and the welding defect data is analyzed based on the defect detection model.

6. The method for detecting welding defects based on images and ultrasonic waves according to claim 5, characterized in that The loss function includes a box regression loss function, a classification loss function and an object loss function, and a total loss function is obtained based on a weighted sum of the box regression loss function, the classification loss function and the object loss function.

7. A welding defect detection system based on images and ultrasonic waves, characterized in that, The system includes: a memory and a processor, wherein the memory includes a program of a welding defect detection method based on images and ultrasound, and when the program of the welding defect detection method based on images and ultrasound is executed by the processor, the following steps are implemented: Acquire an ultrasonic image, perform noise reduction processing on the ultrasonic image, and perform enhancement processing on the ultrasonic image after noise reduction processing to obtain an enhanced image; Performing region segmentation on the enhanced image to obtain a number of sub-region images, and extracting image features of each sub-region image; Analyze the image features based on the set feature interval to obtain welding features, and select welding images based on the welding features; Construct a defect detection model, input the welding image into the defect detection model, and output the welding defect data; The welding defect data is transmitted to the terminal in real time according to the predetermined method.

8. The welding defect detection system based on images and ultrasonic waves according to claim 7, wherein Acquire an ultrasonic image and perform noise reduction on the ultrasonic image, specifically including: Acquire ultrasonic images and extract ultrasonic image features; Compare the ultrasonic image features with the set features to obtain the feature deviation rate; Determine whether the characteristic deviation rate is greater than or equal to a set characteristic deviation rate threshold; If it is greater than or equal to the set feature deviation rate threshold, a noise feature is generated and the noise feature is removed; If it is less than the set feature deviation rate threshold, the ultrasonic image features are judged to be normal.

9. The welding defect detection system based on image and ultrasonic according to claim 8, characterized in that The ultrasonic image after noise reduction is enhanced to obtain an enhanced image, specifically including: Acquire the ultrasonic image after noise reduction processing, and perform grayscale processing on the ultrasonic image after noise reduction processing to obtain a grayscale image; Analyze the grayscale value of the image based on the grayscale image, compare the image grayscale value with the set grayscale threshold, and obtain the grayscale difference; Screen out background features and artifact features based on grayscale difference; The grayscale values of the workpiece features are enhanced and the grayscale values of the background features are blurred to obtain an enhanced image.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a welding defect detection method program based on image and ultrasound. When the welding defect detection method program based on image and ultrasound is executed by a processor, the steps of the welding defect detection method based on image and ultrasound are implemented as described in any one of claims 1 to 6.

Images