A weld defect detection method, system, and storage medium
By combining image slicing and semantic segmentation neural networks with edge detection, the problems of low accuracy and high false positive rate in the detection of internal weld structures are solved, and the accurate detection and size calculation of weld defects are realized.
Patent Information
- Application Number
- CN202310213730.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-28
- Publication Date
- 2025-11-28
- Estimated Expiration
- 2043-02-28
AI Technical Summary
Existing technologies for detecting internal weld structures suffer from low accuracy, high false positive rate, and inability to accurately present specific information about weld defects, especially the difficulty in distinguishing between defects of incomplete fusion and incomplete penetration.
Image slicing technology is used to preprocess weld seam images, and a semantic segmentation neural network model is used for defect detection. By combining edge detection subnetwork and morphological analysis processing, weld seam defects are instantiated, and accurate weld seam defect detection is achieved by fusing mask images and morphological processing.
It improves the precision and accuracy of weld defect detection, reduces the misjudgment rate of incomplete fusion and incomplete penetration defects, and ensures the accuracy and reliability of weld defect size calculation.
Smart Images
Figure CN116309409B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of industrial weld detection, and in particular to a weld defect detection method, system and storage medium. BACKGROUND
[0002] Welding technology is widely used in key fields such as infrastructure manufacturing, aerospace, metallurgical industry, petrochemical industry, national defense science and technology, and equipment manufacturing. Welding can be divided into gas welding, resistance welding, arc welding, induction welding, laser welding according to the process, and can be divided into automatic welding, semi-automatic welding and manual welding according to the welding method. When welding, the welded part may have appearance defects and internal defects due to the influence of production equipment and production process, such as weld bumps, undercut, welding deformation and concave appearance defects that can be directly identified by the naked eye, but internal defects such as incomplete fusion, incomplete penetration, cracks, slag inclusion and porosity will reduce the quality of the welded product. The failure of welding may cause corrosion, fatigue failure in daily application, or even system collapse, and even catastrophic consequences. Therefore, in addition to the inspection of the external defects of the weld, in order to protect the economic benefits and the personal safety, effective non-destructive testing of the internal structure of the weld is needed to avoid unqualified welded parts entering the market or affecting production.
[0003] However, the existing technology for detecting the internal structure of the weld has the defects of low recognition accuracy and high misjudgment rate, and the specific unqualified information of the weld defect cannot be accurately presented. SUMMARY
[0004] To solve the above problems, the technical scheme provided by the present application is as follows:
[0005] A weld defect detection method, comprising:
[0006] obtaining a target weld image, and preprocessing the target weld image to obtain a preprocessed target weld image;
[0007] slicing the preprocessed target weld image to obtain a sliced weld image, and inputting the sliced weld image into a semantic segmentation neural network model to obtain a defect mask image of each slice;
[0008] fusing the defect mask images of each slice to obtain a fusion mask image corresponding to the pixel points in the target weld image;
[0009] instantiating the semantic segmentation result of the fusion mask image to obtain weld defect information on the target weld image;
[0010] converting the weld defect information on the target weld image into actual weld defect parameters, performing level assessment on the actual weld defect parameters, and outputting a weld detection result.
[0011] A weld defect detection system comprises a weld defect detection terminal, a platform user terminal, a management server and a database;
[0012] The weld defect detection terminal comprises a memory and a processor, and the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the above-mentioned weld defect detection method.
[0013] The management server comprises a weld defect image management module, a defect detection result management module, a query module, a user management module and a permission management module.
[0014] The database stores weld defect detection information and images.
[0015] A storage medium, the storage medium stores computer program instructions, and the computer program instructions are executed by the processor to realize the above-mentioned weld defect detection method.
[0016] Compared with the prior art, the technical scheme provided by the present application has the following beneficial effects:
[0017] The weld defect detection method considers that the X-ray image size of the weld defect is generally large, mostly around 4964x775, and on the one hand due to the limitation of computer computing power, and on the other hand if the original weld defect X-ray image is directly adjusted to a smaller size, a lot of information will be lost, thereby reducing the detection accuracy, therefore, the image slicing method is designed, the target weld image is sliced according to a certain size, and the upper, lower, left and right edges of the sliced image and the adjacent slices will have a part of overlap during cutting, and these redundant information can have corresponding fusion auxiliary effect when the information is fused after the slicing image is segmented.
[0018] The semantic segmentation network in the weld defect detection method can not only fuse the ability of the Swin Transformer module to extract global image features and the ability of the convnet module to extract local image features, but also further assist the boundary extraction subnetwork to focus on the shape and boundary information of the target object, realize more accurate boundary segmentation, and thus make the subsequent weld defect size calculation more accurate.
[0019] The weld defect detection method of the present application instantiates the weld defect semantic segmentation result of the fusion mask image, which realizes the instantiation of each weld defect by using image morphological analysis processing in the post-processing stage of semantic segmentation. The advantage of this algorithm is that the minimum oblique rectangle of the defect closed contour line is obtained, instead of the minimum positive rectangle, so as to ensure that the maximum value of the actual length and width of the weld defect is obtained. The minimum positive rectangle obtained by ordinary instance segmentation will cause measurement error when the maximum value of the actual length of the weld defect is not in the horizontal or vertical direction. At this point, the subsequent size calculation of the present application is more accurate than the detection accuracy of the instance segmentation or the minimum positive rectangle target anchor box output by the target detection model.
[0020] For the two specific weld defects of incomplete fusion and incomplete penetration, their appearances on the X-ray image are very similar, and sometimes even the on-site inspectors have difficulty in distinguishing them. If only the results of neural network semantic segmentation are used as the type determination results of incomplete fusion and incomplete penetration, the misjudgment rate of the two will be high. Therefore, the weld defect detection method of the present application further detects the straightness of the upper and lower contour lines of the incomplete fusion and incomplete penetration defect connected domain and the color uniformity of the defect connected domain after instantiating the weld defect, combines the preliminary results of semantic segmentation determination, and further makes a fine judgment on the types of incomplete fusion and incomplete penetration weld defects to reduce the misjudgment rate of the final classification results. BRIEF DESCRIPTION OF DRAWINGS
[0021] Figure 1 The weld defect detection method flowchart of the embodiment of the present application.
[0022] Figure 2 The target weld image slicing schematic diagram of the embodiment of the present application.
[0023] Figure 3 The semantic segmentation neural network model WeldSeg network structure diagram of the embodiment of the present application.
[0024] Figure 4 The Swin Transformer block structure diagram of the embodiment of the present application.
[0025] Figure 5 The convnext block structure diagram of the embodiment of the present application.
[0026] Figure 6 The incomplete penetration defect image schematic diagram of the embodiment of the present application.
[0027] Figure 7 The incomplete fusion defect image schematic diagram of the embodiment of the present application.
[0028] Figure 8 The distance from the central axis to the upper and lower contour lines in the incomplete fusion defect area seeking schematic diagram of the embodiment of the present application.
[0029] Figure 9 This is a schematic diagram showing the distance from the centerline to the upper and lower contour lines of the incomplete penetration defect area in an embodiment of the present invention.
[0030] Figure 10 This is a schematic diagram of the segmentation of the non-fused and incompletely penetrated areas in an embodiment of the present invention.
[0031] Figure 11 This is a flowchart illustrating the weld defect assessment level according to an embodiment of the present invention.
[0032] Figure 12 This is a schematic diagram of a weld defect detection system according to an embodiment of the present invention.
[0033] Figure 13 This is a schematic diagram of the management server functional module according to an embodiment of the present invention.
[0034] Figure 14 This is a sequence diagram showing the upload of weld defect detection results and the download of PDF detection reports in an embodiment of the present invention.
[0035] Figure 15 This is a timing diagram of the upload and download of weld defect images according to an embodiment of the present invention. Detailed Implementation
[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0037] Example 1
[0038] Combined with appendix Figure 1 The present invention provides a method for detecting weld defects, comprising:
[0039] Acquire a target weld image, and preprocess the target weld image to obtain a preprocessed target weld image;
[0040] The preprocessed target weld image is sliced to obtain sliced weld images. The sliced weld images are then input into a semantic segmentation neural network model to obtain defect mask images for each slice.
[0041] The defect mask images of each slice are fused to obtain a fused mask image corresponding to the pixels in the target weld image;
[0042] Instance the semantic segmentation result of the fusion mask image to obtain the weld defect information on the target weld image;
[0043] Convert the weld defect information on the target weld image into actual weld defect parameters, and grade the actual weld defect parameters to output a weld detection result.
[0044] In the above embodiments, the source of the target weld image is an X-ray machine as the acquisition device, which can be a film imaging instrument for shooting high-definition digital images after developing the film, or a digital image acquisition card for directly acquiring X-ray digital images.
[0045] In the above embodiments, grading the actual weld defect parameters is performed according to a certain detection standard, such as the national X-ray detection standard.
[0046] In this embodiment, the slicing of the target weld image includes:
[0047] The target weld image is sliced according to a preset slice size, and the slice weld image and the adjacent edges of the slice weld image have an overlapping area.
[0048] Each slice weld image is associated with the position information of the slice weld image on the target weld image, and the associated slice weld image is input into a semantic segmentation neural network model for semantic segmentation.
[0049] In the above embodiments, the size of the directly obtained X-ray image of the weld defect is generally large, mostly around 4964x775, and due to the limitation of computer computing power, the input image in the current deep learning is generally resized. For image classification tasks, the image size is usually small, mostly 256x256, and due to the requirement of image details, the size of the downstream tasks such as semantic segmentation, target detection, and instance segmentation is relatively large, but is generally limited to the range of 1024x768. For our input image resolution, if we consider resizing directly to a fixed size such as 256x256, a lot of information will be lost, resulting in reduced detection accuracy. Therefore, the present application adopts a method as shown in the attached Figure 2The image slice method crops the target weld image according to a slice size of 256x256 (the size of 256x256 is only an implementation manner and does not limit the technical solution). In order to ensure the consistency of the slice image size, the upper, lower, left and right edges of the slice image and the adjacent slice will have a part of overlap during the cropping, and these redundant information can have a corresponding fusion auxiliary effect when the information is fused after the slice image is segmented. Each cropped slice weld image is sent into a semantic segmentation neural network model WeldSeg for semantic segmentation after saving the corresponding position serial number.
[0050] In the embodiment, as shown in the accompanying drawings, Figure 3 The semantic segmentation neural network model WeldSeg includes a semantic segmentation network and an edge detection subnetwork; the semantic segmentation network includes four feature layers, each feature layer includes two semantic segmentation paths, at least two convnet blocks are arranged at the encoding end of the first semantic segmentation path, at least one convnet block is arranged at the decoding end of the first semantic segmentation path, at least two SwinTransformer blocks are arranged at the encoding end of the second semantic segmentation path, and at least one SwinTransformer block is arranged at the decoding end of the second semantic segmentation path; the edge detection subnetwork extracts the boundary features of the slice weld image; the output layer of the semantic segmentation network and the boundary features of the edge detection subnetwork are fused to obtain the defect mask image;
[0051] In the semantic segmentation neural network model WeldSeg, an edge detection loss function and a semantic segmentation loss function are arranged,
[0052] The edge detection loss function adopts a cross-entropy loss function, and the expression is as follows:
[0053]
[0054] In the formula, y edgepre1 represents a predicted value, y edgelab1 represents a corresponding true value,
[0055] The semantic segmentation loss function adopts a selected cross-entropy loss and a Dice loss as a semantic segmentation network loss function, and the expression is as follows:
[0056]
[0057] In the formula, y edgepre2 represents a predicted value, y seglab2TP represents the number of pixel points of positive samples identified as positive samples, TN represents the number of pixel points of negative samples predicted as negative samples, FP represents the number of pixel points of negative samples identified as positive samples, and FN represents the number of pixel points of positive samples identified as negative samples.
[0058] In the above embodiment, the edge detection sub-network focuses on the shape and boundary information of the target object, and predicts whether the pixel point is a defect edge.
[0059] The related parameter settings of the Swin Transformer block and the convnet block in the semantic segmentation neural network model WeldSeg are shown in Table 1.
[0060] Table 1 Related parameters of Swin Transformer block and convnet block in WeldSeg network
[0061]
[0062] In the above embodiment, as shown in FIG. 1, the edge detection sub-network is used to detect the edge of the target object, and the semantic segmentation sub-network is used to detect the defect type of the target object. Figure 3 to FIG. 6, the edge detection sub-network is used to detect the edge of the target object, and the semantic segmentation sub-network is used to detect the defect type of the target object. Figure 5As shown, the encoding end of the first semantic segmentation path in the first feature layer of the semantic segmentation network is provided with two convnet blocks, and the encoding end of the second semantic segmentation path in the first feature layer of the semantic segmentation network is provided with two Swin Transformer blocks; the encoding end of the first semantic segmentation path in the second feature layer of the semantic segmentation network is provided with two convnet blocks, and the encoding end of the second semantic segmentation path in the second feature layer of the semantic segmentation network is provided with two Swin Transformer blocks; the encoding end of the first semantic segmentation path in the third feature layer of the semantic segmentation network is provided with six convnet blocks, and the encoding end of the second semantic segmentation path in the third feature layer of the semantic segmentation network is provided with six Swin Transformer blocks; the encoding end of the first semantic segmentation path in the fourth feature layer of the semantic segmentation network is provided with two convnet blocks, and the encoding end of the second semantic segmentation path in the fourth feature layer of the semantic segmentation network is provided with two Swin Transformer blocks; preferably, the number of convnet blocks on the first semantic segmentation path in the third feature layer is three times the number of convnet blocks on the first semantic segmentation path in the first / second / fourth feature layer, and the number of Swin Transformer blocks on the second semantic segmentation path in the third feature layer is three times the number of Swin Transformer blocks on the second semantic segmentation path in the first / second / fourth feature layer; the decoding end of the first semantic segmentation path in the fourth to first feature layer of the semantic segmentation network is provided with one convnet block, and the decoding end of the second semantic segmentation path in the fourth to first feature layer of the semantic segmentation network is provided with one Swin Transformer block; the adjacent layers adopt a splicing manner.
[0063] In the above embodiment, the semantic segmentation neural network model WeldSeg divides the weld defects into five types: cracks, incomplete penetration, incomplete fusion, round defects, and tungsten / copper clamping when constructing the data set of the weld defect image, and uses the LabelMe label making tool to make the corresponding pixel-level defect image label, thereby forming the weld defect image data set for training, verification and testing. In addition, the number of collected weld defect images is limited, and the proportion of various weld defects is not balanced. Therefore, it is necessary to consider expanding the data set by image enhancement. In addition to the commonly used data enhancement methods, such as 1. geometric transformation methods, such as flipping, rotating, cropping, random erasing, scaling, translation, and jittering; 2. pixel transformation methods, such as adding salt and pepper noise, Gaussian noise, Gaussian blur, adjusting HSV contrast, adjusting brightness, saturation, histogram equalization, adjusting white balance, and the like, the data set can also be expanded by using Cutout, Mixup, Cutmix, RandAugment, Random Erasing, mosaic technology, and GAN network generated defect image.
[0064] In the above embodiment, the loss function is set until the edge detection loss function and the semantic segmentation loss function both satisfy the preset condition, and the semantic segmentation neural network model is established according to the trained semantic segmentation network and the edge detection subnetwork.
[0065] In the above embodiment, the semantic segmentation network not only has the ability to fuse the Swin Transformer module to extract global image features and the convnet module to extract local image features, but also further uses the boundary extraction subnetwork to focus on the shape and boundary information of the target object, so as to realize more accurate boundary segmentation, thereby enabling the subsequent weld defect size calculation to be more accurate.
[0066] In the present embodiment, the defect mask images of each slice are fused according to their position information on the target weld image, and when the defect mask images are restored to the corresponding positions of the target weld image, they are classified and fused as follows:
[0067] a. If the defect mask image has no overlapping pixel points with other defect mask images, the segmentation result of the defect mask image is directly corresponded;
[0068] b. If the defect mask image has overlapping pixel points with other defect mask images, if the segmentation result types of the two defect mask images are the same, the defect type is directly corresponded;
[0069] c. If the pixel points in the defect mask image overlap with other defect mask images, if the two defect mask images correspond to one background point and one defect point, then the defect point type is processed;
[0070] d. If the pixel points in the defect mask image overlap with other defect mask images, if the two defect mask images correspond to two different types of defect points, further query the pixel points in the neighborhood range of the two pixel points, and select the defect type with the most points in the neighborhood range as the final defect type.
[0071] In the above embodiment, since the semantic segmentation neural network model identifies the slice weld seam image, and the slice weld seam image and the adjacent slice weld seam image have overlapping regions, when the defect mask images of each slice are fused, there may be a case that the overlapping region defect type recognition is not the same. Therefore, various cases that occur during fusion need to be classified and processed.
[0072] In the above embodiment, if the pixel points in the defect mask image overlap with other defect mask images, if the two defect mask images correspond to two different types of defect points, specifically, the pixel points in the neighborhood range of the two pixel points can be queried, and the final defect type of the region can be determined scientifically and reasonably.
[0073] In this embodiment, the weld defect semantic segmentation result of the fusion mask image is instantiated, and the instantiation result of the weld defect includes crack, incomplete penetration, incomplete fusion, circular defect, and copper / tungsten inclusion. The fusion mask image is subjected to morphological processing open operation. The eight-neighborhood connected algorithm is used to find each defect connected domain which is independent of each other, and mark the pixel points, types and serial numbers thereof. Each defect connected domain is selected to obtain the contour line of each defect connected domain. According to the contour line of the defect connected domain, the minimum oblique rectangle of each defect connected domain is obtained. The length, width, center point, central axis and area of each minimum oblique rectangle are obtained as the defect information of the defect connected domain. The defect information of the defect connected domain is mapped and associated with each weld defect.
[0074] In the above embodiment, when the weld defect semantic segmentation result of the fusion mask image is instantiated, the actual longest and widest directions of each weld defect are not necessarily in the x or y axis direction of the image. Therefore, the method is to obtain the minimum oblique rectangle (minAreaRect) of the defect closed contour line, rather than the minimum positive rectangle (boundingRect), so as to ensure that the actual maximum length and width of the weld defect are obtained, and the subsequent size calculation is more accurate than the detection accuracy of the target anchor box (which belongs to the minimum positive rectangle) output by the target detection model.
[0075] In the embodiment, if the instantiated result of the weld defect includes any one of incomplete penetration and incomplete fusion, the incomplete penetration and / or incomplete fusion weld defect is calibrated, and the incomplete penetration and incomplete fusion weld defects are calibrated, including detecting the straightness of the upper and lower contour lines of the defect connected domain and detecting the color uniformity of the defect connected domain.
[0076] As shown in FIGS. 1 to 3, the straightness of the upper and lower contour lines of the defect connected domain includes: Figure 8 Figure 9 As shown in FIGS. 1 to 3, the straightness of the upper and lower contour lines of the defect connected domain includes:
[0077] Taking the defect area axis as a starting point, the upper and lower parts are divided, and the vertical distance averages d avg上 and d avg下 of the upper and lower contour lines to the axis are respectively obtained.
[0078] The vertical distance standard deviation between the upper contour line and the axis is obtained, as shown in the following formula (3),
[0079]
[0080] i is the pixel point of the upper contour line;
[0081] The vertical distance standard deviation between the lower contour line and the axis is obtained, as shown in the following formula (4),
[0082]
[0083] j is the pixel point of the lower contour line;
[0084] A threshold thr is set. If △E 上 >thr&&△E 下 <=thr or △E 下 >thr&&△E 上 <=thr, it is considered to meet one of the characteristics of the incomplete fusion defect; if △E 下 <=thr&&△E 上 <=thr, it is considered to meet one of the characteristics of the incomplete penetration defect.
[0085] As shown in FIGS. 1 to 3, the straightness of the upper and lower contour lines of the defect connected domain includes:Figure 10 The color uniformity of the detected defect connected domain includes:
[0086] The detected un-fused / un-welded defect region is converted from RGB to CIELAB space;
[0087] The mean values A 亮度 , A 红绿 , and A 蓝绿 of the entire un-fused / un-welded defect region in the brightness, red-green, and blue-green channels are calculated;
[0088] The entire un-welded / un-fused defect region is divided into small blocks, and the mean values A k亮度 , A k红绿 , and A k蓝绿 of each small block in the brightness, red-green, and blue-green channels are calculated, where k is the serial number of the small block;
[0089] The color difference △E k between the color mean value of the entire defect region and the color mean value of each small block is calculated using the CMC color difference calculation formula;
[0090] The color uniformity of different parts of the defect region is calculated as follows:
[0091]
[0092] The color uniformity of different parts of the defect region is calculated as follows:
[0093] A threshold value tr is set, and when , it is considered that the un-fused color feature is met, otherwise it is considered that the un-welded color feature is met;
[0094] After obtaining the straightness of the upper and lower contour lines of the defect connected domain and the color uniformity of the defect connected domain, if the straightness of the upper and lower contour lines and the color uniformity both determine that the weld defect is of the un-fused type, but the instantiation result of the semantic segmentation neural network model is of the un-welded type, the weld defect is corrected to be of the un-fused type; if the straightness of the upper and lower contour lines and the color uniformity both determine that the weld defect is of the un-welded type, but the instantiation result of the semantic segmentation neural network model is of the un-fused type, the weld defect is corrected to be of the un-welded type.
[0095] In the above embodiments, Figure 6 is a schematic diagram of an un-welded image, Figure 7The image features of the incomplete fusion are mainly thin black lines, which are straight at the upper and lower ends and have uniform blackness. The image features of the incomplete fusion are also thin black lines, but the shapes thereof are mostly crescent-shaped, that is, linear on the side close to the base material and arc-shaped (sometimes dentiform) on the side close to the center of the weld, and the general blackness feature is light and uneven, and the blackness of the center of the defect is generally large and gradually lightens towards the edge. Thus, two indexes are set: 1. straightness of the upper and lower contour lines of the defect area, and 2. color uniformity of the defect area, and the two indexes of the incomplete fusion and incomplete penetration defects are calculated to finely calibrate the defect types.
[0096] In the above embodiment, the crack, tungsten / copper clamping, circular and other defects in various defect types have relatively obvious features on the X-ray image and are relatively easy to distinguish, but the incomplete fusion and incomplete penetration have very similar performances on the X-ray image, and sometimes even the on-site inspectors have difficulty in distinguishing. If only the results of the neural network semantic segmentation are used as the determination results of the incomplete fusion and incomplete penetration types, the misjudgment rate will be high. Therefore, after the instantiation processing of the weld defects, if the defects have incomplete fusion and incomplete penetration instances, the features of the incomplete fusion and incomplete penetration in the defect shape and color can be further combined to perform once more fine measurement calibration, so as to reduce the misjudgment rate.
[0097] Specifically, after the determination results of the straightness of the upper and lower contour lines of the incomplete fusion or incomplete penetration defect connected domain and the color uniformity of the defect connected domain are obtained, if the straightness of the upper and lower contour lines and the color uniformity both determine that the weld defect is of the incomplete fusion type, and the instantiation result of the semantic segmentation neural network model is of the incomplete penetration type, the weld defect is corrected as the incomplete fusion type again, otherwise the instantiation result of the semantic segmentation neural network model is used as the weld defect result output; if the straightness of the upper and lower contour lines and the color uniformity both determine that the weld defect is of the incomplete penetration type, and the instantiation result of the semantic segmentation neural network model is of the incomplete fusion type, the weld defect is corrected as the incomplete penetration type again, otherwise the instantiation result of the semantic segmentation neural network model is used as the weld defect result output.
[0098] In the embodiment, after the determination of all the weld defects is completed, the specific defect detection result is output according to the defect information of the weld defects, and the camera calibration is performed by using the Zhang Zhengyou calibration algorithm:
[0099] A plurality of images of the calibration board placed in various directions are collected;
[0100] The coordinates of a plurality of points on the calibration board image are taken, the corresponding coordinates on the pixel plane are found, and the homography matrix is obtained by solving the linear equations;
[0101] The camera internal parameters are obtained by linear solving according to the camera linear model;
[0102] According to the optimization solution of the nonlinear model, the precise solution of the camera external parameter is obtained;
[0103] According to the calibrated camera internal parameter and the camera external parameter, the actual size of the weld defect on the target weld image corresponding to the world coordinate system is obtained.
[0104] In the above embodiment, the weld defect detection system is to obtain the size, type, position and other information of the actual weld defect, and therefore it is necessary to convert the pixel-level defect into the size corresponding to the world coordinate system, which requires camera calibration in advance. In the present application, the Zhang Zhengyou calibration algorithm with high calibration accuracy is adopted, and according to the calibration parameter, the size of the image pixel point corresponding to the world coordinate system can be obtained. The length, width and area information of the weld defect obtained in the last section can be converted into the actual size of the weld defect.
[0105] In the above embodiment, after obtaining the actual size information of the weld defect, the defect level is evaluated according to the national X-ray detection standard, and the detection result is output, as shown in the attached Figure 11 The weld defect type is classified, and if the weld defect type is a circular defect or a copper / tungsten clamping, the weld defect type is further evaluated as defect level I-IV according to the aspect ratio of the weld defect type; if the weld defect type is incomplete penetration, incomplete fusion or crack, it is evaluated as defect level IV.
[0106] In the present embodiment, the pre-processing of the target weld image includes at least one of image noise reduction, image weld region cropping and image contrast enhancement,
[0107] The image noise reduction excludes the common salt and pepper noise in the weld image by using median filtering;
[0108] The image weld region cropping performs binary segmentation by using OTUS threshold segmentation method; the foreground region after segmentation is processed by eight-neighbor connected domain processing one by one, and only the area information of the largest region is retained; the minimum circumscribed rectangle parallel to the coordinate axis of the region is calculated; the minimum circumscribed rectangle is dilated; the coordinates of the top-left corner and the bottom-right corner of the minimum circumscribed rectangle are calculated; the circumscribed rectangle is extended along the horizontal coordinate axis direction, so that its width is equal to the width of the weld image; the weld image is cropped according to the position of the dilated rectangle region;
[0109] The image contrast enhancement generates an enhanced image by analyzing the scene illumination field T of the target weld image and with the help of the camera response model; and the target weld image and the enhanced image are further fused to obtain an expected image according to a weight matrix.
[0110] In the above embodiment, the X-ray weld seam image (target weld seam image) includes a weld seam area and some artificial lead characters arranged for marking in the periphery, and generally, in order to prevent the lead character part in the periphery from being mistaken as a defect in subsequent detection, positioning and cutting of the weld seam area need to be performed first, and therefore, the image weld seam area cutting is adopted to preprocess the X-ray weld seam image.
[0111] In the above embodiment, due to the influence of external factors, part of the weld seam defect images collected on site has the characteristics of low contrast and blurred edge, and if these images are not subjected to contrast enhancement processing, it is more difficult to obtain accurate detection results subsequently, and the application intends to use the camera response model and Retinex theory to enhance the contrast of the image, and the application of the image contrast enhancement on the X-ray weld seam image, and the specific steps are as follows:
[0112] The expression of the expected image is as follows formula (6),
[0113]
[0114] In the formula, W is a weight matrix, P represents different exposure images, P1 represents the original target weld seam image, and P2 can be considered as an enhanced image, and then two images are fused to generate the final expected image R;
[0115] The enhanced image P2 can be generated through the camera response model, as follows formula (7),
[0116]
[0117] In the formula, g is a brightness transform function (BTF), and k is an exposure rate. The camera response model can be regarded as a gamma model, and since the camera response curve is fixed, the parameters a and b can be obtained by fitting the curve, and then given the input image P1 and the exposure rate k, the enhanced image P2 can be obtained;
[0118] In order to obtain the exposure rate k, the scene illumination field T needs to be obtained first, and first, a brightness map L is initialized, as follows formula (8),
[0119]
[0120] An energy equation is set, and T is obtained by optimizing the energy equation, as follows formula (9),
[0121]
[0122] In the formula, ||*||2 and ||*||1 are L2 and L1 norms, is a first derivative, and lambda is a control parameter;
[0123] Define M as formula (10) below,
[0124]
[0125] Based on the above formula (9) and formula (10), the following energy equation is simplified as formula (11) below,
[0126]
[0127] Thus, the T value corresponding to the minimum value can be obtained;
[0128] The exposure K in formula (7) is inversely proportional to T, that is, formula (12) is obtained,
[0129] K=1. / T formula (12)
[0130] In the formula,. / represents matrix pixel division;
[0131] W=T α Formula (13)
[0132] Alpha in formula (13) is a weight control factor;
[0133] Finally, the original target weld image P1 and the enhanced image P2 are fused to obtain the expected image R, as formula (14) below,
[0134] R=P1*W+P2*(1-W) formula (14).
[0135] In the above embodiment, if the pre-processing includes image noise reduction, image weld area clipping and image contrast enhancement, the image noise reduction is performed first, then the image weld area clipping is performed, and finally the image contrast enhancement is performed.
[0136] Embodiment 2
[0137] Combined with the attached Figure 12 and the attached Figure 13 , the technical scheme of the present application is a weld defect detection system, which comprises a weld defect detection end, a platform user end, a management server and a database;
[0138] The weld defect detection end comprises a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the weld defect detection method in embodiment 1;
[0139] The management server comprises a weld defect image management module, a defect detection result management module, a query module, a user management module and a permission management module;
[0140] The database stores weld defect detection information and images.
[0141] In the above embodiments, the weld defect detection terminal receives the target weld image output by the weld imager; the platform user terminal includes mobile terminals such as mobile phones and PCs; the management server and the database are built in the cloud.
[0142] Appendix Figure 14 This is a sequence diagram showing the upload of weld defect detection results and the download of PDF inspection reports. First, the weld defect detection terminal sends a POST login request via HTTP protocol. After the server verifies the user information and permissions, the detection terminal can upload the weld inspection results to the server and save them in a MySQL database. Then, platform users can log in to the management system via a browser or mobile phone to view defect images and defect detection information online, and can also request to download inspection reports. After receiving the request, the server queries the database to find the corresponding weld image and inspection results, generates a comprehensive PDF inspection report at the business layer, and transmits it to the user terminal through file data stream, so that the user terminal can save it locally for viewing.
[0143] Appendix Figure 15 This is a sequence diagram of the upload and download of weld defect images (when the user is logged in). Image file information, such as film number, corresponding defect detection result number, and generation time, is stored in the database. The file itself is uploaded to a designated location on the server as a data stream via a file upload plugin. If a user wishes to download an inspection image after logging in, they can submit a download request. Upon receiving the POST request, the server first queries the MySQL database to confirm the image information and the image file's location. Then, the control layer locates the corresponding image file based on its location and transmits it to the user's local storage via a file data stream.
[0144] Example 3
[0145] A storage medium storing computer program instructions, which, when executed by a processor, implement the weld defect detection method described in Embodiment 1.
[0146] The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0147] It is also noted that, in this disclosure, relational terms such as first and second, and the like, can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.
[0148] The above description of disclosed embodiments provides enabling concepts for practicing or using the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method of detecting a weld defect, characterized by, The method comprises the following steps: acquiring a target weld seam image, and preprocessing the target weld seam image to obtain a preprocessed target weld seam image; slicing the preprocessed target weld seam image to obtain a slice weld seam image, and inputting the slice weld seam image into a semantic segmentation neural network model to obtain a defect mask image of each slice; the semantic segmentation neural network model comprises a semantic segmentation network and an edge detection subnetwork; the semantic segmentation network comprises four feature layers, each feature layer comprises two semantic segmentation paths, at least two convnet blocks are arranged at the encoding end of the first semantic segmentation path, at least one convnet block is arranged at the decoding end of the first semantic segmentation path, at least two Swin Transformer blocks are arranged at the encoding end of the second semantic segmentation path, and at least one Swin Transformer block is arranged at the decoding end of the second semantic segmentation path; the number of convnet blocks on the first semantic segmentation path in the third feature layer is greater than the number of convnet blocks on the first / second / fourth feature layer, and the number of Swin Transformer blocks on the second semantic segmentation path in the third feature layer is greater than the number of Swin Transformer blocks on the first / second / fourth feature layer; the edge detection subnetwork extracts the boundary features of the slice weld seam image; the output layer of the semantic segmentation network and the boundary features of the edge detection subnetwork are fused to obtain the defect mask image; fuse the defect mask images of each slice to obtain a fusion mask image corresponding to the pixel points in the target weld seam image; instance the semantic segmentation result of the fusion mask image, and the instance result of the weld defect comprises a crack, incomplete penetration, incomplete fusion, a circular defect, and copper / tungsten clamping; perform an opening operation on the fusion mask image through morphological processing; find each defect connected domain which is independent of each other through an eight-neighbor connected algorithm, and mark the pixel points, types and serial numbers thereof; select each defect connected domain, and obtain the contour line of each defect connected domain; according to the contour line of the defect connected domain, obtain the minimum oblique rectangle of each defect connected domain; obtain the length, width, center point, central axis and area of each minimum oblique rectangle as the defect information of the defect connected domain; map and correlate the defect information of the defect connected domain with each weld defect; if the instance result of the weld defect includes any one of incomplete penetration and incomplete fusion, calibrate the incomplete penetration and / or incomplete fusion weld defect, including detecting the straightness of the upper and lower contour lines of the defect connected domain and detecting the color uniformity of the defect connected domain; convert the weld defect information on the target weld seam image into actual weld defect parameters, perform level assessment on the actual weld defect parameters, and output a weld detection result.
2. The method of claim 1, wherein the slicing of the target weld seam image comprises the following steps: The target weld image is sliced according to a preset slice size, and the slice weld image and an adjacent edge of the slice weld image have an overlapping area; Each slice weld image is associated with position information of the slice weld image on the target weld image, and the associated slice weld image is input into a semantic segmentation neural network model for semantic segmentation.
3. A method of detecting a weld defect according to claim 2, wherein An edge detection loss function and a semantic segmentation loss function are set in the semantic segmentation neural network model, wherein the edge detection loss function adopts a cross-entropy loss function, and the expression is L edge = -∑y edgepre1 lgy edgelab1 , wherein y edgepre1 represents a predicted value, and y edgelab1 represents a corresponding true value, Wherein the semantic segmentation loss function adopts the cross-entropy loss and the Dice loss as the semantic segmentation network loss function, and the expression is In the formula, y edgepre2 represents the predicted value, y seglab2 represents the corresponding true value, TP represents the number of pixel points of positive samples identified as positive samples, TN represents the number of pixel points of negative samples predicted as negative samples, FP represents the number of pixel points of negative samples identified as positive samples, and FN represents the number of pixel points of positive samples identified as negative samples.
4. A method of detecting a weld defect according to claim 3, wherein The defect mask images of each slice are fused according to the position information of the defect mask images on the target weld image, and the defect mask images are classified and fused when the defect mask images are restored to the corresponding positions of the target weld image: a. If the defect mask image has no overlapping pixel points with other defect mask images, the segmentation result of the defect mask image is directly corresponded; b. If the defect mask image has overlapping pixel points with other defect mask images, if the segmentation result types of the two defect mask images are the same, the defect type is directly corresponded; c. If the defect mask image has overlapping pixel points with other defect mask images, if the segmentation result types of the two defect mask images are one background point and one defect point, the defect point type is processed; d. If the defect mask image has overlapping pixel points with other defect mask images, if the segmentation result types of the two defect mask images are two different types of defect points, the pixel points in the neighborhood range of the corresponding pixels of the two defect mask images need to be further queried, and the defect type with the most points in the neighborhood range is selected as the final defect type.
5. A method of detecting a weld defect according to claim 4, wherein Detecting straightness of upper and lower contour lines of the defect connected domain includes: With the axis in the defect region as the starting point, divide into upper and lower two parts, respectively calculate the vertical distance average d of the axis to each point on the upper and lower contour lines avg上 and d avg下 ; calculating the standard deviation of the perpendicular distance between the upper contour line and the center axis i is an upper contour line pixel point; calculating the standard deviation of the perpendicular distance between the lower contour line and the center axis j is a pixel point of the lower contour line; Set threshold thr, if meet ΔE 上 > thr && ΔE 下 <= thr or ΔE 下 > thr && ΔE 上 <= thr, can be considered to meet one of the characteristics of the incomplete fusion defects; if meet ΔE 下 <= thr && ΔE 上 <= thr, can be considered to meet one of the characteristics of the incomplete penetration defects; Detecting color uniformity of the defect connected domain includes: Converting the detected incomplete fusion / incomplete penetration defect area from RGB to CIELAB space; Calculate the mean value A of the entire un-fused / un-welded defect area on the brightness, red-green and blue-green three channels 亮度 , A 红绿 , A 蓝绿 ; Divide the whole non-penetration / non-fusion defect area into small blocks, and calculate the mean value A of each small block in the brightness, red-green and blue-green three channels k亮度 , A k红绿 , A k蓝绿 , k is the block number The color difference DE of the color mean value of the entire defect area and the color mean value of each small block is calculated by using the CMC color difference calculation formula k ; The color uniformity of different parts of the defect area is calculated: That is, the color uniformity of different parts of the defect area, the greater the value, the worse the uniformity, the smaller the value, the better the uniformity; A threshold value tr is set, when the un-melted color feature is met, otherwise it is considered to meet the lack of penetration color feature; After obtaining the determination results of the straightness of the upper and lower contour lines of the defect connected domain and the color uniformity of the defect connected domain, if the straightness of the upper and lower contour lines and the color uniformity both determine that the weld defect is of the incomplete fusion type, but the instantiation result of the semantic segmentation neural network model is of the incomplete penetration type, the weld defect is corrected to be of the incomplete fusion type; if the straightness of the upper and lower contour lines and the color uniformity both determine that the weld defect is of the incomplete penetration type, but the instantiation result of the semantic segmentation neural network model is of the incomplete fusion type, the weld defect is corrected to be of the incomplete penetration type.
6. A method of detecting a weld defect according to claim 5, wherein After the determination of all weld defects is completed, a specific defect detection result is output according to the defect information of the weld defects, and Zhang Zhengyou calibration algorithm is used for camera calibration: Multiple images of the calibration board placed in each direction are collected; Multiple point coordinates on the calibration board image are taken, corresponding coordinates on the pixel plane are found, and a homography matrix is obtained by solving linear equations; Linear solving is performed according to the linear model of the camera to obtain the internal parameters of the camera; Optimization solution is performed according to the nonlinear model to obtain the accurate solution of the external parameters of the camera; According to the calibrated internal parameters and external parameters of the camera, the actual size of the weld defect on the target weld image in the world coordinate system is obtained.
7. A method of detecting a weld defect according to claim 6, wherein After the actual size of the weld defect is obtained, the defect level is evaluated according to the national X-ray detection standard, and the detection result is output: the weld defect type is classified, if the weld defect type is a circular defect or copper / tungsten clamping, the weld defect type is further evaluated as defect level I-IV according to the aspect ratio of the weld defect type; if the weld defect type is incomplete penetration, incomplete fusion or crack, it is evaluated as defect level IV.
8. A method of detecting a weld defect according to any one of claims 1 to 7, characterized in that, The preprocessing of the target weld image includes at least one of image noise reduction, image weld region cropping and image contrast enhancement, The image noise reduction excludes common salt and pepper noise in the weld image by using median filtering; The image weld region cropping performs binary segmentation by using OTUS threshold segmentation method; The foreground region after segmentation is processed by eight-neighbor connected domain processing one by one, only the area of the largest region information is retained; the minimum circumscribed rectangle parallel to the coordinate axis of the region is calculated; the minimum circumscribed rectangle is dilated; the coordinates of the top-left corner and the bottom-right corner of the minimum circumscribed rectangle are calculated; the circumscribed rectangle is extended along the horizontal coordinate axis, so that its width is equal to the width of the weld image; the weld image is cropped according to the position of the dilated rectangle region; The image contrast enhancement generates an enhanced image by analyzing the scene illumination field T of the target weld image and with the help of the camera response model, and further fuses the target weld image and the enhanced image according to a weight matrix to obtain an expected image, including: The expression of the desired image is where W is the weight matrix, P represents different exposure images, P1 represents the original target weld seam image, and P2 is an enhanced image, and the two images are fused to generate the final desired image R. The enhanced image P2 is generated by a camera response model, where g is a luminance transfer function, k is the exposure rate, the camera response model is considered as a gamma model, and since the camera response curve is fixed, parameters a and b are obtained by curve fitting. Given the input image P1 and the exposure rate k, the enhanced image P2 can be obtained. To obtain the exposure rate k, the scene illumination field T needs to be obtained first. First, initialize a luminance map L, An energy equation is set up, and T is obtained by optimizing the energy equation, where ||*||2 and ||*||1 are L2 and L1 norms, is the first derivative, and λ is a control parameter; M is defined as Simplify the energy equation as follows, Find the value of T corresponding to the minimum. The exposure K is inversely proportional to T, i.e. k = 1. / T, where. represents matrix element by element division; W = T α where a is a weight control factor; Fusing the original target weld image P1 and the enhanced image P2 can obtain the expected image R, R=P1*W+P2*(1-W).
9. A weld seam defect detection system characterized by, It comprises a weld defect detection end, a platform user end, a management server and a database; The weld defect detection end comprises a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the weld defect detection method in any one of claims 1 to 8; The management server comprises a weld defect image management module, a defect detection result management module, a query module, a user management module and a permission management module; The database stores weld defect detection information and images.
10. A storage medium, characterized by The storage medium stores computer program instructions, and the computer program instructions are executed by the processor to implement the weld defect detection method in any one of claims 1 to 8. The storage medium stores computer program instructions, and the computer program instructions are executed by the processor to implement the weld defect detection method in any one of claims 1 to 8.