Weld joint visual intelligent detection equipment and method

Abstract

The invention relates to the technical field of weld joint detection, and provides a visual intelligent weld joint detection device and method, and the detection method comprises the steps: obtaining a 2D image of a weld joint; weld joint 3D shape data are obtained, and weld joint point cloud data are generated; generating a training data set according to the welding seam 2D image and the welding seam point cloud data; constructing a neural network model, and training the neural network model through the training data set; performing intelligent detection on the weld seam image according to the trained neural network model; wherein a loss function of the neural network model comprises a 2D positioning loss item, a classification task loss item and a 3D frame regression loss item. According to the method, by obtaining the loss function including the 2D positioning loss item, the classification task loss item and the 3D frame regression loss item, multi-task collaborative optimization can be realized, classification robustness, 2D positioning precision and 3D geometric reconstruction capability are considered, and the method is suitable for scene requirements of high precision and multi-modal fusion in industrial detection.

Description

Technical Field

[0001] The present invention relates to the technical field of weld detection, and in particular to a weld visual intelligent detection device and method. Background Art

[0002] Weld visual inspection technology is gradually replacing traditional inspection methods by leveraging its non-contact and high-precision advantages. It is an intelligent, integrated inspection method developed to overcome the limitations of traditional inspection methods, such as low manual visual inspection efficiency, expensive non-destructive testing equipment, complex operation, and poor real-time performance. Weld visual inspection technology relies on optical hardware such as high-resolution visual cameras and laser scanning to acquire weld morphology data. It combines image processing and deep learning algorithms to measure geometric parameters (quantification of geometric features such as weld width, excess height, and undercut) and identify weld defects (classification and location of cracks, lack of penetration, lack of fusion, porosity, and slag inclusions). It also gradually incorporates 3D reconstruction models, multi-sensor fusion technology, and visual fill lights to address dark areas, reflections, and smoke interference in complex industrial environments.

[0003] The vision-based weld recognition method, system, device, and medium disclosed in application number CN202411220801.9 utilizes the Swin Transformer to establish cross-window remote dependencies, extract features from two different modalities: RGB images and depth images, and fully integrate the complementary information of different modalities through multimodal feature fusion and feature enhancement, thereby enhancing and supplementing local details and improving weld recognition performance. However, this vision-based weld recognition method does not take into account classification robustness, 2D positioning accuracy, and 3D geometric reconstruction capabilities. Its weld detection and recognition accuracy still needs to be improved, making it difficult to meet the high-precision, multimodal fusion requirements of industrial inspection scenarios. Summary of the Invention

[0004] Based on this, in order to solve the problem that the existing vision-based weld recognition method does not take into account classification robustness, 2D positioning accuracy and 3D geometric reconstruction capabilities, the weld detection and recognition accuracy still needs to be improved, and it is difficult to meet the high-precision and multi-modal fusion scene requirements in industrial inspection, the present invention provides a weld visual intelligent detection device and method, which uses weld 2D images and weld 3D shape data to generate a training data set to train the neural network model, and obtains a loss function including a 2D positioning loss term, a classification task loss term and a 3D bounding box regression loss term, which can achieve multi-task collaborative optimization, taking into account classification robustness, 2D positioning accuracy and 3D geometric reconstruction capabilities, and is suitable for the high-precision and multi-modal fusion scene requirements in industrial inspection. The specific technical solution is as follows:

[0005] A weld visual intelligent detection device, comprising:

[0006] Vision camera, used to obtain 2D images of welds;

[0007] Laser scanner, used to obtain 3D topography data of welds and generate weld point cloud data;

[0008] The processing center is used to generate a training data set based on the weld 2D image and weld point cloud data, build a neural network model, train the neural network model using the training data set, and perform intelligent detection on the weld image based on the trained neural network model;

[0009] Among them, the loss function of the neural network model includes 2D positioning loss term, classification task loss term and 3D bounding box regression loss term.

[0010] The weld seam visual intelligent inspection equipment can achieve multi-task collaborative optimization by obtaining a loss function including a 2D positioning loss term, a classification task loss term, and a 3D bounding box regression loss term, taking into account classification robustness, 2D positioning accuracy, and 3D geometric reconstruction capabilities. It is suitable for high-precision, multimodal fusion scenarios in industrial inspections.

[0011] Preferably, the detection device further comprises:

[0012] Fill light, configured to be triggered synchronously with the vision camera and laser scanner;

[0013] The environment adaptation module is used to automatically adjust the brightness of the fill light according to the ambient light intensity feedback.

[0014] Preferably, the processing center includes:

[0015] 3D point cloud reconstruction module, used to register multi-view point clouds using an improved ICP algorithm and fuse high-resolution image texture information;

[0016] The feature extraction module is used to project the point cloud along the normal direction of the weld, calculate the maximum width after extracting the cross-sectional contour, and calculate the height difference and edge mutation gradient based on the reference plane fitting.

[0017] Preferably, the processing center further includes:

[0018] The incremental learning module is used to dynamically update the weights of the classifier in the neural network model based on a preset number of new defective weld samples.

[0019] Preferably, the backbone network of the neural network model adopts the GhostNet lightweight structure, and the loss function L=λ1·L CIoU +λ2·L Focal +λ3·L 3D ;

[0020] Among them, L CI oU、LF ocal、L 3D They represent the 2D positioning loss term, the classification task loss term, and the 3D bounding box regression loss term, respectively. λ1, λ2, and λ3 represent the weight coefficients of the 2D positioning loss term, the classification task loss term, and the 3D bounding box regression loss term, respectively.

[0021] Preferably, the 3D bounding box regression loss term

[0022] Among them, ||c pred 、c gt ||2 represents the centroid position loss item, c pred 、c gt Represents the centroid coordinates of the predicted 3D box and the real 3D box, respectively, ||s pred 、s gt ||1 represents the size regression loss item, s pred 、s gt Represents the size of the predicted 3D box and the real 3D box, represents the directional regression loss item, tr(·) represents the trace of the matrix, R pred 、R gt Represents the selection matrix of the predicted 3D box and the real 3D box, λ c ,λ s ,λ θ They represent the weight coefficients of the centroid position loss, size regression loss, and direction regression loss respectively.

[0023] A weld seam visual intelligent detection method, applied to the weld seam visual intelligent detection device, comprises:

[0024] Get 2D image of weld seam;

[0025] Obtain weld 3D topography data and generate weld point cloud data;

[0026] Generate a training data set based on the weld 2D image and weld point cloud data;

[0027] Build a neural network model and train it using a training data set;

[0028] Intelligent detection of weld images based on the trained neural network model;

[0029] Among them, the loss function of the neural network model includes 2D positioning loss term, classification task loss term and 3D bounding box regression loss term.

[0030] Preferably, the detection method further comprises:

[0031] Use the improved ICP algorithm to register multi-view point clouds and fuse high-resolution image texture information;

[0032] The point cloud is projected along the normal direction of the weld, the maximum width is calculated after the cross-sectional contour is extracted, and the height difference and edge mutation gradient are calculated based on the reference plane fitting.

[0033] Preferably, the constructing of the neural network model includes:

[0034] Improved based on YOLOv10, using the GhostNet lightweight structure as the backbone network;

[0035] Construct loss function L = λ1·L CIoU +λ2·L Focal +λ3·L 3D ;

[0036] Among them, L CIoU 、L Focal 、L 3D They represent the 2D positioning loss term, the classification task loss term, and the 3D bounding box regression loss term, respectively. λ1, λ2, and λ3 represent the weight coefficients of the 2D positioning loss term, the classification task loss term, and the 3D bounding box regression loss term, respectively.

[0037] Preferably, the detection method further comprises:

[0038] Get the centroid position loss, size regression loss, and direction regression loss.

[0039] Construct a 3D bounding box regression loss item based on the centroid position loss item, size regression loss item, and direction regression loss item

[0040] Among them, ||c pred 、c gt ||2 represents the centroid position loss item, c pred 、c gt Represents the centroid coordinates of the predicted 3D box and the real 3D box, respectively, ||s pred 、s gt ||1 represents the size regression loss item, s pred 、s gt Represents the size of the predicted 3D box and the real 3D box, represents the directional regression loss item, tr(·) represents the trace of the matrix, R pred 、R gt Represents the selection matrix of the predicted 3D box and the real 3D box, λ c ,λ s ,λ θThey represent the weight coefficients of the centroid position loss, size regression loss, and direction regression loss respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The present invention can be further understood from the following description in conjunction with the accompanying drawings. The components in the figures are not necessarily drawn to scale, but rather the emphasis is placed on illustrating the principles of the embodiments. In different views, the same reference numerals designate corresponding parts.

[0042] Figure 1 This is a schematic diagram of the overall structure of a weld visual intelligent detection device in one embodiment of the present invention;

[0043] Figure 2 Schematic diagram of the structural relationship between the visual camera, fill light and detection object in one embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of the overall process of a weld visual intelligent detection method in one embodiment of the present invention;

[0045] Figure 4 This is a schematic diagram of a process of a weld visual intelligent detection method in another embodiment of the present invention. Figure 1 ;

[0046] Figure 5 This is a schematic diagram of a process of a weld visual intelligent detection method in another embodiment of the present invention. Figure 2 ;

[0047] Figure 6 This is a schematic diagram of weld defect types.

[0048] Description of reference numerals:

[0049] 1. Vision camera; 2. Fill light; 3. Linear module; 4. Telescopic rod. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with its embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0051] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly attached to the other element or there may be an intermediate element. When an element is referred to as being "connected to" another element, it may be directly connected to the other element or there may be an intermediate element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only implementation methods.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used herein in the specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0053] The "first" and "second" in the present invention do not represent specific quantities and orders, but are only used to distinguish names.

[0054] like Figure 1 As shown, a weld visual intelligent detection device includes a visual camera, a laser scanner and a processing center.

[0055] like Figure 2 As shown, the visual camera 1 is used to obtain the 2D image of the weld and feed it back to the processing center. It can be a high-resolution industrial camera (20 million pixels, global shutter), equipped with a polarizing filter and a ring LED strobe fill light 2, and uses strobe synchronization technology to eliminate smoke interference and reflection problems.

[0056] Specifically, the visual cameras include multiple cameras installed around the inspection object to obtain 2D images of the welds of the inspection object at multiple different angles, and at least one visual camera adjusts its relative position relationship with the inspection object through a moving component such as a linear module 3 moving along the X-axis or Y-axis.

[0057] The fill light can be fixedly mounted on the base or housing of the visual camera via a telescopic rod 4 to adjust the relative distance between the fill light and the visual camera, thereby adjusting the ambient light intensity when the visual camera acquires a 2D image of the weld, to cope with factors such as dark areas, reflections, and smoke interference in complex industrial environments.

[0058] The laser scanner is used to acquire 3D weld topography data, generate weld point cloud data based on this data, and feed it back to the processing center. The laser scanner can be a three-dimensional laser scanner with an accuracy of ±0.01mm. By acquiring 3D weld topography data, it generates a high-density point cloud.

[0059] Preferably, the weld visual intelligent detection equipment also includes a multispectral sensor, which obtains multiple spectral band images of the weld through the multispectral sensor, fuses visible light and near-infrared bands, and enhances the contrast of subtle surface defects (such as tiny cracks).

[0060] The processing center is used to receive weld 2D images and weld point cloud data, generate a training data set based on the weld 2D images and weld point cloud data, build a neural network model, train the neural network model through the training data set, and perform intelligent detection of weld images based on the trained neural network model.

[0061] Among them, the loss function of the neural network model includes 2D positioning loss term, classification task loss term and 3D bounding box regression loss term.

[0062] The preprocessing of the training data set (such as cleaning, denoising, enhancement, and labeling, etc.) belongs to conventional technical means in this field and will not be described in detail here.

[0063] The neural network model is preferably a multi-scale feature fusion network. Specifically, the neural network model is improved based on YOLOv10, with the backbone network adopting the GhostNet lightweight structure and introducing a 3D point cloud feature channel.

[0064] GhostNet significantly reduces the number of parameters and computational overhead through its GhostModule and Ghost Bottleneck structures. GhostModule generates basic features through a small number of ordinary convolutions, then expands the number of channels through depthwise separable convolutions, reducing explicit parameters. Ghost Bottleneck employs a "one-to-two" strategy, where input features are processed by the GhostModule and then added to the residual edge, increasing network depth while controlling computational costs. In YOLOv10, GhostNet serves as the backbone network, extracting 2D semantic features (F_2D_RGB) from RGB images. Its lightweight nature makes it suitable for edge computing scenarios in industrial inspection.

[0065] Point cloud data is acquired through a laser scanner and 3D point cloud is generated through the following process:

[0066] 1. Point cloud preprocessing: Use the ICP algorithm to align multi-view point clouds and integrate high-resolution image texture information.

[0067] 2. Projection and encoding: Project the point cloud into a depth map and extract geometric features through 3D CNN or PointNet++.

[0068] 3. Feature alignment: The coordinate systems of the point cloud and the image are aligned through the spatial transformer network (STN) to ensure the spatial consistency of multimodal features.

[0069] The 3D point cloud feature channel can be expressed as in, Represents 2D image features, extracted from RGB images by GhostNet, containing appearance information such as texture and color. Represents 3D point cloud features, which are directly encoded by depth map or point cloud and contain geometric structure (such as weld width, residual height) and spatial relationship information. The fused multimodal features are represented by channel concatenation to preserve the complementary information of 2D texture and 3D geometry, thereby enhancing the detection rate of small defects (such as pores).

[0070] In order to balance the importance of 2D and 3D features, an attention mechanism can be introduced, namely

[0071] where α=σ(W·[F 2D ; F 3D ]), α represents the dynamic weight coefficient, which is generated by the attention mechanism and its value range is limited to [0,1] by the sigmoid function σ. Its core function is to measure the contribution of 2D features in the current context, while (1-α) corresponds to the contribution of 3D features. This nonlinear mapping ensures the flexibility of the fusion process. W is a learnable weight matrix responsible for converting the concatenated feature vector [F 2D ; F 3D ] is mapped to the attention score space, whose dimension is determined by the number of channels of the input feature. For example, if F 2D is C1 dimension, F 3D If the dimension is C2, then the dimension of W is (C1+C2)×1, and the correlation strength between features is optimized through the training process.

[0072] Here, by combining the lightweight characteristics of GhostNet with the geometric prior of 3D point cloud, it solves the problem of multimodal feature fusion in complex industrial environments. In addition, based on the formula α=σ(W·[F 2D ; F 3D ]), the 2D image texture and 3D morphology data can be weightedly fused through the attention mechanism to enhance the detection rate of small defects (such as pores).

[0073] In summary, the weld visual intelligent inspection equipment can achieve multi-task collaborative optimization by obtaining a loss function including 2D positioning loss terms, classification task loss terms, and 3D bounding box regression loss terms, taking into account classification robustness, 2D positioning accuracy, and 3D geometric reconstruction capabilities. It is suitable for high-precision, multimodal fusion scenarios in industrial inspections.

[0074] As a preferred technical solution, the detection device also includes a fill light and an environment adaptation module.

[0075] The fill light is configured to be triggered synchronously with the vision camera and laser scanner to avoid motion blur and smoke dynamic interference during the welding process.

[0076] The environment adaptation module is used to automatically adjust the brightness of the fill light based on ambient light intensity feedback. This allows us to build a dynamic fill light strategy, improve the quality of the collected weld 2D images and weld point cloud data, and thus improve the quality of the training dataset.

[0077] Specifically, the dynamic fill light strategy can be expressed as I LED =k·(I target -I ambient ); where k represents the proportional coefficient, which can be understood as the sensitivity coefficient of ambient light compensation. It is dimensionless and controls the response speed of fill light intensity as the ambient light changes. The larger the value, the more aggressive the compensation. target It represents the target illumination threshold, which can be understood as the preset optimal illumination intensity required for weld detection. The unit is lx. It is set according to the camera imaging quality (such as signal-to-noise ratio, dynamic range) and the reflective characteristics of the weld material. It can be determined in combination with prior experimental data. am bient represents the ambient light intensity feedback value, which can be understood as the ambient light intensity collected by the sensor in real time. The unit is lx. Its function is to reflect the actual light interference on site (such as smoke and dust obstruction, natural light changes). The multispectral sensor can be used to fuse visible light and infrared band data to improve robustness. LED Indicates the target brightness value of the fill light in lumens (lm) or lux (lx). Its function is to ensure stable and uniform lighting conditions in the weld area by adjusting the LED brightness in real time.

[0078] By setting the dynamic fill light strategy and collaborating with environmental perception and hardware, the detection stability in complex industrial scenarios can be significantly improved.

[0079] As a preferred technical solution, the processing center includes a 3D point cloud reconstruction module and a feature extraction module.

[0080] The 3D point cloud reconstruction module is used to register multi-view point clouds using the improved ICP (Iterative Closest Point) algorithm and fuse high-resolution image texture information.

[0081] Among them, R is the rotation matrix, which represents the rigid body rotation in 3D space. It is a 3×3 orthogonal matrix (R T R=I,determinant|R|=1), its function is to transform the source point cloud p iRotate the source point cloud to the target point cloud coordinate system to eliminate angular deviation. t is the translation vector, representing the translation operation in 3D space. It is a 3×1 vector that translates the rotated source point cloud to the target point cloud coordinate system to eliminate positional deviation.

[0082] p i is the i-th point in the source point cloud (point cloud to be registered), with coordinates (x i ,y i ,z i ).q i is the target point cloud with p i The corresponding nearest neighbor points are determined by KD-Tree or brute force search, which aims to establish the spatial correspondence between point clouds by minimizing the sum of the squares of the distances between them.

[0083] ω i is a dynamic weight, which can be understood as a weight factor (ω) dynamically adjusted based on the image edge confidence. i ∈[0,1]), the role is if p i The corresponding high-resolution image area has significant edge features (such as weld edges and texture mutation areas), and a higher ω is assigned. i The algorithm prioritizes key areas by setting a value. n is the number of valid point pairs, which can be understood as the number of point pairs involved in the registration. It is usually determined by filtering (such as voxel downsampling) or removing outliers.

[0084] In the traditional ICP algorithm, the weight of all point pairs is 1, which easily leads to noise sensitivity and local optimality problems. The improved ICP algorithm uses ω i The introduction of image information can achieve feature enhancement (combining image edge detection (such as the Canny operator) and dynamically assigning weights to make the algorithm pay more attention to high-confidence areas (such as weld geometric boundaries)) and optimize anti-interference performance (reducing the weight of non-feature areas (such as flat surfaces) to suppress the influence of noise and redundant data).

[0085] Here, an improved ICP algorithm is used to align multi-view point clouds and integrate high-resolution image texture information. In weld detection, the measurement accuracy of geometric parameters such as width and residual height can be improved by enhancing the weight of the weld edge; under the interference of smoke and reflections, the robustness of the alignment can be improved by suppressing the weight of low-confidence areas.

[0086] That is to say, the improved ICP algorithm solves the problem of insufficient sensitivity of traditional methods to feature areas by fusing image information and dynamically adjusting weights, and achieves significant improvements in accuracy and robustness.

[0087] The feature extraction module is used to project the point cloud along the normal direction of the weld, calculate the maximum width after extracting the cross-sectional contour, and calculate the height difference and edge mutation gradient based on the reference plane fitting.

[0088] Specifically, after extracting the cross-sectional profile, the maximum width W'=max(d(x i ,x j )),x i ,x j ∈S profile Among them, S profile represents the set of two-dimensional cross-sectional contour points obtained by projecting the 3D point cloud along the normal direction of the weld, x i ,x j Represents the coordinates of any two points on the cross-sectional profile, usually expressed in two-dimensional plane coordinates, d(x i ,x j ) represents the Euclidean distance between two points. By traversing all point pairs, the maximum distance is found to be the weld width. W' represents the final measured value of the weld width, which reflects the maximum lateral extension range of the weld.

[0089] Height difference and edge mutation gradient Among them, z plane Represents the reference plane height value fitted by the RANSAC (Random Sample Consensus) algorithm, z i It represents the height coordinate of the i-th point in the point cloud in the vertical direction (Z axis), N represents the number of valid points involved in the calculation, Δh represents the quantitative value of the residual height, and represents the average height difference of the weld surface relative to the reference plane.

[0090] Here, the RANSAC algorithm is used to eliminate noise, allowing the detection equipment to adapt to complex industrial environments such as smoke, dust, and reflections; its multimodal fusion based on line laser scanning and high-resolution camera improves the point cloud density and accuracy.

[0091] Preferably, the processing center also includes an incremental learning module. This module is used to dynamically update the classifier weights in the neural network model based on a preset number of new defective weld samples. This allows the classifier weights to be dynamically updated with a small number of new defective samples to adapt to changes in production line processes.

[0092] As an optimal technical solution, the backbone network of the neural network model adopts the GhostNet lightweight structure, and the loss function L=λ1·L CIoU +λ2·L Focal +λ3·L 3D ;

[0093] Among them, L CIoU 、L Focal 、L 3DThey represent the 2D positioning loss term, the classification task loss term, and the 3D bounding box regression loss term, respectively. λ1, λ2, and λ3 represent the weight coefficients of the 2D positioning loss term, the classification task loss term, and the 3D bounding box regression loss term, respectively.

[0094] Preferably, the 3D bounding box regression loss term

[0095] Among them, ||c pred 、c gt ||2 represents the centroid position loss item, c pred 、c gt Represents the centroid coordinates of the predicted 3D box and the real 3D box, respectively, ||s pred 、s gt ||1 represents the size regression loss item, s pred 、s gt Represents the size of the predicted 3D box and the real 3D box, represents the directional regression loss item, tr(·) represents the trace of the matrix, R pred 、R gt Represents the selection matrix of the predicted 3D box and the real 3D box, λ c ,λ s ,λ θ They represent the weight coefficients of the centroid position loss, size regression loss, and direction regression loss respectively.

[0096] ||||2 represents the L2 norm (Euclidean distance), which is used to quantify the spatial offset error between the predicted center of mass and the true center of mass. It reflects the model's positioning accuracy of the target center. The L2 loss is more sensitive to larger errors and is suitable for precise regression.

[0097] ||||1 represents the L1 norm (sum of absolute errors), which is used to calculate the dimension-by-dimension deviation between the predicted size and the true value. L1 loss is robust to outliers and can effectively reduce model oscillations caused by large errors in individual dimensions, making it suitable for stable size regression.

[0098] The angular deviation is calculated by the similarity of the rotation matrices. Formula It is equivalent to calculating the angular difference (in radians) between two rotation matrices. It directly measures the geometric difference between the predicted direction and the true direction, and is suitable for precise alignment of object orientations in complex scenes.

[0099] The classification task loss item is used to solve the problem of category imbalance and enhance the detection capability of small sample defects (such as cracks and pores). The 2D positioning loss item is used to optimize the position and size accuracy of the target detection box while considering the overlapping area, center point distance and aspect ratio. The 3D bounding box regression loss item is used to accurately regress the center of mass position, size and orientation of the 3D bounding box in the laser scanning point cloud data.

[0100] The weight coefficients λ for the centroid position loss, size regression loss, and direction regression loss c ,λ s ,λ θ Used to balance the contribution ratio of different loss terms to avoid a single task dominating the optimization process. For example: increase λ c When , the centroid positioning accuracy is optimized first; increasing λ θ When , the importance of direction regression is emphasized. In scenes with sparse point clouds, λ can be appropriately reduced. s To reduce the impact of dimensional errors on overall losses.

[0101] In summary, based on the composite loss function L = λ1·L CIoU +λ2·L Focal +λ3·L 3D Through multi-task collaborative optimization, it takes into account classification robustness, 2D positioning accuracy and 3D geometric reconstruction capabilities, and is suitable for high-precision and multi-modal fusion scenario requirements in industrial inspection.

[0102] An embodiment of the present invention further provides a weld seam visual intelligent detection method, which is applied to the weld seam visual intelligent detection device, such as Figure 3 As shown, it includes the following steps:

[0103] S1: Acquire a 2D image of the weld. Specifically, a high-resolution industrial camera (e.g., 20 megapixels, global shutter) can be used to capture the weld 2D image. This can be combined with a polarizing filter and a ring-shaped LED strobe light, using strobe synchronization technology to eliminate smoke interference and reflections.

[0104] S2, obtaining 3D weld morphology data and generating weld point cloud data. Specifically, the 3D weld morphology data can be obtained by a 3D laser scanner to generate a high-density point cloud.

[0105] S3, generating a training data set based on the weld 2D image and weld point cloud data.

[0106] S4, building a neural network model, and training the neural network model using a training data set.

[0107] Specifically, the construction of the neural network model includes the following steps: based on the improvement of YOLOv10, the GhostNet lightweight structure is used as the backbone network, and the 3D point cloud feature channel is introduced. in, Represents 2D image features, extracted from RGB images by GhostNet, containing appearance information such as texture and color. Represents 3D point cloud features, which are directly encoded by depth map or point cloud and contain geometric structure (such as weld width, residual height) and spatial relationship information. The fused multimodal features are represented by channel concatenation to preserve the complementary information of 2D texture and 3D geometry, thereby enhancing the detection rate of small defects (such as pores).

[0108] After constructing the loss function, the neural network model is trained using the training data set until the set accuracy or the maximum number of iterations is reached.

[0109] S5, intelligently inspect the weld image based on the trained neural network model. Specifically, the trained neural network image is used to identify the acquired weld 2D image and weld 3D topography data, and detect defect information (including type and quantity) of the sample to be inspected.

[0110] Among them, the loss function of the neural network model includes 2D positioning loss term, classification task loss term and 3D bounding box regression loss term. It should be noted that the training data set is generated based on the weld 2D image and weld point cloud data. Here, in addition to obtaining the weld 2D images and weld 3D topography data of multiple inspection objects through the visual camera and 3D laser scanner connected to the processing center signal, and generating the training data set based on the obtained weld 2D images and weld 3D topography data, it is also possible to use big data technology to obtain a large amount of relevant and rich weld 2D images and weld 3D topography data through the factory database of the corresponding inspection products, and then generate the training data set.

[0111] The inspection object can be understood as multiple products associated with the sample to be inspected, including standard products, good products and defective products. There are various types of weld defects on defective products, such as Figure 6 As shown, weld defects include but are not limited to pores, incomplete penetration and cracks. The sample to be tested can be understood as the target test product.

[0112] Preferably, the loss function L = λ1·L CIoU +λ2·L Focal +λ3·L 3D Among them, L CIoU 、L Focal 、L 3DThey represent the 2D positioning loss term, the classification task loss term, and the 3D bounding box regression loss term, respectively. λ1, λ2, and λ3 represent the weight coefficients of the 2D positioning loss term, the classification task loss term, and the 3D bounding box regression loss term, respectively.

[0113] Classification task loss term L Focal = -α(1-p) γ lg(p) is used to address class imbalance and enhance the detection of small sample defects (such as cracks and pores). Here, p represents the confidence level of the model in predicting the presence of a defect (in the range [0, 1]). α is a class weight parameter, which is used to reduce the loss contribution of the majority class (such as normal welds) and increase the weight of the minority class (such as pores). γ is a focusing parameter (usually set to γ greater than or equal to 2), which is used to suppress the gradient contribution of simple samples (high-confidence predictions) and make the model focus more on difficult samples.

[0114] 2D positioning loss It is used to optimize the position and size accuracy of the target detection frame, while considering the overlapping area, center point distance and aspect ratio; IoU represents the intersection over union ratio of the predicted box and the real box, which is used to measure the accuracy of the overlapping area, ρ represents the Euclidean distance between the center point of the predicted box and the real box, c represents the diagonal length of the minimum closed area surrounding the two boxes, and v represents the aspect ratio consistency parameter, which is calculated as Used to constrain the aspect ratio error.

[0115] w gt 、h gt Represents the width and height of the real box, w pred 、h pred They represent the width and height of the prediction box respectively, and β represents the weight coefficient, which is used to balance the weight parameter of the aspect ratio consistency parameter.

[0116] The weight coefficients for the 2D positioning loss, classification loss, and 3D bounding box regression loss are used to balance the contribution of different tasks, preventing a single task from dominating the optimization process. In weld inspection, if positioning accuracy is a priority, λ1 > λ2 > λ3 can be set. Furthermore, weights can be adjusted based on the task stage. For example, increasing the weight coefficient for the classification loss in the early stages of training to stabilize classification, and increasing the weight coefficient for the 3D bounding box regression loss in the later stages to optimize geometric parameters, can be used.

[0117] Typically, λ1+λ2+λ3=1 to prevent gradient explosion.

[0118] Preferably, if Figure 4 Said detection method further comprises:

[0119] S6, obtains the centroid position loss item, the size regression loss item, and the direction regression loss item.

[0120] S7, constructs the 3D bounding box regression loss item based on the centroid position loss item, size regression loss item, and direction regression loss item

[0121] Among them, ||c pred 、c gt ||2 represents the centroid position loss item, c pred 、c gt Represents the centroid coordinates of the predicted 3D box and the real 3D box, respectively, ||s pred 、s gt ||1 represents the size regression loss item, s pred 、s gt Represents the size of the predicted 3D box and the real 3D box, represents the directional regression loss item, tr(·) represents the trace of the matrix, R pred 、R gt Represents the selection matrix of the predicted 3D box and the real 3D box, λ c ,λ s ,λ θ They represent the weight coefficients of the centroid position loss, size regression loss, and direction regression loss respectively.

[0122] The weight coefficients λ for the centroid position loss, size regression loss, and direction regression loss c ,λ s ,λ θ Used to balance the contribution ratio of different loss terms to avoid a single task dominating the optimization process. For example: increase λ c When , give priority to optimizing the center of mass positioning accuracy; increase λ θ When , the importance of direction regression is emphasized. In scenes with sparse point clouds, λ can be appropriately reduced. s To reduce the impact of dimensional errors on overall losses.

[0123] Preferably, the detection method further includes constructing a virtual welding environment (such as implemented by the Unity3D engine) to simulate different lighting, smoke and material reflection scenes to accelerate the iteration of the neural network training process, improve training efficiency and recognition accuracy.

[0124] Based on the detection method and the traditional YOLOV5 neural network model, the weld seam intelligent detection of the sample to be tested is performed, and the results are shown in the following table:

[0125] index Methods Traditional method (YOLOv5) Defect detection rate (mAP) 98.2％ 92.5％ Geometric parameter error ≤0.05mm ≤0.15mm Single frame processing time 120ms 250ms Success rate of anti-smoke interference 95％ 70％

[0126] It can be seen from the above table that the detection method described in this application outperforms the traditional YOLOV5 neural network model in terms of defect detection rate, geometric parameter error, and anti-smoke interference success rate.

[0127] In summary, the weld seam visual intelligent inspection method can achieve multi-task collaborative optimization by obtaining a loss function including 2D positioning loss term, classification task loss term and 3D bounding box regression loss term, taking into account classification robustness, 2D positioning accuracy and 3D geometric reconstruction capability, and is suitable for high-precision and multimodal fusion scenario requirements in industrial inspection.

[0128] As a preferred technical solution, Figure 5 As shown, the detection method further includes:

[0129] S8, uses the improved ICP algorithm to register multi-view point clouds and fuse high-resolution image texture information, specifically

[0130] Among them, R is the rotation matrix, t is the translation vector, and p i is the i-th point in the source point cloud (point cloud to be registered), with coordinates (x i ,y i ,z i ).q i is the target point cloud with p i The corresponding nearest neighbor point is determined by KD-Tree or brute force search, ω i is a dynamic weight.

[0131] In the traditional ICP algorithm, the weight of all point pairs is 1, which easily leads to noise sensitivity and local optimality problems. The improved ICP algorithm uses ω i The introduction of image information can achieve feature enhancement (combining image edge detection (such as the Canny operator) and dynamically assigning weights to make the algorithm pay more attention to high-confidence areas (such as weld geometric boundaries)) and optimize anti-interference performance (reducing the weight of non-feature areas (such as flat surfaces) to suppress the influence of noise and redundant data).

[0132] Here, an improved ICP algorithm is used to align multi-view point clouds and integrate high-resolution image texture information. In weld detection, the measurement accuracy of geometric parameters such as width and residual height can be improved by enhancing the weight of the weld edge; under the interference of smoke and reflections, the robustness of the alignment can be improved by suppressing the weight of low-confidence areas.

[0133] That is to say, the improved ICP algorithm solves the problem of insufficient sensitivity of traditional methods to feature areas by fusing image information and dynamically adjusting weights, and achieves significant improvements in accuracy and robustness.

[0134] S9, project the point cloud along the normal direction of the weld, extract the cross-sectional contour, calculate the maximum width, and calculate the height difference and edge mutation gradient based on the reference plane fitting.

[0135] Specifically, after extracting the cross-sectional profile, the maximum width W'=max(d(x i ,x j )),x i ,x j ∈S profile Among them, S profile represents the set of two-dimensional cross-sectional contour points obtained by projecting the 3D point cloud along the normal direction of the weld, x i ,x j Represents the coordinates of any two points on the cross-sectional profile, d(x i ,x j ) represents the Euclidean distance between two points, and W' represents the final measured value of the weld width, which reflects the maximum lateral extension of the weld.

[0136] Height difference and edge mutation gradient Among them, z plane Represents the reference plane height value fitted by the RANSAC (Random Sample Consensus) algorithm, z i It represents the height coordinate of the i-th point in the point cloud in the vertical direction (Z axis), N represents the number of valid points involved in the calculation, Δh represents the quantitative value of the residual height, and represents the average height difference of the weld surface relative to the reference plane.

[0137] Here, the RANSAC algorithm is used to eliminate noise, allowing the detection equipment to adapt to complex industrial environments such as smoke, dust, and reflections; its multimodal fusion based on line laser scanning and high-resolution camera improves the point cloud density and accuracy.

[0138] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0139] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A weld seam visual intelligent detection device, characterized in that: include: Vision camera, used to obtain 2D images of welds; Laser scanner, used to obtain 3D topography data of welds and generate weld point cloud data; The processing center is used to generate a training data set based on the weld 2D image and weld point cloud data, build a neural network model, train the neural network model using the training data set, and perform intelligent detection on the weld image based on the trained neural network model; Among them, the loss function of the neural network model includes 2D positioning loss term, classification task loss term and 3D bounding box regression loss term.

2. The weld seam visual intelligent detection device according to claim 1, characterized in that: The detection device also includes: Fill light, configured to be triggered synchronously with the vision camera and laser scanner; The environment adaptation module is used to automatically adjust the brightness of the fill light according to the ambient light intensity feedback.

3. The weld seam visual intelligent detection device according to claim 2, characterized in that: The processing center includes: 3D point cloud reconstruction module, used to register multi-view point clouds using an improved ICP algorithm and fuse high-resolution image texture information; The feature extraction module is used to project the point cloud along the normal direction of the weld, calculate the maximum width after extracting the cross-sectional contour, and calculate the height difference and edge mutation gradient based on the reference plane fitting.

4. The weld seam visual intelligent detection device according to claim 3, characterized in that: The processing center also includes: The incremental learning module is used to dynamically update the weights of the classifier in the neural network model based on a preset number of new defective weld samples.

5. The intelligent visual inspection device for welds according to claim 4, characterized in that: The backbone network of the neural network model adopts the GhostNet lightweight structure, and the loss function L = λ1·L CIoU +λ2·L Focal +λ3·L 3D ; Among them, L CIoU 、L Focal 、L 3D They represent the 2D positioning loss term, the classification task loss term, and the 3D bounding box regression loss term, respectively. λ1, λ2, and λ3 represent the weight coefficients of the 2D positioning loss term, the classification task loss term, and the 3D bounding box regression loss term, respectively.

6. The weld seam visual intelligent detection device according to claim 5, characterized in that: The 3D bounding box regression loss term Among them, ||c pred 、c gt ||2 represents the centroid position loss item, c pred 、c gt Represents the centroid coordinates of the predicted 3D box and the real 3D box, respectively, ||s pred 、s gt ||1 represents the size regression loss item, s pred 、s gt Represents the size of the predicted 3D box and the real 3D box, represents the directional regression loss item, tr(·) represents the trace of the matrix, R pred 、R gt Represents the selection matrix of the predicted 3D box and the real 3D box, λ c ,λ s ,λ θ They represent the weight coefficients of the centroid position loss, size regression loss, and direction regression loss respectively.

7. A weld seam visual intelligent detection method, applied to the weld seam visual intelligent detection device according to any one of claims 1 to 6, characterized in that: The detection method comprises: Get 2D image of weld seam; Obtain weld 3D topography data and generate weld point cloud data; Generate a training data set based on the weld 2D image and weld point cloud data; Build a neural network model and train it using a training data set; Intelligent detection of weld images based on the trained neural network model; Among them, the loss function of the neural network model includes 2D positioning loss term, classification task loss term and 3D bounding box regression loss term.

8. A weld seam visual intelligent detection method according to claim 7, characterized in that: The detection method further comprises: Use the improved ICP algorithm to register multi-view point clouds and fuse high-resolution image texture information; The point cloud is projected along the normal direction of the weld, the maximum width is calculated after the cross-sectional contour is extracted, and the height difference and edge mutation gradient are calculated based on the reference plane fitting.

9. A weld seam visual intelligent detection method according to claim 8, characterized in that: The construction of the neural network model includes: Improved based on YOLOv10, using the GhostNet lightweight structure as the backbone network; Construct loss function L = λ1·L CIoU +λ2·L Focal +λ3·L 3D ; Among them, L CIoU 、L Focal 、L 3D They represent the 2D positioning loss term, the classification task loss term, and the 3D bounding box regression loss term, respectively. λ1, λ2, and λ3 represent the weight coefficients of the 2D positioning loss term, the classification task loss term, and the 3D bounding box regression loss term, respectively.

10. A weld seam visual intelligent detection method according to claim 9, characterized in that: The detection method further comprises: Get the centroid position loss, size regression loss, and direction regression loss. Construct a 3D bounding box regression loss item based on the centroid position loss item, size regression loss item, and direction regression loss item Among them, ||c pred 、c gt ||2 represents the centroid position loss item, c pred 、c gt Represents the centroid coordinates of the predicted 3D box and the real 3D box, respectively, ||s pred 、s gt ||1 represents the size regression loss item, s pred 、s gt Represents the size of the predicted 3D box and the real 3D box, represents the directional regression loss item, tr(·) represents the trace of the matrix, R pred 、R gt Represents the selection matrix of the predicted 3D box and the real 3D box, λ c ,λ s ,λ θ They represent the weight coefficients of the centroid position loss, size regression loss, and direction regression loss respectively.

Images