Weld joint positioning and defect detection system and method based on deep learning

Abstract

According to the weld joint positioning and defect detection system and method based on deep learning, the target detection pyramid network structure suitable for different image sizes is applied to weld joint detection, and the adaptability of intelligent weld joint detection to the image sizes is improved; meanwhile, a defect labeling mode of a free rectangular frame is selected, so that the identification accuracy of positioning is improved; welding seam positioning and defect identification are carried out by adopting a deep learning method based on an FPN network structure through marking defects of welding seams, air holes, welding beadings and pits in an image data set, and training of target detection and semantic segmentation functions is completed; according to the method, model compression is carried out by adopting double-correlation knowledge distillation, correlation between spatial positions and correlation between channels in a feature map are considered at the same time, and two types of correlation information are migrated from a complex model (teacher model) to a compact model (student model), so that compression of the model is realized, and balance between segmentation precision and reasoning speed is achieved.

Description

Technical Field

[0001] The present invention relates to the field of weld positioning and detection, and in particular to a weld positioning and defect detection system and method based on deep learning. Background Art

[0002] As the "tailor" of industry, welding technology has been applied in aerospace, machinery manufacturing, construction engineering, transportation, petrochemical, microelectronics and other industrial manufacturing fields. According to statistics from relevant departments, more than 45% of the annual output of steel and metal materials at home and abroad are processed into products through welding technology.

[0003] The welding quality of the weld has a decisive influence on the product quality. In the actual metal welding process, various defects such as cracks, lack of fusion or incomplete penetration, spatter, weld bead, undercut, porosity and other defects will occur due to factors such as materials and processes. The welding defects on the weld will seriously affect the quality and performance of industrial products while reducing the appearance. The welding quality directly determines the performance of metal parts and thus affects the service life of industrial products.

[0004] At present, the identification and post-processing of weld defects are mostly done by manpower. On the one hand, the manufacturing efficiency is low and the quality is unstable. On the other hand, the working environment is relatively bad, which affects the physical and mental health of workers. Therefore, it is particularly important to conduct fast and accurate defect detection of welds. By inspecting the quality of welds in welded structures, it can effectively prevent defective products from entering the market, and provide protection for the economy and safety from the root. After entering the 21st century, computer technology has been widely popularized, which has promoted the research progress in the field of deep learning at home and abroad. Image detection technology based on machine vision has become a new direction, which can improve the quality of weld image detection in a more efficient, more accurate and more intelligent way. Deep learning is a branch with great development potential in the field of machine vision. It extracts image features and processes them through training models, and then completes target classification;

[0005] Weld defects occupy a small proportion of pixels in the image and have less feature information. The detection of weld defects belongs to the small target detection task. The existing deep learning models mostly extract weld features through a hierarchical structure, which makes it difficult to take into account the feature map size and feature information volume. The effect of direct application in the field of weld defect detection is not ideal. Therefore, how to better apply deep learning to vision-based weld positioning and defect detection has become a key issue in weld defect detection. Summary of the invention

[0006] In view of the above situation, in order to overcome the defects of the prior art, the present invention provides a weld positioning and defect detection method based on deep learning, which effectively solves the problem that the existing deep learning model is difficult to take into account the feature map size and feature information amount for weld detection, and the effect of direct application in the field of weld defect detection is not ideal.

[0007] The technical solution of the present invention is as follows:

[0008] The weld positioning and defect detection system based on deep learning includes a lower computer, a data transmission layer, and a host computer. The lower computer includes a monocular camera, a binocular camera, a PLC, a robot motion control card, and an industrial robot. The data transmission layer is a switch. The host computer is an industrial computer and a keyboard, a mouse, a touch screen, and a display. The host computer is responsible for human-computer interaction and weld image processing, and controls the communication with the lower computer through the switch. The PLC in the lower computer receives the host computer instruction and transmits it to the robot motion control card, or the host computer transmits it to the robot motion control card through the switch. The robot motion control card realizes industrial robot welding by controlling the motion axis. At the same time, the PLC triggers the monocular camera and the binocular camera at the same time to collect the weld surface image.

[0009] The weld positioning and defect detection method based on deep learning includes the following steps:

[0010] Step 1: Weld image acquisition and position calibration based on line structured light;

[0011] Step 2: Acquisition of high-density point cloud images and generation of depth images of the weld surface based on surface structured light. Specifically, the Z coordinate of the height information of the high-density point cloud is converted into the image pixel grayscale, and the X and Y coordinates of the spatial information of the point cloud are converted into pixel indexes. The high-density point cloud image preprocessing includes abnormal point removal, point cloud filtering, point cloud data calibration, generation of a two-dimensional grid matrix and grayscale conversion.

[0012] Step 3: Based on deep learning, weld defect recognition and positioning, the fully convolutional neural network FCN is used to achieve pixel positioning segmentation of the target, that is, semantic segmentation. The semantic segmentation task can classify each pixel when searching for objects in the image, and mark pixels representing features such as color and texture as the same category. The result of semantic segmentation is to output an image. The pixel value represents the classification result of the corresponding image in the input image. The size of the input and output images is the same. The network can automatically learn useful features from the image data, complete image annotation with the help of annotation software, interactively mark pixels and output labeled data to train the neural network;

[0013] Step 4: Model compression based on dual-correlation knowledge distillation;

[0014] Step 5: Identification of pits and porosity defects in welds, weld bumps and unqualified defects.

[0015] The present invention has the following beneficial effects: 1) In view of the problem that the pixels of welds and weld defects account for a very small proportion in the image, while the working image taken by a high-resolution camera contains a very large amount of information, a pyramid network structure FPN is innovatively adopted to solve the problem of target detection of different image sizes, that is, the high-dimensional features and low-dimensional features of the convolutional neural network are integrated to make the semantics contained in the network richer, which is conducive to the detection and recognition of small targets.

[0016] 2) Models with high segmentation accuracy are often complex in structure and have a huge number of parameters, which means that the model training and reasoning processes require more powerful hardware resources and are more time-consuming. Currently, the semantic segmentation models that perform at the top in terms of accuracy usually have extremely high requirements for computing resources. When detecting and locating weld defects, the use of high-precision models will be greatly limited and face the contradiction between segmentation accuracy and reasoning speed. To address the above issues, we creatively use dual-correlation knowledge distillation for model compression to achieve a balance between segmentation accuracy and reasoning speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The hardware structure block diagram of the system of the present invention;

[0018] Figure 2 This is a flow chart of line structured light image acquisition and shape dimension detection of the method of the present invention;

[0019] Figure 3 This is a flow chart of high-density point cloud image acquisition and surface defect detection in the method of the present invention;

[0020] Figure 4 The overall process of weld positioning and defect identification of the method of the present invention is as follows;

[0021] Figure 5 It is a flow chart of the overall steps of the present invention. Figure 6 The process flow of the image volume extraction step of the present invention; Figure 7 This is a schematic diagram of X / pixel of the method of the present invention; Figure 8 This is a schematic diagram of the welding motherboard fitting of the present invention; Fig. 9 This is a flow chart of outlier removal in the present invention; Fig.10 The surface morphology of welds with different defect types and the corresponding depth map of the present invention; Fig.11 This is the head network diagram of the present invention; Fig.12 Annotate the training flow chart for the present invention; Fig.13It is a free rectangular rectangular marking diagram of the present invention; Fig.14 This is a network structure diagram of the present invention; Fig.15 It is the overall framework diagram of the double-correlation distillation method of the present invention. DETAILED DESCRIPTION

[0022] For the above and other technical contents, features and effects of the present invention, please refer to the following attached Figures 1 to 15 The detailed description of the embodiments will clearly show that the structural contents mentioned in the following embodiments are all based on the drawings in the specification.

[0023] Exemplary embodiments of the present invention will be described below with reference to the accompanying drawings.

[0024] Embodiment 1, a weld positioning and defect detection system based on deep learning, such as Figure 1 As shown, it includes a lower computer, a data transmission layer, and an upper computer. The lower computer includes a monocular camera, a binocular camera, a PLC, a robot motion control card, and an industrial robot. The data transmission layer is a switch. The upper computer is an industrial computer and a keyboard, a mouse, a touch screen, and a display. It is characterized in that the upper computer is responsible for human-computer interaction and weld image processing, and controls the communication with the lower computer through the switch. The PLC in the lower computer receives the upper computer instruction and transmits it to the robot motion control card, or the upper computer transmits it to the robot motion control card through the switch. The motion control card has a professional communication interface. The robot motion control card realizes industrial robot welding by controlling the motion axis. At the same time, the PLC triggers the monocular camera and the binocular camera at the same time to collect the weld surface image. The monocular camera and the binocular camera are equipped with a special LED light source for fill light. The operation process is as follows: (1) Equipment initialization: Connect and initialize the monocular camera and binocular camera, industrial robot welding equipment, and test the communication stability between various hardware devices (2) Detection parameter setting: According to the welding requirements, import the current welding parameters, such as welding speed, etc. (3) Weld dimension detection: After the welding workpiece is clamped, the host computer sends a command to the industrial robot to start welding, and at the same time triggers the camera to start taking pictures, collect weld images, use the weld dimension detection algorithm to extract and detect weld feature points, calculate the weld geometry, and store the measurement results (4) Weld surface defect detection: Trigger the monocular camera and the binocular camera at the same time, collect weld surface images, use the defect detection deep learning algorithm to detect and locate weld defects from the depth image, if a defect is detected, process the point cloud data of the defect location to calculate the defect size, and store the detection results.

[0025] Embodiment 2, a weld positioning and defect detection method based on deep learning, comprises the following steps;

[0026] Step 1: Weld image acquisition and position calibration based on line structured light. Detection principle: First, project the line structured light onto the surface of the weld to be tested. According to the changes in the weld surface morphology, the line structured light is modulated into light stripes of different shapes. Then, the light stripe image is captured by an industrial camera to obtain the weld surface contour. The pixel coordinates of the stripe feature points can be extracted according to the stripe shape in the weld line structured light stripe image. Then, according to the calibration results of the line structured light camera, the pixel coordinates of the feature points can be converted into three-dimensional coordinates in the world coordinate system, thereby realizing the detection of weld dimensions such as weld width, excess height, misalignment, and joint angle.

[0027] Detection method: Based on the line structured light camera SDK, the camera status is read and the camera parameters are set. The PLC enables and triggers the camera acquisition through the IO control module. After obtaining the weld line structured light stripe image, the image is preprocessed, and then the center line and feature points of the line structured light stripe are extracted to complete the weld size detection. The processing result output is completed through the human-computer interaction and visualization module, and the size detection data is saved through the data management module. The complete process of weld size detection is as follows: Figure 2 As shown;

[0028] Step 2: Acquisition of high-density point cloud images and generation of depth images of the weld surface based on surface structured light.

[0029] Working principle: Use a binocular 3D depth camera to shoot the weld surface to obtain high-precision weld surface morphology, that is, high-density point cloud data of the weld surface. First, project the modulated structured light image onto the surface of the object to be tested, and then calculate the parallax between the original projected image and the observed image after deformation of the surface of the object to be tested, and finally obtain high-density point cloud data of the surface of the object to be tested;

[0030] Workflow: Such as Figure 3 As shown, based on the binocular structured light depth camera SDK, the camera status is read and the camera parameters are set. The camera is enabled and triggered through the IO control module of the PLC to obtain the high-density point cloud on the weld surface. Then, the point cloud data is preprocessed by removing noise, filling gaps, filtering, etc., and then the point cloud image is gray-scale converted into a grayscale image, i.e., a depth image. Specifically, the Z coordinate of the height information of the high-density point cloud is converted into the image pixel grayscale, and the X and Y coordinates of the spatial information of the point cloud are converted into pixel indexes. The preprocessing of the high-density point cloud image includes abnormal point removal, point cloud filtering, point cloud data calibration, generation of a two-dimensional grid matrix, and grayscale conversion.

[0031] Step 3: Weld defect recognition and positioning based on deep learning. The grayscale image is input into the FPN deep learning model to detect weld surface defects. If defects exist, the defect category and defect location are output. Finally, the point cloud data and depth image of the defect area are extracted according to the category and location of the defect to measure the size of the defect. Specifically, the fully convolutional neural network FCN is used to achieve pixel positioning and segmentation of the target, that is, semantic segmentation. The semantic segmentation task can classify each pixel when searching for objects in the image, and mark the pixels representing features such as color and texture as the same category. The structure of the network is as follows: Fig.14 As shown in the figure, the result of semantic segmentation is to output an image. The pixel value represents the classification result of the corresponding image in the input image. The size of the input and output images is the same. The network can automatically learn useful features from the image data. With the help of annotation software, image annotation is completed. The pixels are interactively labeled and the labeled data is output to train the neural network. The biggest feature of semantic segmentation image annotation is the extremely high annotation accuracy. It can achieve pixel-level annotation, which can fundamentally solve the problems of edge not fitting the target and too much background information.

[0032] Step 4: Model compression is achieved based on dual-association knowledge distillation. An effective way to find a balance between segmentation accuracy and inference speed is model compression. Knowledge distillation is a commonly used model compression method. After using knowledge distillation technology to improve its segmentation accuracy, a trade-off between segmentation accuracy and inference speed is also achieved on the compressed model.

[0033] Step 5: Identification of pits and pores in welds, weld bumps and unqualified defects, such as Figure 4 As shown in the figure, after collecting the workpiece image, the weld is located by line structured light, the depth image of the weld is obtained by a binocular structured light camera, and the pits and pores are extracted from the located weld with the help of semantic segmentation method. The other part of the defect location and recognition task is completed by target detection method (the target detection workflow mainly includes three parts: data set preparation, training, and reasoning. Data set preparation includes image collection and image annotation. Training includes selecting a framework and creating a model. Training the model based on a specific task and evaluating the performance of the trained model. Reasoning refers to applying the trained model to a new image set, using FPN). This step locates and identifies weld nodules and seriously unqualified welds.

[0034] Embodiment 3, based on Embodiment 2: Step 1 specifically comprises the following steps:

[0035] Step 1A: Construction of mathematical model of line structured light camera: Line structured light camera consists of a line laser emitter and an industrial camera. The line laser emitter emits a straight line of laser light to form a laser light plane. The light plane intersects with the surface of the object to be measured to form a light strip on the surface of the object to be measured. The position of the light strip in the world coordinate system can be obtained according to the imaging position of the light strip in the camera and the principle of triangulation. The relationship between the world coordinate system, the camera coordinate system and the image coordinate system is constructed according to the ideal pinhole imaging camera model, such as Figure 5 As shown in the figure, let point Q be a point on the laser light strip on the surface of the object to be measured, and let this point be in the world coordinate system Ow-X w Y w Z w The coordinates under are Q(X w ,Y w ,Z w ), its coordinates in the pixel coordinate system are Q′(u,v), and its coordinate transformation process mainly includes the following steps: 1. Conversion from world coordinates to camera coordinates 2. Conversion from camera coordinates to image coordinates 3. Conversion from image coordinates to pixel coordinates;

[0036] Step 1B: Extraction of sub-pixel coordinates of the center line of the line structured light stripe: including image preprocessing and extraction of sub-pixel coordinates of the center line of the light stripe, wherein the image preprocessing includes five steps: image filtering, image segmentation, morphological processing, width filtering and region of interest (ROI) extraction;

[0037] The sub-pixel coordinates of the center line of the light stripe are extracted. After image preprocessing, the line structured light stripe is still larger than one pixel width. In order to improve the accuracy of weld size detection, the center line of the line structured light stripe needs to be extracted to obtain the coordinates of the center point of the line structured light stripe expressed in sub-pixel coordinates;

[0038] Step 1C: Line structured light stripe feature point extraction and weld size detection: including weld feature point recognition and weld size detection;

[0039] The weld feature point recognition, such as Figure 7 As shown, the least squares method is first used to fit the center line of the line structured light stripe, as shown in Figure 8 As shown in the figure, after the welding mother plate is fitted into a straight line, in order to realize the detection of the geometric dimensions of the weld shape, it is necessary to extract the characteristic points related to the weld size calculation from the center line of the structured light stripe, and then use the threshold method to roughly extract the weld characteristic points, and finally use the linear interpolation method to accurately extract the weld characteristic points;

[0040] After extracting the weld feature points, the weld size detection obtains the weld width, excess height, misalignment, joint angle and other external dimensions according to the calibration results of the line structured light camera and the definition of the weld external geometric dimensions, and records and displays the weld geometric dimensions in real time.

[0041] Embodiment 4, based on embodiment 2: step 2 specifically comprises the following steps:

[0042] Step 2A: Obtain the preprocessed point cloud data and depth image, reconstruct the defect morphology in three dimensions and measure the defect size, convert the high-density point cloud data into a depth image for preprocessing the point cloud data, that is, convert the high-density point cloud data into a grayscale image with unchanged relative position and encoded by height, namely, a depth image. In the weld depth image, the height information (Z coordinate) of the high-density point cloud is converted into image pixel grayscale, and the spatial information (X and Y coordinates) of the point cloud is converted into pixel index. The converted weld depth image can reduce the data volume to about one-third of the original high-density point cloud data, and will be used as image data for the next step of training the deep learning network. The preprocessing method includes five steps: outlier removal, point cloud filtering, point cloud data calibration, generation of a two-dimensional grid matrix and grayscale conversion. Fig. 9 The five steps of the high-density point cloud data preprocessing method are shown;

[0043] It is necessary to reconstruct the defect morphology and measure the defect size in three dimensions based on the point cloud data and depth image of the defect location, such as Fig.10 As shown, the weld surface morphology and corresponding depth map of different defect types;

[0044] The object detection workflow mainly consists of three parts: preparing the dataset, training, and inference. Preparing the dataset includes collecting images and annotating them. Training includes selecting a framework and creating a model, training the model based on a specific task, and evaluating the performance of the trained model. Inference refers to applying the trained model on a new set of images.

[0045] In 2017, TsungYiLi et al. proposed the Feature Pyramid Network (FPN) structure. The skip-layer structure between network layers is more conducive to feature extraction. This network structure solves the problem of object detection of different image sizes.

[0046] FPN can fuse the high-dimensional features of convolutional neural networks with low-dimensional features, making the semantics contained in the network richer. The difference between FPN and the traditional Faster R-CNN is that it has multiple prediction layers, and the targets at the ends of the tower structure will have higher resolution, which helps to identify small targets after they have high-resolution features;

[0047] When mapping regions to different feature layers according to their size, small regions are mapped to shallow layers, and large regions are mapped to deep layers. The allocation principle used by the network structure FPN is shown in the formula:

[0048]

[0049] Where k0 is a constant representing the number of prediction layers assigned, w and h represent the width and height of the prediction box, and 224 represents the commonly used size of the pre-trained model in the dataset. The formula can be used to map the prediction box of size w×h to the feature map. After being assigned, the prediction box enters the ROIPooling layer to complete the connection of the output results.

[0050] FPN includes two tasks: finding instances and classifying them. Its network consists of three main parts. The first part is called the core network, also known as the backbone network, which consists of pre-trained networks. Its main function is to generate various feature maps. These feature maps encode different types of information according to different proportions and determine their depth in the network. The second part combines core layers of different levels together. The third part selects additional networks from each level, called head networks, such as Fig.11 As shown;

[0051] Step 2B: Based on step 2A, image annotation and training are performed. Annotation requires providing real different types of weld images and defect area images, and listing this information in detail for the computer. Through repeated training and learning, with the help of continuous inspection and correction by staff, the model training is finally completed. The annotation training process is as follows: Fig.12 As shown in the figure, the process includes model testing, data annotation, model training, and product evaluation. The target detection function supports two annotation methods: axis-parallel rectangle and free rectangle. Axis-parallel rectangle is annotated with a rectangle parallel to the coordinate axis, and free rectangle is annotated with a rectangle that can adjust the direction. This annotation method can annotate the target object at a better angle, reduce the proportion of non-object pixels, and calculate the target size. The two annotation methods are as follows Fig.13 As shown in the white rectangle, this study uses the free rectangle annotation method. Accurate defect annotation helps network training, allowing the network to converge quickly and improve recognition accuracy.

[0052] Embodiment 5, based on Embodiment 2: Step 3 implements model compression based on dual-correlation knowledge distillation. Semantic segmentation is a basic task in image understanding. Its goal is to classify the input image pixel by pixel. With the advancement of semantic segmentation algorithms, the accuracy of semantic segmentation tasks has also been greatly improved. Currently, the top semantic segmentation models in terms of accuracy usually have extremely high requirements for computing resources, which limits the application scenarios of these high-precision models. For embedded devices or mobile devices with limited computing power, the time required for single-image reasoning using these high-precision models is often unbearable. Therefore, in application scenarios such as autonomous driving, virtual reality, and robots, there is a contradiction between segmentation accuracy and reasoning speed;

[0053] An effective way to find a balance between segmentation accuracy and inference speed is model compression. Knowledge distillation is a commonly used model compression method. The main body of knowledge distillation includes two parts: a complex model with high accuracy and large number of parameters and computation, and a compact model with low accuracy and small number of parameters and computation. The former is also often called the teacher model, and the latter is correspondingly called the student model. The main idea is to use the category probability output by the teacher model as additional supervision information to guide the training process of the student model, so that the student model can not only obtain knowledge from the true value label provided by the data set itself, but also from the predicted value of the teacher model. Since the teacher model usually has more layers, the stacking of more convolutional layers makes the effective receptive field of the teacher model larger than that of the student model. Therefore, the teacher model can capture more contextual information from the same data. Passing the additional information learned by the teacher model to the student model that cannot obtain this part of information can improve the final classification accuracy of the student model in practice. Since the student model naturally has the advantages of simple model and fast inference speed, the use of knowledge distillation technology to improve its segmentation accuracy also achieves a trade-off between segmentation accuracy and inference speed on the student model.

[0054] The dual-correlation distillation method is used to fully explore the knowledge contained in the teacher model and transfer this knowledge to the student model. The dual-correlation distillation method can simultaneously model the correlation between spatial positions and the correlation between channels in the teacher model feature map. After these contextual correlation information are passed to the student model, the student model can better imitate the teacher model at the feature distribution level, thereby improving the segmentation accuracy of the student model. The classification accuracy of the student model can be further improved. After the distillation is completed, the gap between the student model and the teacher model is significantly reduced.

[0055] The overall framework of the dual correlation distillation method is as follows Fig.15As shown in the figure, the dual-association distillation method includes a teacher model and a student model. Since the feature maps of the intermediate layers are not aligned, the question of which two intermediate feature maps should be more consistent is not involved. Therefore, the student model does not need to have a network structure similar to the teacher model. In order to fully transfer the information contained in the trained teacher model to the student model, the information in the teacher model is mined from three dimensions: feature map, score map and overall segmentation result, and this information is passed to the student model as knowledge, so that the student model has more learning basis. The feature map used in this project is the last feature map, that is, the input feature map of the last convolutional layer, with a dimension of W×H×C, where W represents the width of the feature map, H represents the height of the feature map, and C represents the number of channels of the feature map; the score map is the output result of the last convolutional layer, with a dimension of W×H×c, where c represents the number of semantic categories, which is generally much smaller than C; the overall segmentation result is the result of argmaxing the score map with a dimension of W×H×c. The role of argmax is to retain the subscript corresponding to the maximum value in the c channels corresponding to each spatial position, so the dimension of the overall segmentation result is W×H×1. In order to allow the student model to absorb as much knowledge as possible from the teacher model, in addition to the multi-classification cross entropy loss commonly used in semantic segmentation tasks, four additional knowledge distillation loss functions are assembled into the solution. They are: spatial association loss acting on feature maps, channel association loss acting on feature maps, pixel-wise similarity loss acting on score maps, and adversarial learning loss acting on the final segmentation result. The specific steps include:

[0056] Step 4A: Bi-correlation similarity alignment in feature maps:

[0057] If knowledge distillation technology can be used to guide the feature extraction process of the student model, the feature extraction ability of the student model will be closer to that of the teacher model, and the improved feature extraction ability will help produce better segmentation results. Therefore, spatial correlation similarity loss and channel correlation similarity loss are used to transfer the rich contextual correlation information contained in the feature map of the teacher model to the student model;

[0058] Taking the teacher model as an example to explain the specific calculation process, given the feature map F′∈R that needs to calculate the spatial correlation information C ×H×W , where W and H represent the width and height of the feature map respectively, and C represents the number of channels of the feature map. First, adjust its shape to get F∈R C×N , where N = H × W, represents the number of pixels in the feature map, after which the transpose of F T Perform matrix multiplication with F to obtain a matrix of dimension M×N, and then perform a Softmax on the result of the matrix multiplication to obtain the spatial association graph S∈R N×N, the process can be described by the following formula:

[0059]

[0060] S ij Indicates the degree of association between the i-th position and the j-th position in the feature map. The higher the degree of association between two spatial positions in the feature map, the larger the corresponding value in the spatial association map.

[0061] Minimize the difference between the spatial association graphs calculated by the teacher model t and the student model s. To achieve this effect, this project uses the conventional mean square error to calculate the spatial association similarity loss L S , L S The calculation process can be described in the following formula:

[0062]

[0063] Where N represents the number of pixels in the feature map, and Respectively represent the strength of the correlation between the jth spatial position and the ith spatial position in the student model feature map and the teacher model feature map;

[0064] A channel-related similarity loss L C It is used to measure the difference in the correlation information between channels in the feature graph of the teacher model and the student model. The channel correlation similarity loss function L C The definition is as follows:

[0065]

[0066] Where C represents the number of channels of the feature map, and Respectively represent the degree of correlation between the Sth channel and the tth channel in the student model feature map and the teacher model feature map. The higher the correlation between the two channels in the feature map, the larger the corresponding value in the channel correlation map.

[0067] Step 4B: Pixel-wise similarity alignment in score map:

[0068] It is not enough to simply require the student model to align with the feature distribution of the teacher model. It is also necessary to impose constraints at the score map level to align the score map of the student model with the teacher model. The semantic segmentation task is regarded as a pixel-level multi-classification task separated by pixels. The output probability of each category of the teacher model at each pixel position is directly passed to the student model as a soft label. The pixel-by-pixel similarity loss function L P The definition is as follows:

[0069]

[0070] in represents the category probability predicted by the teacher model t at the i-th pixel position, represents the category probability predicted by the student model s at the i-th pixel position. They are all 1×1×c vectors, c represents the number of semantic categories, N=H×W represents the number of pixels in the feature map, and KL(·) represents the KL divergence between the two probability distributions. The specific calculation method of KL divergence is as follows:

[0071]

[0072] Where N represents the number of pixels, c represents the number of semantic categories, represents the probability that the teacher model predicts the i-th position of the input feature as category j, It represents the probability that the student model predicts the i-th position of the input feature as category j. The effect of using KL divergence to calculate the pixel-by-pixel similarity loss is to make the student model closer to the category probabilities with larger values ​​in the teacher model. Since the final segmentation result is determined by these category probabilities with larger values, the output results of the two are also closer.

[0073] Step 4C: Overall similarity alignment among segmentation results:

[0074] The student model is regarded as a generator. Its goal is to generate a semantic segmentation result for the input image so that this result is as similar as possible to the segmentation result of the teacher model. At the same time, a discriminator is trained to distinguish the segmentation result of the teacher model from the segmentation result of the student model as much as possible. The difference and connection between the segmentation result here and the score map mentioned above is that the dimension of the segmentation result is 1×H×W, and the dimension of the score map is C×H×W. After argmax is performed on the score map, the segmentation result is obtained;

[0075] Adversarial learning loss function L adv It can be formulated as follows:

[0076]

[0077] Where I represents input, Z S represents the output of the student model (Z s ~P S (Z S )), D means using the discriminator for discrimination;

[0078] Discriminator loss function L d It can be formulated as follows:

[0079]

[0080] Where I represents input, Z trepresents the output of the teacher model (Z t ~P t (Z t )), D means using the discriminator for discrimination;

[0081] For the dual-association distillation framework proposed in this project, the overall loss function consists of the four distillation losses mentioned above and a multi-classification cross entropy loss for semantic segmentation:

[0082] L=L CE +λ1L P -λ2L Adv +λ3(L S +L C ) (9)

[0084] Among them, λ1, λ2, λ3 can be set according to the situation, such as 10, 0.1 and 25, L CE The calculation is performed as follows:

[0085]

[0086] Where N represents the number of pixels, c represents the number of categories, and p S represents the segmentation result of the dual correlation distillation method, y S Represents the true value of semantic segmentation. If the true value of the semantic segmentation of the pixel at the i-th position is j, then is 1, otherwise is 0;

[0087] Step 4D: Evaluation criteria:

[0088] In order to evaluate the improvement of the student model before and after distillation in each semantic category and the difference between the student model and the teacher model in each semantic category, the F harmonic mean F of a specific semantic category i is calculated using the following formula 1,j :

[0089]

[0090] Where P represents precision, R represents recall, β=1, and the average F harmonic mean of all categories, meanF1, is the arithmetic average of the F harmonic means of each category:

[0091]

[0092] Among them, C represents the number of categories, and the F harmonic mean and average intersection-over-union accuracy evaluation indicators are both between 0 and 1 in value, and the higher the value, the more accurate the segmentation result.

[0093] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A weld positioning and defect detection system based on deep learning, comprising a lower computer, a data transmission layer, and an upper computer, wherein the lower computer comprises a monocular camera, a binocular camera, a PLC, a robot motion control card, and an industrial robot, the data transmission layer is a switch, and the upper computer is an industrial computer and a keyboard, a mouse, a touch screen, and a display, characterized in that: The host computer is responsible for human-machine interaction and weld image processing, and communicates with the slave computer through a switch for control. The PLC in the slave computer receives instructions from the host computer and transmits them to the robot motion control card, or the host computer transmits them to the robot motion control card through the switch. The robot motion control card realizes industrial robot welding by controlling the motion axis. At the same time, the PLC triggers the monocular camera and the binocular camera at the same time to collect the weld surface image.

2. A weld positioning and defect detection system based on deep learning, characterized in that: The motion control card has a professional communication interface and realizes communication with the host computer through a switch.

3. The weld positioning and defect detection system based on deep learning is characterized by: The monocular camera and binocular camera are equipped with a dedicated LED light source.

4. A weld positioning and defect detection method based on deep learning, characterized in that: The steps include: Step 1: Weld image acquisition and position calibration based on line structured light; Step 2: Acquisition of high-density point cloud images and generation of depth images of the weld surface based on surface structured light. Specifically, the Z coordinate of the height information of the high-density point cloud is converted into the image pixel grayscale, and the X and Y coordinates of the spatial information of the point cloud are converted into pixel indexes. The high-density point cloud image preprocessing includes abnormal point removal, point cloud filtering, point cloud data calibration, generation of a two-dimensional grid matrix and grayscale conversion. Step 3: Based on deep learning, weld defect recognition and positioning, the fully convolutional neural network FCN is used to achieve pixel positioning segmentation of the target, that is, semantic segmentation. The semantic segmentation task can classify each pixel when searching for objects in the image, and mark pixels representing features such as color and texture as the same category. The result of semantic segmentation is to output an image. The pixel value represents the classification result of the corresponding image in the input image. The size of the input and output images is the same. The network can automatically learn useful features from the image data, complete image annotation with the help of annotation software, interactively mark pixels and output labeled data to train the neural network; Step 4: Model compression based on dual-correlation knowledge distillation; Step 5: Identification of pits and porosity defects in welds, weld bumps and unqualified defects.

5. The weld positioning and defect detection method based on deep learning according to claim 1, characterized in that: The step 1 specifically comprises the following steps: Step 1A: Construction of mathematical model of line structured light camera: Line structured light camera consists of a line laser emitter and an industrial camera. The line laser emitter emits a straight line of laser light to form a laser light plane. The light plane intersects with the surface of the object to be measured to form a light strip on the surface of the object to be measured. The position of the light strip in the world coordinate system can be obtained based on the imaging position of the light strip in the camera and the principle of triangulation. The relationship between the world coordinate system, the camera coordinate system and the image coordinate system is constructed based on the ideal pinhole imaging camera model. Step 1B: Extraction of sub-pixel coordinates of the center line of the line structured light stripe: including image preprocessing and extraction of sub-pixel coordinates of the center line of the light stripe, wherein the image preprocessing includes five steps: image filtering, image segmentation, morphological processing, width filtering and extraction of regions of interest; The sub-pixel coordinate extraction of the center line of the light stripe is to extract the center line of the line structure light stripe to obtain the center point coordinate of the line structure light stripe expressed by sub-pixel coordinates; Step 1C: Line structured light stripe feature point extraction and weld size detection: including weld feature point recognition and weld size detection; The weld feature point recognition first uses the least square method to fit the center line of the line structured light stripe, then uses the threshold method to roughly extract the weld feature points, and finally uses the linear interpolation method to accurately extract the weld feature points; After extracting the weld feature points, the weld size detection obtains the weld width, excess height, misalignment, joint angle and other external dimensions according to the calibration results of the line structured light camera and the definition of the weld external geometric dimensions, and records and displays the weld geometric dimensions in real time.

6. The weld positioning and defect detection method based on deep learning according to claim 1, characterized in that: The step 2 specifically comprises the following steps: Step 2A: Obtain the preprocessed point cloud data and depth image, reconstruct the defect morphology in three dimensions and measure the defect size. Specifically, when mapping the area to different feature layers according to the size, the small area is mapped to the shallow layer, and the large area is mapped to the deep layer. The allocation principle used by the network structure FPN is shown in the formula: Where k0 is a constant representing the number of prediction layers assigned, w and h represent the width and height of the prediction box, and 224 represents the commonly used size of the pre-trained model in the dataset. The formula can be used to map the prediction box of size w×h to the feature map. After being assigned, the prediction box enters the ROIPooling layer to complete the connection of the output results. FPN includes two tasks: finding instances and classifying them. Its network consists of three main parts. The first part is called the core network, also known as the backbone network, which is composed of pre-trained networks. Its main function is to generate various feature maps (FeatureMap). These feature maps encode different types of information according to different proportions and determine their depth in the network. The second part combines core layers of different levels together. The third part selects additional networks from each level, called head networks; Step 2B: Based on step 2A, image annotation and training are performed, using free rectangle annotation. The process includes model testing, data annotation, model training, and product evaluation.

7. The weld positioning and defect detection method based on deep learning according to claim 1, characterized in that: The step 3 realizes model compression based on dual-association knowledge distillation, including a teacher model and a student model. In order to fully transfer the information contained in the trained teacher model to the student model, the information in the teacher model is mined from three dimensions: feature map, score map and overall segmentation result, and the information is passed to the student model as knowledge, so that the student model has more learning basis. Specifically, the following steps are included: Step 4A: Bi-correlation similarity alignment in feature maps: Given a feature map F′∈R that needs to calculate spatial correlation information C×H×W , where W and H represent the width and height of the feature map respectively, and C represents the number of channels of the feature map. First, adjust its shape to get F∈R C×N , where N = H × W, represents the number of pixels in the feature map, after which the transpose of F T Perform matrix multiplication with F to obtain a matrix of dimension M×N, and then perform a Softmax on the result of the matrix multiplication to obtain the spatial association graph S∈R N×N , the process can be described by the following formula: S ij Indicates the degree of association between the i-th position and the j-th position in the feature map. The higher the degree of association between two spatial positions in the feature map, the larger the corresponding value in the spatial association map. Minimize the difference between the spatial association graphs calculated by the teacher model t and the student model s. To achieve this effect, this project uses the conventional mean square error to calculate the spatial association similarity loss L S , L S The calculation process can be described in the following formula: Where N represents the number of pixels in the feature map, and Respectively represent the strength of the correlation between the jth spatial position and the ith spatial position in the student model feature map and the teacher model feature map; A channel-related similarity loss L C It is used to measure the difference in the correlation information between channels in the feature graph of the teacher model and the student model. The channel correlation similarity loss function L C The definition is as follows: Where C represents the number of channels of the feature map, and Respectively represent the degree of correlation between the Sth channel and the tth channel in the student model feature map and the teacher model feature map. The higher the correlation between the two channels in the feature map, the larger the corresponding value in the channel correlation map. Step 4B: Pixel-wise similarity alignment in score map: The semantic segmentation task is regarded as a pixel-by-pixel multi-classification task. The output probability of each category of the teacher model at each pixel position is directly passed to the student model as a soft label. The pixel-by-pixel similarity loss function L P The definition is as follows: in represents the category probability predicted by the teacher model t at the i-th pixel position, represents the category probability predicted by the student model s at the i-th pixel position. They are all 1×1×c vectors, c represents the number of semantic categories, N=H×W represents the number of pixels in the feature map, and KL(·) represents the KL divergence between the two probability distributions. The specific calculation method of KL divergence is as follows: Where N represents the number of pixels, c represents the number of semantic categories, represents the probability that the teacher model predicts the i-th position of the input feature as category j, It represents the probability that the student model predicts the i-th position of the input feature as category j. The effect of using KL divergence to calculate the pixel-by-pixel similarity loss is to make the student model closer to the category probability with a larger value in the teacher model. Step 4C: Overall similarity alignment among segmentation results: The student model is regarded as a generator. Its goal is to generate a semantic segmentation result for the input image so that this result is as similar as possible to the segmentation result of the teacher model. At the same time, a discriminator is trained to distinguish the segmentation result of the teacher model from the segmentation result of the student model as much as possible. The difference and connection between the segmentation result here and the score map mentioned above is that the dimension of the segmentation result is 1×H×W, and the dimension of the score map is C×H×W. After argmax is performed on the score map, the segmentation result is obtained; Adversarial learning loss function L adv It can be formulated as follows: Where I represents input, Z S represents the output of the student model (Z s ~P S (Z S )), D means using the discriminator for discrimination; Discriminator loss function L d It can be formulated as follows: Where I represents input, Z t represents the output of the teacher model (Z t ~P t (Z t )), D means using the discriminator for discrimination; For the dual-association distillation framework proposed in this project, the overall loss function consists of the four distillation losses mentioned above and a multi-classification cross entropy loss for semantic segmentation: L=L CE +λ1L P -λ2L Adv +λ3(L S +L C ) (9) Among them, λ1, λ2, λ3 can be set according to the situation, such as 10, 0.1 and 25, L CE The calculation is performed as follows: Where N represents the number of pixels, c represents the number of categories, and p S represents the segmentation result of the dual correlation distillation method, y S Represents the true value of semantic segmentation. If the true value of the semantic segmentation of the pixel at the i-th position is j, then is 1, otherwise is 0; Step 4D: Evaluation criteria: In order to evaluate the improvement of the student model before and after distillation in each semantic category and the difference between the student model and the teacher model in each semantic category, the F harmonic mean F of a specific semantic category i is calculated using the following formula 1,j : Where P represents precision, R represents recall, β=1, and the average F harmonic mean of all categories, meanF1, is the arithmetic average of the F harmonic means of each category: Among them, C represents the number of categories, and the F harmonic mean and average intersection-over-union accuracy evaluation indicators are both between 0 and 1 in value, and the higher the value, the more accurate the segmentation result.

Images