Multi-modal welding seam accurate identification method and system based on MaxViT visual Transform

Abstract

The invention discloses a MaxViT visual Transformer-based multi-modal weld joint accurate identification method and system, and relates to the technical field of weld joint identification, the identification method comprises the following steps: S1, data acquisition and modeling: selecting a 20 million pixel industrial camera (supporting RGB three-channel imaging, and having a frame rate of greater than or equal to 30fps) and a laser radar module (having a point cloud resolution of less than or equal to 0.5 mm, and ranging precision of + / -0.1 mm), integrating the system, and performing data acquisition and modeling; an RGB image and point cloud data of a to-be-welded part are synchronously collected, the RGB image records color and texture information, the point cloud data records three-dimensional space coordinate information, and the data are stored in an industrial server in real time. The interference of uneven illumination, strong light reflection, smoke noise, point cloud abnormal points and the like on a welding site is effectively overcome, and the harsh requirement for high-precision automatic welding is met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of weld seam recognition, in particular to a multi-modal weld seam accurate recognition method and system based on MaxViT visual Transformer. BACKGROUND

[0002] Weld seam accurate recognition is a crucial link in weld quality control, especially in automation and intelligent manufacturing, accurate recognition of weld seams can greatly improve production efficiency and weld quality.

[0003] In the current industrial production process, traditional weld seam detection mainly relies on manual visual inspection or sampling non-destructive testing, not only has the problems of low efficiency, strong subjectivity and high missed detection rate, but the core pain point is the lack of accuracy: manual detection is difficult to accurately quantify the weld seam position, direction and geometric shape, and sampling detection cannot cover the whole production line; the existing automatic recognition method based on conventional machine vision also has the problems of insufficient feature extraction and low class distinction due to the large difference in features of multiple categories of weld seams (welding points, straight weld seams and arc weld seams), which makes it difficult to meet the requirements of modern production lines for recognition accuracy and consistency.

[0004] Therefore, a multi-modal visual neural network-based multi-category weld seam accurate recognition method and system is provided, which combines a multi-modal data processing flow and solves the problem of insufficient accuracy by using a self-developed MultiVisionModels neural network to realize automatic and high-precision recognition and positioning of the weld seam area of a welding workpiece. SUMMARY

[0005] To overcome the shortcomings of the prior art, the present application provides a multi-modal weld seam accurate recognition method and system based on MaxViT visual Transformer, which solves the problems raised in the background art.

[0006] To achieve the above purpose, the present application realizes the following technical scheme: a multi-modal weld seam accurate recognition method based on MaxViT visual Transformer, comprising the following steps: S1, data acquisition and modeling: a 20 million pixel industrial camera (supporting RGB three-channel imaging, frame rate ≥30fps) and a laser radar module (point cloud resolution ≤0.5mm, ranging accuracy ±0.1mm) are selected, the system is integrated, and the RGB image and point cloud data of the part to be welded are synchronously collected, the RGB image records color and texture information, the point cloud data records three-dimensional space coordinate information, and the data is stored in an industrial-grade server in real time; S2, multi-source interference suppression preprocessing: adaptive light equalization and salt and pepper noise filtering are performed on the collected RGB image, the Retinex algorithm is used to eliminate the strong light / shadow interference of the welding scene, and the median filter is used to remove the pixel abnormal points; the outlier rejection and downsampling optimization are performed on the point cloud data, the statistical filtering algorithm is used to remove the abnormal points with a ranging error of ±0.1 mm, and the voxel grid downsampling is used to maintain the integrity of the three-dimensional structure while reducing the data amount; S3, data fusion and labeling: the preprocessed point cloud three-dimensional coordinate information is mapped to the corresponding pixel of the RGB image, the fusion data of "RGB+three-dimensional coordinates" is generated and saved in TIFF format; the x-AnyLabeling labeling tool is used for pixel-level labeling, the weld is divided into point (welding point), line (straight weld) and arc (arc weld) according to the geometric shape, and the labeling result is stored in JSON format, forming a standardized data set associated with JSON+TIFF; S4, neural network architecture optimization: a dual-branch multi-modal neural network based on MaxViT is constructed, which respectively extracts features from RGB images and point cloud data, and realizes adaptive fusion through a learnable cross-modal attention mechanism, which includes: RGB image branch: a hybrid structure of convolutional neural network (CNN) and Transformer encoder is used; the input image is first subjected to preliminary feature extraction by ResNet-50 convolutional backbone network to generate multi-scale two-dimensional feature maps; the high-level feature maps are flattened and projected into serialized tokens and input into the multi-layer Transformer encoder; each Transformer block contains a multi-head self-attention mechanism and a feedforward network, which captures long-range semantic dependency relationships in the image through the self-attention mechanism to enhance the understanding ability of complex scenes; finally, the branch outputs a set of global feature representations with rich semantic information; Point cloud branch: multi-scale three-dimensional feature extraction is performed based on the MaxViT module; the original point cloud is divided into multiple local neighborhoods through the hierarchical sampling and grouping strategy of PointNet++; the MaxViT module is introduced in each level to realize efficient feature aggregation across spatial dimensions by alternately performing local multi-scale convolution and global axial attention; this design not only preserves the fine-grained geometric details of the point cloud, but also enhances the perception ability of large-scale structures; finally, the point cloud branch outputs multi-scale three-dimensional feature representations; fusion layer: a learnable cross-modal attention mechanism (Learnable Cross-Modal Attention, LCMA) is introduced, which takes the RGB feature and point cloud feature as the key (Key) and value (Value), and introduces a learnable query vector , the fusion feature is calculated by cross-modal attention:

[0007] wherein , is the spliced key and value matrix; the attention weight reflects the relative importance of different modalities in the current task; in addition, a bidirectional cross attention (Bi-directional Cross Attention) is introduced, so that the RGB feature can focus on the key structure in the point cloud, and vice versa; finally, the fused feature is sent to the subsequent task head for weld recognition, and the neural network automatically learns the optimal combination strategy of the two modalities, so that it is not dependent on the pre-defined fusion weight or simple splicing / addition operation, improves the expression ability and adaptability, and can dynamically adjust the modal weight according to different inputs, improve the robustness and accuracy of the model in complex scenes; S5, adaptive multi-task learning training: a dynamic weight distribution strategy is adopted to automatically adjust the loss weights of classification, positioning and geometric parameter regression of three tasks through a neural network; a curriculum learning mechanism is introduced during training, and the samples are trained in a hierarchical manner from easy to difficult to improve the convergence stability of the model; S6, feature extraction and category recognition: divide the labeled data set into training set, validation set and test set according to the ratio of 7:2:1, and input it into the self-developed MultiVisionModels neural network; the training adopts AdamW optimizer (initial learning rate 1e-4, weight decay 1e-5), batch size 32, and trains for 50 rounds, and the loss function is a weighted fusion of cross-entropy loss, Dice loss and geometric loss; the model outputs high-resolution feature map and category label; S7, class customization post-processing: for weld points, after local maximum value detection and NMS to remove redundant points, the center coordinates (X, Y, Z) and confidence score ≥0.8 are output; for straight welds, the key points are selected by interval NMS, and the start / end point coordinates and average confidence score are output; for arc welds, after nine-grid key point screening, least squares circle fitting and NMS processing, the center coordinates, radius, arc starting coordinates and confidence score are output; S8, dynamic precision calibration and feedback: introduce the weld material parameters (metal / non-metal) and environmental parameters (temperature 0-50℃, illumination 100-10000lux) to build a calibration model to dynamically correct the geometric parameters output by the post-processing; compare the corrected recognition results with the standard weld sample data, calculate the deviation value and feed it back to the loss function weight adjustment module of the MultiVisionModels model to realize self-adaptive optimization of the model; S9, data output and application: generate a structured report containing workpiece ID, identification time, category, corrected geometric parameters and confidence (≥0.95 is valid), and transmit it to the robot control system through Profinet / Modbus protocol to provide millimeter-level path planning data support.

[0008] According to the above technical scheme, the Retinex algorithm in S2 adopts a multi-scale Gaussian wrap function, and the wrap scale parameters are set to 8, 16 and 32 pixels. The adaptive adjustment of the illumination compensation intensity is performed. The number threshold of the neighborhood points of the statistical filtering is set to 15, and the outlier judgment threshold is 2.5 times the data standard deviation.

[0009] According to the above technical scheme, the cross-modal attention fusion layer in S4 assigns weights by calculating the cosine similarity of the RGB feature and the point cloud feature, and the weight distribution range is 0.1-0.9. When the similarity is greater than or equal to 0.7, the corresponding modal feature weight is enhanced.

[0010] According to the above technical scheme, the calculation formula of the cross-entropy loss in S6 is:

[0011] Wherein, CE_Loss represents the cross-entropy loss; N represents the number of pixels in the image; i represents the i-th pixel; C represents the type of the pixel; j represents the j-th type; yij is a sign function, which takes the value of 0 or 1, if the true class of the i-th pixel is j, then 1, otherwise 0; pij represents the predicted probability of the i-th pixel belonging to the j-th type; The calculation formula of the Dice loss is:

[0012] Wherein, Dice_Loss represents the Dice loss, |X| represents the sum of the true labels, |Y| represents the sum of the predicted results, and |X∩Y| represents the intersection of the true labels and the predicted results. For the welding spot and the straight weld, the cross-entropy loss is used; for the arc weld, the fusion loss is used, and the calculation formula is:

[0013] Wherein, L represents the fusion loss, and w represents the weighting coefficient.

[0014] According to the above technical scheme, the calibration model in S8 adopts the gradient boosting tree algorithm, takes the material type, temperature and light intensity as the input features, and takes the geometric parameter deviation as the output label. The model training iteration number is greater than or equal to 100 rounds.

[0015] The multi-modal welding seam accurate identification system based on MaxViT visual Transformer comprises: Multimodal acquisition module: Composed of a 20-megapixel industrial camera, a LiDAR module and an adjustable bracket, it enables synchronous acquisition and industrial-grade storage of RGB and point cloud data; Multi-source interference preprocessing module: Configures image enhancement unit and point cloud optimization unit to perform adaptive illumination equalization, noise filtering and outlier removal, and downsampling processing respectively; Fusion annotation module: Configures a data registration engine and intelligent verification unit to complete point cloud-RGB spatial mapping and TIFF format storage, integrates x-AnyLabeling annotation tool and cross-validation module to ensure annotation accuracy ≥99%; MaxViT Feature Recognition Module: Built-in MultiVisionModels model as described in claim 1, including RGB feature extraction submodule, MaxViT point cloud feature submodule, cross-modal fusion submodule and classification submodule, with an accuracy of ≥98% for three types of weld seam recognition; The classification post-processing module includes a weld point positioning unit, a straight weld fitting unit, and an arc weld modeling unit, which respectively perform key point screening, parameter calculation, and confidence quantification for the corresponding category of welds, with geometric parameter measurement error ≤ ±0.3mm; Dynamic calibration feedback module: integrates parameter sensing unit, calibration calculation unit and model optimization unit, corrects geometric data based on material and environmental parameters and feeds back to optimize model; Intelligent output module: integrates a data formatting engine and industrial protocol interface to generate structured reports and output robot control signals through Profinet / Modbus protocol, supporting trajectory adjustment adapted to weld type.

[0016] According to the above technical solution, the median filtering window size of the multi-source interference preprocessing module can be automatically switched according to the image noise intensity, with a switching range of 3×3-7×7 pixels; the voxel size of the voxel grid downsampling is set to 0.3-0.8mm to adapt to different precision requirements.

[0017] According to the above technical solution, the dynamic calibration feedback module is equipped with a temperature sensor and a light sensor, with a sampling frequency of ≥10Hz and a parameter transmission delay of ≤50ms to ensure real-time calibration.

[0018] This invention provides a method and system for accurate multimodal weld seam recognition based on MaxViT vision Transformer. It offers the following advantages: (1) The identification method and system effectively overcome interference such as uneven lighting, strong light reflection, smoke and dust noise and point cloud anomalies at the welding site through the dual technical means of "multi-source interference suppression preprocessing" and "MaxViT dual-branch multimodal neural network". The MaxViT model fully leverages the advantages of Transformer in long-distance dependency modeling, accurately extracts the global spatial features of the weld, and improves the overall identification accuracy of welds with significant morphological differences, such as weld points, straight welds and arc welds, to over 98%. The geometric parameter measurement error is controlled within ±0.3mm, which meets the stringent requirements of high-precision automated welding.

[0019] (2) The identification method and system introduces a "dynamic accuracy calibration and feedback mechanism". By sensing environmental parameters such as material, temperature and light in real time, the gradient boosting tree model is used to dynamically compensate and correct the identification results, and the deviation is fed back to the model for self-optimization. This enables the system to adapt to different working conditions, avoids the decline in identification performance caused by environmental changes, and significantly improves the stability and reliability of long-term operation in complex industrial sites.

[0020] (3) The identification method and system adopts a “category-customized post-processing strategy” and designs differentiated algorithms for different weld geometric characteristics: accurately locate the center of the weld point, fit the start and end points of the straight weld, and perform circular parameter regression for the arc weld. It not only outputs the category label of the weld, but also provides complete structured information including accurate coordinates, size and confidence, providing direct, usable and high-precision data support for subsequent robot path planning, and realizing a seamless connection from “identification” to “usability”.

[0021] (4) The identification method and system construct a complete automated closed-loop system from multimodal data acquisition, fusion annotation, intelligent identification, dynamic calibration to the final control signal output. The modules work together, and in particular, the calibration feedback mechanism forms an autonomous cycle of "identification-evaluation-optimization", which greatly reduces the dependence on external human intervention. Through standard industrial protocols (such as Profinet / Modbus) and integration with the robot control system, the key technology link from visual perception to robot execution is opened up, which effectively improves the automation level and production efficiency of intelligent manufacturing production lines. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the identification method of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0024] Please see Figure 1 One embodiment of the present invention is: a method for accurate identification of multimodal welds based on MaxViT visual Transformer, comprising the following steps: S1. Data Acquisition and Modeling: A 20-megapixel industrial camera (supporting RGB three-channel imaging, frame rate ≥30fps) and a LiDAR module (point cloud resolution ≤0.5mm, ranging accuracy ±0.1mm) are selected and integrated into a system to simultaneously acquire RGB images and point cloud data of the part to be welded. The RGB images record color and texture information, and the point cloud data records three-dimensional spatial coordinate information. The data is stored in real time on an industrial-grade server. S2. Multi-source interference suppression preprocessing: Adaptive illumination equalization and salt-and-pepper noise filtering are performed on the acquired RGB images. The Retinex algorithm is used to eliminate strong light / shadow interference in the welding scene, and median filtering is used to remove outlier pixels. Outlier removal and downsampling optimization are performed on the point cloud data. Outlier points with ranging errors exceeding ±0.1mm are removed based on the statistical filtering algorithm. Voxel mesh downsampling is used to maintain the integrity of the three-dimensional structure while reducing the amount of data. S3. Data Fusion and Labeling: The preprocessed point cloud 3D coordinate information is mapped to the corresponding pixels of the RGB image to generate fused data of "RGB + 3D coordinates" and saved in TIFF format; the x-AnyLabeling labeling tool is used for pixel-level labeling, and the weld is divided into three categories according to its geometric shape: point (weld point), line (straight weld), and arc (arc weld). The labeling results are stored in JSON format to form a standardized dataset associated with JSON + TIFF. S4. Neural Network Architecture Optimization: Construct a MaxViT-based dual-branch multimodal neural network to extract features from RGB images and point cloud data respectively, and achieve adaptive fusion through a learnable cross-modal attention mechanism, specifically including: RGB Image Branch: Employs a hybrid structure of stacked Convolutional Neural Network (CNN) and Transformer encoder. The input image first undergoes preliminary feature extraction through a ResNet-50 convolutional backbone network, generating multi-scale two-dimensional feature maps. The high-level feature maps are flattened and projected into serialized tokens, which are then input into a multi-layer Transformer encoder. Each Transformer block contains a multi-head self-attention mechanism and a feedforward network. The self-attention mechanism captures long-distance semantic dependencies in the image, enhancing the understanding of complex scenes. Finally, this branch outputs a set of global feature representations with rich semantic information. Point cloud branch: Multi-scale 3D feature extraction is performed based on the MaxViT module; the original point cloud is divided into multiple local neighborhoods through the hierarchical sampling and grouping strategy of PointNet++; the MaxViT module is introduced in each level, and efficient feature aggregation across spatial dimensions is achieved by alternately performing local multi-scale convolution and global axial attention; this design not only preserves the fine-grained geometric details of the point cloud, but also enhances the perception ability of large-scale structures; finally, the point cloud branch outputs a multi-scale 3D feature representation. Fusion layer: Introduces a learnable cross-modal attention mechanism (LCMA) given RGB features. and point cloud features This is used as the key and value, and a learnable query vector is introduced. fused features are computed through cross-modal attention:

[0025] in , The concatenated key-value matrix; attention weights This reflects the relative importance of different modalities in the current task; in addition, bidirectional cross attention is introduced, enabling RGB features to focus on key structures in the point cloud, and vice versa; finally, the fused features... It is then fed into the subsequent task head for weld seam identification; S5. Adaptive Multi-Task Learning and Training: A dynamic weight allocation strategy is adopted, which automatically adjusts the loss weights of the three tasks of classification, localization, and geometric parameter regression through the neural network; a course learning mechanism is introduced during the training process, and training is carried out in stages according to the difficulty of the samples, starting with the easier ones and progressing to the more difficult ones to improve the convergence stability of the model. S6. Feature Extraction and Class Recognition: The labeled dataset is divided into training, validation, and test sets in a 7:2:1 ratio and input into the self-developed MultiVisionModels neural network. Training is performed using the AdamW optimizer (initial learning rate 1e-4, weight decay 1e-5), batch size 32, and 50 training rounds. The loss function is a weighted fusion of cross-entropy loss, Dice loss, and geometric loss. The model outputs a high-resolution feature map of the weld seam and class labels. S7. Customized Post-Processing by Category: For weld joints, after local maximum detection and NMS to remove redundant points, the center coordinates (X, Y, Z) and a confidence score of ≥0.8 are output; for straight welds, key points are filtered through interval NMS, and the start / end coordinates and mean confidence score are output; for curved welds, after nine-grid key point filtering, least squares circle fitting, and NMS processing, the center coordinates, radius, arc start coordinates, and confidence score are output. S8. Dynamic Accuracy Calibration and Feedback: A calibration model is constructed by introducing weld material parameters (metal / non-metal) and environmental parameters (temperature 0-50℃, illumination 100-10000 lux) to dynamically correct the geometric parameters output by post-processing; the corrected identification results are compared with standard weld template data, the deviation value is calculated and fed back to the loss function weight adjustment module of the MultiVisionModels model to achieve adaptive optimization of the model; S9. Data Output and Application: Generates a structured report containing workpiece ID, recognition time, category, corrected geometric parameters, and confidence level (≥0.95 is valid), which is transmitted to the robot control system via Profinet / Modbus protocol, providing millimeter-level path planning data support.

[0026] In S2, the Retinex algorithm employs a multi-scale Gaussian wrap function with wrap scale parameters set to 8, 16, and 32 pixels to adaptively adjust the illumination compensation intensity. The threshold for the number of neighboring points in the statistical filter is set to 15, and the threshold for outlier detection is 2.5 times the standard deviation of the data. The multi-scale Gaussian wrap function covers illumination interference of different intensities, avoiding feature loss caused by overexposure or shadows. The statistical filter threshold is scientifically set to effectively remove outliers while preserving the true structure, thereby improving the reliability of 3D data.

[0027] In S4, the cross-modal attention fusion layer assigns weights by calculating the cosine similarity between RGB features and point cloud features. The weight assignment ranges from 0.1 to 0.9. When the similarity is ≥0.7, the weight of the corresponding modal feature is enhanced. By dynamically adjusting the weights of RGB and point cloud features through similarity, the contribution of key modalities is strengthened, the distinguishability of weld seam features is improved, the fusion balance is ensured, and the generalization ability of the model is improved, making it suitable for industrial real-time scenarios.

[0028] The formula for calculating cross-entropy loss in S6 is:

[0029] Where CE_Loss represents the cross-entropy loss; N represents the number of pixels in the image; i represents the i-th pixel; C represents the type of the pixel; j represents the j-th type; yij is the sign function, which takes the value 0 or 1. If the true class of the i-th pixel is j, it is 1, otherwise it is 0; pij represents the predicted probability that the i-th pixel belongs to the j-th type. The formula for calculating Dice loss is:

[0030] Where Dice_Loss represents the Dice loss, |X| represents the summation over the true labels, |Y| represents the summation over the predicted results, and |X∩Y| represents the intersection of the true labels and the predicted results; For weld points and straight welds, cross-entropy loss is used; for curved welds, fusion loss is used, and the calculation formula is as follows:

[0031] Where L represents the fusion loss and w represents the weighting coefficient.

[0032] The calibration model in S8 uses the gradient boosting tree algorithm, with material type, temperature, and light intensity as input features and geometric parameter deviation as output labels. The model is trained for ≥100 rounds. The GBT model captures the complex relationship between material, temperature, light and geometric deviation, and achieves high-precision dynamic correction.

[0033] A multimodal weld seam precision recognition system based on MaxViT vision Transformer includes: Multimodal acquisition module: Composed of a 20-megapixel industrial camera, a LiDAR module and an adjustable bracket, it enables synchronous acquisition and industrial-grade storage of RGB and point cloud data; Multi-source interference preprocessing module: Configures image enhancement unit and point cloud optimization unit to perform adaptive illumination equalization, noise filtering and outlier removal, and downsampling processing respectively; Fusion annotation module: Configures a data registration engine and intelligent verification unit to complete point cloud-RGB spatial mapping and TIFF format storage, integrates x-AnyLabeling annotation tool and cross-validation module to ensure annotation accuracy ≥99%; MaxViT Feature Recognition Module: Built-in MultiVisionModels model as described in claim 1, including RGB feature extraction submodule, MaxViT point cloud feature submodule, cross-modal fusion submodule and classification submodule, with an accuracy of ≥98% for three types of weld seam recognition; The classification post-processing module includes a weld point positioning unit, a straight weld fitting unit, and an arc weld modeling unit, which respectively perform key point screening, parameter calculation, and confidence quantification for the corresponding category of welds, with geometric parameter measurement error ≤ ±0.3mm; Dynamic calibration feedback module: integrates parameter sensing unit, calibration calculation unit and model optimization unit, corrects geometric data based on material and environmental parameters and feeds back to optimize model; Intelligent output module: integrates a data formatting engine and industrial protocol interface to generate structured reports and output robot control signals through Profinet / Modbus protocol, supporting trajectory adjustment adapted to weld type.

[0034] The median filter window size of the multi-source interference preprocessing module can be automatically switched according to the image noise intensity, with a switching range of 3×3-7×7 pixels; the voxel size of the voxel grid downsampling is set to 0.3-0.8mm to adapt to different precision requirements. The filter window is dynamically adjusted according to the noise intensity to balance the denoising effect and detail preservation, reducing the computational load while ensuring accuracy.

[0035] The dynamic calibration feedback module is equipped with a temperature sensor and a light sensor, with a sampling frequency of ≥10Hz and a parameter transmission delay of ≤50ms, ensuring real-time calibration and meeting the timing requirements of continuous operation of welding robots.

[0036] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for accurate multimodal weld seam recognition based on MaxViT vision Transformer, characterized in that, Includes the following steps: S1. Data Acquisition and Modeling: A 20-megapixel industrial camera (supporting RGB three-channel imaging, frame rate ≥30fps) and a LiDAR module (point cloud resolution ≤0.5mm, ranging accuracy ±0.1mm) are selected and integrated into a system to simultaneously acquire RGB images and point cloud data of the part to be welded. The RGB images record color and texture information, and the point cloud data records three-dimensional spatial coordinate information. The data is stored in real time on an industrial-grade server. S2. Multi-source interference suppression preprocessing: Adaptive illumination equalization and salt-and-pepper noise filtering are performed on the acquired RGB images. The Retinex algorithm is used to eliminate strong light / shadow interference in the welding scene, and median filtering is used to remove abnormal pixels. Outlier removal and downsampling optimization are performed on point cloud data. Outliers with ranging errors exceeding ±0.1mm are removed based on statistical filtering algorithm. Voxel mesh downsampling is used to maintain the integrity of the three-dimensional structure while reducing the amount of data. S3. Data Fusion and Labeling: The preprocessed point cloud 3D coordinate information is mapped to the corresponding pixels of the RGB image to generate fused data of "RGB + 3D coordinates" and saved in TIFF format; the x-AnyLabeling labeling tool is used for pixel-level labeling, and the weld is divided into three categories according to its geometric shape: point (weld point), line (straight weld), and arc (arc weld). The labeling results are stored in JSON format to form a standardized dataset associated with JSON + TIFF. S4. Neural Network Architecture Optimization: Construct a MaxViT-based dual-branch multimodal neural network to extract features from RGB images and point cloud data respectively, and achieve adaptive fusion through a learnable cross-modal attention mechanism, specifically including: RGB Image Branch: Employs a hybrid structure of stacked Convolutional Neural Network (CNN) and Transformer encoder; the input image is first subjected to preliminary feature extraction through ResNet-50 convolutional backbone network to generate multi-scale two-dimensional feature maps; the high-level feature maps are flattened and projected into serialized tokens, which are then input into the multi-layer Transformer encoder; finally, this branch outputs a set of global feature representations with rich semantic information. Point cloud branch: Multi-scale 3D feature extraction is performed based on the MaxViT module; the original point cloud is divided into multiple local neighborhoods through the PointNet++ hierarchical sampling and grouping strategy; the MaxViT module is introduced in each level, and finally, the point cloud branch outputs a multi-scale 3D feature representation. Fusion layer: Introduces a learnable cross-modal attention mechanism (LCMA) given RGB features. and point cloud features This is used as the key and value, and a learnable query vector is introduced. fused features are computed through cross-modal attention: in , The concatenated key-value matrix; attention weights This reflects the relative importance of different modalities in the current task; in addition, bidirectional cross attention is introduced, enabling RGB features to focus on key structures in the point cloud, and vice versa; finally, the fused features... It is then fed into the subsequent task head for weld seam identification; S5. Adaptive Multi-Task Learning and Training: A dynamic weight allocation strategy is adopted, which automatically adjusts the loss weights of the three tasks of classification, localization, and geometric parameter regression through the neural network; a course learning mechanism is introduced during the training process, and training is carried out in stages according to the difficulty of the samples, starting with the easier ones and progressing to the more difficult ones to improve the convergence stability of the model. S6. Feature Extraction and Class Recognition: The labeled dataset is divided into training, validation, and test sets in a 7:2:1 ratio and input into the self-developed MultiVisionModels neural network. Training is performed using the AdamW optimizer (initial learning rate 1e-4, weight decay 1e-5), batch size 32, and 50 training rounds. The loss function is a weighted fusion of cross-entropy loss, Dice loss, and geometric loss. The model outputs a high-resolution feature map of the weld seam and class labels. S7. Customized Post-Processing by Category: For weld joints, after local maximum detection and NMS to remove redundant points, the center coordinates (X, Y, Z) and a confidence score of ≥0.8 are output; for straight welds, key points are filtered through interval NMS, and the start / end coordinates and mean confidence score are output; for curved welds, after nine-grid key point filtering, least squares circle fitting, and NMS processing, the center coordinates, radius, arc start coordinates, and confidence score are output. S8. Dynamic Accuracy Calibration and Feedback: A calibration model is constructed by introducing weld material parameters (metal / non-metal) and environmental parameters (temperature 0-50℃, illumination 100-10000 lux) to dynamically correct the geometric parameters output by post-processing; the corrected identification results are compared with standard weld template data, the deviation value is calculated and fed back to the loss function weight adjustment module of the MultiVisionModels model to achieve adaptive optimization of the model; S9. Data Output and Application: Generates a structured report containing workpiece ID, recognition time, category, corrected geometric parameters, and confidence level (≥0.95 is valid), which is transmitted to the robot control system via Profinet / Modbus protocol, providing millimeter-level path planning data support.

2. The method for accurate multimodal weld seam recognition based on MaxViT vision Transformer according to claim 1, characterized in that: The Retinex algorithm described in S2 uses a multi-scale Gaussian wrap function with wrap scale parameters set to 8, 16, and 32 pixels to adaptively adjust the illumination compensation intensity; the threshold for the number of neighboring points in the statistical filter is set to 15, and the threshold for outlier determination is 2.5 times the standard deviation of the data.

3. The method for accurate multimodal weld seam recognition based on MaxViT visual Transformer according to claim 2, characterized in that: The cross-modal attention fusion layer described in S4 assigns weights by calculating the cosine similarity between RGB features and point cloud features. The weight assignment ranges from 0.1 to 0.9, and the weights of the corresponding modal features are enhanced when the similarity is ≥0.

7.

4. The method for accurate multimodal weld seam recognition based on MaxViT visual Transformer according to claim 3, characterized in that: The formula for calculating the cross-entropy loss described in S6 is as follows: Where CE_Loss represents the cross-entropy loss; N represents the number of pixels in the image; i represents the i-th pixel; C represents the type of the pixel; j represents the j-th type; yij is the sign function, which takes the value 0 or 1. If the true class of the i-th pixel is j, it is 1, otherwise it is 0; pij represents the predicted probability that the i-th pixel belongs to the j-th type. The formula for calculating the Dice loss is as follows: Where Dice_Loss represents the Dice loss, |X| represents the summation over the true labels, |Y| represents the summation over the predicted results, and |X∩Y| represents the intersection of the true labels and the predicted results; For weld points and straight welds, cross-entropy loss is used; for curved welds, fusion loss is used, and the calculation formula is as follows: Where L represents the fusion loss and w represents the weighting coefficient.

5. The method for accurate multimodal weld seam recognition based on MaxViT vision Transformer according to claim 4, characterized in that: The calibration model described in S8 uses the gradient boosting tree algorithm, with material type, temperature, and light intensity as input features and geometric parameter deviation as output labels. The model is trained for ≥100 iterations.

6. A multimodal weld seam precision recognition system based on MaxViT vision transformer, characterized in that, include: Multimodal acquisition module: Composed of a 20-megapixel industrial camera, a LiDAR module and an adjustable bracket, it enables synchronous acquisition and industrial-grade storage of RGB and point cloud data; Multi-source interference preprocessing module: Configures image enhancement unit and point cloud optimization unit to perform adaptive illumination equalization, noise filtering and outlier removal, and downsampling processing respectively; Fusion annotation module: Configures a data registration engine and intelligent verification unit to complete point cloud-RGB spatial mapping and TIFF format storage, integrates x-AnyLabeling annotation tool and cross-validation module to ensure annotation accuracy ≥99%; MaxViT Feature Recognition Module: Built-in MultiVisionModels model as described in claim 1, including RGB feature extraction submodule, MaxViT point cloud feature submodule, cross-modal fusion submodule and classification submodule, with an accuracy of ≥98% for three types of weld seam recognition; The classification post-processing module includes a weld point positioning unit, a straight weld fitting unit, and an arc weld modeling unit, which respectively perform key point screening, parameter calculation, and confidence quantification for the corresponding category of welds, with geometric parameter measurement error ≤ ±0.3mm; Dynamic calibration feedback module: integrates parameter sensing unit, calibration calculation unit and model optimization unit, corrects geometric data based on material and environmental parameters and feeds back to optimize model; Intelligent output module: integrates a data formatting engine and industrial protocol interface to generate structured reports and output robot control signals through Profinet / Modbus protocol, supporting trajectory adjustment adapted to weld type.

7. The method and system for accurate multimodal weld seam recognition based on MaxViT vision Transformer according to claim 6, characterized in that: The median filter window size of the multi-source interference preprocessing module can be automatically switched according to the image noise intensity, with a switching range of 3×3-7×7 pixels; the voxel size of the voxel grid downsampling is set to 0.3-0.8mm to adapt to different precision requirements.

8. The method and system for accurate multimodal weld seam recognition based on MaxViT vision Transformer according to claim 7, characterized in that: The dynamic calibration feedback module is equipped with a temperature sensor and a light sensor, with a sampling frequency of ≥10Hz and a parameter transmission delay of ≤50ms to ensure real-time calibration.

Images