A welding seam welded area segmentation method and system based on a semantic segmentation network and deep geometric constraints

Abstract

This invention discloses a method and system for segmenting welded areas based on semantic segmentation networks and depth geometric constraints, belonging to the field of welded area extraction technology. The method includes the following steps: acquiring RGB and depth images of the workpiece to be welded; acquiring rectangular detection boxes and a preliminary mask for marking the welded areas; constructing an energy model; using the energy model to perform global pixel-level optimization on the preliminary mask to determine the mask boundary, obtaining an optimized mask; calculating the gradient field, normal direction, and local curvature of the depth image in the boundary neighborhood, interpolating the depth variation curve along the normal direction to obtain an interpolation curve; and using the interpolation curve to perform depth interpolation processing on the optimized mask to obtain the three-dimensional boundary information of the weld. The method provided by this invention simultaneously considers feature representation, geometric alignment, and engineering usability, realizing a complete boundary inference process from coarse network segmentation to fine depth geometric localization.

Description

Technical Field

[0001] This invention relates to the field of intelligent welding technology, and in particular to a method and system for segmenting the welded area of ​​a weld seam based on semantic segmentation networks and depth geometric constraints. Background Technology

[0002] In complex industrial environments, there is a technical challenge in obtaining high-precision, stable, and realistically three-dimensionally consistent welded areas. Welded areas typically exhibit weakened texture, strong localized reflections, surface inhomogeneity due to molten pool resolidification, and fragmented ablation marks during imaging. Furthermore, welded areas often occupy a small scale in actual images, being small targets with a width of only a few pixels. Existing segmentation models are prone to insufficient information, weakened boundary responses, and neglect of small target details during feature extraction.

[0003] While the existing YOLO series possesses good detection capabilities, its feature fusion method, convolution scale, and segmentation head structure are not designed for the characteristics of the welded area: "strong reflection, weak texture, small scale, and abrupt local changes." This makes it difficult for shallow details and deep semantics to effectively coordinate, causing the detection frame and mask in the welded area to easily become blurred, broken, or offset at the boundary. Furthermore, the subtle height fluctuations in the welded area and the true three-dimensional structure of the weld after formation cannot be effectively represented by two-dimensional RGB images alone, making it difficult to meet the geometric accuracy requirements of industrial tasks such as weld width measurement, excess height assessment, and welding path planning.

[0004] Therefore, it is necessary to propose a method and system for segmenting the welded area of ​​a weld seam based on semantic segmentation networks and deep geometric constraints to solve or at least alleviate some of the above-mentioned defects. Summary of the Invention

[0005] The main objective of this invention is to provide a method and system for segmenting welded areas based on semantic segmentation networks and deep geometric constraints, aiming to solve the technical problem of low segmentation and recognition accuracy of welded areas in the prior art.

[0006] To achieve the above objectives, this invention provides a method for segmenting welded areas based on semantic segmentation networks and depth geometric constraints, comprising the following steps:

[0007] S10: Acquire the RGB image and depth image of the workpiece to be welded; S20: Enhance the RGB image to obtain an enhanced image, perform image recognition on the enhanced image using a semantic segmentation model, and obtain detection output and segmentation output results. The detection output is a rectangular detection box used to mark the welded area of ​​the weld, and the segmentation output is a preliminary mask used to mark the welded area; S30: Spatially align the enhanced image and the depth image, and construct an energy model based on color consistency, depth geometric difference, and boundary smoothness constraints; S40: Use the energy model to perform global pixel-level optimization on the preliminary mask to determine the mask boundary, and obtain an optimized mask; S50: Calculate and acquire the gradient field, normal direction, and local curvature of the depth image in the boundary neighborhood, interpolate the depth change curve along the normal direction, and determine the coordinates of the points where the depth change rate reaches an extreme value or where the geometric thickness jumps as the coordinates of the real space boundary points, and obtain the interpolation curve; S60: Use the interpolation curve to perform depth interpolation processing on the optimized mask to obtain the three-dimensional boundary information of the weld.

[0008] Furthermore, using the formula Constructing an energy model, ,in, This represents the unified energy function corresponding to the energy model. This is a metric used to measure the difference between the initial segmentation probability and color consistency. The pixel coordinates of the target pixel to be optimized. For characterization Confidence level of belonging to the welded area of ​​the weld seam. For neighborhood weights, express The pixel coordinates of the neighboring pixels, and Used to describe the local relationships between pixels. , Representing pixel coordinates Pixel coordinates Depth values ​​in the depth image, The smoothing coefficient representing the depth difference; This is the value for the regularization term to suppress local noise and maintain overall boundary smoothness. , , These are the weighting coefficients.

[0009] Further, in step S20, the enhanced image is subjected to image recognition using an improved YOLO12 segmentation model, which includes a backbone network, a neck feature fusion network, and a segmentation detection head. The enhanced image is processed by the backbone network to obtain feature maps p2, p3, p4, and p5. The neck feature fusion network processes feature map p5 to obtain upsampled feature E. The upsampled feature E is concatenated with feature map p4 output by the backbone network and processed to obtain an initial fused feature map p4+. The initial fused feature map p4+ is processed to obtain upsampled feature O. The upsampled feature O is concatenated with feature map p3 output by the backbone network and processed to obtain an initial fused feature map p3+. The initial fused feature map p3+ is processed to obtain upsampled feature G. Feature G is concatenated with and processed by the feature map p2 output by the backbone network to obtain a super-fused feature map P2; after processing, the super-fused feature map P2 is processed to obtain output feature H, which is fused with and processed by the initial fused feature map p3+ to obtain a super-fused feature map P3; after convolution, the super-fused feature map P3 is concatenated with and processed by the initial fused feature map p4+ to obtain a super-fused feature map P4; after processing, the super-fused feature map P4 is processed to obtain output feature J, which is fused with and processed by the feature map p5 to obtain a super-fused feature map P5; the segmentation detection head includes a segment-MSFA segmentation head module, which is used to process the super-fused feature maps P2, P3, P4, and P5 to obtain a sequence {Z2, Z3, Z4, Z5} composed of scale-enhanced features. The sequence {Z2, Z3, Z4, Z5} simultaneously enters the mask modeling and target detection of the segmentation detection head, and outputs the detection output result and the segmentation output result.

[0010] Furthermore, the segment-MSFA segmentation head module has a corresponding edge attention weight w, wherein, g(·) is a nonlinear mapping consisting of 3×3 convolution, channel shrinkage, and channel recovery. The function is Sigmoid; the final output of the segment-MSFA segmentation head module is... .

[0011] Furthermore, the spatial resolutions of feature maps p2, p3, p4, and p5 output by the backbone network are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the RGB image, respectively.

[0012] Furthermore, the number of channels of the initial fused feature map p4+, ​​the initial fused feature map p3+, and the feature map p2 in the neck feature fusion network are 512, 256, and 128, respectively.

[0013] Furthermore, the neck feature fusion network includes a C3 module, which concatenates the P2 feature map output from the backbone network with the features passed down from the initial fusion feature map p3+ in the neck feature fusion network after an upsampling operation, and uses the C3 module to perform feature processing to obtain the super fusion feature map P2.

[0014] Furthermore, the segment-MSFA segmentation head module consists of three parallel branches: 1×1 convolution, 3×3 depthwise convolution, and 5×5 depthwise stacked convolution.

[0015] Furthermore, the present invention also provides a weld seam welded area segmentation system based on semantic segmentation network and depth geometric constraints, including an RGB-D camera and a processing device. The RGB-D camera is used to acquire RGB images and depth images of the workpiece to be welded, and the processing device is used to implement the steps of the above-mentioned weld seam welded area segmentation method based on semantic segmentation network and depth geometric constraints.

[0016] Compared with existing technologies, the welded area segmentation method based on semantic segmentation networks and depth geometric constraints provided by this invention has the following advantages:

[0017] This invention provides a method for segmenting welded areas based on semantic segmentation networks and depth geometric constraints. First, it acquires RGB and depth images of the workpiece to be welded. After enhancing the RGB image to obtain an enhanced image, a semantic segmentation model is used to perform image recognition on the enhanced image to obtain rectangular detection boxes and preliminary masks marking the welded areas. Next, an energy model is constructed based on color consistency, depth geometric differences, and boundary smoothness constraints. This energy model is then used to perform global pixel-level optimization of the initial mask, ensuring that the segmentation boundary satisfies consistency in both texture and depth structure. Then, based on the gradient field, local normal direction, and curvature distribution of the depth image, sub-pixel-level geometric interpolation is performed on the boundary points corresponding to the optimized mask along the depth gradient direction, ensuring that the final boundary is precisely aligned with the actual 3D abrupt change position in depth space, thus obtaining the 3D boundary information of the weld. This achieves a complete boundary inference process from coarse network segmentation to fine depth geometric localization. The method of this invention considers feature representation, geometric alignment, and engineering usability simultaneously. Based on enhancing small target and boundary response, it introduces a depth-guided energy model for pixel-level boundary optimization and combines geometric constraints of local normal direction and curvature to achieve sub-pixel-level boundary fine localization. This constructs a complete, deployable, and precision-controllable three-stage boundary refinement process. Considering small target recognition, boundary extraction, and spatial geometric consistency, it obtains a high-precision welded area contour, providing accurate geometric basis for subsequent applications such as weld width, reinforcement height, deformation, and defect analysis. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a method for segmenting welded areas based on semantic segmentation networks and depth geometric constraints in one embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of the improved YOLO12 segmentation model;

[0021] Figure 3This is a schematic diagram of the output results of the improved YOLO12 segmentation model in one embodiment of the present invention; where a is a schematic diagram of bounding box accuracy, the horizontal axis is the training epoch, and the vertical axis is the bounding box accuracy, the vertical axis represents how many of the bounding boxes predicted as positive are true; b is a schematic diagram of overall bounding box accuracy, the horizontal axis is the training epoch, and the vertical axis is the mask segmentation accuracy, the vertical axis refers to the proportion of instances of the mask that the model predicts as welded regions that actually belong to the welded regions; c is a schematic diagram of mask segmentation accuracy, the horizontal axis is the training epoch, and the vertical axis is the bounding box mAP50, representing the average mask segmentation accuracy of the detection task under the condition of IoU=0.5; d is a schematic diagram of overall mask segmentation accuracy, the horizontal axis is the training epoch, and the vertical axis is the bounding box mAP50, representing the overall average mask segmentation accuracy of the detection task under the condition of IoU=0.5.

[0022] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0023] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0025] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a specific posture. If the specific posture changes, the directional indication will also change accordingly.

[0026] Furthermore, the use of terms such as "first" and "second" in this invention is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. When the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed by this invention.

[0027] Research reveals that current technologies for segmentation and detection of welded areas generally follow two technical routes. One approach relies primarily on two-dimensional visual information, using deep neural networks for pixel-level segmentation or target detection. Typical techniques include semantic segmentation methods based on U-Net and FCN, as well as YOLO-based detection and segmentation models. Existing methods typically enhance the network's ability to represent small-scale targets and boundary details by improving the backbone network structure, introducing multi-scale feature fusion mechanisms, or employing attention modules. The other approach introduces depth information as a supplementary modality into the system. This involves cascading RGB-D images as input, using RGB images to guide depth map restoration, employing depth-guided optimizations such as GraphCut or DenseCRF, or constructing a 3D reconstruction and segmentation process using point cloud data and voxel data. This leverages the height variations and thickness abrupt changes reflected in the depth field to improve the accuracy of boundary localization and shape restoration. Existing methods have several drawbacks: relying solely on 2D RGB networks is prone to boundary blurring, false connectivity, or breaks under conditions of strong reflectivity on metal surfaces, thin textures, or high noise. Furthermore, the general multi-scale / attention module design, geared towards natural image tasks, cannot maintain stable responses in small-scale welded areas, leading to offsets or discontinuities between the detection box and mask boundaries in real-world industrial samples. Simply feeding the depth map as an additional channel along with the RGB values ​​into the convolutional network or stitching it together in shallow layers often smooths out the subtle depth gradients during the stacking of multiple depth convolutions, preventing the explicit representation of geometric abrupt changes in depth. The mask cannot be easily preserved and utilized, making it difficult to directly use for precise boundary geometry correction; traditional morphology-based post-processing or fixed-neighborhood-based smoothing methods for mask correction have poor adaptability, rely on manually set structural elements and thresholds, and cannot simultaneously take into account the shape diversity of different welded workpieces and different shooting conditions, often resulting in over-smoothing or failure to eliminate false boundaries; fusion methods based on 3D point clouds or voxels can provide more direct geometric information and achieve good results in some work, but their computational complexity, the engineering overhead of point cloud registration / densification, and dependence on the accuracy of sensing equipment pose practical obstacles to their real-time performance and deployment in conventional industrial sites. In summary, existing 2D segmentation methods lack boundary stability under weak texture and strong reflective conditions; existing simple multimodal fusion methods cannot preserve key geometric features such as depth gradients; and existing point cloud or voxel-based 3D methods are computationally expensive and rely on hardware precision, making them difficult to deploy in engineering. For the segmentation and recognition of welded areas, it is difficult to simultaneously solve key technical problems such as maintaining pixel-level accuracy of small-scale welded area boundaries, explicitly preserving depth geometric information in the network or post-processing for real boundary alignment, and real-time performance and resistance to illumination interference in industrial scenarios within a unified technical system.

[0028] Please refer to the appendix. Figure 1 , Figure 2 as well as Figure 3 This invention provides a method for segmenting the welded area of ​​a weld seam based on a semantic segmentation network and depth geometric constraints, comprising the following steps:

[0029] S10, acquire the RGB image and depth image of the part to be welded;

[0030] S20, the RGB image is enhanced to obtain an enhanced image, and the enhanced image is recognized using a semantic segmentation model to obtain detection output results and segmentation output results. The detection output results are rectangular detection boxes used to mark the welded areas of the weld, and the segmentation output results are preliminary masks used to mark irregular welded areas.

[0031] S30, Spatially align the enhanced image with the depth image, and construct an energy model based on color consistency, depth geometric difference, and boundary smoothness constraints;

[0032] S40, the energy model is used to perform global pixel-level optimization on the preliminary mask to determine the mask boundary, and an optimized mask is obtained;

[0033] S50, calculate and obtain the gradient field, normal direction and local curvature of the depth image in the boundary neighborhood, interpolate the depth change curve along the normal direction, determine the coordinates of the location point where the depth change rate reaches the extreme value or the geometric thickness jumps as the coordinates of the real space boundary point, and obtain the interpolation curve;

[0034] S60, the optimized mask is subjected to depth interpolation using the interpolation curve to obtain the three-dimensional boundary information of the weld.

[0035] The weld seam segmentation method based on semantic segmentation network and depth geometric constraints provided by this invention first acquires the RGB image and depth image of the workpiece to be welded. After enhancing the RGB image to obtain an enhanced image, the semantic segmentation model is used to perform image recognition on the enhanced image to obtain a rectangular detection box and a preliminary mask marking the weld seam area. Then, an energy model is constructed based on color consistency, depth geometric difference, and boundary smoothness constraints. The energy model is used to perform global pixel-level optimization on the initial mask so that the segmentation boundary satisfies consistency in both texture and depth structure. Then, based on the gradient field, local normal direction, and curvature distribution of the depth image, sub-pixel-level geometric interpolation is performed on the boundary points corresponding to the optimized mask along the depth gradient direction so that the final boundary is accurately aligned with the real three-dimensional abrupt change position in the depth space, thereby obtaining the three-dimensional boundary information of the weld seam. This realizes a complete boundary inference process from coarse network segmentation to fine depth geometric localization. The method of this invention considers feature representation, geometric alignment, and engineering usability simultaneously. Based on enhancing small target and boundary response, it introduces a depth-guided energy model for pixel-level boundary optimization and combines geometric constraints of local normal direction and curvature to achieve sub-pixel-level boundary fine localization. This constructs a complete, deployable, and precision-controllable three-stage boundary refinement process. Considering small target recognition, boundary extraction, and spatial geometric consistency, it obtains a high-precision welded area contour, providing accurate geometric basis for subsequent applications such as weld width, reinforcement height, deformation, and defect analysis.

[0036] Optionally, in the solution of the present invention, type labels are marked on the RGB image, the type labels include soldered area labels and unsoldered area labels, and the RGB image corresponding to the soldered area label is enhanced to obtain an enhanced image; a dataset is constructed based on the enhanced image and type labels in the training data, and the model is trained, validated and tested to obtain a semantic segmentation model.

[0037] In one specific embodiment of the present invention, the Retinex algorithm is used for enhancement processing. The basic theory of the Retinex algorithm is that the color of an object is determined by its ability to reflect long-wave (red), medium-wave (green), and short-wave (blue) light, rather than by the absolute value of the reflected light intensity. The color of the object is not affected by the non-uniformity of illumination and has consistency. That is, Retinex is based on color consistency (color constancy). The Retinex algorithm used in the scheme of the present invention is different from the traditional linear and non-linear methods that can only enhance a certain type of image features. Retinex can achieve a balance in three aspects: dynamic range compression, edge enhancement, and color constancy, and can adaptively enhance various types of images.

[0038] Understandably, a depth map is a grayscale image that contains distance information for each pixel from the camera, used to describe the three-dimensional structure of a scene.

[0039] In an optional embodiment of the present invention, the RGB-D camera fixed on the robotic arm is calibrated using hand-eye calibration to obtain the coordinate transformation relationship between the Camera Coordinate System (CCS) and the Tool Coordinate System (TCS), i.e., the hand-eye calibration matrix. The pose transformation matrix in the robot controller parameters is read in real time to represent the coordinate transformation relationship between the TCS and the robot base coordinate system (BCS). Subsequently, the intrinsic parameter matrices, extrinsic parameter rotation matrix, and translation matrix of the RGB camera and depth camera are calibrated using the Zhang Zhengyou checkerboard calibration method. Next, RGB and depth images of a self-made complex 3D welded component are captured from different angles using the calibrated camera, and irregular polygonal annotations of welded and unwelded areas are added to the RGB images using Labelme. In specific implementation, to address the common problems of strong reflection, local overexposure, and uneven illumination in metal weld images, an image preprocessing method based on Retinex theory is used to suppress illumination and enhance details in the acquired RGB images. In the actual training process, in order to expand the dataset, data augmentation techniques were used to perform enhancement operations such as rotation, flipping, brightness and contrast adjustment, and noise addition on the original dataset. The final images and their corresponding labels were divided into training set and validation set in a ratio of 8:2.

[0040] In one specific embodiment of the present invention, the folder of RGB images is input into the Labelme image annotation tool, and labeled with Labelme irregular polygons. The weld seam area is labeled as "unweld", and the welded area is labeled as "weld". For the segmentation of the welded area, the Color Recovery Multi-Scale (MSRCR) algorithm in the Retinex algorithm is used to preprocess the RGB images with the "weld" label to solve the problem of dark images and metallic reflections. The preprocessed images are then subjected to enhancement operations such as rotation, flipping, brightness and contrast adjustment, and noise addition. All the final images and their corresponding labels are divided into training and validation sets in a ratio of 8:2 for model training, validation, and testing to obtain the semantic segmentation model.

[0041] Furthermore, using the formula Constructing an energy model, ,in, This represents the unified energy function corresponding to the energy model. This is a metric used to measure the difference between the initial segmentation probability and color consistency, representing the pixel coordinates of the target pixel to be optimized. For characterization Confidence level of belonging to the welded area of ​​the weld seam. , For neighborhood weights, express The pixel coordinates of the neighboring pixels, and Used to describe the local relationships between pixels. , Representing pixel coordinates Pixel coordinates Depth values ​​in the depth image, The smoothing coefficient representing the depth difference; This is the value for the regularization term to suppress local noise and maintain overall boundary smoothness. , , These are the weighting coefficients. In the scheme of this invention, after accurately segmenting the welded area based on the depth map information corresponding to the depth image and obtaining the preliminary mask of the welded area of ​​the semantic segmentation model, in order to overcome the problems of boundary jaggedness, local noise, and sensitivity to metal reflection caused by relying solely on RGB features, depth information is further introduced to construct a joint energy model to optimize the segmentation boundary at the pixel level. Specifically, the RGB image and the depth image are spatially registered through coordinate transformation relationship, and the color confidence, geometric thickness change, and local surface morphology are obtained under the same pixel coordinates. Using the preliminary mask M as the optimization variable, a unified energy function containing data consistency terms, depth geometric constraint terms, and smoothing regularization terms is constructed. Conventional optimization techniques such as graph cut and conditional random fields are used to optimize the energy function. The solution is minimized to obtain the pixel-level optimal segmentation result that conforms to the RGB image and depth geometry.

[0042] In one specific embodiment of the present invention Let Ω represent the pixel positions in the image domain, M represent the mask variable, and M represent the entire pixel mapping. This represents the value of the mask at pixel p; The sigmoid output S(p) is used as the foreground probability, and S(p) is the foreground confidence output by the segmentation network. Indicates the optimization of mask M at the pixel The value at the corresponding position, , Used to characterize this pixel The confidence score for belonging to the welded area is a pixel-level label variable solved in the energy optimization model, i.e., the final output; Measuring the mask at the pixel level Consistency between the selected values ​​and the RGB (or network segmentation confidence) information is the main driving force for ensuring that the results are consistent with the initial segmentation and color texture evidence from the network. Initially, a logarithmic form is adopted, which is natural for probabilities and facilitates direct initialization using model output; neighborhood interaction weights. Inconsistency penalty, , Data item weights control the priority of consistency with network / color evidence; The depth consistency term weight controls the strength of depth-driven boundary correction; The regularization term weight controls the intensity of boundary smoothing and denoising.

[0043] In the present invention, after obtaining the energy-optimized mask boundary based on the energy function, sub-pixel-level boundary reconstruction is performed based on the continuity characteristics of the depth field to further meet the geometric accuracy requirements in scenarios such as weld thickness, reinforcement height, and bevel formation. Specifically, since the energy optimization result is still limited by the pixel grid, its boundary can only fall at the grid points of discrete pixels, which cannot accurately describe the real weld geometry. By calculating the gradient, normal direction, and local curvature of the depth image in the boundary neighborhood, the depth change curve is interpolated along the normal direction to determine the position where the depth change rate reaches the extreme value or the geometric thickness jumps as the coordinates of the real spatial boundary point, realizing continuous boundary positioning across pixels. Through this geometric interpolation, the discrete pixel-level boundary can be elevated to a continuous boundary consistent with the real weld formation, thereby obtaining a high-precision welded area contour, providing accurate geometric basis for subsequent applications such as weld width, reinforcement height, deformation, and defect analysis.

[0044] Please refer to this again. Figure 2 Furthermore, in step S20, the enhanced image is subjected to image recognition using an improved YOLO12 segmentation model, which includes a backbone network, a neck feature fusion network, and a segmentation detection head; the enhanced image is processed by the backbone network to obtain feature maps p2, p3, p4, and p5.

[0045] The neck feature fusion network processes the feature map p5 to obtain upsampled feature E. The upsampled feature E is then concatenated with the feature map p4 output by the backbone network and processed to obtain an initial fused feature map p4+. The initial fused feature map p4+ is then processed to obtain upsampled feature O. The upsampled feature O is then concatenated with the feature map p3 output by the backbone network and processed to obtain an initial fused feature map p3+. The initial fused feature map p3+ is then processed to obtain upsampled feature G. The upsampled feature G is then concatenated with the feature map p3 output by the backbone network and processed to obtain an initial fused feature map p3+. The feature map p2 is concatenated and processed to obtain a super-fused feature map P2; the super-fused feature map P2 is processed to obtain an output feature H, which is then fused with the initial fused feature map p3+ and processed to obtain a super-fused feature map P3; the super-fused feature map P3 is convolved and concatenated with the initial fused feature map p4+ and processed to obtain a super-fused feature map P4; the super-fused feature map P4 is processed to obtain an output feature J, which is then fused with the feature map p5 and processed to obtain a super-fused feature map P5;

[0046] The segmentation detection head includes a segment-MSFA segmentation head module. The segmentation detection head is used to process super-fusion feature maps P2, P3, P4, and P5 to obtain a sequence {Z2, Z3, Z4, Z5} composed of scale-enhanced features. The sequence {Z2, Z3, Z4, Z5} is simultaneously entered into the mask modeling and target detection of the segmentation detection head, and the detection output result and segmentation output result are output.

[0047] Furthermore, the segment-MSFA segmentation head module has a corresponding edge attention weight w, wherein, g(·) is a nonlinear mapping consisting of 3×3 convolution, channel shrinkage, and channel recovery. The function is Sigmoid; the final output of the segment-MSFA segmentation head module is... .

[0048] Furthermore, the spatial resolutions of feature maps p2, p3, p4, and p5 output by the backbone network are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the RGB image, respectively.

[0049] Furthermore, the number of channels of the initial fused feature map p4+, ​​the initial fused feature map p3+, and the feature map p2 in the neck feature fusion network are 512, 256, and 128, respectively.

[0050] Furthermore, the neck feature fusion network includes a C3 module, which concatenates the P2 feature map output from the backbone network with the features passed down from the initial fusion feature map p3+ in the neck feature fusion network after an upsampling operation, and uses the C3 module to perform feature processing to obtain the super fusion feature map P2.

[0051] Furthermore, the segment-MSFA segmentation head module consists of three parallel branches: 1×1 convolution, 3×3 depthwise convolution, and 5×5 depthwise stacked convolution.

[0052] In the solution of this invention, the improved YOLOv12 segmentation model is used for processing. The improved YOLOv12 segmentation model (network) includes three parts: backbone network, neck feature fusion network, and segmentation detection head, and the network modules are interconnected.

[0053] In practice, the input RGB image first enters the backbone network of the improved YOLOv12 segmentation model. This backbone network, centered on a multi-layered Conv, C3k2, and A2C2f structure, extracts multi-scale representation features in a progressively deeper order. The first step involves the first Conv module, which uses a 3×3 convolutional kernel with a stride of 2. Its function is to initially construct the feature channels (expanding the RGB image from 3 channels to 64 channels) and perform the first spatial compression (reducing the image size to half, generating feature map p1). The output feature map p1 retains the basic details of the image while reducing the complexity of subsequent calculations. Next, feature map p1 is input into the second Conv module, which uses a 3×3 convolutional kernel with a stride of 2 to further expand the feature channels to 128 channels and compress the spatial size to one-quarter, ultimately outputting a shallow, high-resolution feature map. Feature map Focusing on fine-grained details of small targets provides core foundational features for subsequent small target detection; subsequently, feature maps... The first C3k2 module is input. A C3k2 module consists of an input convolution, two parallel convolutional sequences, and an output convolution. The input convolution unifies the feature dimensions, while the two parallel convolutional sequences extract local features from different receptive fields (one focusing on detail preservation, the other on simple semantic integration). The output convolution then fuses and summarizes the two parallel features, ensuring no loss of feature maps. Under the premise of detail, mid-shallow semantic features are constructed to enhance the expressive power of the features. The final output is a mid-shallow enhanced feature map p2 with 256 channels. Feature map p2 is then input into the third Conv module. A 3×3 convolution kernel with a stride of 2 maintains the feature channels at 256 channels and compresses the spatial size to 1 / 8 to generate the feature map. The role of this module is to enhance the abstraction of features based on the shallow semantics, laying the foundation for medium-level target detection. Feature map p3 is then fed into the second C3k2 module, which has the same structure as the first C3k2 module. Through the synergistic effect of input convolution, dual parallel convolution sequences, and output convolution, it further deepens the semantic information of the p3 feature, integrates more local and global related features, and outputs a mid-level semantically enhanced feature map p3 with 512 channels, improving the feature recognition of medium-level targets. After that, feature map p3 is input into the fourth Conv module. The setting of 3×3 convolution kernel and stride 2 keeps the feature channels at 512 channels, compressing the spatial size to 1 / 16, and generating the feature map. The core function of this module is to balance the level of abstraction of features with computational efficiency, providing mid-level semantic support for large object detection; feature maps Next, we move to the first A2C2f module. A2C2f is a residual-enhanced feature extraction module based on C2f. Its core is replacing the traditional module with a "double ABlock concatenation group." ABlock uses AAttn region attention (calculating attention across regions of the feature map, balancing dependency capture and efficiency) and MLP for feature enhancement. It first reduces the dimensionality of the input features through a 1×1 convolution to obtain initial hidden features, then continuously enhances them through multiple ABlock groups and concatenates all features. Finally, it increases the dimensionality through a 1×1 convolution, combining a learnable layer scaling factor with the original input for residual fusion. This achieves efficient feature extraction while enhancing key information, maintaining a feature channel count of 512. The output enhanced feature map p4 possesses both semantic depth and detailed relevance. Subsequently, feature map p4 is input into the fifth Conv module. A 3×3 convolution kernel and stride of 2 expand the feature channels to 1024 channels, compressing the spatial size to 1 / 32, generating the feature map. This module maximizes the level of feature abstraction, extracts global semantic information from the image, and provides high-level feature support for ultra-large object detection; feature maps Finally, the data is fed into the second A2C2f module, which also employs a multi-branch, attention-based structure. This module focuses on enhancing the integration of global semantic features, suppressing background noise through an attention mechanism, and focusing on high-level semantic associations of core target regions in the image. It outputs a high-level globally enhanced feature map p5 with 1024 channels. At this point, the backbone network completes multi-scale feature extraction from shallow to deep, outputting core feature maps at four scales: p2 (1 / 4), p3 (1 / 8), p4 (1 / 16), and p5 (1 / 32). Each convolution in the backbone network generates a feature map, which is then enhanced by either the C3K2 or A2C2f module before being input into the neck network for feature fusion.

[0054] In practice, the multi-scale feature maps p2, p3, p4, and p5 output by the backbone network are fed into the neck feature fusion network. The neck feature fusion network adopts an improved U-shaped structure to achieve accurate fusion of high-level semantics and low-level details. First, the p5 high-level feature map (1024 channels) output by the backbone network enters the first nn.Upsample module. This module uses nearest neighbor interpolation to upsample the feature map by a factor of 2, restoring the spatial size to 1 / 16. Its purpose is to improve the spatial resolution of the features and supplement the detailed information of the high-level features, outputting the upsampled feature E. Feature E and the p4 feature map output by the backbone network are concatenated by the Concat module along the channel dimension (fusion of high-level semantics and mid-level details). The concatenated feature map is input into the first A2C2f module. This module further integrates and fuses the features through a multi-branch structure, suppresses redundant information, and outputs a p4+ fused feature map (11 layers) with 512 channels. This feature map retains the semantic features of the large target and supplements the mid-level details. Then, the p4+ fused feature map enters the second nn.Upsample module, which upsamples it by a factor of 2 using nearest neighbor interpolation. The first module, with a spatial size of 1 / 8, outputs an upsampled feature O. This upsampled feature O is then concatenated with the p3 feature map output from the backbone network via a Concat module (fusing mid-level semantics with mid-to-low-level details). The concatenated feature map is then input into the second A2C2f module for further feature optimization, outputting a p3+ fused feature map (14 layers) with 256 channels, providing accurate semantic and detail fusion features for medium-sized target detection. To enhance small target detection, the p3+ fused feature map enters the third nn.Upsample module, upsampled by 2 times to 1 / 4 of the spatial size, outputting an upsampled feature G. This feature G is then concatenated with the p2 feature map (core detail features of small targets) output from the backbone network via a Concat module, fusing fine-grained details and mid-to-low-level semantics of small targets. The concatenated feature map is then input into the C3 module (C3 is chosen here to avoid dimensionality mismatch due to specific dimensionality requirements of A2C2f). The structure consists of an input convolution, two lightweight convolutional branches, and an output convolution. It enhances the recognition and detail preservation of shallow texture features in a residual manner and outputs a P2 final fusion feature map (17 layers) with 128 channels. The P2 final fusion feature map is the core dependency for the detection of very small objects and retains the richest fine-grained information such as the edges and textures of small objects.

[0055] In practical implementation, the process enters the downsampling fusion stage (supplementing high-level semantics to low-level features): The final fused feature map of P2 is input into the sixth Conv module. A 3×3 convolutional kernel and a stride of 2 expand the channels to 256, compressing the spatial size to 1 / 8, and outputting feature H. Feature H is concatenated with the previous p3+ fused feature map (14 layers) through the Concat module, fusing small target details and mid-level semantics. After concatenation, it is input into the third A2C2f module to optimize the feature integration effect, outputting the final fused feature map of P3 (20 layers) with 256 channels, adapting to the needs of small target detection. The final fused feature map of P3 is then input into the seventh Conv module. A 3×3 convolutional kernel and a stride of 2 expand the channels to 512, compressing the spatial size to 1 / 16, and outputting feature I. Feature I is then concatenated with the p4 fused feature map (11 layers) through the Concat module. The `at` module concatenates the target details and high-level semantics, and then inputs the concatenated data into the fourth `A2C2f` module to further enhance the recognizability of large target features. The output is a final fused feature map (23 layers) with 512 channels, adapted for medium target detection. The final fused feature map of P4 enters the eighth `Conv` module, where a 3×3 convolution kernel and a stride of 2 maintain the number of channels at 512, compressing the spatial size to 1 / 32, and outputting feature J. Feature J is concatenated with the previous high-level enhanced feature map of p5 (8 layers) through the `Concat` module to fuse the large target details and global semantics. The concatenated data is then input into the third `C3k2` module (this module enables residual connections to improve gradient propagation efficiency), which maximizes the semantic feature expression of ultra-large targets. The output is a final fused feature map (26 layers) with 1024 channels, adapted for large target detection.

[0056] In practice, the four final fused feature maps (P2-17 layers, P3-20 layers, P4-23 layers, and P5-26 layers) output by the neck feature fusion network are input into the Segment-MSFA module, and the final output is an instance segmentation mask and object detection box with the same size as the input RGB image.

[0057] Understandably, please refer to this again. Figure 2 In the present invention, the main improvements of the improved YOLO12 segmentation model compared to the existing YOLO12 network include:

[0058] Improvements to the backbone network layer and neck feature fusion network include: on the basis of the standard PAN structure, a high-resolution p2 feature map is extracted from the second layer of the backbone network; through a single upsampling operation, the p2 feature map is concatenated with the features passed down from higher layers in the neck feature fusion network, and the C3 module is used for feature processing to finally generate a new predictive feature map P2 specifically for small object detection. The C3 module addresses the issue that conventional advanced feature extraction modules (such as A2C2f) often incorporate complex attention mechanisms (such as ABblock) or grouped convolutions, which impose strict constraints on the channel dimension of input features for computational efficiency and performance. ABblock, for example, requires the network input channels to be a multiple of 32. Directly applying such modules to heterogeneous feature bodies in the P2 layer can lead to structural incompatibility or forced dimensionality pruning, thereby compromising valuable spatial details preserved for small targets. The C3 module, however, has no special restrictions on the number of input channels and can seamlessly process heterogeneous features of arbitrary channel dimensions generated after P2 layer concatenation, avoiding information loss. Furthermore, the C3 module is generally more lightweight than A2C2f, which includes complex attention mechanisms. Using C3 on high-resolution feature maps, which have the highest computational cost, achieves effective optimization of overall computational resources without sacrificing small target detection performance. The proposed layered adaptive feature extraction strategy is based on the idea that the deep and middle layers (p3, p4) of the network have relatively regular features, making them suitable for using the computationally intensive A2C2f module with complex attention to maximize the extraction of semantic information. In the shallowest layer (p2) of the network, which is most sensitive to details, the innovative C3 module with a more flexible structure and greater versatility is adopted to solve the heterogeneity problem of shallow features in multi-scale feature fusion. By strategically deploying the C3 module in the p2 layer, high-fidelity extraction of detailed information of small targets is achieved, while ensuring the generalization and computational efficiency of the entire network.

[0059] Understandably, in the fields of object detection and instance segmentation, YOLO series models generally employ multi-scale feature fusion structures to enhance object representation capabilities. Specifically, YOLOv8 typically uses C2f for lightweight cross-layer fusion on mid-to-deep features such as P3 and P4; YOLOv11 further introduces C3k2, enhancing mid-to-deep semantic modeling capabilities through large kernel convolutions and multi-branch structures; YOLOv12 and related improved networks tend to use A2C2f modules with attention mechanisms to achieve stronger channel selection and context aggregation in deep features. Although the aforementioned publicly available techniques have achieved significant results in improving semantic feature representation, expanding the effective receptive field, and optimizing global modeling capabilities, the design of these modules is based on the default premise that the input feature channel structure is regular, the spatial resolution is low, and the semantics are stable. Therefore, they are widely used for deep feature maps such as P3 to P5, but rarely for processing shallow, high-resolution feature maps. Within this mainstream technical framework, shallow P2 features, due to their highest spatial resolution, low channel dimension, and significant channel heterogeneity after concatenation with upsampled features, are structurally inconsistent with the design assumptions of mid-to-deep fusion modules such as C2f, C3k2, and A2C2f. Specifically, the multi-branch recombination structure of C2f and C3k2 leads to a sharp increase in computational cost under high-resolution input; the large kernel convolution in C3k2 further amplifies the computational cost of shallow convolutions and affects the stability of details; the attention block built into A2C2f requires the number of input channels to meet a fixed multiple for grouping, otherwise it will cause structural conflicts such as attention segmentation misalignment and grouped convolution dimension mismatch. Based on this situation, existing technologies generally only retain simple convolutions in the p2 layer or do not include them in the prediction branch, thus forming a technical bias within the industry that "shallow high-resolution features are difficult to be compatible with complex feature fusion modules." This invention, through a systematic analysis of the underlying causes of the aforementioned biases, points out that the root of the problem lies not in the P2 features themselves, but in the fact that existing feature fusion modules are designed for mid-to-deep semantic features. Their structures implicitly assume stability in the number of channels, alignment of convolutional structures, and attention block partitioning rules, making them difficult to adapt to the non-uniform channel structure formed after concatenating shallow, high-resolution features. Based on this understanding, this invention proposes a C3 module in the P2 fusion path, which has a simpler structure, no channel constraints, low computational cost, and is more detail-preserving. This module does not rely on grouped convolutions or attention mappings, has no fixed requirements on the input channel structure, and can stably handle heterogeneous inputs formed by concatenating shallow p2 features and upsampled features. Simultaneously, its shallow convolution depth results in a computational cost far lower than C3k2 and A2C2f when deployed on high-resolution feature maps, ensuring the real-time performance and deployment efficiency of the entire network.By introducing the C3 module into the shallow path and continuing to use structures such as C2f, C3k2, and A2C2f in mid-to-deep semantic fusion positions that are suitable for their characteristics, this invention breaks through the inherent limitations of the existing YOLO series on the use of shallow features. It enables high-resolution P2 features to participate in the prediction head construction without structural conflicts or additional computational burden, thereby significantly improving the performance of small object detection, the accuracy of instance segmentation boundaries, and the fidelity of shallow details, while maintaining the stability and computational efficiency of the overall network architecture.

[0060] The improvements to the segmentation detection head include: adopting a segment-MSFA segmentation head module. The MSFA (Multi-Scal Rendering with Edge Attention) segmentation head module consists of three parallel branches, which use 1×1, 3×3 depthwise convolutions and 5×5 depthwise stacked convolutions to capture multi-scale features. The output features are concatenated in the channel dimension and then fused by a 1×1 convolution to obtain a comprehensive feature map. An edge attention branch is introduced into the fused features, which uses convolution and sigmoid activation to generate an edge response weight map. The final output is enhanced through an adaptive linear enhancement method to achieve feature enhancement and boundary correction, thereby obtaining a smoother segmentation result in the details of the weld edge. The MSFA segmentation head module proposed in this invention is not a general stack of multi-scale convolutions, but is specifically designed for the structural characteristics of weld seam images. Firstly, in terms of multi-scale representation, this invention uses a fixed three-scale combination of "1×1 linear convolution + depthwise 3×3 texture convolution + cascaded depthwise 3×3 equivalent 5×5 structure," which is then output to the fusion module to integrate features from different receptive fields. This structural design purpose is significantly different from the general scale expansion of traditional Inception-type algorithms. Secondly, in the edge attention branch of this invention, the edge attention weight is... , The final output of the MSFA segmentation head module is: , Edge attention path output adopts " The controlled linear enhancement method of this invention enhances the boundaries of fused features. Unlike the simple multiplicative weights of existing attention mechanisms, it does not suppress the main features, ensuring the continuity and stability of the slender weld seam boundaries during enhancement. Furthermore, the MSFA segmentation head module of this invention, as a lightweight pluggable structure, is embedded in the YOLO segmentation detection head, allowing the detection branch, prototype generation, and mask coefficient branch to infer based on the edge-enhanced features, forming an integrated small target segmentation enhancement framework, rather than independent convolutional branches. This invention is significantly different from existing technologies in terms of structural logic, boundary representation, and task adaptability. The MSFA segmentation head module is introduced to enhance the spatial details and local texture representation of multi-scale feature maps. Based on the input feature map, the MSFA segmentation head module sequentially extracts features through a set of convolutional transformation subunits composed of different receptive fields. The first subunit is a one-dimensional convolutional transformation, consisting of a one-by-one convolution, a normalization layer, and a nonlinear activation layer, mainly used to adjust the channel distribution and extract local responses. The second subunit consists of depthwise separable three-by-three convolutions and one-by-one convolutions in sequence, enhancing local structural features by separating spatial convolution and channel convolution. The third subunit consists of two consecutive depthwise convolutions and one-by-one convolutions, used to construct a relatively larger effective receptive field. The outputs of the three subunits are concatenated along the channel dimension and then fed into a one-by-one convolution and a normalization layer to complete multi-scale fusion, thereby generating the fused spatial enhancement feature. Subsequently, this fused feature is input into the boundary awareness branch. In the edge enhancement path, the fused feature O is first input into a nonlinear mapping g(·) consisting of three-by-three convolutions, channel compression, and channel recovery. This mapping is used to extract high-frequency information such as local gradient abrupt changes and boundary texture changes. Its output is processed by the Sigmoid function to obtain normalized edge attention weights. Unlike traditional attention mechanisms that directly perform pointwise multiplicative inhibition on features using w, MSFA employs... The final output is generated through a linear enhancement method. Because... The value of is limited to between zero and one. This form ensures that the lower bound of the enhancement operation is always F, while the enhancement strength is determined according to... The size is locally adaptively adjusted, so that only the boundary region is amplified without weakening the main structural information. Through this "guaranteed gain" mechanism, the module emphasizes edge details while avoiding the feature suppression problem that may occur in conventional attention, so that the extracted features have both boundary sensitivity and main stability.

[0061] In the scheme of this invention, in the multi-scale segmentation detection head, four sets of feature maps P2, p3, p4, and p5 from the neck feature fusion network at different resolutions are respectively used as inputs and fed into the MSFA segmentation head module corresponding to their channel number; for input features of any scale... The MSFA segmentation head module internally feeds the same input into a 1x1 convolution branch, a 3x3 depthwise separable convolution branch, and an equivalent large receptive field branch formed by stacking two levels of depthwise convolutions to extract fine-grained, mid-scale, and wide-domain structural information. The outputs of the three branches are concatenated along the channel dimension and fused by a 1x1 convolution to form the enhanced feature output at that scale. The sequence {z2, z3, z4, z5}, composed of enhancement features at all scales, simultaneously enters both the mask modeling and object detection paths of the segmentation detection head. The shallowest enhancement feature z2 serves as input for mask prototype generation, entering the Proto module. Proto generates a fixed number of basic mask prototypes in the spatial domain of z2 through multiple convolutional layers. These prototypes serve as a shared basis for instance mask decoding. Meanwhile, enhancement features at all scales... The corresponding coefficient prediction branch is input separately. This branch consists of two levels of 3x3 convolutions and one level of 1x1 convolution. Through spatial compression, mask combination coefficients corresponding to each scale are generated. The coefficient maps generated at different scales are spliced ​​in the spatial dimension to form a unified mask coefficient matrix. This matrix is ​​multiplied with the prototype features output by the Proto module in a linear combination manner, thus completing the reconstruction of the instance mask at the pixel level. The object detection branch directly receives the enhanced multi-scale feature sequence {z2, z3, z4, z5} and generates bounding boxes, class probabilities, and other structured prediction information for each scale through the Detect module. Thus, the multi-scale segmentation detection head achieves collaborative modeling of enhanced feature extraction, mask prototype construction, coefficient generation, and detection output within a unified framework.

[0062] In this invention, the Segment_MSFA segmentation detection head includes a parallel enhancement structure that takes multi-scale features as input. Its core is to set independent MSFA enhancement modules for feature maps from different network stages to perform multi-scale texture modeling and edge response enhancement for those scale features. The enhanced features are used simultaneously in the detection sub-network and the mask generation sub-network, ensuring that the detection results and instance segmentation process share a consistent multi-scale enhanced feature representation. The Segment_MSFA segmentation detection head includes a prototype generation path and a mask coefficient generation path. The prototype generation path is based only on the shallowest feature map, which is first enhanced by the MSFA segmentation head module and then a fixed number of mask prototypes are extracted by the prototype generation network. The mask coefficient generation path is based on enhanced features at all scales, extracting corresponding mask coefficients through convolutional modules and concatenating them in the spatial dimension to form a multi-scale joint mask coefficient matrix. The segmentation result is obtained by linearly combining the enhanced prototypes and coefficients and then upsampling. In the detection part, this structure uses multi-scale features processed by MSFA to replace the original features as input to generate target bounding boxes and class confidence scores, thereby achieving a tight coupling mechanism between the detection and segmentation tasks by sharing edge enhancement features. This coupling structure can significantly improve the detection recall and boundary consistency of small-scale weld regions. During the training phase, mask-related losses are calculated only for predicted instances that match the real targets; during the inference phase, several candidate targets can be selected for mask reconstruction based on confidence scores to reduce computational and storage overhead. This structure allows the multi-scale edge enhancement characteristics of MSFA to be embedded in both detection and segmentation paths simultaneously, achieving consistent optimization at the feature level and improving the boundary accuracy, structural integrity, and adaptability to small targets in welded regions.

[0063] In this invention, an improved YOLO12 segmentation model is used to output detection results and segmentation results. The detection output is rectangular bounding boxes. The model predicts the position of the target in an anchor-free manner, and the output format is [x1, y1, x2, y2] or center point form. After decoding, the actual pixel coordinates and class probabilities (ClassScores) are obtained. The class confidence score is output for each rectangular bounding box. In this task, the number of classes is the welded area of ​​the weld. The segmentation output is an instance mask.

[0064] The improved YOLOv12 segmentation network proposed in this invention exhibits a stable convergence trend and good training characteristics on the welded area dataset. Whether it is the precision and recall of the detection box or the precision and recall of the mask segmentation, they all show typical curve characteristics of rapid increase in the early iteration stage and then stable convergence at a high level. This indicates that the network can effectively learn the appearance features and morphological structure of the weld area in a short time, and no obvious oscillation, overfitting or performance degradation occurs during the training process. All mAP indicators also show a highly consistent growth trend.

[0065] Please refer to the figure. The innovations of the welded area segmentation method based on semantic segmentation network and depth geometric constraints in this invention include: the multi-scale spatial fusion structure and edge attention design of the MSFA segmentation head module. The MSFA segmentation head module includes a feature enhancement module composed of 1×1 convolution, 3×3 depthwise convolution, two-level 3×3 depthwise convolution, and edge attention generation branches. Its multi-scale parallel structure and fusion method are independently innovative. The multi-branch mask generation and fusion mechanism of the MSFA segmentation head module inserts MSFA into the prototype generation and mask coefficient generation paths. The module's structural layout allows enhanced features to directly participate in the segmentation head output. The boundary refinement process, combining RGB segmentation and depth geometry, utilizes a method that integrates RGB segmentation results with depth geometric information. Using the improved YOLO segmentation network output as the initial mask, the segmentation results are aligned with the depth image. A constructed energy function, fusing color consistency, depth difference weights, and boundary smoothness constraints, is then used to globally optimize the initial mask at the pixel level. This ensures that the segmentation boundary satisfies consistency in both texture and depth structure. Furthermore, based on the gradient field, local normal direction, and curvature distribution of the depth map, sub-pixel-level geometric interpolation is performed on the optimized boundary points along the depth gradient direction. This ensures that the final boundary is precisely aligned with the actual 3D abrupt change locations in depth space, thus achieving a complete boundary inference process from coarse network segmentation to fine depth geometric localization.

[0066] The beneficial effects of the welded area segmentation method based on semantic segmentation network and depth geometric constraints of the present invention include:

[0067] Please refer to Table 1 for the output results of the improved YOLO12 segmentation model, and Figure 3The present invention improves the detection and segmentation accuracy of weld seam areas and significantly enhances the boundary recognition capability in weak texture and strong reflective scenes. By introducing the MSFA multi-scale spatial fusion module and the reconstructed Segment-MSFA segmentation head, the model can form a collaborative expression between shallow texture and deep semantics, which fully enhances the edge features of slender weld seams. Experimental results verify that the model is significantly better than traditional methods in complex lighting, weak texture and small-scale target scenes.

[0068] Table 1

[0069]

[0070] This method achieves boundary reconstruction from 2D appearance to 3D geometry consistency, making the weld contour closer to the real physical structure. Based on the preliminary mask of the initial YOLO segmentation results, an energy function that integrates RGB confidence, depth gradient weights and smoothing terms is introduced for boundary optimization. Furthermore, sub-pixel interpolation is performed using the gradient, normal and curvature of the depth map to make the final boundary fit the weld thickness abrupt change. This achieves precise alignment from the 2D mask to the depth geometry, providing high-precision geometric information for weld height measurement, width evaluation and robot localization tasks.

[0071] While maintaining high accuracy, it also maintains a deployable lightweight structure, enabling real-time recognition and stable inference in engineering applications. The optimized MSFA and Segment_MSFA maintain good feature representation efficiency, significantly improving overall performance while keeping inference speed stable. It can run in real time on edge computing devices such as industrial cameras and robotic arm controllers, providing direct application value for automatic weld detection and spatial positioning.

[0072] The present invention also provides a weld seam welded area segmentation system based on semantic segmentation network and depth geometric constraints, including an RGB-D camera and a processing device. The RGB-D camera is used to acquire RGB images and depth images of the workpiece to be welded, and the processing device is used to implement the steps of the above-mentioned weld seam welded area segmentation method based on semantic segmentation network and depth geometric constraints.

[0073] Although the steps are described in the claims and specification using S-numbers such as S10, S20, S30, etc., those skilled in the art should understand that, unless otherwise expressly stated, these numbers are for ease of description only and do not constitute a mandatory limitation on the order of execution of the steps. In some embodiments, some steps may be performed in parallel, in a different order, or omitted without departing from the core technical solution of the present invention, as long as the technical effects of the present invention can be achieved.

[0074] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for weld area segmentation based on semantic segmentation network and depth geometric constraint, characterized in that, Includes the following steps: Obtain the RGB image and depth image of the part to be welded; The RGB image is enhanced to obtain an enhanced image. The enhanced image is then used to perform image recognition using a semantic segmentation model to obtain detection output results and segmentation output results. The detection output results are rectangular detection boxes used to mark the welded areas of the weld, and the segmentation output results are preliminary masks used to mark the welded areas. The enhanced image is spatially aligned with the depth image, and an energy model is constructed based on color consistency, depth geometric difference, and boundary smoothness constraints. adopting formula constructing an energy model, wherein, denotes a unified energy function corresponding to the energy model, is a value for measuring the difference between the preliminary segmentation probability and the color consistency, denotes the pixel coordinates of the target pixel to be optimized, is a value for representing the confidence degree belonging to the welded area of the weld, , is a neighborhood weight, denotes the pixel coordinates of the neighborhood pixels of , and are used to describe the local correlation between pixels, , respectively denote the depth values of the pixel coordinates and the pixel coordinates in the depth image, denotes a smoothing coefficient of the depth difference; is a value of a regularization term for suppressing local noise and maintaining overall smoothing of the boundary, , , are weight coefficients, respectively. The energy model is used to perform global pixel-level optimization on the initial mask to determine the mask boundary, thus obtaining the optimized mask; The gradient field, normal direction, and local curvature of the depth image in the boundary neighborhood are calculated and obtained. The depth change curve is interpolated along the normal direction. The coordinates of the points where the depth change rate reaches an extreme value or where the geometric thickness jumps are determined as the coordinates of the real space boundary points, and the interpolation curve is obtained. The optimized mask is subjected to depth interpolation using the interpolation curve to obtain the three-dimensional boundary information of the weld.

2. The welded area segmentation method based on semantic segmentation network and depth geometric constraints according to claim 1, characterized in that, The enhanced image is used for image recognition using an improved YOLO12 segmentation model, which includes a backbone network, a neck feature fusion network, and a segmentation detection head. The enhanced image is processed by the backbone network to obtain feature map p2, feature map p3, feature map p4, and feature map p5; The neck feature fusion network processes the feature map p5 to obtain upsampled feature E. The upsampled feature E is then concatenated with the feature map p4 output by the backbone network and processed to obtain an initial fused feature map p4+. The initial fused feature map p4+ is then processed to obtain upsampled feature O. The upsampled feature O is then concatenated with the feature map p3 output by the backbone network and processed to obtain an initial fused feature map p3+. The initial fused feature map p3+ is then processed to obtain upsampled feature G. The upsampled feature G is then concatenated with the feature map p3 output by the backbone network and processed to obtain an initial fused feature map p3+. The feature map p2 is concatenated and processed to obtain a super-fused feature map P2; the super-fused feature map P2 is processed to obtain an output feature H, which is then fused with the initial fused feature map p3+ and processed to obtain a super-fused feature map P3; the super-fused feature map P3 is convolved and concatenated with the initial fused feature map p4+ and processed to obtain a super-fused feature map P4; the super-fused feature map P4 is processed to obtain an output feature J, which is then fused with the feature map p5 and processed to obtain a super-fused feature map P5; The segmentation detection head includes a segment-MSFA segmentation head module. The segmentation detection head is used to process super-fusion feature maps P2, P3, P4, and P5 to obtain a sequence {Z2, Z3, Z4, Z5} composed of scale-enhanced features. The sequence {Z2, Z3, Z4, Z5} is simultaneously entered into the mask modeling and target detection of the segmentation detection head, and the detection output result and segmentation output result are output.

3. The welded area segmentation method based on semantic segmentation network and depth geometric constraints according to claim 2, characterized in that, The segment-MSFA segmentation head module has corresponding edge attention weights. ,in, g(·) is a nonlinear mapping consisting of 3×3 convolution, channel shrinkage, and channel recovery. For the Sigmoid function; The final output of the segment-MSFA segmentation head module is .

4. The welded area segmentation method based on semantic segmentation network and depth geometric constraints according to claim 2, characterized in that, The spatial resolutions of feature maps p2, p3, p4, and p5 output by the backbone network are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the RGB image, respectively.

5. The welded area segmentation method based on semantic segmentation network and depth geometric constraints according to claim 2, characterized in that, The number of channels in the initial fused feature map p4+, ​​the initial fused feature map p3+, and the feature map p2 in the neck feature fusion network are 512, 256, and 128, respectively.

6. The welded area segmentation method based on semantic segmentation network and depth geometric constraints according to claim 4, characterized in that, The neck feature fusion network includes a C3 module. The feature map p2 output from the backbone network is upsampled once and then concatenated with the features passed down from the initial fusion feature map p3+ in the neck feature fusion network. The C3 module is used to perform feature processing to obtain the super fusion feature map P2.

7. The welded area segmentation method based on semantic segmentation network and depth geometric constraints according to claim 4, characterized in that, The segment-MSFA segmentation head module consists of three parallel branches: 1×1 convolution, 3×3 depthwise convolution, and 5×5 depthwise stacked convolution.

8. A weld seam segmentation system based on semantic segmentation networks and deep geometric constraints, characterized in that, It includes an RGB-D camera and a processing device, wherein the RGB-D camera is used to acquire RGB images and depth images of the workpiece to be welded. The processing apparatus is used to implement the steps of the weld seam segmentation method based on semantic segmentation network and depth geometric constraints as described in any one of claims 1 to 7.

Images