Weld joint corrosion feature extraction method and system based on deep learning and SfM

Abstract

The invention relates to the field of weld defect detection, in particular to a weld corrosion feature extraction method and system based on deep learning and SfM, and the method comprises the steps: obtaining a multi-angle feature recognition image of a target recognition structure, and taking the obtained feature recognition image as the input information of a recognition optimization network for completing training, determining a rust semantic tag of each feature recognition image; generating a target three-dimensional point cloud for the target recognition structure based on the corrosion information in the corrosion semantic tag of each feature recognition image; dense slicing is conducted on the generated target three-dimensional point cloud in the length direction of the weld joint to obtain a plurality of section point clouds, and weld joint basic parameters and corrosion geometric parameters of the target recognition structure are determined based on the obtained section point clouds; and outputting a corrosion feature fusion result of the target recognition structure. According to the method, the extraction quality and extraction efficiency of the welding seam corrosion characteristics are improved.

Description

Technical Field

[0001] This invention relates to the field of weld defect detection, and in particular to a method and system for extracting weld corrosion features based on deep learning and SfM. Background Technology

[0002] Traditional weld corrosion detection relies heavily on manual operation and requires sophisticated surface pretreatment, making it difficult to effectively quantify corrosion features, geometric parameters, and three-dimensional spatial information. Existing deep learning technologies for weld corrosion detection primarily focus on identifying corrosion regions in two-dimensional images, failing to effectively capture the three-dimensional characteristics of corrosion defects and thus unable to meet the practical needs for acquiring feature information about weld corrosion. Therefore, improving the effectiveness of weld corrosion feature acquisition results while minimizing the consumption of human resources is a pressing issue for those skilled in the art.

[0003] Chinese patent application publication number CN118657664A discloses a method for extracting weld features, including the following steps: acquiring weld images using an industrial CCD camera; transmitting the acquired weld images to an image processing system, performing image preprocessing on the uploaded weld images, including grayscale processing and image enhancement to make the weld images clearer; performing edge detection, image segmentation, erosion operations, and dilation operations on the weld images to extract the weld contour image; quantizing, selecting features, and reducing the dimensionality of the weld contour image to obtain weld features; and finally, visualizing the weld features for easy understanding and analysis by operators. However, the above scheme has the following drawbacks: it fails to effectively reconstruct three-dimensional information of weld corrosion defects based on the acquired two-dimensional weld images, resulting in poor completeness of the defect feature information obtained during the weld corrosion detection process. Summary of the Invention

[0004] To address this issue, the present invention provides a method and system for extracting weld corrosion features based on deep learning and SfM, which overcomes the problem that existing technologies fail to effectively reconstruct three-dimensional information of weld corrosion defects from the acquired two-dimensional weld images, resulting in poor completeness of defect feature information acquisition during the weld corrosion detection process.

[0005] To achieve the above objectives, this invention provides a method for extracting weld corrosion features based on deep learning and SfM, comprising: The feature recognition images of the target recognition structure from multiple angles are obtained, and the obtained feature recognition images are used as input information for the trained recognition optimization network to determine the rust semantic label of each feature recognition image. Based on the corrosion information within the corrosion semantic labels of the images identified by each feature, a 3D point cloud of the target is generated for the target structure. Key feature points are determined by using the binary mask of each feature recognition image, and bundle adjustment and camera parameter optimization are performed based on the key feature points to determine the sparse point cloud. The system obtains the point cloud holes that exist in the sparse point cloud, determines the distribution state of each point cloud hole based on the distribution mask parameters and distribution mask difference parameters, and determines the corresponding point cloud hole reconstruction and repair method based on the distribution state of each point cloud hole in order to determine the dense point cloud, perform filtering processing, and output the target 3D point cloud. Determine whether to adjust the acquisition execution parameters for the corresponding analysis sub-region based on the reconstruction execution distribution parameters of each analysis sub-region; Dense slices are made along the length of the weld to obtain several cross-sectional point clouds, and the weld basic parameters and corrosion geometric parameters of the target identification structure are determined based on the obtained cross-sectional point clouds. Output the fusion result of the corrosion features of the target recognition structure.

[0006] Furthermore, the process of determining the sparse point cloud includes: Along the length of the weld, the scanning area corresponding to the target identification structure is divided into several overlapping sub-regions based on the partitioning interval parameters; Bundle adjustment is performed on the feature recognition images corresponding to each analysis sub-region to obtain the local camera optimization parameters for the corresponding analysis sub-region. Based on the local camera optimization parameters corresponding to each analysis sub-region, the global camera optimization parameters of the scanning region corresponding to the target recognition structure are determined to obtain sparse point clouds.

[0007] Furthermore, the distribution state of the point cloud holes includes a first-class distribution state, a second-class distribution state, and a third-class distribution state; Point cloud holes in a type I distribution state are point cloud holes whose distribution mask parameters are less than or equal to the preset distribution mask parameters; Point cloud holes in the second type of distribution state are point cloud holes whose distribution mask parameter is greater than the preset distribution mask parameter and whose distribution mask difference parameter is less than or equal to the preset distribution mask difference parameter. Point cloud holes in the three distribution states are those where the distribution mask parameter is greater than the preset distribution mask parameter and the distribution mask difference parameter is greater than the preset distribution mask difference parameter.

[0008] Furthermore, the distribution mask parameters and distribution mask difference parameters of any point cloud hole are determined based on the binary mask and geometric feature parameters of the hole boundary points.

[0009] Furthermore, Poisson reconstruction is performed on point cloud holes in a certain distribution state to generate dense point clouds corresponding to those point cloud holes. Edge constraint processing is performed on point cloud holes in three distribution states, and triangulation is performed based on the semantic boundary line corresponding to the point cloud hole as a hard constraint.

[0010] Furthermore, feature parameters are reconstructed for point cloud holes that are in a type II distribution state or a type III distribution state and have completed edge constraint processing. Based on the spatial vector parameters of the hole boundary points of the point cloud hole, the dense point cloud corresponding to the point cloud hole is synthesized.

[0011] Furthermore, under the condition that the reconstruction is completed, the reconstruction execution distribution parameters for each analysis sub-region are detected, and the reconstruction execution distribution parameters are determined based on the execution evaluation parameters of each point cloud hole; The reconstruction is completed when the target recognition structure's three-dimensional point cloud is constructed, meaning that all holes in the point cloud are reconstructed and repaired.

[0012] Furthermore, based on the reference execution evaluation parameters, the acquisition execution parameters of the analysis sub-regions whose reconstruction execution distribution parameters are greater than the preset reconstruction execution distribution parameters are increased and adjusted; The increase in the collected execution parameters is positively correlated with the reference execution evaluation parameters.

[0013] Furthermore, the training process of the recognition optimization network includes: Construct a dataset of weld corrosion images; A CBAM attention module is embedded in the YOLOv8 base network to enhance the feature extraction of the rusted area. The CBAM attention module is embedded between the output of the C2f module of the YOLOv8 backbone network and the input of the neck SPPF module. By combining channel attention and spatial attention, the characteristics of the rusted area are enhanced and background noise is suppressed. Based on formula (1), optimize the segmentation boundary accuracy and sample imbalance. , in, for The corresponding weighting coefficients, for The corresponding weighting coefficients; The feature recognition images obtained from the target recognition structure are used as input information for the recognition optimization network to complete the training. The rust semantic label of each feature recognition image includes the binary mask of the rust region, the bounding box coordinates, and the confidence level.

[0014] This invention also provides a weld corrosion feature extraction system based on deep learning and SfM, comprising: A multi-view acquisition module is used to acquire feature recognition images of the target structure from multiple angles; The corrosion analysis module, which is connected to the multi-view acquisition module, is used to use the acquired feature recognition images as input information for the trained recognition optimization network in order to determine the corrosion semantic label of each feature recognition image. A 3D reconstruction module, which is connected to the corrosion analysis module, includes a sparse point cloud construction unit and a dense point cloud construction unit, used to generate a target 3D point cloud for the target recognition structure based on the corrosion information in the corrosion semantic labels of each feature recognition image. The sparse point cloud construction unit is used to perform bundle adjustment and camera parameter optimization based on key feature points to determine the sparse point cloud. The dense point cloud construction unit is used to determine the reconstruction and repair method of the corresponding point cloud holes based on the distribution state of each point cloud hole, so as to determine the dense point cloud. The distribution state of the point cloud holes is determined according to the distribution mask parameter and the distribution mask difference parameter. The reconstruction evaluation module, which is connected to the three-dimensional reconstruction module, is used to determine whether to adjust the acquisition execution parameters for the corresponding analysis sub-region based on the reconstruction execution distribution parameters of each analysis sub-region. The slicing execution module, which is connected to the three-dimensional reconstruction module, is used to perform dense slicing on the generated target three-dimensional point cloud to obtain several cross-sectional point clouds, and to determine the weld basic parameters and corrosion geometric parameters of the target identification structure based on the obtained cross-sectional point clouds.

[0015] Compared with existing technologies, the advantages of this invention are that the technical solution of this invention is based on the YOLOv8-CBAM network to accurately segment the rust region in the multi-view images acquired for the target of weld rust detection. Furthermore, by optimizing the network and fusing the dual loss function, the accuracy of the segmentation results of the rust region is improved, providing targeted guidance for the SfM reconstruction process and avoiding data analysis redundancy in the SfM 3D reconstruction analysis process. This improves the analysis efficiency and accuracy of the 3D reconstruction results, and the invention improves the extraction quality and efficiency of weld rust features.

[0016] Furthermore, this invention filters key feature points based on rust semantic tags and combines bundle adjustment to obtain sparse point clouds. This not only reduces resource consumption in the actual construction process but also improves the effectiveness of the obtained sparse point clouds. Moreover, it determines the actual state of point cloud holes in the sparse point cloud based on distribution mask parameters and distribution mask difference parameters, thereby determining targeted reconstruction methods for each point cloud hole. This makes the reconstruction process for point cloud holes more consistent with the actual situation. This invention improves the processing efficiency of the reconstruction process while ensuring the construction effect of dense point clouds.

[0017] Furthermore, in this invention, the distribution state of point cloud holes in sparse point clouds is determined based on distribution mask parameters and distribution mask difference parameters. The distribution mask parameters and distribution mask difference parameters characterize the correlation between the location of the point cloud hole and the rust and weld areas of the target recognition structure, as well as the complexity of the location. This determines the reconstruction process for different point cloud holes, ensuring reconstruction efficiency for simple holes while improving the fidelity for relatively complex point cloud holes. This invention improves the construction efficiency and quality of the target 3D point cloud, thereby improving the extraction quality and efficiency of weld rust features.

[0018] Furthermore, this invention performs three-dimensional reconstruction based on the acquired multi-view weld corrosion defects and extracts cross-sections from the acquired 3D point cloud model to effectively extract the features of the corrosion defects. Moreover, by performing a "two-dimensional-three-dimensional-two-dimensional" transformation on the acquired weld corrosion defect image, the feature analysis results for the weld corrosion defects are made more complete. In addition to covering the basic geometric parameters of the weld itself, such as the weld toe radius and side angle, it also includes quantitative indicators such as the slice area and depth of the corrosion region. This invention improves the completeness of the feature analysis results for the corrosion region in weld corrosion detection. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the weld corrosion feature extraction method based on deep learning and SfM of the present invention; Figure 2 This is a flowchart illustrating how the distribution state of each point cloud hole is determined based on distribution mask parameters and distribution mask difference parameters according to the present invention. Figure 3 This is a flowchart illustrating the reconstruction and repair method for point cloud holes based on their distribution. Figure 4 This is a module connection diagram of the weld corrosion feature extraction system based on deep learning and SfM of the present invention. Detailed Implementation

[0020] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0021] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0022] It should be noted that in the description of this invention, the terms "upper", "lower", "left", "right", "inner", "outer", etc., which indicate directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and is not intended to indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0023] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0024] Please see Figures 1 to 3 As shown, this embodiment of the invention provides a method for extracting weld corrosion features based on deep learning and SfM, including: The feature recognition images of the target recognition structure from multiple angles are obtained, and the obtained feature recognition images are used as input information for the trained recognition optimization network to determine the rust semantic label of each feature recognition image. Based on the corrosion information within the corrosion semantic labels of the images identified by each feature, a 3D point cloud of the target is generated for the target structure. Key feature points are determined by using the binary mask of each feature recognition image, and bundle adjustment and camera parameter optimization are performed based on the key feature points to determine the sparse point cloud. The system obtains the point cloud holes that exist in the sparse point cloud, determines the distribution state of each point cloud hole based on the distribution mask parameters and distribution mask difference parameters, and determines the corresponding point cloud hole reconstruction and repair method based on the distribution state of each point cloud hole in order to determine the dense point cloud, perform filtering processing, and output the target 3D point cloud. Determine whether to adjust the acquisition execution parameters for the corresponding analysis sub-region based on the reconstruction execution distribution parameters of each analysis sub-region; Dense slices are made along the length of the weld to obtain several cross-sectional point clouds, and the weld basic parameters and corrosion geometric parameters of the target identification structure are determined based on the obtained cross-sectional point clouds. Output the fusion result of the corrosion features of the target recognition structure.

[0025] In this invention, three-dimensional feature information is extracted during weld corrosion detection. This further improves the effectiveness of the acquired weld corrosion features without consuming additional manpower. The object being detected for weld corrosion is designated as the target recognition structure. Due to the inherent structural issues of the target recognition structure, spatial occlusion can easily lead to a loss of geometric information when acquiring images of it. Therefore, in addition to analyzing the corrosion status of the images acquired from the target recognition structure, three-dimensional reconstruction is performed to obtain the three-dimensional feature information of the weld corrosion status of the target recognition structure. In this invention, an industrial camera and laser positioning component are mounted on a movable gimbal to acquire images of the visible surface of the target recognition structure from all angles. The acquired images are designated as feature recognition images of the target recognition structure, and each feature recognition image is associated with shooting coordinate information.

[0026] Dense slicing is performed on the generated target 3D point cloud along the weld length direction. A unified local coordinate system is constructed with the weld length direction as the slicing reference direction to ensure that the slicing plane is perpendicular to the weld extension direction. In the target 3D point cloud, weld feature points with more than 10 uniform distributions are selected along the weld extension direction, such as the weld height apex. A straight line is fitted using the least squares method and set as the X-axis of the local coordinate system. The center of symmetry of the weld section is selected and the plane where the section is located is fitted. The normal vector of the plane is set as the Y-axis and the Z-axis is the weld height direction. The intersection of the center line of the weld start end and the surface of the target recognition structure is recorded as the origin of the coordinate system. All point coordinates of the global dense point cloud are transformed to this local coordinate system. Along the positive X-axis direction of the local coordinate system, starting from the origin, slicing planes perpendicular to the X-axis are generated sequentially. The initial cross-sectional point cloud of each slicing plane is extracted using a planar clipping algorithm. The initial cross-sectional point cloud is projected onto the corresponding slicing plane and recorded as the cross-sectional point cloud of the corresponding slicing plane. The categories of weld basic parameters include weld width, weld reinforcement, fusion width, and bevel angle. Determining each weld basic parameter is a matter well understood by those skilled in the art. For example, for a single cross-sectional point cloud, the weld width is the maximum span of the weld outline within that cross-sectional point cloud along the Y-axis, which will not be elaborated upon here. The categories of corrosion geometric parameters include, but are not limited to, cross-sectional corrosion coverage area, cross-sectional corrosion thickness, corrosion depth, and corrosion protrusion height. When determining each category of corrosion geometric parameters, it is necessary to first segment the corrosion region and the weld body region in the cross-sectional point cloud. Determining each corrosion geometric parameter is a matter well understood by those skilled in the art. For example, for a single cross-sectional point cloud, the cross-sectional corrosion thickness is the vertical distance between the highest point of the segmented corrosion region and the surface of the target recognition structure, which will not be elaborated upon here. The output corrosion feature fusion result of the target recognition structure includes: a target 3D point cloud with corrosion semantic labels, weld basic parameters, and corrosion geometric parameters.

[0027] Specifically, the process of determining the sparse point cloud includes: Along the length of the weld, the scanning area corresponding to the target identification structure is divided into several overlapping sub-regions based on the partitioning interval parameters; Bundle adjustment is performed on the feature recognition images corresponding to each analysis sub-region to obtain the local camera optimization parameters for the corresponding analysis sub-region. Based on the local camera optimization parameters corresponding to each analysis sub-region, the global camera optimization parameters of the scanning region corresponding to the target recognition structure are determined to obtain sparse point clouds.

[0028] In this process, when dividing the analysis sub-regions, the length of each sub-region along the weld direction is the division interval parameter. The area of ​​the scanning region corresponding to each sub-region is consistent. There is an overlap between the scanning regions corresponding to any two adjacent sub-regions, and the corresponding adjacent overlap ratio parameter is consistent. For any two overlapping sub-regions, the adjacent overlap ratio parameter is... , To analyze the area of ​​the scanned region corresponding to the sub-region, The area of ​​the overlapping region between the scanning regions corresponding to the two overlapping analysis sub-regions mentioned above. The values ​​of the division interval parameter and the adjacent overlap ratio parameter can be set by the user according to the actual situation. One value for the division interval parameter is 5% of the total scanning length of the target recognition structure in the weld direction. Another value for the adjacent overlap ratio parameter is 20%. For a single analysis sub-region, when performing local optimization based on bundle adjustment, this analysis sub-region The set of feature recognition images corresponding to this sub-region of analysis , To determine the number of feature recognition images corresponding to the analysis sub-region, when extracting key feature points, the SIFT algorithm is applied to each feature recognition image for detection, and the set of SIFT feature points obtained for the j-th feature recognition image is determined. , The number of SIFT feature points in the feature recognition image is given. Each SIFT feature point corresponds to spatial information including coordinates, scale, and orientation. For each SIFT feature point, its corresponding pixel value in the rust binary mask is checked. If the corresponding pixel value is a foreground or rust pixel, the SIFT feature point is retained and recorded as a key feature point. If the corresponding pixel value is a background pixel, it is discarded. The retained key feature points are used as input for bundle adjustment. For the feature recognition images existing in the analysis sub-region, feature point matching is performed based on the SIFT descriptor to establish cross-image correspondences, i.e., to obtain key feature points with consistent spatial information existing between different feature recognition images. The initial intrinsic parameters obtained from camera calibration during the pre-processing stage are used as the initial values ​​of the camera intrinsic parameters for this analysis sub-region. The initial values ​​of the camera extrinsic parameters are solved based on the key feature points and the EPnP algorithm. The initial values ​​of the camera extrinsic parameters include the rotation matrix and translation vector. The objective function constructed by bundle adjustment is... ,in, To analyze the number of feature recognition images corresponding to sub-regions, To analyze the number of key feature points within a sub-region, Let v be the actual pixel coordinates of the v-th key feature point in the u-th image. For the camera intrinsic parameter matrix, as well as Let u be the extrinsic parameter of the feature recognition image. For the first The three-dimensional coordinates of the key feature points The camera projection function is used; local camera parameter optimization is performed and verified. Intrinsic parameters, extrinsic parameters, and the 3D coordinates of key feature points are used as optimization variables and input into the Ceres Solver open-source library. Convergence conditions are set (RMS reprojection error ≤ 1.0 pixel, maximum iterations ≤ 100 times). Sparse BA solution is performed. After the solution is completed, the RMS reprojection error of the analysis sub-region is calculated. If RMS > 1.5 pixels, the feature points of the sub-region are re-extracted and the above steps are repeated until the accuracy requirements are met. The local camera optimization parameters of the analysis sub-region are saved, including the optimized intrinsic parameter correction values, the optimized extrinsic parameters for each image, and the local 3D coordinates of key feature points within the analysis sub-region. The first analysis sub-region at the weld start end is selected as the global baseline, and its local camera optimization parameters are directly used as the global initial parameters. For the i-th analysis sub-region... (i≥2), extract the sub-regions that are analyzed in the previous analysis. Common feature points within the overlapping region are solved using the ICP algorithm. Local coordinate system to The transformation matrix of the local coordinate system will The local camera optimization parameters are transformed to the global coordinate system using a transformation matrix, resulting in... The initial values ​​of the global camera parameters are determined. After the initial transformation of all analysis sub-regions, a global bundle adjustment objective function is constructed. With the goal of minimizing the global reprojection error, the initial values ​​of the global camera parameters for all sub-regions are jointly optimized. The weight allocation is inversely proportional to the local reprojection error of the sub-region; the higher the local accuracy, the greater the weight. Globally unified camera intrinsic parameters and global extrinsic parameters of the feature recognition image are saved. The global sparse point cloud generation process includes: based on the optimized global camera parameters, the 3D coordinates of key feature points in all analysis sub-regions are solved using a linear triangulation algorithm to obtain the global 3D coordinates of each key feature point. The global 3D coordinates of all key feature points are then summarized to form the initial global sparse point cloud.

[0029] Specifically, the distribution states of the point cloud holes include three types: type I, type II, and type III. Point cloud holes in a type I distribution state are point cloud holes whose distribution mask parameters are less than or equal to the preset distribution mask parameters; Point cloud holes in the second type of distribution state are point cloud holes whose distribution mask parameter is greater than the preset distribution mask parameter and whose distribution mask difference parameter is less than or equal to the preset distribution mask difference parameter. Point cloud holes in the three distribution states are those where the distribution mask parameter is greater than the preset distribution mask parameter and the distribution mask difference parameter is greater than the preset distribution mask difference parameter.

[0030] Specifically, the distribution mask parameters and distribution mask difference parameters of any point cloud hole are determined based on the binary mask and geometric feature parameters of the hole boundary points.

[0031] If the target recognition structure completes the construction of the sparse point cloud, then the point cloud holes existing in the determined sparse point cloud are obtained. When obtaining point cloud holes, the nearest neighbor density of each point in the obtained sparse point cloud is detected. Here, k=20 is selected to determine the number of points in the neighborhood radius of each point in the sparse point cloud, and this number is recorded as the local density of that point. For any point existing in the sparse point cloud, if the local density of that point is less than a preset local density parameter, then that point is recorded as a potential hole point. The preset local density parameter is the average local density of all points existing in the sparse point cloud. The value is the product of the scene evaluation coefficient and the scene evaluation coefficient. The scene evaluation coefficient can be set by the user based on the actual working scenario. The more complex the texture of the rusted area of ​​the target recognition structure, the smaller the scene evaluation coefficient value. One possible scene evaluation coefficient value is 0.3. A connected component analysis algorithm is used to cluster potential void points to form continuous void regions. Each connected component corresponds to a point cloud void. The boundary points of each point cloud void are extracted and denoted as the void boundary points of the corresponding point cloud void. For a single point cloud void, the distribution mask parameter is... , This is the average of the mask values ​​of all boundary points corresponding to the point cloud hole. To evaluate the mask value, the distributed mask difference parameter is the average of the feature difference indices of each geometric feature parameter. For a single geometric feature parameter, the feature difference index... , This is the average value of the geometric feature parameter for each key feature point corresponding to the hole in the point cloud. The standard deviation between the values ​​of the geometric feature parameter corresponding to each key feature point in the point cloud hole is used to determine the category of the geometric feature parameter of the distribution mask difference parameter, including but not limited to the normal vector angle and curvature. How to determine the normal vector angle and curvature of each key feature point is a content that is easy for those skilled in the art to understand, and will not be elaborated here. A value of evaluation mask value is provided, and the value of evaluation mask value is 1. The values ​​of the preset distribution mask parameter and the preset distribution mask difference parameter are understood to be such that the higher the user's requirements for the quality of the rust feature information of the target recognition structure, the smaller the value of the preset distribution mask parameter and the smaller the value of the preset distribution mask difference parameter. The distribution mask parameter represents the degree of rust in the area corresponding to the point cloud hole in the constructed sparse point cloud, and the distribution mask difference parameter represents the degree of feature difference between the point cloud hole and the surrounding point cloud. One preset distribution mask parameter value is 0.4, and another preset distribution mask difference parameter value is 0.2.

[0032] Specifically, Poisson reconstruction is performed on point cloud holes in a certain distribution state to generate dense point clouds corresponding to the point cloud holes. Edge constraint processing is performed on point cloud holes in three distribution states, and triangulation is performed based on the semantic boundary line corresponding to the point cloud hole as a hard constraint.

[0033] For a single point cloud cavity, if the cavity exists in a certain distribution state, it indicates that the area corresponding to the cavity is likely unrelated to the rusted area. In this case, the accuracy requirement for reconstructing and repairing the cavity is low. Therefore, Poisson reconstruction can meet the actual needs and effectively improve the efficiency of cavity reconstruction and repair. How to perform Poisson reconstruction for such cavities is easy for those skilled in the art to understand and will not be elaborated here. For a single point cloud hole, if the point cloud hole is in the third type of distribution state, it indicates that the area corresponding to the point cloud hole is likely related to the rust area, and the geometric features of the boundary points of the point cloud hole are significantly different. Since directly reconstructing such point cloud holes is prone to edge distortion, in order to ensure the reconstruction quality of the point cloud hole, the semantic boundary line of the point cloud hole needs to be triangulated as a hard constraint to ensure the processing effect of the subsequent reconstruction process. When performing edge constraint processing on any point cloud hole in the three distribution states, the hole boundary points are used as input. The binary mask of the rust region of each hole boundary point is used to determine whether the corresponding hole boundary point is in the rust region. For any hole boundary point, if the binary mask of the rust region of the nearest neighbor point in space is different from the binary mask of the rust region of the hole boundary point, then the hole boundary point is recorded as a semantic boundary point. All semantic boundary points are clustered and connected by spline curve fitting or minimum spanning tree to form a set of semantic boundary lines. The obtained sparse point cloud and the semantic boundary line sampling points are merged to form a triangulated input point set. Constrained Delaunay triangulation is performed, and the semantic boundary lines are used as constraint edges that must be included in the triangulation. A triangulation satisfying the Delaunay property is constructed, while ensuring that all constraint edges exist as edges of the triangulation.

[0034] Specifically, feature parameters are reconstructed for point cloud holes that are in a type II distribution state or a type III distribution state and have completed edge constraint processing. Based on the spatial vector parameters of the hole boundary points of the point cloud hole, the dense point cloud corresponding to the point cloud hole is synthesized.

[0035] For a single point cloud hole, if the point cloud hole is in a type II distribution state, it indicates that the area corresponding to the point cloud hole is likely related to the rusted area, and the difference between the geometric features of the boundary points of the point cloud hole is small. When reconstructing point cloud holes in type II distribution state and type III distribution state after edge constraint processing, it is necessary to retain the consistency of their set features with respect to the surrounding area to avoid distortion of the reconstruction results. Therefore, feature parameter reconstruction is performed on point cloud holes in type II distribution state and type III distribution state after edge constraint processing. For any point cloud cavity in a binary or tri-class distribution state with completed edge constraint processing, feature parameter reconstruction involves extracting the set of cavity boundary points. Various spatial vector parameters are calculated for each boundary point, including normal vector, curvature, and neighborhood distance distribution vector. Using these boundary points as support, a tetrahedral mesh framework is constructed inside the cavity. For example, the MarchingTetrahedra algorithm is used to determine the spatial index of each point to be generated within the cavity. An inverse distance weighted interpolation method is employed to predict the spatial vector parameters of each point to be generated within the mesh framework, based on the spatial vector parameters of the boundary points. The weights are inversely proportional to the distance from the point to be generated to the boundary points, ensuring a smooth transition of parameters from the boundary to the interior while preserving the rust-like morphology. Based on the predicted internal point parameters... Within the mesh framework, a dense point cloud conforming to parametric characteristics is generated. For example, raised point clusters are generated in high-curvature regions, and peeling slopes are generated in regions with abrupt changes in normal vectors. The sampling density of the dense point cloud generated for the void in the surrounding rusted region is adjusted to ensure uniformity by comparing it with the point cloud density of the surrounding rusted region. Local feature calibration is performed, and the dense point cloud generated for the void is merged with the boundary point cloud and the surrounding normal point cloud to complete the reconstruction. The calculation of the normal vector, curvature, and neighborhood distance distribution vector for each void boundary point is easily understood by those skilled in the art. For example, the neighborhood distance distribution vector is used to characterize the spatial structure of the local neighborhood of the void boundary point. Its construction method is as follows: for any void boundary point, calculate its Euclidean distance to the K nearest neighbors in the global point cloud, and sort and statistically analyze this distance sequence (e.g., take the first M distance values) to form an M-dimensional descriptor. During reconstruction, interpolation of these vectors ensures that the point cloud generated inside the void maintains consistency with the boundary region in terms of local density and arrangement pattern, which will not be elaborated upon here.

[0036] Specifically, once the reconstruction is complete, the reconstruction execution distribution parameters for each analysis sub-region are detected. These reconstruction execution distribution parameters are determined based on the execution evaluation parameters of each point cloud hole. The reconstruction completion condition is that the target 3D point cloud of the target recognition structure is completed, that is, all holes in the point cloud are reconstructed and repaired.

[0037] Specifically, based on the reference execution evaluation parameters, the acquisition execution parameters of the analysis sub-regions whose reconstruction execution distribution parameters are greater than the preset reconstruction execution distribution parameters are increased and adjusted. The increase in the collected execution parameters is positively correlated with the reference execution evaluation parameters.

[0038] Specifically, for a single analysis sub-region, the reconstruction execution distribution parameters are... , This represents the number of relevant point cloud holes present in the analyzed sub-region. To identify the number of point cloud holes in the target structure. The reference execution evaluation parameters are the average of the execution evaluation parameters of the relevant point cloud holes existing in the analysis sub-region. For a single point cloud hole, if the point cloud hole maps to the area where the target recognition structure exists in the analysis sub-region, then the point cloud hole is recorded as the relevant point cloud hole of the analysis sub-region. The execution evaluation parameters are the sum of the distribution mask parameters and the distribution mask difference parameters of the point cloud hole. For a single analysis sub-region, if the reconstruction execution distribution parameter of the analysis sub-region is greater than the preset reconstruction execution distribution parameter, it indicates that there is a significant lack of point cloud information in the corresponding part of the analysis sub-region during the construction of dense point clouds. Therefore, it indicates that the weld corrosion image acquired for this analysis sub-region under the same conditions cannot meet the actual construction requirements. Therefore, targeted optimizations are made to the acquisition execution parameters for this analysis sub-region to optimize the acquisition of the corresponding part of the analysis sub-region during subsequent weld corrosion image acquisition. The acquisition execution parameter is the pixel overlap ratio of two adjacent feature recognition images acquired for this analysis sub-region along the weld length direction. The adjusted acquisition execution parameter... , These are the unadjusted acquisition execution parameters. For reference, the unadjusted acquisition execution parameters are used as a reference. It can be understood that the higher the user's requirements for the quality of acquiring the corrosion feature information of the target identification structure, the larger the value of the unadjusted acquisition execution parameters. One example of an unadjusted acquisition execution parameter value is provided: 20%. The value of the preset reconstruction execution distribution parameter can be understood as follows: the higher the user's requirements for the quality of the rust feature information of the target recognition structure, the smaller the value of the preset reconstruction execution distribution parameter. The reconstruction execution distribution parameter represents the degree of missing point cloud information in the process of three-dimensional reconstruction of the analysis sub-region. A preset reconstruction execution distribution parameter value of 0.9 is provided.

[0039] Specifically, the training process of the recognition optimization network includes: Construct a dataset of weld corrosion images; A CBAM attention module is embedded in the YOLOv8 base network to enhance the feature extraction of the rusted area. The CBAM attention module is embedded between the output of the C2f module of the YOLOv8 backbone network and the input of the neck SPPF module. By combining channel attention and spatial attention, the characteristics of the rusted area are enhanced and background noise is suppressed. Based on formula (1), optimize the segmentation boundary accuracy and sample imbalance. , in, for The corresponding weighting coefficients, for The corresponding weighting coefficients; The feature recognition images obtained from the target recognition structure are used as input information for the recognition optimization network to complete the training. The rust semantic label of each feature recognition image includes the binary mask of the rust region, the bounding box coordinates, and the confidence level.

[0040] The weld corrosion image dataset is a collection of feature recognition images containing corrosion regions collected from several target recognition structures that have completed weld corrosion feature extraction. Annotation tools (such as LabelImg) are used to annotate the corrosion regions in each image, generating a corrosion semantic label containing a binary mask of the corrosion region, bounding box coordinates, and confidence labels. Accurate boundaries should be ensured during annotation, especially for fine-grained division of the transition region between corrosion edges and the background. The annotated images are randomly divided into training, validation, and test sets in a 7:2:1 ratio. In the YOLOv8 backbone network, the CBAM attention module is embedded between the output of the last C2f module and the input of the SPPF module. Specifically, the output feature map of C2f passes sequentially through the channel attention submodule and spatial attention submodule in CBAM before being input to the SPPF module. The channel dimensions of the feature map are weighted, strengthening corrosion-related channel features and suppressing irrelevant channels. Through the synergy of channel attention and spatial attention, corrosion region features are enhanced and background noise is suppressed. In terms of spatial dimension, the location of the rusted area is highlighted, while the background noise is weakened. Focus loss is used to address the imbalance problem. The Dessian loss is used to optimize the segmentation boundary accuracy. The weighted composite loss function is expressed as follows: To alleviate the imbalance in pixel count between the rusted area and the background, as well as The corresponding weighting coefficients need to be set through grid search or experience and optimized on the validation set to provide a certain level of performance. as well as The corresponding weighting coefficient values, The corresponding weighting coefficient is 0.6. The corresponding weight coefficient is 0.4. This invention provides a training process setup scheme: Images in the weld corrosion image dataset are preprocessed, including size unification, normalization, and data augmentation (random rotation, brightness and contrast adjustment, Mosaic enhancement, etc.). Pre-trained YOLOv8 weights are used as initialization. Iterative training is performed on the training set, using AdamW as the optimizer with an initial learning rate of 1e-3, adjusted using a cosine annealing strategy, and the batch size is set according to GPU memory. Model performance is monitored on the validation set, with key evaluation metrics including mAP@0.5 for the detection task and [missing data - likely a score or value]. The IoU and Dice coefficients for the cutting task were calculated. The training iterations were set to 300 rounds, with an initial learning rate of 0.01, adaptively decaying to 1e-5. The final model achieved a rust region segmentation accuracy ≥95% and a localization error ≤1 pixel. Users can adjust the training process according to their specific circumstances; this is easily understood by those skilled in the art and will not be elaborated upon here. The rust semantic label of the feature recognition image includes a binary mask rule for the rust region. Foreground pixel values ​​are recorded as 255, corresponding to pixels in the rust region and weld contour, while background pixel values ​​are recorded as 0, corresponding to pixels in the non-rust region and weld contour.

[0041] Please see Figure 4 The diagram shown is a module connection diagram of the weld corrosion feature extraction system based on deep learning and SfM of the present invention. The present invention also provides a weld corrosion feature extraction system based on deep learning and SfM, comprising: A multi-view acquisition module is used to acquire feature recognition images of the target structure from multiple angles; The corrosion analysis module, which is connected to the multi-view acquisition module, is used to use the acquired feature recognition images as input information for the trained recognition optimization network in order to determine the corrosion semantic label of each feature recognition image. A 3D reconstruction module, which is connected to the corrosion analysis module, includes a sparse point cloud construction unit and a dense point cloud construction unit, used to generate a target 3D point cloud for the target recognition structure based on the corrosion information in the corrosion semantic labels of each feature recognition image. The sparse point cloud construction unit is used to perform bundle adjustment and camera parameter optimization based on key feature points to determine the sparse point cloud. The dense point cloud construction unit is used to determine the reconstruction and repair method of the corresponding point cloud holes based on the distribution state of each point cloud hole, so as to determine the dense point cloud. The distribution state of the point cloud holes is determined according to the distribution mask parameter and the distribution mask difference parameter. The reconstruction evaluation module, which is connected to the three-dimensional reconstruction module, is used to determine whether to adjust the acquisition execution parameters for the corresponding analysis sub-region based on the reconstruction execution distribution parameters of each analysis sub-region. The slicing execution module, which is connected to the three-dimensional reconstruction module, is used to perform dense slicing on the generated target three-dimensional point cloud to obtain several cross-sectional point clouds, and to determine the weld basic parameters and corrosion geometric parameters of the target identification structure based on the obtained cross-sectional point clouds.

[0042] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A method for extracting weld corrosion features based on deep learning and SfM, characterized in that, The method comprises the following steps: acquiring multi-angle feature recognition images of the target recognition structure, and taking the acquired feature recognition images as input information of the recognition optimization network trained to determine the rust semantic labels of each feature recognition image; generating a target three-dimensional point cloud for the target recognition structure based on the rust information in the rust semantic labels of each feature recognition image, wherein determining key feature points according to the binary mask of each feature recognition image, performing bundle adjustment and camera parameter optimization based on the key feature points to determine a sparse point cloud; acquiring point cloud holes existing in the determined sparse point cloud, determining the distribution state of each point cloud hole according to the distribution mask parameter and the distribution mask difference parameter, determining the reconstruction repair mode of the corresponding point cloud hole based on the distribution state of each point cloud hole to determine a dense point cloud, and performing filtering processing to output the target three-dimensional point cloud; determining whether to adjust the acquisition execution parameter of the corresponding analysis sub-region according to the reconstruction execution distribution parameter of each analysis sub-region; performing dense slicing on the generated target three-dimensional point cloud along the length direction of the weld to acquire a plurality of cross-section point clouds, and determining the weld base parameter and the rust geometric parameter of the target recognition structure based on the acquired cross-section point clouds; outputting the rust feature fusion result of the target recognition structure.

2. The deep learning and SfM based weld seam rust feature extraction method according to claim 1, characterized in that, The determination process of the sparse point cloud comprises: dividing the scanning region corresponding to the target recognition structure into a plurality of analysis sub-regions with overlap based on a division interval parameter along the length direction of the weld; performing bundle adjustment on the feature recognition image corresponding to each analysis sub-region to acquire local camera optimization parameters of the corresponding analysis sub-region; determining global camera optimization parameters of the scanning region corresponding to the target recognition structure based on the local camera optimization parameters of each analysis sub-region to acquire a sparse point cloud. 3.The deep learning and SfM based weld seam rust feature extraction method of claim 2, characterized in that, The distribution state of the point cloud hole comprises a first distribution state, a second distribution state, and a third distribution state; the point cloud hole in the first distribution state is a point cloud hole with a distribution mask parameter less than or equal to a preset distribution mask parameter; the point cloud hole in the second distribution state is a point cloud hole with a distribution mask parameter greater than the preset distribution mask parameter and a distribution mask difference parameter less than or equal to a preset distribution mask difference parameter; the point cloud hole in the third distribution state is a point cloud hole with a distribution mask parameter greater than the preset distribution mask parameter and a distribution mask difference parameter greater than the preset distribution mask difference parameter.

4. The deep learning and SfM based weld seam rust feature extraction method according to claim 3, characterized in that, The distribution mask parameter and the distribution mask difference parameter of any point cloud hole are determined according to the binary mask and the geometric feature parameter of the hole boundary point.

5. The deep learning and SfM based weld seam rust feature extraction method according to claim 3, characterized in that, Poisson reconstruction is performed on the point cloud hole in the first distribution state to generate a dense point cloud corresponding to the point cloud hole; edge constraint processing is performed on the point cloud hole in the third distribution state, and the corresponding semantic boundary line of the point cloud hole is taken as a hard constraint for triangulation.

6. The deep learning and SfM based weld seam rust feature extraction method according to claim 5, characterized in that, feature parameter reconstruction is performed on the point cloud hole in the second distribution state, or the point cloud hole in the third distribution state and the edge constraint processing is completed, and the spatial vector parameter of the hole boundary point of the point cloud hole is synthesized to generate a dense point cloud corresponding to the point cloud hole.

7. The deep learning and SfM based weld seam rust feature extraction method according to claim 6, characterized in that, The reconstruction execution distribution parameters of each analysis sub-region are detected under the reconstruction completion condition, and the reconstruction execution distribution parameters are determined according to the execution evaluation parameters of each point cloud hole; The reconstruction completion condition is that the target recognition structure target three-dimensional point cloud is completed, that is, each point cloud hole is completed. 8.The deep learning and SfM based weld seam rust feature extraction method of claim 1, wherein, The acquisition execution parameters of the analysis sub-region with the reconstruction execution distribution parameter greater than the preset reconstruction execution distribution parameter are adjusted based on the reference execution evaluation parameter; The increase value of the acquisition execution parameter and the reference execution evaluation parameter are in a positive correlation relationship.

9. The deep learning and SfM based weld seam corrosion feature extraction method according to claim 8, characterized in that, The training process of the recognition optimization network includes: Constructing a weld corrosion image dataset; Embedding a CBAM attention module in the YOLOv8 basic network to strengthen the corrosion area feature extraction, and the CBAM attention module is embedded between the output end of the C2f module of the YOLOv8 backbone network and the input end of the neck SPPF module; According to formula (1), the segmentation boundary accuracy and the sample imbalance degree are optimized, ， wherein is a corresponding weight coefficient, is a corresponding weight coefficient; The feature recognition images obtained for the target recognition structure are used as the input information of the trained recognition optimization network, and the corrosion semantic labels of the obtained feature recognition images include corrosion area binary masks, boundary box coordinates and confidence.

10. A feature extraction system applying the deep learning and SfM based weld seam rust feature extraction method according to any one of claims 1 to 9, characterized in that, It includes: A multi-view acquisition module is used to obtain multi-angle feature recognition images of the target recognition structure; An corrosion analysis module connected with the multi-view acquisition module is used to use the obtained feature recognition images as the input information of the trained recognition optimization network to determine the corrosion semantic labels of each feature recognition image; A three-dimensional reconstruction module connected with the corrosion analysis module includes a sparse point cloud construction unit and a dense point cloud construction unit, which is used to generate a target three-dimensional point cloud for the target recognition structure based on the corrosion information in the corrosion semantic labels of each feature recognition image; The sparse point cloud construction unit is used to perform bundle adjustment and camera parameter optimization based on key feature points to determine a sparse point cloud; The dense point cloud construction unit is used to determine the reconstruction repair mode of each point cloud hole based on the distribution state of the point cloud hole to determine a dense point cloud, and the distribution state of the point cloud hole is determined according to the distribution mask parameter and the distribution mask difference parameter; A reconstruction evaluation module connected with the three-dimensional reconstruction module is used to determine whether to adjust the acquisition execution parameter of the corresponding analysis sub-region according to the reconstruction execution distribution parameter of each analysis sub-region; A slice execution module connected with the three-dimensional reconstruction module is used to perform dense slicing on the generated target three-dimensional point cloud to obtain a plurality of cross-sectional point clouds, and determine the weld base parameters and corrosion geometric parameters of the target recognition structure based on the obtained cross-sectional point clouds.

Images