A weld seam rust feature extraction method and system based on deep learning and SfM

Abstract

The present application relates to the field of weld defect detection, and more particularly to a weld corrosion feature extraction method and system based on deep learning and SfM, the method comprising: acquiring multi-angle feature recognition images of a target recognition structure, and taking the acquired feature recognition images as input information of a recognition optimization network that has completed training, to determine corrosion semantic labels of each feature recognition image; generating a target three-dimensional point cloud for the target recognition structure based on corrosion information within the corrosion semantic labels of each feature recognition image; performing dense slicing on the generated target three-dimensional point cloud along the length direction of the weld to acquire a plurality of cross-sectional point clouds, and determining weld base parameters and corrosion geometric parameters of the target recognition structure based on the acquired cross-sectional point clouds; and outputting a corrosion feature fusion result of the target recognition structure. The present application improves the extraction quality and efficiency of weld corrosion features.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of weld defect detection, in particular to a weld corrosion feature extraction method and system based on deep learning and SfM. BACKGROUND

[0002] Traditional weld corrosion detection mainly relies on manual operation and has high requirements for surface pretreatment, and cannot effectively quantify corrosion features, geometric parameters and three-dimensional spatial information. The existing weld corrosion detection through deep learning technology mainly identifies the corrosion area of the two-dimensional image, and cannot effectively obtain the feature description of the corrosion defect area in the three-dimensional space, which cannot meet the actual demand for obtaining feature information of weld corrosion features. Therefore, how to improve the effectiveness of the acquisition result of the weld corrosion feature under the premise of avoiding the consumption of human resources is a problem to be solved by those skilled in the art.

[0003] Chinese patent application publication No. CN118657664A discloses a weld feature extraction method, comprising the following steps: using an industrial CCD camera to collect a weld image; transmitting the collected weld image to an image processing system, performing image preprocessing on the weld image uploaded to the image processing system, and making the weld image clearer through gray scale processing and image enhancement; performing edge detection, image segmentation, erosion operation and inflation operation on the weld image to extract a weld contour image; quantizing the weld contour image, selecting features and reducing dimensions to obtain weld features; and finally visualizing the weld features for easy understanding and analysis by an operator. However, the above-mentioned scheme has the following defects: it cannot effectively reconstruct three-dimensional information of weld corrosion defects according to the obtained two-dimensional weld image, resulting in poor completeness of the defect feature information obtained in the weld corrosion detection process. SUMMARY

[0004] Therefore, the present application provides a weld corrosion feature extraction method and system based on deep learning and SfM, to overcome the problem that the prior art cannot effectively reconstruct three-dimensional information of weld corrosion defects according to the obtained two-dimensional weld image, resulting in poor completeness of the defect feature information obtained in the weld corrosion detection process.

[0005] To achieve the above-mentioned purpose, the present application provides a weld corrosion feature extraction method based on deep learning and SfM, comprising:

[0006] obtaining multi-angle feature recognition images of a target recognition structure, and taking the obtained feature recognition images as input information of a trained recognition optimization network to determine corrosion semantic labels of each feature recognition image;

[0007] generate a target three-dimensional point cloud for the target recognition structure based on rust information in a rust semantic label of each feature recognition image, wherein

[0008] determine key feature points according to a binary mask of each feature recognition image, perform bundle adjustment and camera parameter optimization based on the key feature points to determine a sparse point cloud;

[0009] obtain point cloud holes existing in the determined sparse point cloud, determine the distribution state of each point cloud hole according to the distribution mask parameter and the distribution mask difference parameter, and determine the reconstruction repair mode of the corresponding point cloud hole based on the distribution state of each point cloud hole to determine a dense point cloud, and perform filtering processing to output the target three-dimensional point cloud;

[0010] determine whether to adjust the acquisition execution parameter of the corresponding analysis sub-region according to the reconstruction execution distribution parameter of each analysis sub-region;

[0011] perform dense slicing on the generated target three-dimensional point cloud along the length direction of the weld to obtain a plurality of cross-sectional point clouds, and determine the weld base parameter and the rust geometric parameter of the target recognition structure based on the obtained each cross-sectional point cloud;

[0012] output the rust feature fusion result of the target recognition structure.

[0013] Further, the determination process of the sparse point cloud comprises:

[0014] based on the division interval parameter, divide the scanning region corresponding to the target recognition structure into a plurality of analysis sub-regions with overlap along the length direction of the weld;

[0015] perform bundle adjustment on the feature recognition image corresponding to each analysis sub-region to obtain the local camera optimization parameter of the corresponding analysis sub-region;

[0016] determine the global camera optimization parameter of the scanning region corresponding to the target recognition structure based on the local camera optimization parameter corresponding to each analysis sub-region to obtain the sparse point cloud.

[0017] Further, the distribution state of the point cloud hole comprises a first distribution state, a second distribution state and a third distribution state;

[0018] The point cloud hole in the first distribution state is a point cloud hole with a distribution mask parameter less than or equal to a preset distribution mask parameter;

[0019] The point cloud hole in the second distribution state is a point cloud hole with a distribution mask parameter greater than a preset distribution mask parameter and a distribution mask difference parameter less than or equal to a preset distribution mask difference parameter;

[0020] The point cloud hole in the third type of distribution state is a point cloud hole with a distribution mask parameter greater than a preset distribution mask parameter and a distribution mask difference parameter greater than a preset distribution mask difference parameter.

[0021] Further, the distribution mask parameter and the distribution mask difference parameter of any point cloud hole are determined according to the binary mask and the geometric feature parameter of the hole boundary point.

[0022] Further, Poisson reconstruction is performed on the point cloud hole in the first type of distribution state to generate a dense point cloud corresponding to the point cloud hole.

[0023] Edge constraint processing is performed on the point cloud hole in the third type of distribution state, and the point cloud hole corresponding semantic boundary line is used as a hard constraint for triangulation.

[0024] Further, feature parameter reconstruction is performed on the point cloud hole in the second type of distribution state, or the point cloud hole in the third type of distribution state and the edge constraint processing is completed, and the dense point cloud corresponding to the point cloud hole is synthesized based on the spatial vector parameter of the hole boundary point of the point cloud hole.

[0025] Further, under the reconstruction completion condition, the reconstruction execution distribution parameter of each analysis sub-region is detected, and the reconstruction execution distribution parameter is determined according to the execution evaluation parameter of each point cloud hole.

[0026] The reconstruction completion condition is that the target recognition structure target three-dimensional point cloud is completed, that is, each point cloud hole is completed.

[0027] Further, based on the reference execution evaluation parameter, the acquisition execution parameter of the analysis sub-region with the reconstruction execution distribution parameter greater than the preset reconstruction execution distribution parameter is increased.

[0028] The increase value of the acquisition execution parameter and the reference execution evaluation parameter are in a positive correlation.

[0029] Further, the training process of the recognition optimization network comprises:

[0030] A weld corrosion image dataset is constructed.

[0031] A CBAM attention module is embedded in the YOLOv8 basic network to strengthen the feature extraction of the corrosion area, and the CBAM attention module is embedded between the output end of the C2f module of the YOLOv8 backbone network and the input end of the neck SPPF module.

[0032] Through the cooperation of channel attention and spatial attention, the corrosion area features are strengthened and the background noise is suppressed.

[0033] According to formula (1), the segmentation boundary precision and the sample imbalance degree are optimized.

[0034] ,

[0035] wherein, is a corresponding weight coefficient, is a corresponding weight coefficient;

[0036] The rust semantic label of each feature recognition image obtained for the target recognition structure includes a rust area binary mask, a bounding box coordinate, and a confidence.

[0037] The application also provides a weld rust feature extraction system based on deep learning and SfM, comprising:

[0038] A multi-view acquisition module is configured to acquire multi-angle feature recognition images of the target recognition structure.

[0039] A rust analysis module is connected to the multi-view acquisition module and configured to use the acquired feature recognition images as input information of the trained recognition optimization network to determine the rust semantic label of each feature recognition image.

[0040] A three-dimensional reconstruction module is connected to the rust analysis module and comprises a sparse point cloud construction unit and a dense point cloud construction unit, configured to generate a target three-dimensional point cloud for the target recognition structure based on the rust information in the rust semantic label of each feature recognition image.

[0041] The sparse point cloud construction unit is configured to perform bundle adjustment and camera parameter optimization based on key feature points to determine a sparse point cloud.

[0042] The dense point cloud construction unit is configured to determine a reconstruction repair method for a corresponding point cloud hole based on the distribution state of the point cloud hole to determine a dense point cloud, wherein the distribution state of the point cloud hole is determined according to a distribution mask parameter and a distribution mask difference parameter.

[0043] A reconstruction evaluation module is connected to the three-dimensional reconstruction module and configured to determine whether to adjust the acquisition execution parameter for the corresponding analysis sub-region according to the reconstruction execution distribution parameter of each analysis sub-region.

[0044] A slice execution module is connected to the three-dimensional reconstruction module and configured to perform dense slicing on the generated target three-dimensional point cloud to obtain a plurality of cross-sectional point clouds, and determine the weld base parameter and the rust geometric parameter of the target recognition structure based on the acquired cross-sectional point clouds.

[0045] Compared with the prior art, the present application has the beneficial effects that the technical scheme of the present application is based on the YOLOv8-CBAM network to perform accurate segmentation of the corrosion area for the multi-view images obtained for the weld corrosion detection target, and by optimizing the network to fuse a double loss function, the accuracy of the segmentation result for the corrosion area is improved, targeted guidance is provided for the SfM reconstruction process, data analysis redundancy in the SfM three-dimensional reconstruction analysis process is avoided, and the analysis efficiency and accuracy of the three-dimensional reconstruction result are improved, and the extraction quality and efficiency of the weld corrosion feature are improved.

[0046] Further, in the present application, key feature points are screened based on corrosion semantic labels, sparse point clouds are obtained by combining bundle adjustment, and in addition to reducing resource consumption in the actual construction process, the effectiveness of the obtained sparse point clouds is also improved, and the actual state of the point cloud holes in the sparse point clouds is determined according to the distribution mask parameters and the distribution mask difference parameters, so as to determine the targeted reconstruction means for each point cloud hole, so that the reconstruction process for the point cloud holes is more in line with the actual situation, and the processing efficiency of the reconstruction process is improved while ensuring the construction effect of the dense point clouds.

[0047] Further, in the present application, the distribution state of the point cloud holes in the sparse point clouds is determined according to the distribution mask parameters and the distribution mask difference parameters, the correlation degree of the position of the point cloud holes relative to the corrosion area and the weld area of the target recognition structure and the complexity degree of the position are represented by the distribution mask parameters and the distribution mask difference parameters, so as to determine the reconstruction process of different point cloud holes, which not only ensures the reconstruction efficiency of simple holes, but also improves the fidelity of relatively complex point cloud holes, and the construction efficiency and quality of the target three-dimensional point cloud are improved, thereby improving the extraction quality and efficiency of the weld corrosion feature.

[0048] Further, in the present application, the multi-view weld corrosion defect is reconstructed, and the section is extracted for the obtained 3D point cloud model to realize effective extraction of the corrosion defect feature, and by converting the obtained weld corrosion defect image into "two-dimensional-three-dimensional-two-dimensional", the integrity of the feature analysis result of the weld corrosion defect is improved, which not only includes the basic geometric parameters such as the weld toe radius and the side angle of the weld itself, but also includes quantitative indexes such as the corrosion area section area and the depth, and the integrity of the feature analysis result of the corrosion area in the weld corrosion detection is improved. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 a schematic diagram of the weld corrosion feature extraction method based on deep learning and SfM of the present application;

[0050] Figure 2A flowchart for determining the distribution state of each point cloud hole according to the distribution mask parameter and the distribution mask difference parameter of the present application;

[0051] Figure 3 A flowchart for determining the reconstruction repair method of the corresponding point cloud hole based on the distribution state of each point cloud hole of the present application;

[0052] Figure 4 A module connection diagram of the weld corrosion feature extraction system based on deep learning and SfM of the present application. DETAILED DESCRIPTION

[0053] In order to make the objects and advantages of the present application clearer, the present application will be further described below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0054] The preferred embodiments of the present application will be described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the protection scope of the present application.

[0055] It should be noted that in the description of the present application, the terms "upper", "lower", "left", "right", "inner", "outer" and the like indicate the direction or positional relationship terms based on the direction or positional relationship shown in the drawings, which are only for the convenience of description and do not indicate or imply that the device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0056] In addition, it should also be noted that in the description of the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication inside two elements. Those skilled in the art can understand the specific meaning of the above terms in the present application according to the specific circumstances.

[0057] Please refer to Figures 1 to 3 The embodiment of the present application provides a weld corrosion feature extraction method based on deep learning and SfM, which comprises:

[0058] Obtain multi-angle feature recognition images of the target recognition structure, and take the obtained feature recognition images as input information of the recognition optimization network completed training, to determine the corrosion semantic label of each feature recognition image;

[0059] Generate a target three-dimensional point cloud for the target recognition structure based on the corrosion information in the corrosion semantic label of each feature recognition image, wherein,

[0060] According to the binary mask of the feature recognition image, the key feature points are determined, the bundle adjustment and camera parameter optimization are performed based on the key feature points, so as to determine the sparse point cloud;

[0061] The point cloud holes existing in the determined sparse point cloud are obtained, the distribution state of each point cloud hole is determined according to the distribution mask parameter and the distribution mask difference parameter, and the reconstruction repair mode of the corresponding point cloud hole is determined based on the distribution state of each point cloud hole, so as to determine the dense point cloud, and the filtering processing is performed, and the target three-dimensional point cloud is output;

[0062] According to the reconstruction execution distribution parameter of each analysis sub-region, it is determined whether to adjust the acquisition execution parameter of the corresponding analysis sub-region;

[0063] The generated target three-dimensional point cloud is densely sliced along the length direction of the weld, so as to obtain a plurality of cross-section point clouds, and the weld base parameter and the corrosion geometric parameter of the target recognition structure are determined based on the obtained each cross-section point cloud;

[0064] The corrosion feature fusion result of the target recognition structure is output.

[0065] In the present application, the three-dimensional feature information is extracted in the process of weld corrosion detection, without additional consumption of human resources, the effectiveness of the obtained weld corrosion feature is further improved, the object to be detected for weld corrosion is recorded as a target recognition structure, due to the structure problem of the target recognition structure, the geometric information acquisition is easy to be lost due to the space occlusion when the image of the target recognition structure is acquired, therefore, in addition to the analysis of the corrosion condition of the target recognition structure, the three-dimensional reconstruction is performed to obtain the three-dimensional feature information of the weld corrosion condition of the target recognition structure, in the present application, the industrial camera and the laser positioning assembly are carried by the movable holder to perform full-angle image acquisition on the visible surface of the target recognition structure, the acquired image is recorded as a feature recognition image of the target recognition structure, and any feature recognition image is associated with shooting coordinate information.

[0066] Dense slices are performed on the generated target three-dimensional point cloud along the length direction of the weld, the length direction of the weld is taken as the slice reference direction, a unified local coordinate system is constructed, the slice plane is ensured to be perpendicular to the extension direction of the weld, in the target three-dimensional point cloud, more than 10 evenly distributed weld feature points, such as weld crown points, are selected along the extension direction of the weld, a straight line is fitted by using the least square method, the straight line is taken as the X-axis of the local coordinate system, the center of symmetry of the weld section is selected, the plane where the section is located is fitted, the normal vector of the plane is taken as the Y-axis, and the Z-axis is the height direction of the weld; the intersection point of the center line of the starting end of the weld and the surface of the target recognition structure is recorded as the origin of the coordinate system, all point coordinates of the global dense point cloud are converted to the local coordinate system, the slice planes perpendicular to the X-axis are sequentially generated from the origin along the positive direction of the X-axis, the initial section point cloud of each slice plane is obtained by using a plane clipping algorithm, the initial section point cloud is projected to the corresponding slice plane, and the section point cloud of the corresponding slice plane is recorded;

[0067] The categories of the weld base parameters include a weld width, a weld crown, a fusion width and a bevel angle, how to determine each weld base parameter is a content mastered by those skilled in the art, for example, for a single section point cloud, the weld width is the maximum span of the contour line of the weld in the Y-axis direction in the section point cloud, and details are not repeated herein; the categories of the corrosion geometric parameters include but are not limited to a section corrosion coverage area, a section corrosion thickness, a corrosion depth and a corrosion protrusion height, when each category of the corrosion geometric parameter is determined, the corrosion area and the weld body area need to be segmented in the section point cloud, how to determine each corrosion geometric parameter is a content mastered by those skilled in the art, for example, for a single section point cloud, the section corrosion thickness is the vertical distance between the highest point of the segmented corrosion area and the body surface of the target recognition structure, and details are not repeated herein; the output corrosion feature fusion result of the target recognition structure includes the target three-dimensional point cloud with a corrosion semantic label, the weld base parameters and the corrosion geometric parameters.

[0068] Specifically, the determination process of the sparse point cloud includes:

[0069] The scanning region corresponding to the target recognition structure is divided into a plurality of analysis sub-regions with overlap based on a division interval parameter along the length direction of the weld;

[0070] Beam adjustment is performed on each feature recognition image corresponding to the analysis sub-region to obtain the local camera optimization parameter of the corresponding analysis sub-region;

[0071] The global camera optimization parameter of the scanning region corresponding to the target recognition structure is determined based on the local camera optimization parameter corresponding to each analysis sub-region to obtain the sparse point cloud.

[0072] Wherein, when dividing the analysis sub-regions, the length of each analysis sub-region in the weld direction is the division interval parameter, the area of the scanning region corresponding to each analysis sub-region is consistent, there is an overlapping part between the scanning regions corresponding to any two adjacent analysis sub-regions, and the corresponding adjacent overlap ratio parameter is consistent, for any two analysis sub-regions with overlap, the adjacent overlap ratio parameter , is the area of the scanning region corresponding to the analysis sub-region, is the area of the overlapping region between the scanning regions corresponding to the above two analysis sub-regions with overlap, the values of the division interval parameter and the adjacent overlap ratio parameter can be set by the user according to the actual situation, a value of the division interval parameter is provided, which is 5% of the total scanning length of the target recognition structure in the weld direction, and a value of the adjacent overlap ratio parameter is provided, which is 20%;

[0073] For a single analysis sub-region, when performing local optimization based on bundle adjustment, the analysis sub-region , the set of feature recognition images corresponding to the analysis sub-region , is the number of feature recognition images corresponding to the analysis sub-region, when extracting key feature points, the SIFT algorithm is applied to each feature recognition image for detection, and the set of SIFT feature points obtained for the jth feature recognition image is , is the number of SIFT feature points existing in the feature recognition image, the spatial information corresponding to each SIFT feature point includes coordinate, scale and direction information, each SIFT feature point is checked for its corresponding pixel value in the rust binary mask, if the corresponding pixel value is foreground or rust pixel, the SIFT feature point is retained and recorded as a key feature point, if the corresponding pixel value is a background pixel, the SIFT feature point is removed, and the retained key feature points are used as the input for performing bundle adjustment, for the feature recognition images existing in the analysis sub-region, feature point matching is performed based on the SIFT descriptor to establish the corresponding relationship across images, i.e. to obtain key feature points with consistent spatial information between different feature recognition images; the initial intrinsic parameter obtained in the pre-preparation stage is used as the initial value of the camera intrinsic parameter of the analysis sub-region; the key feature points and the EPnP algorithm are used to solve the initial value of the camera extrinsic parameter, which includes a rotation matrix and a translation vector, the objective function constructed by bundle adjustment , wherein, is the number of feature recognition images corresponding to the analysis sub-region, is the number of key feature points in the analysis sub-region, is the actual pixel coordinate of the vth key feature point on the uth image, For the camera intrinsic parameter matrix, as well as Let u be the extrinsic parameter of the feature recognition image. For the first The three-dimensional coordinates of the key feature points The camera projection function is used; local camera parameter optimization is performed and verified. Intrinsic parameters, extrinsic parameters, and the 3D coordinates of key feature points are used as optimization variables and input into the Ceres Solver open-source library. Convergence conditions are set (RMS reprojection error ≤ 1.0 pixel, maximum iterations ≤ 100 times). Sparse BA solution is performed. After the solution is completed, the RMS reprojection error of the analysis sub-region is calculated. If RMS > 1.5 pixels, the feature points of the sub-region are re-extracted and the above steps are repeated until the accuracy requirements are met. The local camera optimization parameters of the analysis sub-region are saved, including the optimized intrinsic parameter correction values, the optimized extrinsic parameters for each image, and the local 3D coordinates of key feature points within the analysis sub-region. The first analysis sub-region at the weld start end is selected as the global baseline, and its local camera optimization parameters are directly used as the global initial parameters. For the i-th analysis sub-region... (i≥2), extract the sub-regions that are analyzed in the previous analysis. Common feature points within the overlapping region are solved using the ICP algorithm. Local coordinate system to The transformation matrix of the local coordinate system will The local camera optimization parameters are transformed to the global coordinate system using a transformation matrix, resulting in... The initial values ​​of the global camera parameters are determined. After the initial transformation of all analysis sub-regions, a global bundle adjustment objective function is constructed. With the goal of minimizing the global reprojection error, the initial values ​​of the global camera parameters for all sub-regions are jointly optimized. The weight allocation is inversely proportional to the local reprojection error of the sub-region; the higher the local accuracy, the greater the weight. Globally unified camera intrinsic parameters and global extrinsic parameters of the feature recognition image are saved. The global sparse point cloud generation process includes: based on the optimized global camera parameters, the 3D coordinates of key feature points in all analysis sub-regions are solved using a linear triangulation algorithm to obtain the global 3D coordinates of each key feature point. The global 3D coordinates of all key feature points are then summarized to form the initial global sparse point cloud.

[0074] Specifically, the distribution states of the point cloud holes include three types: type I, type II, and type III.

[0075] Point cloud holes in a type I distribution state are point cloud holes whose distribution mask parameters are less than or equal to the preset distribution mask parameters;

[0076] The point cloud hole in the second distribution state is a point cloud hole with a distribution mask parameter greater than a preset distribution mask parameter and a distribution mask difference parameter less than or equal to a preset distribution mask difference parameter.

[0077] The point cloud hole in the third distribution state is a point cloud hole with a distribution mask parameter greater than a preset distribution mask parameter and a distribution mask difference parameter greater than a preset distribution mask difference parameter.

[0078] Specifically, the distribution mask parameter and the distribution mask difference parameter of any point cloud hole are determined according to the binary mask and the geometric feature parameter of the hole boundary point.

[0079] If the target recognition structure completes the construction of the sparse point cloud, the determined point cloud hole existing in the sparse point cloud is obtained, and the neighborhood density of each point in the obtained sparse point cloud is detected when the point cloud hole is obtained. The value of k is selected as 20, the number of point clouds in the neighborhood radius of each point in the sparse point cloud is determined, and is recorded as the local density of the point. For any point existing in the sparse point cloud, if the local density of the point is less than a preset local density parameter, the point is recorded as a potential hole point. The preset local density parameter is the product of the average value of the local density of each point existing in the sparse point cloud and a scene evaluation coefficient. The value of the scene evaluation coefficient can be set by the user according to the actual working scene. The more complex the texture of the rust area of the target recognition structure is, the smaller the value of the scene evaluation coefficient is. A value of the scene evaluation coefficient is provided, and the value of the scene evaluation coefficient is 0.3. The potential hole points are clustered by using a connected domain analysis algorithm to form continuous hole regions. Each connected domain corresponds to a point cloud hole. The boundary points of each point cloud hole are extracted and recorded as the hole boundary points corresponding to the point cloud hole. For a single point cloud hole, the distribution mask parameter is , is the average value of the mask values of each boundary point corresponding to the point cloud hole, is the average value of the mask values of each boundary point corresponding to the point cloud hole, , is the average value of the mask values of each boundary point corresponding to the point cloud hole, is the average value of the mask values of each boundary point corresponding to the point cloud hole, The category of the geometric feature parameter used to determine the distribution mask difference parameter includes but is not limited to the normal vector angle and the curvature. How to determine the normal vector angle and the curvature of each key feature point is easy to understand for those skilled in the art, and is not described here. A value of the evaluation mask value is provided, and the value of the evaluation mask value is 1.

[0080] The values of the preset distribution mask parameter and the preset distribution mask difference parameter can be understood as follows: the higher the user's requirement for the quality of the rust feature information of the target recognition structure, the smaller the value of the preset distribution mask parameter, and the smaller the value of the preset distribution mask difference parameter. The distribution mask parameter represents the rust degree of the point cloud hole existing in the constructed sparse point cloud. The distribution mask difference parameter represents the feature difference degree between the point cloud hole and the surrounding point cloud. A value of the preset distribution mask parameter is provided, which is 0.4. A value of the preset distribution mask difference parameter is provided, which is 0.2.

[0081] Specifically, Poisson reconstruction is performed on the point cloud hole in the first distribution state to generate a dense point cloud corresponding to the point cloud hole.

[0082] Edge constraint processing is performed on the point cloud hole in the third distribution state, and the semantic boundary line corresponding to the point cloud hole is used as a hard constraint for triangulation.

[0083] For a single point cloud hole, if the point cloud hole is in the first distribution state, it indicates that the region corresponding to the point cloud hole is more likely to be unrelated to the rust region. At this time, the accuracy requirement for the reconstruction and repair of the point cloud hole is low. Therefore, Poisson reconstruction is performed on the point cloud hole to meet the actual demand, effectively improving the reconstruction and repair efficiency of the point cloud hole. How to perform Poisson reconstruction on such a point cloud hole is easily understood by those skilled in the art, and will not be described here.

[0084] For a single point cloud hole, if the point cloud hole is in the third distribution state, it indicates that the region corresponding to the point cloud hole is more likely to be related to the rust region, and the difference degree between the geometric features of the boundary points existing in the point cloud hole is large. Since direct reconstruction of such a point cloud hole is prone to edge distortion, in order to ensure the reconstruction quality of the point cloud hole, the semantic boundary line of the point cloud hole is used as a hard constraint for triangulation processing to ensure the processing effect of the subsequent reconstruction process.

[0085] For edge constraint processing of any point cloud hole in the three-class distribution state, the hole boundary points of the point cloud hole are taken as input, and whether the corresponding hole boundary point is in the corrosion area is determined according to the corrosion area binary mask of each hole boundary point. For any hole boundary point, if the corrosion area binary mask of the nearest neighbor point existing in space of the hole boundary point is different from the corrosion area binary mask of the hole boundary point, the hole boundary point is recorded as a semantic boundary point. All semantic boundary points are clustered, and a semantic boundary line set is formed through spline curve fitting or minimum spanning tree connection. The obtained sparse point cloud and semantic boundary line sampling points are combined to form a triangulation input point set, and constraint Delaunay triangulation is performed. The semantic boundary line is taken as a constraint edge that must be included in the triangulation network, and a triangulation network satisfying the Delaunay property is constructed, while ensuring that all constraint edges exist as edges of the triangulation network.

[0086] Specifically, for the point cloud hole in the two-class distribution state, or the point cloud hole in the three-class distribution state and completing the edge constraint processing, feature parameter reconstruction is performed, and the spatial vector parameters of the hole boundary points of the point cloud hole are synthesized to obtain the corresponding dense point cloud.

[0087] For a single point cloud hole, if the point cloud hole is in the two-class distribution state, it indicates that the region corresponding to the point cloud hole is more likely to be related to the corrosion area, and the difference between the geometric features of the boundary points existing in the point cloud hole is small. For the point cloud hole in the two-class distribution state and the point cloud hole in the three-class distribution state and completing the edge constraint processing, the consistency of the collective features of the surrounding area needs to be preserved to avoid distortion of the reconstruction result. Therefore, feature parameter reconstruction is performed for the point cloud hole in the two-class distribution state and the point cloud hole in the three-class distribution state and completing the edge constraint processing.

[0088] For any point cloud hole in a second-class distribution state, or in a third-class distribution state and completing edge constraint processing, when feature parameter reconstruction is performed, a set of hole boundary points of the point cloud hole is extracted, and each type of spatial vector parameter of each hole boundary point is calculated. The types of spatial vector parameters include: normal vector, curvature, and neighborhood distance distribution vector. A tetrahedral mesh skeleton inside the point cloud hole is constructed based on the hole boundary points, for example, the Marching Tetrahedra algorithm is used to determine the spatial position index of each to-be-generated point inside the point cloud hole. The inverse distance weighted interpolation method is used to predict the spatial vector parameters of each to-be-generated point in the mesh skeleton based on the spatial vector parameters of the boundary points. The weight is inversely proportional to the distance from the to-be-generated point to the boundary point, which ensures smooth transition of the parameters from the boundary to the inside and preserves the corrosion morphology. According to the predicted internal point parameters, a dense point cloud with parameters meeting the characteristics is generated in the mesh skeleton, for example, a convex point cluster is generated in a high-curvature area, and a stripping slope is generated in a normal vector mutation area. By comparing the point cloud density of the surrounding corrosion area, the sampling density of the dense point cloud generated for the point cloud hole is adjusted to ensure uniformity. Local feature calibration is performed, and the dense point cloud generated for the point cloud hole is fused with the boundary point cloud and the surrounding normal point cloud to complete reconstruction. How to calculate the normal vector, curvature, and neighborhood distance distribution vector of each hole boundary point is easily understood by those skilled in the art, for example, the neighborhood distance distribution vector is used to represent the spatial structure of the local neighborhood of the hole boundary point, and its construction method is as follows: for any hole boundary point, the Euclidean distances between the hole boundary point and K nearest neighbor points in the global point cloud are calculated, and the distance sequence is sorted and counted (such as taking the first M distance values) to form an M-dimensional descriptor. During reconstruction, by interpolating these vectors, it can be ensured that the point cloud generated inside the hole is consistent with the boundary area in terms of local density and arrangement pattern, which will not be described here.

[0089] Specifically, under the reconstruction completion condition, the reconstruction execution distribution parameter of each analysis sub-region is detected, and the reconstruction execution distribution parameter is determined according to the execution evaluation parameter of each point cloud hole.

[0090] The reconstruction completion condition is that the target three-dimensional point cloud of the target recognition structure is completed, that is, each point cloud hole is completed.

[0091] Specifically, based on the reference execution evaluation parameter, the acquisition execution parameter of the analysis sub-region with the reconstruction execution distribution parameter greater than the preset reconstruction execution distribution parameter is increased.

[0092] The increase value of the acquisition execution parameter and the reference execution evaluation parameter are in a positive correlation.

[0093] For a single analysis sub-region, the reconstruction execution distribution parameter , a number of relevant point cloud holes existing in the analysis sub-region, a number of point cloud holes existing in the target recognition structure, a reference execution evaluation parameter, the reference execution evaluation parameter being an average value of execution evaluation parameters of relevant point cloud holes existing in the analysis sub-region, for a single point cloud hole, if the point cloud hole is mapped to a region where the target recognition structure exists corresponding to the analysis sub-region, the point cloud hole is recorded as a relevant point cloud hole of the analysis sub-region, and the execution evaluation parameter is the sum of the distribution mask parameter and the distribution mask difference parameter of the point cloud hole;

[0094] For a single analysis sub-region, if the reconstruction execution distribution parameter of the analysis sub-region is greater than the preset reconstruction execution distribution parameter, it indicates that there is a large degree of missing point cloud information in the corresponding part of the analysis sub-region in the construction process of the dense point cloud, so that the weld corrosion image obtained for the analysis sub-region cannot meet the actual construction requirements under the same conditions, and therefore the acquisition execution parameter for the analysis sub-region is adjusted to optimize the acquisition of the analysis sub-region corresponding part for the analysis sub-region in subsequent weld corrosion image acquisition, the acquisition execution parameter being the pixel overlap ratio of the adjacent two feature recognition images obtained for the analysis sub-region in the weld length direction, the adjusted acquisition execution parameter , the unadjusted acquisition execution parameter, a reference execution evaluation parameter, the unadjusted acquisition execution parameter, it can be understood that the higher the user's requirement for the quality of the rust feature information of the target recognition structure, the greater the value of the unadjusted acquisition execution parameter, and a value of the unadjusted acquisition execution parameter is provided, the value of the unadjusted acquisition execution parameter being 20%;

[0095] the value of the preset reconstruction execution distribution parameter, it can be understood that the higher the user's requirement for the quality of the rust feature information of the target recognition structure, the smaller the value of the preset reconstruction execution distribution parameter, and the reconstruction execution distribution parameter represents the degree of missing point cloud information in the three-dimensional reconstruction process of the analysis sub-region, a value of the preset reconstruction execution distribution parameter is provided, the value of the preset reconstruction execution distribution parameter being 0.9.

[0096] Specifically, the training process of the recognition optimization network includes:

[0097] constructing a weld corrosion image dataset;

[0098] A CBAM attention module is embedded in the YOLOv8 base network to enhance the feature extraction of the rusted area. The CBAM attention module is embedded between the output of the C2f module of the YOLOv8 backbone network and the input of the neck SPPF module.

[0099] By combining channel attention and spatial attention, the characteristics of the rusted area are enhanced and background noise is suppressed;

[0100] The segmentation boundary accuracy and sample imbalance are optimized according to formula (1).

[0101] ,

[0102] in, for The corresponding weighting coefficients, for The corresponding weighting coefficients;

[0103] The feature recognition images obtained from the target recognition structure are used as input information for the recognition optimization network to complete the training. The rust semantic label of each feature recognition image includes the binary mask of the rust region, the bounding box coordinates, and the confidence level.

[0104] The weld corrosion image dataset is a collection of feature recognition images containing corrosion regions collected from several target recognition structures that have completed weld corrosion feature extraction. Annotation tools (such as LabelImg) are used to annotate the corrosion regions in each image, generating a corrosion semantic label containing a binary mask of the corrosion region, bounding box coordinates, and confidence labels. Accurate boundaries should be ensured during annotation, especially for fine-grained division of the transition region between corrosion edges and the background. The annotated images are randomly divided into training, validation, and test sets in a 7:2:1 ratio. In the YOLOv8 backbone network, the CBAM attention module is embedded between the output of the last C2f module and the input of the SPPF module. Specifically, the output feature map of C2f passes sequentially through the channel attention submodule and spatial attention submodule in CBAM before being input to the SPPF module. The channel dimensions of the feature map are weighted, strengthening corrosion-related channel features and suppressing irrelevant channels. Through the synergy of channel attention and spatial attention, corrosion region features are enhanced and background noise is suppressed. In terms of spatial dimension, the location of the rusted area is highlighted, while the background noise is weakened. Focus loss is used to address the imbalance problem. The Dessian loss is used to optimize the segmentation boundary accuracy. The weighted composite loss function is expressed as follows: To alleviate the imbalance in pixel count between the rusted area and the background, as well as The corresponding weighting coefficients need to be set through grid search or experience and optimized on the validation set to provide a certain level of performance. as well as The corresponding weighting coefficient values, The corresponding weighting coefficient is 0.6. The corresponding weight coefficient is 0.4. This invention provides a training process setup scheme: Images in the weld corrosion image dataset are preprocessed, including size unification, normalization, and data augmentation (random rotation, brightness and contrast adjustment, Mosaic enhancement, etc.). Pre-trained YOLOv8 weights are used as initialization. Iterative training is performed on the training set, using AdamW as the optimizer with an initial learning rate of 1e-3, adjusted using a cosine annealing strategy, and the batch size is set according to GPU memory. Model performance is monitored on the validation set, with key evaluation metrics including mAP@0.5 for the detection task and [missing data - likely a score or value]. The IoU and Dice coefficients for the cutting task were calculated. The training iterations were set to 300 rounds, with an initial learning rate of 0.01, adaptively decaying to 1e-5. The final model achieved a rust region segmentation accuracy ≥95% and a localization error ≤1 pixel. Users can adjust the training process according to their specific circumstances; this is easily understood by those skilled in the art and will not be elaborated upon here. The rust semantic label of the feature recognition image includes a binary mask rule for the rust region. Foreground pixel values ​​are recorded as 255, corresponding to pixels in the rust region and weld contour, while background pixel values ​​are recorded as 0, corresponding to pixels in the non-rust region and weld contour.

[0105] Please see Figure 4 The diagram shown is a module connection diagram of the weld corrosion feature extraction system based on deep learning and SfM of the present invention. The present invention also provides a weld corrosion feature extraction system based on deep learning and SfM, comprising:

[0106] A multi-view acquisition module is used to acquire feature recognition images of the target structure from multiple angles;

[0107] The corrosion analysis module, which is connected to the multi-view acquisition module, is used to use the acquired feature recognition images as input information for the trained recognition optimization network in order to determine the corrosion semantic label of each feature recognition image.

[0108] A 3D reconstruction module, which is connected to the corrosion analysis module, includes a sparse point cloud construction unit and a dense point cloud construction unit, used to generate a target 3D point cloud for the target recognition structure based on the corrosion information in the corrosion semantic labels of each feature recognition image.

[0109] The sparse point cloud construction unit is used to perform bundle adjustment and camera parameter optimization based on key feature points to determine the sparse point cloud.

[0110] The dense point cloud construction unit is configured to determine a reconstruction repair mode of a corresponding point cloud hole based on a distribution state of the point cloud hole, to determine the dense point cloud, and the distribution state of the point cloud hole is determined according to a distribution mask parameter and a distribution mask difference parameter.

[0111] The reconstruction evaluation module is connected with the three-dimensional reconstruction module, and is configured to determine whether to adjust a collection execution parameter of a corresponding analysis sub-region according to a reconstruction execution distribution parameter of each analysis sub-region.

[0112] The slice execution module is connected with the three-dimensional reconstruction module, and is configured to perform dense slicing on the generated target three-dimensional point cloud to obtain a plurality of cross-sectional point clouds, and determine a weld seam basic parameter and a corrosion geometric parameter of a target identification structure based on each obtained cross-sectional point cloud.

[0113] So far, the technical solutions of the present application have been described in combination with the preferred embodiments shown in the drawings, but those skilled in the art can easily understand that the protection scope of the present application is obviously not limited to these specific embodiments. Those skilled in the art can make equivalent changes or replacements to the related technical features without departing from the principles of the present application, and the technical solutions after the changes or replacements will all fall within the protection scope of the present application.

Claims

1. A method for extracting weld corrosion features based on deep learning and SfM, characterized in that, include: The feature recognition images of the target recognition structure from multiple angles are obtained, and the obtained feature recognition images are used as input information for the trained recognition optimization network to determine the rust semantic label of each feature recognition image. Based on the corrosion information within the corrosion semantic labels of the images identified by each feature, a 3D point cloud of the target is generated for the target structure. Key feature points are determined by using the binary mask of each feature recognition image, and bundle adjustment and camera parameter optimization are performed based on the key feature points to determine the sparse point cloud. The system obtains the point cloud holes that exist in the sparse point cloud, determines the distribution state of each point cloud hole based on the distribution mask parameters and distribution mask difference parameters, and determines the corresponding point cloud hole reconstruction and repair method based on the distribution state of each point cloud hole in order to determine the dense point cloud, perform filtering processing, and output the target 3D point cloud. Determine whether to adjust the acquisition execution parameters for the corresponding analysis sub-region based on the reconstruction execution distribution parameters of each analysis sub-region; Dense slices are made along the length of the weld to obtain several cross-sectional point clouds, and the weld basic parameters and corrosion geometric parameters of the target identification structure are determined based on the obtained cross-sectional point clouds. Output the fusion result of the corrosion features of the target recognition structure.

2. The method for extracting weld corrosion features based on deep learning and SfM according to claim 1, characterized in that, The process of determining the sparse point cloud includes: Along the length of the weld, the scanning area corresponding to the target identification structure is divided into several overlapping sub-regions based on the partitioning interval parameters; Bundle adjustment is performed on the feature recognition images corresponding to each analysis sub-region to obtain the local camera optimization parameters for the corresponding analysis sub-region. Based on the local camera optimization parameters corresponding to each analysis sub-region, the global camera optimization parameters of the scanning region corresponding to the target recognition structure are determined to obtain sparse point clouds.

3. The weld corrosion feature extraction method based on deep learning and SfM according to claim 2, characterized in that, The distribution states of the point cloud holes include three types: type I, type II, and type III. Point cloud holes in a type I distribution state are point cloud holes whose distribution mask parameters are less than or equal to the preset distribution mask parameters; Point cloud holes in the second type of distribution state are point cloud holes whose distribution mask parameter is greater than the preset distribution mask parameter and whose distribution mask difference parameter is less than or equal to the preset distribution mask difference parameter. Point cloud holes in the three distribution states are those where the distribution mask parameter is greater than the preset distribution mask parameter and the distribution mask difference parameter is greater than the preset distribution mask difference parameter.

4. The weld corrosion feature extraction method based on deep learning and SfM according to claim 3, characterized in that, The distribution mask parameters and distribution mask difference parameters of any point cloud hole are determined based on the binary mask of the hole boundary points and the geometric feature parameters.

5. The weld corrosion feature extraction method based on deep learning and SfM according to claim 3, characterized in that, Poisson reconstruction is performed on point cloud holes in a certain distribution state to generate dense point clouds corresponding to the point cloud holes. Edge constraint processing is performed on point cloud holes in three distribution states, and triangulation is performed based on the semantic boundary line corresponding to the point cloud hole as a hard constraint.

6. The weld corrosion feature extraction method based on deep learning and SfM according to claim 5, characterized in that, For point cloud holes that are in a Class II distribution state or a Class III distribution state and have completed edge constraint processing, feature parameters are reconstructed, and dense point clouds corresponding to the point cloud holes are synthesized based on the spatial vector parameters of the hole boundary points.

7. The weld corrosion feature extraction method based on deep learning and SfM according to claim 6, characterized in that, Once reconstruction is complete, the reconstruction execution distribution parameters for each analysis sub-region are detected. These reconstruction execution distribution parameters are determined based on the execution evaluation parameters of each point cloud hole. The reconstruction is completed when the target recognition structure's three-dimensional point cloud is constructed, meaning that all holes in the point cloud are reconstructed and repaired.

8. The method for extracting weld corrosion features based on deep learning and SfM according to claim 1, characterized in that, Based on the reference execution evaluation parameters, the acquisition execution parameters of the analysis sub-regions whose reconstruction execution distribution parameters are greater than the preset reconstruction execution distribution parameters are increased and adjusted; The increase in the collected execution parameters is positively correlated with the reference execution evaluation parameters.

9. The method for extracting weld corrosion features based on deep learning and SfM according to claim 8, characterized in that, The training process of the recognition optimization network includes: Construct a dataset of weld corrosion images; A CBAM attention module is embedded in the YOLOv8 base network to enhance the feature extraction of the rusted area. The CBAM attention module is embedded between the output of the C2f module of the YOLOv8 backbone network and the input of the neck SPPF module. Based on formula (1), optimize the segmentation boundary accuracy and sample imbalance. ， in, for The corresponding weighting coefficients, for The corresponding weighting coefficients; The feature recognition images obtained from the target recognition structure are used as input information for the recognition optimization network to complete the training. The rust semantic label of each feature recognition image includes the binary mask of the rust region, the bounding box coordinates, and the confidence level.

10. A feature extraction system applying the deep learning and SfM-based weld corrosion feature extraction method according to any one of claims 1 to 9, characterized in that, include: A multi-view acquisition module is used to acquire feature recognition images of the target structure from multiple angles; The corrosion analysis module, which is connected to the multi-view acquisition module, is used to use the acquired feature recognition images as input information for the trained recognition optimization network in order to determine the corrosion semantic label of each feature recognition image. A 3D reconstruction module, which is connected to the corrosion analysis module, includes a sparse point cloud construction unit and a dense point cloud construction unit, used to generate a target 3D point cloud for the target recognition structure based on the corrosion information in the corrosion semantic labels of each feature recognition image. The sparse point cloud construction unit is used to perform bundle adjustment and camera parameter optimization based on key feature points to determine the sparse point cloud. The dense point cloud construction unit is used to determine the reconstruction and repair method of the corresponding point cloud holes based on the distribution state of each point cloud hole, so as to determine the dense point cloud. The distribution state of the point cloud holes is determined according to the distribution mask parameter and the distribution mask difference parameter. The reconstruction evaluation module, which is connected to the three-dimensional reconstruction module, is used to determine whether to adjust the acquisition execution parameters for the corresponding analysis sub-region based on the reconstruction execution distribution parameters of each analysis sub-region. The slicing execution module, which is connected to the three-dimensional reconstruction module, is used to perform dense slicing on the generated target three-dimensional point cloud to obtain several cross-sectional point clouds, and to determine the weld basic parameters and corrosion geometric parameters of the target identification structure based on the obtained cross-sectional point clouds.

Images