Method and system for identifying damage at steel-wood composite beam-column joints using machine vision

Abstract

The present invention discloses a method and system for damage identification at steel-wood composite beam-column joints using machine vision. The method comprises the following steps: S1: acquiring a multimodal image sequence of visible light, near-infrared, and thermal imaging of the steel-wood composite beam-column joint, and performing denoising processing using adaptive anisotropic diffusion filtering to obtain a denoised multimodal image sequence; S2: performing Gaussian pyramid decomposition and Laplacian pyramid feature fusion on the denoised multimodal image sequence tensor to obtain a fused feature map; S3: performing adaptive threshold segmentation and morphological processing on the fused feature map to obtain a final binary map; S4: training a deep learning network to obtain a trained deep learning network; and S5: using the trained deep learning network to perform damage identification. The present invention can accurately identify damage at steel-wood composite beam-column joints, obtain a segmentation probability map, annotate the segmentation results, and visualize the damaged area.

Description

Technical Field

[0001] The present invention relates to the technical field of steel-wood composite beam-column node damage, and in particular to a steel-wood composite beam-column node damage identification method and system using machine vision. Background Art

[0002] With the rapid development of building industrialization, steel-timber composite structures have been widely used in the construction industry due to their environmental friendliness, high efficiency, and cost-effectiveness. In particular, in new building forms such as prefabricated buildings, green buildings, and high-rise timber structures, steel-timber composite beam-column joints, as key load-bearing components, have a performance that directly impacts the safety, durability, and service life of the overall structure. As important load-bearing components in building structures, the damage state of steel-timber composite beam-column joints directly affects the safety performance of the overall structure. Currently, structural damage identification methods primarily include traditional approaches based on physical quantities such as vibration, strain, and displacement.

[0003] Traditional structural damage identification methods mainly rely on analysis of physical quantities measured by sensors. For example, the invention patent with application number 201910800633.3 discloses a beam structure damage identification method based on support reaction and strain. This method measures the strain curve and support reaction of the damaged structure, calculates the stiffness at each position, and identifies the damage location and extent through stiffness mutation. This type of method requires the arrangement of a large number of sensors on the structure, increasing the detection cost. At the same time, the choice of sensor installation location has a significant impact on the identification results, and damage information can only be obtained at discrete measurement points, making it difficult to achieve continuous monitoring of global damage. Summary of the Invention

[0004] In view of this, the present invention provides a method and system for identifying damage at steel-wood composite beam-column nodes using machine vision. The purpose is to provide a method and system for identifying damage at steel-wood composite beam-column nodes using machine vision, combining adaptive filtering denoising, multi-scale feature fusion and deep learning networks to achieve accurate identification and positioning of damage at steel-wood composite beam-column nodes.

[0005] To achieve the above objectives, the present invention provides a method for identifying damage at steel-wood composite beam-column joints using machine vision, comprising the following steps:

[0006] S1: Obtain visible light, near-infrared, and thermal imaging multimodal image sequences of steel-timber composite beam-column joints, perform denoising on them through adaptive anisotropic diffusion filtering, and obtain a denoised multimodal image sequence. Based on the denoised multimodal image sequence, a denoised multimodal image sequence tensor is constructed.

[0007] S2: Perform Gaussian pyramid decomposition and Laplacian pyramid feature fusion on the denoised multimodal image sequence tensor to obtain a fused feature map;

[0008] S3: Perform adaptive threshold segmentation and morphological processing on the fused feature map to obtain the final binary image;

[0009] S4: The deep learning network is trained to obtain a trained deep learning network;

[0010] S5: Use the trained deep learning network to identify damage, obtain segmentation probability maps, annotate segmentation results, and visualize the damaged area. Specifically, it includes:

[0011] Use the trained deep learning network to extract features from the denoised multimodal image sequence tensor, and perform spatial attention guidance on the extracted features and the final binary image to obtain guided features;

[0012] The guided features and the final binary image are concatenated in the feature dimension and then input into the Transformer module. The self-attention mechanism of the Transformer module is used to perform feature fusion to obtain the fused features.

[0013] The fused features are input into the trained deep learning network to segment damaged pixels and output a segmentation probability map.

[0014] Annotate segmentation results on multimodal image sequences to visualize damaged areas.

[0015] Optionally, the S1 includes:

[0016] S11: Acquiring a multimodal image sequence using a sensor array of three different wavelengths I total , specifically:

[0017] I total ={I vis ,I nir ,I thm};

[0018] Among them, I vis , I nir and I thm These are modal images collected in the visible light band, near-infrared band, and thermal imaging band respectively;

[0019] S12: Using iterative optimization strategy to multimodal image sequence I total Perform adaptive anisotropic diffusion filtering to denoise and obtain the denoised multimodal image sequence I den , specifically:

[0020]

[0021] in, and are the denoised pixel values ​​at the pixel point (x, y) of the mth modal image in the denoised multimodal image sequence at the tth and t+1th iterations respectively; m = 1, 2, 3, are the modality numbers, and the first modality image, the second modality image and the third modality image are I vis , I nir and I thm ; When t = 0, is the pixel value of the mth modality image at the pixel point (x, y) in the multimodal image sequence; η is the iteration step parameter; N is the set of four directions: up, down, left, and right; is the gradient value of the mth modal image at the pixel point (x, y) along the direction d; c d (x,y) is the diffusion coefficient of the pixel point (x,y) in direction d, specifically:

[0022]

[0023] Where K(x,y) is the adaptive local threshold at the pixel point (x,y):

[0024] K(x,y)=μ(x,y)+λ·σ(x,y);

[0025] Where μ(x,y) is The mean of the gradient amplitude in a 7×7 local window centered at the pixel point (x, y); σ(x, y) is The standard deviation of the gradient amplitude within a 7×7 local window centered at the pixel point (x, y); λ is the adjustment parameter;

[0026] The iteration termination condition is:

[0027]

[0028] Among them, ε is the convergence threshold; is the L2 norm of the t-th iteration result of the m-th modality image in the denoised multimodal image sequence; is the L2 norm of the difference between the t+1th and tth iteration results of the mth modal image; after the iteration is completed, the mth modal image in the denoised multimodal image sequence is obtained

[0029] S13: Construct the denoised multimodal image sequence tensor T:

[0030]

[0031] Optionally, the S2 includes:

[0032] S21: The denoised multimodal image sequence tensor is decomposed into a Gaussian pyramid, specifically:

[0033]

[0034] in, is the pixel value of the mth modal image in the denoised multimodal image sequence tensor at the pixel point (x, y) in the layer-th Gaussian pyramid decomposition result image; layer = 1, 2, 3, is the number of Gaussian pyramid layers; G sigma is a Gaussian kernel function with a standard deviation of 1.6; * is a convolution operation; The decomposition result of the mth modal image in the layer-1 Gaussian pyramid in the denoised multimodal image sequence tensor is shown. When layer=1 ↓2 is a 2x downsampling operation in the horizontal and vertical directions;

[0035] S22: Construct a Laplace pyramid, specifically:

[0036]

[0037] in, is the result of the Laplacian pyramid of the mth modality image in the layer-1 layer of the denoised multimodal image sequence tensor; Up is the bilinear interpolation upsampling operation, which expands the input image by 2 times in the horizontal and vertical directions; The Gaussian pyramid decomposition result image of the mth modal image in the layer of the denoised multimodal image sequence tensor;

[0038] S23: Calculate the fusion weight, specifically:

[0039]

[0040] in, is the layer-1 layer, and the fusion weight of the mth modality image at the pixel point (x, y) in the denoised multimodal image sequence tensor; is the saliency map of the kth modality image at the pixel (x, y) in the denoised multimodal image sequence tensor at layer-1, k = 1, 2, 3; (x, y) is the layer-1 layer, and the saliency map of the mth modality image in the denoised multimodal image sequence tensor at the pixel point (x, y) is:

[0041]

[0042] Where Ω is the pixel point contained in the 7×7 neighborhood window centered on the pixel point (x, y), and (i, j) is the element in Ω; for The mean of the pixel values ​​in the 7×7 local window centered at the pixel point (x, y);

[0043] S24: Laplacian pyramid feature fusion and reconstruction to obtain a fused feature map.

[0044] Optionally, the S24 includes:

[0045] S241: Perform weighted fusion on the Laplacian pyramid to obtain the fused Laplacian pyramid:

[0046]

[0047] in, is the pixel value after fusion at the pixel point (x, y) of layer-1; is the pixel value at pixel (x, y) of the result image of the mth modality image in the layer-1 Laplacian pyramid in the denoised multimodal image sequence tensor;

[0048] S242: Reconstruct the fused Laplacian pyramid to obtain the fused feature map F:

[0049]

[0050] Among them, Up layer-1 Indicates a layer-1 2x upsampling operation; F(x,y) is the pixel value of the fused feature map at the pixel point (x,y).

[0051] Optionally, in step S3, adaptive threshold segmentation and morphological processing are performed on the fused feature map to obtain a final binary map, including:

[0052] S31: Calculate the adaptive local threshold of the fused feature map, specifically:

[0053]

[0054] Where Th(x,y) is the adaptive local threshold at the pixel point (x,y); is the mean of the fused feature map F in the 15×15 local window centered at the pixel point (x, y); is the standard deviation of the fused feature map F in the 15×15 local window centered at the pixel point (x, y); α is the threshold adjustment coefficient;

[0055] S32: Perform adaptive local threshold segmentation on the fused feature map, and compare the size relationship between the fused feature map F and the adaptive threshold Th at each pixel (x, y) to obtain the initial binary map M0, specifically:

[0056] When F(x,y)>Th(x,y), M0(x,y)=1;

[0057] When F(x,y)≤Th(x,y), M0(x,y)=0;

[0058] Among them, M0(x,y) is the pixel value of the initial binary image M0 at the pixel point (x,y);

[0059] S33: Perform morphological processing on the initial binary image to obtain a final binary image.

[0060] Optionally, the S33 includes:

[0061] S331: Perform opening operation:

[0062] Performing an opening operation on the initial binary image M0, wherein the opening operation uses a 3×3 rectangular structure element to first perform an erosion operation on the initial binary image M0, and then perform an expansion operation to obtain an image M1 after the opening operation;

[0063] S332: Perform closing operation:

[0064] Perform a closing operation on M1. The closing operation also uses a 3×3 rectangular structure element to first perform an expansion operation on M1, and then perform an erosion operation to obtain an image M2 after the closing operation.

[0065] S333: Perform area filtering:

[0066] Perform area filtering on M2. The area filtering is to calculate the pixel area of ​​each connected area in the image M2 and set the pixel values ​​of the pixels contained in the connected areas with an area less than 20 to 0 to obtain the final binary image M. final .

[0067] Optionally, the S4 includes:

[0068] S41: Calculate segmentation loss Loss seg and consistency constraint loss Loss consist , get the total loss Loss total :

[0069] Loss seg =CE(Prob seg ,Label);

[0070] Loss consist =MSE(Prob seg ,Label);

[0071] Loss total =Loss seg+Loss consist ;

[0072] Among them, CE is cross entropy loss; MSE is mean square error; Label is the true segmentation label; Prob seg is the segmentation probability map;

[0073] S42: Use the stochastic gradient descent algorithm to train the parameters in the deep learning network to reduce the total loss; after reaching the set number of iterations, the trained deep learning network is obtained.

[0074] Optionally, the S5 includes:

[0075] The trained deep learning networks include ResNet50 and UNet.

[0076] The ResNet50 network is used to extract features from the denoised multimodal image sequence tensor to obtain the denoised multimodal image sequence tensor features;

[0077] Combine the denoised multimodal image sequence tensor features with the final binary image M final Perform spatial attention guidance to obtain guided multimodal image sequence tensor features;

[0078] The guided multimodal image sequence tensor features and the final binary map M final After splicing in the feature dimension, it is input into the Transformer module for feature fusion to obtain the fused multimodal image sequence tensor feature Fea fuse ;

[0079] The fused multimodal image sequence tensor features are input into the UNet network for damage pixel segmentation to obtain a segmentation probability map, where each pixel value in the segmentation probability map represents the probability that the location belongs to the damage area;

[0080] The segmentation probability map is binarized to obtain the final segmentation mask, and the segmentation results are annotated on the multimodal image sequence to achieve visualization of the damaged area.

[0081] The present invention also discloses a steel-wood composite beam-column joint damage identification system using machine vision, comprising:

[0082] Denoising module: This module obtains visible light, near-infrared, and thermal imaging multimodal image sequences of steel-timber composite beam-column joints, performs denoising processing on them through adaptive anisotropic diffusion filtering, and obtains a denoised multimodal image sequence. Based on this denoised multimodal image sequence, a denoised multimodal image sequence tensor is constructed.

[0083] Fusion module: Perform Gaussian pyramid decomposition and Laplacian pyramid feature fusion on the denoised multimodal image sequence tensor to obtain a fused feature map;

[0084] Binarization module: performs adaptive threshold segmentation and morphological processing on the fused feature map to obtain the final binary map;

[0085] Recognition model training module: Recognition model training module: trains the deep learning network to obtain a trained deep learning network;

[0086] Identification model application module: Use the trained deep learning network to perform damage identification, obtain segmentation probability maps, and annotate segmentation results to visualize the damaged area.

[0087] Compared with the prior art, the present invention has at least the following beneficial effects:

[0088] This method uses image sequences from three different wavelengths—visible light, near-infrared, and thermal—for damage identification, fully leveraging the complementary information of different modal images. Multimodal images are denoised using an adaptive anisotropic diffusion filtering algorithm. This algorithm adaptively adjusts the diffusion coefficient based on local image features, effectively suppressing noise while preserving edge and detail information. Furthermore, an iterative optimization strategy and adaptive termination conditions ensure the stability and reliability of the denoising effect, laying the foundation for subsequent feature extraction and fusion.

[0089] This method decomposes images into features of different scales through Gaussian pyramid decomposition and combines it with Laplacian pyramid reconstruction to achieve multi-scale representation of these features. An innovative adaptive weight fusion strategy based on local saliency allows for optimal combination of features of different modalities and scales based on their importance. Furthermore, adaptive threshold segmentation and morphological processing methods are employed to effectively extract the preliminary outline of the damaged area, providing valuable prior information for deep learning networks.

[0090] The deep learning network architecture designed in this paper has strong feature extraction and fusion capabilities. Feature extraction is performed using ResNet50, combined with a binary graph-guided spatial attention mechanism to highlight the characteristic expression of damage-related areas. The innovative introduction of the Transformer module enables deep fusion of multimodal features, fully exploring the correlation between features of different modalities. A UNet network is used for damage pixel segmentation, and an optimization objective combining cross-entropy loss and consistency constraints is designed to improve the accuracy and robustness of damage identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 The figure is a flow chart of a method for identifying damage at steel-wood composite beam-column joints using machine vision according to an embodiment of the present invention. DETAILED DESCRIPTION

[0092] The present invention will be further described below with reference to the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention fall within the scope of protection of the present invention.

[0093] Example 1: A method for identifying damage at steel-wood composite beam-column joints using machine vision, such as Figure 1 As shown, the following steps are included:

[0094] S1: Obtain visible light, near infrared, and thermal imaging multimodal image sequences of the steel-wood composite beam-column joint, and perform denoising processing through adaptive anisotropic diffusion filtering to obtain a denoised multimodal image sequence. Then, construct a denoised multimodal image sequence tensor based on the denoised multimodal image sequence:

[0095] S11: Acquiring a multimodal image sequence using a sensor array of three different wavelengths I total , specifically:

[0096] I total ={I vis ,I nir ,I thm};

[0097] Among them, I vis , I nir and I thm These are modal images collected in the visible light band, near-infrared band, and thermal imaging band respectively;

[0098] S12: Using iterative optimization strategy to multimodal image sequence I total Perform adaptive anisotropic diffusion filtering to denoise and obtain the denoised multimodal image sequence I den , specifically:

[0099]

[0100] in, and are the pixel values ​​at the pixel point (x, y) of the mth modal image in the denoised multimodal image sequence at the tth and t+1th iterations respectively; m = 1, 2, 3, are the modal numbers, and the first modal image, the second modal image and the third modal image are I vis , I nir and I thm ; When t = 0, is the pixel value of the mth modality image at the pixel point (x, y) in the multimodal image sequence; η is the iteration step parameter, and its value range is (0, 1]; N is the set of four directions: up, down, left, and right; is the gradient value of the mth modal image at the pixel point (x, y) along the direction d; c d (x,y) is the diffusion coefficient of the pixel point (x,y) in direction d, specifically:

[0101]

[0102] Where K(x,y) is the adaptive local threshold at the pixel point (x,y):

[0103] K(x,y)=μ(x,y)+λ·σ(x,y);

[0104] Where μ(x,y) is The mean of the gradient amplitude in a 7×7 local window centered at the pixel point (x, y); σ(x, y) is The standard deviation of the gradient amplitude within a 7×7 local window centered at the pixel point (x, y); λ is a tuning parameter with a value range of [0.8, 1.2];

[0105] The iteration termination condition is:

[0106]

[0107] Among them, ε is the convergence threshold, which is set to 0.001; is the L2 norm of the t-th iteration result of the m-th modality image in the denoised multimodal image sequence; is the L2 norm of the difference between the t+1th and tth iteration results of the mth modal image; after the iteration is completed, the mth modal image in the denoised multimodal image sequence is obtained

[0108] S13: Construct the denoised multimodal image sequence tensor T:

[0109]

[0110] This step collects image data from three different wavelengths: visible light, near-infrared, and thermal imaging, leveraging their complementary information. Visible light images provide clear surface details and texture features, near-infrared images penetrate the surface to reveal internal structures, and thermal images reveal stress distribution and thermal anomalies within the material. This multimodal fusion significantly improves the comprehensiveness and reliability of damage detection.

[0111] The adaptive anisotropic diffusion filter employed in this step offers unique advantages in image processing. Through an iterative optimization strategy, this method automatically adjusts the filter strength based on local image features, enhancing denoising in smooth areas while preserving detail information in edge regions. The diffusion coefficient design during the filtering process takes local gradient information into account, while the introduction of an adaptive local threshold allows the algorithm to intelligently adapt to image characteristics in different regions. By setting appropriate iterative termination conditions, the algorithm's convergence is ensured while avoiding loss of detail due to over-filtering.

[0112] S2: Perform Gaussian pyramid decomposition and Laplacian pyramid feature fusion on the denoised multimodal image sequence tensor to obtain a fused feature map:

[0113] S21: Perform Gaussian pyramid decomposition on the denoised multimodal image sequence tensor, specifically:

[0114]

[0115] in, is the pixel value of the mth modal image in the denoised multimodal image sequence tensor at the pixel point (x, y) in the layer-th Gaussian pyramid decomposition result image; layer = 1, 2, 3, is the number of Gaussian pyramid layers; G sigma is a Gaussian kernel function with a standard deviation of 1.6; * is a convolution operation; The decomposition result of the mth modal image in the layer-1 Gaussian pyramid in the denoised multimodal image sequence tensor is shown. When layer=1 ↓2 is a 2x downsampling operation in the horizontal and vertical directions;

[0116] S22: Construct a Laplace pyramid, specifically:

[0117]

[0118] in, is the result of the Laplacian pyramid of the mth modality image in the layer-1 layer of the denoised multimodal image sequence tensor; Up is the bilinear interpolation upsampling operation, which expands the input image by 2 times in the horizontal and vertical directions; The Gaussian pyramid decomposition result image of the mth modal image in the layer of the denoised multimodal image sequence tensor;

[0119] S23: Calculate the fusion weight, specifically:

[0120]

[0121] in, is the layer-1 layer, and the fusion weight of the mth modality image at the pixel point (x, y) in the denoised multimodal image sequence tensor; is the saliency value of the kth modality at pixel (x, y) in the saliency map of the denoised multimodal image sequence tensor at layer-1, k = 1, 2, 3; is the layer-1 layer, and the saliency value of the mth modality in the denoised multimodal image sequence tensor at the pixel point (x, y) is:

[0122]

[0123] Where Ω is the pixel point contained in the 7×7 neighborhood window centered on the pixel point (x, y), and (i, j) is the element in Ω; for The mean of the pixel values ​​in the 7×7 local window centered at the pixel point (x, y);

[0124] S24: Feature fusion and reconstruction to obtain a fused feature map; specifically:

[0125] S241: performing weighted fusion on the Laplacian pyramid to obtain a fused Laplacian pyramid. The weighted fusion operation is to multiply the pixel values ​​at corresponding positions of each modality image in the denoised multimodal image sequence tensor by corresponding weights for each layer of the Laplacian pyramid, and then add the weighted pixel values. Specifically,

[0126]

[0127] in, is the pixel value after fusion at the pixel point (x, y) of layer-1; is the pixel value at pixel (x, y) of the result image of the mth modality image in the layer-1 Laplacian pyramid in the denoised multimodal image sequence tensor;

[0128] S242: Reconstruct the fused Laplacian pyramid to obtain the fused feature map F:

[0129]

[0130] Among them, Up layer-1 Indicates a layer-1 2x upsampling operation; F(x,y) is the pixel value of the final fused feature map at the pixel point (x,y).

[0131] This step achieves effective feature extraction and fusion of multimodal images through Gaussian pyramid decomposition and fusion strategies. First, each modality's image is decomposed into representations of different scales using Gaussian pyramid decomposition. This hierarchical structure captures multi-scale feature information, from detailed to global. The introduction of the Gaussian kernel function ensures smoothness and stability in the feature extraction process, while the pyramid-based downsampling operation effectively reduces computational complexity.

[0132] The Laplacian pyramid constructed on this basis emphasizes image edges and details by calculating the differences between adjacent scale levels. This bandpass filtering property enables the Laplacian pyramid to effectively separate and preserve image information of different frequency components, providing an ideal feature representation for subsequent feature fusion. Furthermore, bilinear interpolation is used for upsampling to ensure smooth image reconstruction.

[0133] S3: Perform adaptive threshold segmentation and morphological processing on the fused feature map to obtain the final binary image:

[0134] S31: Calculate the adaptive local threshold of the fused feature map, specifically:

[0135]

[0136] Where Th(x,y) is the adaptive local threshold at the pixel point (x,y); is the mean of the fused feature map F in the 15×15 local window centered at the pixel point (x, y); is the standard deviation of the fused feature map F within a 15×15 local window centered at the pixel point (x, y); α is the threshold adjustment coefficient, which is 0.1 in this embodiment;

[0137] S32: Adaptive local threshold segmentation is performed on the fused feature map. The initial binary map M0 is obtained by comparing the size relationship between the fused feature map F and the adaptive threshold Th at each pixel (x, y). Specifically:

[0138] When F(x,y)>Th(x,y), M0(x,y)=1;

[0139] When F(x,y)≤Th(x,y), M0(x,y)=0;

[0140] Among them, M0(x,y) is the pixel value of the initial binary image M0 at the pixel point (x,y);

[0141] S33: Perform morphological processing on the initial binary image to obtain a final binary image; specifically:

[0142] S331: Perform opening operation:

[0143] Performing an opening operation on the initial binary image M0, wherein the opening operation uses a 3×3 rectangular structure element to first perform an erosion operation on M0 and then perform an expansion operation to obtain an image M1 after the opening operation;

[0144] S332: Perform closing operation:

[0145] Perform a closing operation on M1. The closing operation also uses a 3×3 rectangular structure element to first perform an expansion operation on M1, and then perform an erosion operation to obtain an image M2 after the closing operation.

[0146] S333: Perform area filtering:

[0147] Perform area filtering on M2. The area filtering is to calculate the pixel area of ​​each connected area in the image M2 and set the pixel values ​​of the pixels contained in the connected area with an area less than 20 to 0 to obtain the final binary image M. final .

[0148] This step uses adaptive threshold segmentation combined with morphological processing to achieve efficient conversion from the fused feature map to the final binary image. By introducing an adaptive local threshold calculation mechanism, the statistical characteristics of local image regions are fully considered, allowing the segmentation process to better adapt to brightness and contrast variations in different image regions. A large local window size ensures the reliability of statistical features, while a reasonable threshold adjustment coefficient effectively controls the false detection rate while maintaining detection sensitivity.

[0149] This step of binarization uses a simple and intuitive threshold comparison method, but it overcomes the limitations of traditional fixed threshold methods by introducing an adaptive local threshold. This adaptive mechanism is particularly well-suited for processing images with uneven lighting or local contrast differences, improving the robustness of the algorithm while maintaining detection accuracy.

[0150] S4: Train the deep learning network to obtain a trained deep learning network;

[0151] S41: Calculate segmentation loss Loss seg and consistency constraint loss Loss consist , get the total loss Loss total :

[0152] Loss seg =CE(Prob seg ,Label);

[0153] Loss consist =MSE(Prob seg ,Label);

[0154] Loss total =Loss seg +Loss consist ;

[0155] Among them, CE is cross entropy loss; MSE is mean square error; Label is the true segmentation label; Prob seg is the segmentation probability map;

[0156] S42: Use the stochastic gradient descent algorithm to train the parameters in the deep learning network to reduce the total loss; after reaching the set number of iterations, the trained deep learning network is obtained.

[0157] S5: Use the trained deep learning network to identify damage, obtain segmentation probability maps, annotate segmentation results, and visualize the damaged area. Specifically, it includes:

[0158] S51: Feature extraction and binary image guidance:

[0159] S511: Use the ResNet50 network to extract features from the denoised multimodal image sequence tensor, specifically:

[0160]

[0161] in, is the mth modality image in the denoised multimodal image sequence tensor Features extracted by ResNet50 network;

[0162] S512: The final binary image M final The spatial attention guidance is performed with the denoised multimodal image sequence tensor features, specifically:

[0163]

[0164] in, is the mth modality image in the denoised multimodal image sequence tensor After M final The features after guidance; ⊙ is element-wise multiplication;

[0165] S52: And the final binary image M final After splicing in the feature dimension, the data is input into the Transformer module, and the feature fusion is performed through the self-attention mechanism of the Transformer module. Specifically:

[0166]

[0167] Fea fuse =Transformer(Feainput );

[0168] Among them, Fea input is the input of the Transformer module layer; Fea fuse is the fused feature;

[0169] S53: Using UNet network for damaged pixel segmentation:

[0170] The fused features are input into the UNet network for damaged pixel segmentation, and the segmentation probability map is output through the Softmax function;

[0171] Prob seg =Softmax(UNet(Fea fuse ));

[0172] Among them, Prob seg is the segmentation probability map, and each pixel value in the segmentation probability map is the probability value of damage; Softmax is the normalized exponential function.

[0173] S54: Binarize the segmentation probability map to obtain the final segmentation mask, and annotate the segmentation results on the multimodal image sequence to achieve visualization of the damaged area.

[0174] This step achieves efficient feature extraction and fusion segmentation of multimodal images through a deep learning network. First, ResNet50, a classic deep convolutional network, is used to extract features from each modality, leveraging its advantages in image feature extraction. ResNet50 is a deep residual network. Its residual connection design not only enables the network to extract deeper feature information, but also effectively alleviates the vanishing gradient problem in deep network training, ensuring the effectiveness and stability of feature extraction.

[0175] This step innovatively introduces a binary image as guidance for the spatial attention mechanism, achieving selective feature enhancement through element-by-element multiplication. This guidance mechanism enables the network to focus on potential damage areas, effectively suppressing interference from background areas and improving the targeted nature of feature extraction.

[0176] Example 2: The present invention also discloses a steel-wood composite beam-column joint damage identification system using machine vision, which includes the following five modules:

[0177] Denoising module: This module obtains visible light, near-infrared, and thermal imaging multimodal image sequences of steel-timber composite beam-column joints, performs denoising processing on them through adaptive anisotropic diffusion filtering, and obtains a denoised multimodal image sequence. Based on this denoised multimodal image sequence, a denoised multimodal image sequence tensor is constructed.

[0178] Fusion module: Perform Gaussian pyramid decomposition and Laplacian pyramid feature fusion on the denoised multimodal image sequence tensor to obtain a fused feature map;

[0179] Binarization module: performs adaptive threshold segmentation and morphological processing on the fused feature map to obtain the final binary map;

[0180] Recognition model training module: trains the deep learning network to obtain a trained deep learning network;

[0181] Identification model application module: Use the trained deep learning network to perform damage identification, obtain segmentation probability maps, and annotate segmentation results to visualize the damaged area.

[0182] It should be noted that the serial numbers of the above-mentioned embodiments of the present invention are for descriptive purposes only and do not represent the advantages or disadvantages of the embodiments. In addition, the terms "including", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method comprising the element.

[0183] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0184] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for identifying damage at steel-wood composite beam-column joints using machine vision, characterized in that: The following steps are involved: S1: Obtain visible light, near-infrared, and thermal imaging multimodal image sequences of steel-timber composite beam-column joints, perform denoising on them through adaptive anisotropic diffusion filtering, and obtain a denoised multimodal image sequence. Based on the denoised multimodal image sequence, a denoised multimodal image sequence tensor is constructed. S2: Perform Gaussian pyramid decomposition and Laplacian pyramid feature fusion on the denoised multimodal image sequence tensor to obtain a fused feature map; S3: Perform adaptive threshold segmentation and morphological processing on the fused feature map to obtain the final binary image; S4: Train the deep learning network to obtain a trained deep learning network; including: S41: Calculate segmentation loss Loss seg and consistency constraint loss Loss consist , get the total loss Loss total : Loss seg =CE(Prob seg ,Label); Loss consist =MSE(Prob seg ,Label); Loss total =Loss seg +Loss consist ; Among them, CE is cross entropy loss; MSE is mean square error; Label is the true segmentation label; Prob seg is the segmentation probability map; S42: Use the stochastic gradient descent algorithm to train the parameters in the deep learning network to reduce the total loss; after reaching the set number of iterations, the trained deep learning network is obtained; S5: Use the trained deep learning network to identify damage, obtain segmentation probability maps, annotate segmentation results, and visualize the damaged area. Specifically, it includes: Use the trained deep learning network to extract features from the denoised multimodal image sequence tensor, and perform spatial attention guidance on the extracted features and the final binary image to obtain guided features; The guided features and the final binary image are concatenated in the feature dimension and then input into the Transformer module. The self-attention mechanism of the Transformer module is used to perform feature fusion to obtain the fused features. The fused features are input into the trained deep learning network to segment damaged pixels and output a segmentation probability map. Annotate segmentation results on multimodal image sequences to visualize damaged areas.

2. The method for identifying damage at steel-wood composite beam-column joints using machine vision according to claim 1, characterized in that: Said S1 comprises the following steps: S11: Acquiring a multimodal image sequence using a sensor array of three different wavelengths I total , specifically: I total ={I vis ,I nir ,I thm }; Among them, I vis , I nir and I thm These are modal images collected in the visible light band, near-infrared band, and thermal imaging band respectively; S12: Using iterative optimization strategy to multimodal image sequence I total Perform adaptive anisotropic diffusion filtering to denoise and obtain the denoised multimodal image sequence I den , specifically: in, and are the denoised pixel values ​​at the pixel point (x, y) of the mth modal image in the denoised multimodal image sequence at the tth and t+1th iterations respectively; m = 1, 2, 3, are modality numbers, and the first modal image, the second modal image and the third modal image are I vis , I nir and I thm ; When t = 0, is the pixel value of the mth modality image at the pixel point (x, y) in the multimodal image sequence; η is the iteration step parameter; N is the set of four directions: up, down, left, and right; is the gradient value of the mth modal image at the pixel point (x, y) along the direction d; c d (x, y) is the diffusion coefficient of the pixel point (x, y) in the direction d, specifically: Where K(x, y) is the adaptive local threshold at the pixel point (x, y): K(x, y)=μ(x, y)+λ·σ(x, y); Where μ(x, y) is The mean of the gradient amplitude in a 7×7 local window centered at the pixel point (x, y); σ(x, y) is The standard deviation of the gradient amplitude within a 7×7 local window centered at the pixel point (x, y); λ is the adjustment parameter; The iteration termination condition is: Among them, ε is the convergence threshold; is the L2 norm of the t-th iteration result of the m-th modality image in the denoised multimodal image sequence; is the L2 norm of the difference between the t+1th and tth iteration results of the mth modal image; after the iteration is completed, the mth modal image in the denoised multimodal image sequence is obtained S13: Construct the denoised multimodal image sequence tensor T:

3. The method for identifying damage at steel-wood composite beam-column joints using machine vision according to claim 2, characterized in that: Said S2 comprises the following steps: S21: Perform Gaussian pyramid decomposition on the denoised multimodal image sequence tensor, specifically: in, is the pixel value of the mth modal image in the denoised multimodal image sequence tensor at the pixel point (x, y) in the layer-th Gaussian pyramid decomposition result image; layer = 1, 2, 3, is the number of Gaussian pyramid layers; G sigma is a Gaussian kernel function with a standard deviation of 1.6; * is a convolution operation; The decomposition result of the mth modal image in the layer-1 Gaussian pyramid in the denoised multimodal image sequence tensor is shown. When layer=1 ↓2 is a 2x downsampling operation in the horizontal and vertical directions; S22: Construct a Laplace pyramid, specifically: in, is the result of the Laplacian pyramid of the mth modality image in the layer-1 layer of the denoised multimodal image sequence tensor; Up is the bilinear interpolation upsampling operation, which expands the input image by 2 times in the horizontal and vertical directions; The Gaussian pyramid decomposition result image of the mth modal image in the layer of the denoised multimodal image sequence tensor; S23: Calculate the fusion weight, specifically: in, is the layer-1 layer, and the fusion weight of the mth modality image at the pixel point (x, y) in the denoised multimodal image sequence tensor; is the saliency map of the kth modality image at the pixel (x, y) in the denoised multimodal image sequence tensor, layer-1, where k = 1, 2, or 3; is the layer-1 layer, and the saliency value of the saliency map of the mth modality image in the denoised multimodal image sequence tensor at the pixel point (x, y) is specifically: Where Ω is the pixel point contained in the 7×7 neighborhood window centered on the pixel point (x, y), and (i, j) is the element in Ω; for The mean of the pixel values ​​in the 7×7 local window centered at the pixel point (x, y); S24: Laplacian pyramid feature fusion and reconstruction to obtain a fused feature map.

4. The method for identifying damage at steel-wood composite beam-column joints using machine vision according to claim 3, characterized in that: The S24 includes the following steps: S241: Perform weighted fusion on the Laplacian pyramid to obtain the fused Laplacian pyramid: in, is the pixel value after fusion at the pixel point (x, y) of layer-1; is the pixel value at pixel (x, y) of the result image of the mth modality image in the layer-1 Laplacian pyramid in the denoised multimodal image sequence tensor; S242: Reconstruct the fused Laplacian pyramid to obtain the fused feature map F: Among them, Up layer-1 Indicates a layer-1 2x upsampling operation; F(x, y) is the pixel value of the fused feature map at the pixel point (x, y).

5. The method for identifying damage at steel-wood composite beam-column joints using machine vision according to claim 3, characterized in that: The S3 includes the following steps: S31: Calculate the adaptive local threshold of the fused feature map, specifically: Among them, Th(x, y) is the adaptive local threshold at the pixel point (x, y) of the fusion feature map; is the mean of the fused feature map F in the 15×15 local window centered at the pixel point (x, y); is the standard deviation of the fused feature map F in the 15×15 local window centered at the pixel point (x, y); α is the threshold adjustment coefficient; S32: Perform adaptive threshold segmentation on the fused feature map, and compare the size relationship between the fused feature map F and the adaptive local threshold Th at each pixel (x, y) to obtain the initial binary map M0, specifically: When F(x,y)>Th(x,y), M0(x,y)=1; When F(x,y)≤Th(x,y), M0(x,y)=0; Among them, M0(x,y) is the pixel value of the initial binary image M0 at the pixel point (x,y); S33: Perform morphological processing on the initial binary image to obtain a final binary image.

6. The method for identifying damage at steel-wood composite beam-column joints using machine vision according to claim 5, characterized in that: The S33 includes the following steps: S331: Perform opening operation: Performing an opening operation on the initial binary image M0, wherein the opening operation uses a 3×3 rectangular structure element to first perform an erosion operation on the initial binary image M0, and then perform an expansion operation to obtain an image M1 after the opening operation; S332: Perform closing operation: Perform a closing operation on M1. The closing operation also uses a 3×3 rectangular structure element to first perform an expansion operation on M1, and then perform an erosion operation to obtain an image M2 after the closing operation. S333: Perform area filtering: Perform area filtering on M2. The area filtering is to calculate the pixel area of ​​each connected region in the image M2 and set the pixel values ​​of the pixels contained in the connected regions with an area less than 20 to 0 to obtain the final binary image M. final .

7. The method for identifying damage at steel-wood composite beam-column joints using machine vision according to claim 6, characterized in that: The S5 comprises the following steps: The trained deep learning networks include ResNet50 and UNet. The ResNet50 network is used to extract features from the denoised multimodal image sequence tensor to obtain the denoised multimodal image sequence tensor features; Combine the denoised multimodal image sequence tensor features with the final binary image M final Perform spatial attention guidance to obtain guided multimodal image sequence tensor features; The guided multimodal image sequence tensor features and the final binary map M final After splicing in the feature dimension, it is input into the Transformer module for feature fusion to obtain the fused multimodal image sequence tensor feature Fea fuse ; The fused multimodal image sequence tensor features are input into the UNet network for damage pixel segmentation to obtain the segmentation probability map Prob seg , where each pixel value of the segmentation probability map represents the probability that the position corresponding to the pixel value belongs to the damaged area; The segmentation probability map is binarized to obtain the final segmentation mask, and the segmentation results are annotated on the multimodal image sequence to achieve visualization of the damaged area.

8. A machine vision-based steel-wood composite beam-column joint damage identification system is characterized by: include: Denoising module: This module obtains visible light, near-infrared, and thermal imaging multimodal image sequences of steel-timber composite beam-column joints, performs denoising processing on them through adaptive anisotropic diffusion filtering, and obtains a denoised multimodal image sequence. Based on this denoised multimodal image sequence, a denoised multimodal image sequence tensor is constructed. Fusion module: Perform Gaussian pyramid decomposition and Laplacian pyramid feature fusion on the denoised multimodal image sequence tensor to obtain a fused feature map; Binarization module: performs adaptive threshold segmentation and morphological processing on the fused feature map to obtain the final binary map; Recognition model training module: trains the deep learning network to obtain a trained deep learning network; Identification model application module: Use the trained deep learning network to perform damage identification, obtain segmentation probability maps, annotate segmentation results, and visualize the damaged area; To realize the steel-wood composite beam-column node damage identification method using machine vision as described in any one of claims 1-7.

Images