Overflow surface defect detection and identification method and system based on multi-modal feature and prompt mechanism

Abstract

The invention relates to the technical field of industrial defect detection, and discloses an overflowing surface defect detection and identification method and system of a multi-modal feature and prompt mechanism, and the method comprises the steps: collecting image data and point cloud data of the surface of a water turbine runner overflowing surface, and converting the point cloud data into a surface defect two-dimensional image; obtaining a high-resolution defect image based on an image super-resolution reconstruction method of a deep recursive network; inputting a defect image into the generative adversarial network, judging a data source, generating a high-quality defect image through an objective function optimization generative model, and expanding a data set for a defect identification detection model to use; and inputting the point cloud data into the PDE-Net to construct a dynamic adjacency graph, and mapping the point cloud data to a specific defect category after graph convolution and multi-layer feature extraction. The method has the beneficial effects that a mixed attention mechanism is introduced in a multi-modal feature extraction stage, global information and local details in a defect image are fully extracted, and the detection capability of complex defects is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial defect detection, and specifically to an overcurrent surface defect detection and recognition method and system with multi-modal features and a prompting mechanism. Background Art

[0002] The runner blades of a water turbine are often exposed to high-speed water flow impact and wear. Long-term operation can cause various defects on their surfaces, such as cracks, corrosion, wear, inclusions, etc. If these surface defects cannot be detected and repaired in time, it may lead to unstable operation of the water turbine or even failures, seriously affecting the safety and economic benefits of the power plant. Traditional visual inspection and simple physical measurements often have difficulty effectively detecting tiny surface defects or internal hidden dangers, and modern water turbines are mostly made of complex alloy materials with complex surface morphologies and high precision. Therefore, there is an urgent need for more accurate and efficient defect detection technologies to ensure their safe operation. Summary of the Invention

[0003] In view of the above existing problems, the present invention is proposed.

[0004] Therefore, the technical problems solved by the present invention are as follows: It is difficult to collect original samples. The water turbine is large in size and complex in structure. The runner blades are complex inclined planes with different curvatures, and the space between the blades is narrow and the light is dim. It is difficult to collect defect pictures of the water turbine blades, and the sample data is very limited. It is difficult to obtain defects. The defects only occupy a small part of the entire blade surface area. The defect area needs to be segmented from a large number of irrelevant background areas to improve the accuracy and speed of the subsequent detection model. It is difficult to identify tiny defects. Specific details of small and thin defects are easily lost during the detection process, causing certain difficulties in defect positioning and identification.

[0005] To solve the above technical problems, the present invention provides the following technical solutions: An overcurrent surface defect detection and recognition method with multi-modal features and a prompting mechanism, including: Collecting image data and point cloud data on the surface of the overcurrent surface of the water turbine runner, and converting the point cloud data into a two-dimensional surface defect image; Based on the image super-resolution reconstruction method of a deep recursive network, obtaining a high-resolution defect image; Inputting the defect image into a generative adversarial network, judging the data source, and optimizing the generative model through an objective function to generate high-quality defect images to expand the data set for use by the defect recognition and detection model; Inputting the point cloud data into PDE-Net (Point Cloud Feature Dynamic Extraction Network) to construct a dynamic adjacency graph, and mapping the category of the defect after graph convolution and multi-layer feature extraction.

[0006] As a preferred solution of the overcurrent surface defect detection and recognition method with multimodal features and prompt mechanism according to the present invention, wherein: the image data and point cloud data include, after completing image acquisition, preprocessing the image, and recording image specification data and image content data; Among them, the image specification data includes the resolution of the surface image of each water turbine runner overcurrent surface; the image content data includes, according to the preset image acquisition frequency, obtaining the surface image of each sampling node of the water turbine runner overcurrent surface; The point cloud data includes point cloud specification data and point cloud content data. After completing the point cloud acquisition of the water turbine runner overcurrent surface, perform point cloud filtering, point cloud downsampling, and point cloud feature extraction on the point cloud data; the point cloud specification data includes the file format of the surface point cloud data of each water turbine runner overcurrent surface; the point cloud content data includes point cloud position information ; Use the orthogonal projection method to project the points in each point cloud , and convert them into image coordinates . The k coordinate of each point in the point cloud can be mapped to the depth value of the corresponding image pixel; Reconstruct the two-dimensional defect image from the point cloud data of the runner overcurrent surface, and perform super-resolution reconstruction of the defect image.

[0007] As a preferred solution of the overcurrent surface defect detection and recognition method with multimodal features and prompt mechanism according to the present invention, wherein: the image super-resolution reconstruction method includes using the output of the current three subnets of the embedding network, inference network, and reconstruction network to reconstruct the high-resolution defect image as the input of the next model; Embedding network, the embedding network extracts feature information from the original defect image and transmits it to the inference network in a fixed form. Set the input defect image , function , , represent the subnet outputs of the embedding network, inference network, and reconstruction network respectively. The entire network model is expressed as: , Inference network, located in the middle of the entire architecture and after the embedding network, used to deduce the pixel values of the high-resolution image. The function takes the output of the embedding network as the input, and outputs after recursive convolution calculation. The formula is expressed as: , Among them, represents the number of recursions; represents the weight parameter; represents the bias term, is the output after the -th recursive convolution calculation; The reconstruction network converts the feature map representing the high-resolution image output by the inference network back to the original image space. It consists of two convolutional layers and can be regarded as the inverse operation of the embedding network. The output of the reconstruction network is used as the input, and the calculation formula is expressed as: , , , where, represents the final output high-resolution defect image, is the output of the inference network after the -th recursion, is the weight parameter after the -th recursion, is the bias term after the -th recursion, is the weight parameter after the -th recursion, is the bias term after the -th recursion; f() is the subnet function chain of the embedding network, inference network, and reconstruction network.

[0008] As a preferred solution of the overcurrent surface defect detection and recognition method of the multi-modal feature and prompt mechanism described in the present invention, wherein: the generative adversarial network includes a generative adversarial network model for augmenting the overcurrent surface defect image dataset of the runner and constructing a large-scale reliable dataset based on the original small-scale dataset; The obtained high-resolution defect image is input into the current network, and at the same time, the real dataset and the new data generated by the generation model are input. According to the output probability value, the source of the input data is judged. If the probability value is greater than 0.5, it is determined to be from the real dataset, and if it is less than 0.5, it is the new data generated by the generation model. The objective function of the defect image generation network is: , where, represents the expected value of the distribution function, represents the distribution of real samples, represents the noise distribution defined on the input of the generator, and through the parameter the input noise z data is mapped to the high-dimensional data space to obtain ; keeping the discriminator parameters fixed, the generation model is optimized, that is, minimizing , O(G(Z)) is the output probability of the discriminator for the newly generated data and O(X) is the output probability of the discriminator for the input defective image X. is the expected value of the sample x sampled from the true data distribution , z is the noise sampled from the noise distribution, and O is the output probability of the discriminator.

[0009] As a preferred solution of the overcurrent surface defect detection and recognition method of the multi-modal feature and prompt mechanism described in the present invention, wherein: the generation of high-quality defective images includes, when processing cloud data in multi-modal feature extraction, constructing a dynamic adjacency graph. PDE-Net uses the K-nearest neighbor algorithm to calculate the Euclidean distance between points for the input point cloud data to determine the nearest neighbors, and recalculates the adjacent points during each layer of feature extraction to realize the construction of the dynamic adjacency graph. At the same time, for each pair of adjacent points and calculate the variable features, which are obtained by splicing the features of the points and the corresponding position features. Among them, the features and positions of the corresponding points are represented by the feature mapping values: , wherein, is the feature mapping value representing the points and ; and represent the feature representations of the points and ; represents the relative position of the two points.

[0010] As a preferred solution of the overcurrent surface defect detection and recognition method of the multi-modal feature and prompt mechanism described in the present invention, wherein: the expansion of the data set includes using a hybrid attention mechanism to extract the global features of the contour, shape, and texture of the image, and enhancing the local learning ability of the model for small-scale defects; The hybrid attention mechanism is composed of the weighted fusion of the global attention mechanism and the local attention mechanism, and the formula is expressed as: , wherein, is the global attention weight, score represents the cosine similarity function, represents the length of the input sequence; , represent the elements of the input sequence , , k; i, j, k are variable indices; The local attention mechanism sets the local context window for each element to 3, calculates the similarity within the current window, and applies the softmax function to obtain the local attention weights. The formula is expressed as: , The hybrid attention mechanism uses the weights of the global attention mechanism and the local attention mechanism to perform a weighted sum on the input to form the context representation as: , , And it uses a weighted method to fuse the context representations of the GA and LA mechanisms to form a new comprehensive feature vector. The calculation formula is expressed as: , Wherein, is the local attention weight, represents a hyperparameter that controls the weights of global and local information; represents the local context window; represents the weighted sum over the entire sequence, represents the weighted sum within the local context window.

[0011] As a preferred solution of the over - flow surface defect detection and recognition method of the multi - modal feature and prompt mechanism of the present invention, wherein: the defect recognition and detection model includes constructing a prompt mechanism in defect detection, inputting an image containing spliced cracks and crack defects into a branch network, learning crack and spliced crack features by designing a loss function. The prompt mechanism uses the Sigmoid activation function to generate a binary classification mask representing the probability that a pixel point is a seam or a crack, and outputs a seam feature vector; Taking the multi - modal fusion feature output by the feature extraction network as the input, combining the spliced crack information and features of the prompt mechanism, enabling the detection network to effectively identify the defect type and distinguish between seam and crack defects. The corresponding loss function is obtained by the weighted sum of the confidence, bounding box, and classification loss functions. By calculating the current loss function, the rationality of the defect annotation box is evaluated, enabling the detection network to adaptively distinguish the quality of the annotation and optimize the training process.

[0012] Another object of the present invention is to provide an over - flow surface defect detection and recognition system for multi - modal features and prompt mechanisms.

[0013] To solve the above - mentioned technical problems, the present invention provides the following technical solution: an over - flow surface defect detection and recognition system for multi - modal features and prompt mechanisms, including: a collection unit, a reconstruction unit, a data enhancement unit, a fusion unit, and an identification unit; The collection unit collects the image data and point cloud data of the water turbine runner over - flow surface, and converts the point cloud data into a two - dimensional image of surface defects; The reconstruction unit obtains a high-resolution defect image through an image super-resolution reconstruction method based on a deep recursive network; The data augmentation unit inputs the high-resolution defect image into a generative adversarial network, determines the data source according to the output probability value, and optimizes the generative model through an objective function to generate high-quality defect images, expanding the dataset for use by the defect recognition and detection model; The fusion unit constructs a dynamic adjacency graph for the point cloud spatial structure features by building a PDE-Net, and maps out the categories of defects after graph convolution and multi-layer feature extraction; The recognition unit combines the information and features of the hint mechanism, enabling the detection network to not only identify the defect type but also distinguish the point cloud stitching seam from the actual defect on the flow surface.

[0014] A computer device includes: a memory and a processor; the memory stores a computer program, including: when the processor executes the computer program, the steps of the method described in any one of the present inventions are implemented.

[0015] A computer-readable storage medium stores a computer program thereon, including: when the computer program is executed by a processor, the steps of the method described in any one of the present inventions are implemented.

[0016] Advantages of the present invention: The present invention fuses multi-modal features and a hint mechanism to construct a defect detection network for the flow surface of a hydraulic turbine runner. In order to extract global features such as the contour, shape, and texture of the flow surface image of the runner and enhance the local learning ability of the model for small-scale defects, the present invention introduces a hybrid attention mechanism in the multi-modal feature extraction stage, fully extracting the global information and local details in the defect image and improving the detection ability for complex defects. Since the gaps generated by the stitching of point cloud data have highly similar structural features to the actual crack defects on the flow surface, in order to accurately identify the actual crack defects on the flow surface, a hint mechanism is embedded in the original defect detection backbone network. The hint mechanism learns the features of the seams and cracks and prompts the seam information to the defect detection network, enabling the network to accurately identify and locate crack defects during the defect detection process. This method overcomes the problems of missed detection of small-scale defects and inaccurate defect type detection in traditional defect detection networks. Description of the Drawings

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1The overall flowchart of the overcurrent surface defect detection and recognition method for multimodal features and prompt mechanism provided by an embodiment of the present invention. Specific embodiments

[0019] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following detailed description of the specific embodiments of the present invention will be made in conjunction with the accompanying drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative work shall fall within the scope of protection of the present invention.

[0020] Example 1, referring to Figure 1 , which is an embodiment of the present invention, provides an overcurrent surface defect detection and recognition method for multimodal features and prompt mechanism, including: S1: Collect image data and point cloud data on the surface of the overcurrent surface of the water turbine runner, and convert the point cloud data into a two-dimensional image of surface defects.

[0021] Furthermore, the image data includes, after completing image acquisition, preprocessing the image, and recording image specification data and image content data. The image data includes, after completing image acquisition, preprocessing the image, and recording image specification data and image content data; the image specification data includes the resolution of each surface image of the overcurrent surface of the water turbine runner.

[0022] Even further, the image content data includes, according to the preset image acquisition frequency, obtaining surface images of the overcurrent surface of the water turbine runner at each sampling node; the point cloud data includes, after completing point cloud acquisition, preprocessing the point cloud, and recording point cloud specification data and point cloud content data. The point cloud data includes, after completing point cloud acquisition for the overcurrent surface of the water turbine runner, performing point cloud filtering, point cloud downsampling, and point cloud feature extraction; the point cloud specification data includes the file format of each surface point cloud data of the overcurrent surface of the water turbine runner; the point cloud content data includes point cloud position information .

[0023] Even further, the orthogonal projection method is used to convert the point cloud data into a two-dimensional defect image, and super-resolution reconstruction based on a deep recursive network is performed on the original defect image, including the setting of an embedding network, an inference network, and a reconstruction network.

[0024] Even further, in order to ensure the accuracy of the two-dimensional image of surface defects converted from the point cloud data, enhance the useful information of the image, restore the detailed texture, and thus improve the defect recognition accuracy, the image super-resolution reconstruction method based on the deep recursive network is used, and the high-resolution defect image obtained after reconstruction is used as the input of the next model. The specific steps are as follows: Step 1, Embedding Network: Extract feature information from the original defect image and transfer the feature information to the inference network in a fixed form. Define the input defect image as , function , , respectively represent the outputs of the three subnets, namely the embedding network, the inference network, and the reconstruction network. Finally, the entire network model is represented by the function as: , Step 2, Inference Network: The inference network is in the middle of the entire architecture and is located after the embedding network. In the reconstructed image, the inference network is used to infer the pixel values of the high-resolution image. The inference network function takes the output of the embedding network as the input, and after recursive convolution calculation, the output is , and its calculation formula is: , where, represents the number of recursions; represents the weight parameter; represents the bias term, is the output after the -th recursive convolution calculation.

[0025] Step 3, Reconstruction Network: The main task of the reconstruction network is to convert the feature map representing the high-resolution image output by the inference network back to the original image space. The reconstruction network is mainly composed of two convolutional layers, and it can be roughly understood that the reconstruction network is the inverse operation of the embedding network. The reconstruction network obtains the output of the inference network as the input, and the calculation formula is as follows: , , , where, represents the finally output high-resolution defect image, is the output of the inference network after the -th recursion, is the weight parameter after the -th recursion, is the bias term after the -th recursion, is the weight parameter after the -th recursion, is the bias term after the -th recursion; f() is the subnet function chain of the embedding network, the inference network, and the reconstruction network.

[0026] It should be noted that for model evaluation and optimization: in the generative adversarial network, analyze indicators such as the similarity distribution and diversity between the generated data and the real data, and adjust the parameters of the generator and discriminator to improve the quality and authenticity of the generated data; for multi-modal feature extraction, calculate indicators such as the accuracy and stability of the feature vectors, and adjust the parameters of the PDE-Net and the hybrid attention mechanism to ensure accurate representation of defect information; for the defect detection network, according to indicators such as the detection accuracy and recall rate of the validation dataset, adjust the parameters of the backbone network and the hint mechanism branch of the HintD-Net (defect detection network) to optimize the defect type recognition and the ability to distinguish between spliced cracks and crack defects; adjust the model hyperparameters (such as learning rate, batch size, number of network layers, etc.) according to the verification results to improve performance and convergence speed.

[0027] S2: An image super-resolution reconstruction method based on a deep recursive network to obtain high-resolution defect images.

[0028] Furthermore, due to the difficulty of collecting defect images of the runner flow surface of the water turbine and the very limited sample data, a generative adversarial network model for generating a large-scale reliable dataset on the original small-scale dataset is constructed for the augmentation of the defect image dataset of the runner flow surface.

[0029] Even further, input the obtained high-resolution defect images into this network, and at the same time input the real dataset and the new data generated by the generative model , and judge the source of the input data according to the output probability value. If the probability value is greater than 0.5, it is determined that it comes from the real dataset, and if it is less than 0.5, it is the new data generated by the generative model. The objective function of the defect image generation network is: , where represents the expected value of the distribution function, represents the distribution of real samples, represents the noise distribution defined on the input of the generator, and through the parameter map the input noise z data to a high-dimensional data space to obtain ; keep the discriminator parameters fixed and optimize the generative model , that is, minimize , O(G(Z)) is the output probability of the discriminator for the newly generated data , O(X) is the output probability of the discriminator for the input defect image X, is the expected value of the sample x sampled from the real data distribution , is the noise z sampled from the noise distribution, and O is the output probability of the discriminator.

[0030] It should be noted that it is extremely difficult to collect data on the defects of turbine blades, and the sample size is extremely limited. Especially in the actual operation and maintenance scenarios of turbines, it is very difficult to obtain a large number of accurate defect pictures. By constructing a generative adversarial network model, a large-scale and reliable dataset can be generated based on the original small-scale dataset. The high-resolution defect images are input into this network, and the data source is judged according to the output probability value. Its objective function involves the real sample distribution and the noise distribution input to the generator. By optimizing the generative model, high-quality defect images can be generated to expand the original dataset and provide richer input data for the defect recognition and detection models. This enables the model to better adapt to the situation of limited data, improves the recognition ability and detection accuracy of the flow-through surface defects of the turbine runner, ensures the safe and stable operation of the turbine, reduces the failure risk caused by the untimely discovery of defects, and is of great significance to the safe production of the hydropower industry.

[0031] S3: Input the defect images into the generative adversarial network, judge the data source, and optimize the generative model through the objective function to generate high-quality defect images, which are expanded to the dataset for the defect recognition and detection models to use.

[0032] Furthermore, extract the multi-modal features of the surface defect data of the flow-through surface of the runner. The size of the flow-through surface of the runner is large, and the existing defects only account for a very small area. Some tiny defects are difficult to distinguish, which belongs to small target detection. In order to more accurately identify the defect types, multi-modal feature extraction is performed on the surface defect data.

[0033] Even further, the multi-modal features mainly include two parts: one is the point cloud spatial structure feature; the other is the defect image feature. Steps 3a - 3b are for extracting the point cloud spatial structure features, and step 3c is for extracting the defect image features.

[0034] Step 3a, construct the point cloud dynamic adjacency graph. In order to fully extract the defect feature information in the point cloud data, a point cloud feature dynamic extraction network (PDE-Net) is constructed. For the input point cloud data, PDE-Net uses the K-nearest neighbor algorithm to calculate the Euclidean distance between points to determine the nearest neighbors. The adjacent points are recalculated during the feature extraction process of each layer, thereby realizing the construction of the dynamic adjacency graph. This dynamic nature enables the model to better adapt to different local structures in the point cloud. For each pair of adjacent points and , calculate their variable features, and obtain them by splicing the features of the points and their relative position features: , where represents the feature mapping value between point and , and Represents a point and the characteristic representation of represents the relative position of two points.

[0035] Step 3b, Graph Convolution and Multi - layer Feature Extraction: PDE - Net uses graph convolution operations to fuse the neighborhood information obtained in step (3a), and uses edge features to update the features of the central point: , where is the updated feature representation of point , is a non - linear activation function is the neighbor set of point . PDE - Net contains multiple graph convolution layers, and each layer dynamically calculates the adjacency graph and extracts features. By stacking multiple layers, the model can capture features from local to global. After feature extraction, global pooling operations are used to obtain the global features of the entire point cloud, compressing the features of different points into a fixed - length feature vector, representing the state of the entire runner flow - through surface. The global feature vector is input into the fully - connected layer of the PDE - Net network, and the fully - connected layer is used to map the global features to specific defect categories, such as cracks, corrosion, and wear, etc.

[0036] Step 3c, Defect Image Feature Extraction. To extract the global features such as the contour, shape, and texture of the runner flow - through surface image, and enhance the local learning ability of the model for small - scale defects, a hybrid attention mechanism (HA) is introduced. The HA mechanism is composed of the weighted fusion of the global attention mechanism (GA) and the local attention mechanism (LA). For each element in the input sequence, GA calculates its similarity with all other elements and applies the softmax function to obtain the global attention weights: , where is the global attention weight, score represents the cosine similarity function represents the length of the input sequence; , represents the elements of the input sequence , , k; i, j, k are variable indices.

[0037] When calculating LA, first set the local context window of each element to 3, calculate the similarity within this window, and apply the softmax function to obtain the local attention weights: , Furthermore, HA uses the weights of the GA and LA mechanisms to perform a weighted sum on the input to form a context representation: , , Furthermore, a weighted method is adopted to fuse the context representations of the GA and LA mechanisms to form a new comprehensive feature vector. The calculation formula is as follows: , where, is the local attention weight, represents a hyperparameter that controls the weights of global and local information; represents the local context window; represents the weighted sum over the entire sequence, represents the weighted sum using the local context window.

[0038] The hybrid mechanism can effectively utilize the global information and local details in the defect image, improve the performance of the model in defect image feature extraction, enhance the robustness of the model, and improve its detection ability for complex defects.

[0039] Step 3d, multi-modal feature fusion. The point cloud spatial structure feature vector obtained in step 3b and the defect image feature vector obtained in step 3c are fused to form a new feature matrix, which is used as the input information for the subsequent runner flow surface surface defect detection network.

[0040] Furthermore, by constructing a point cloud feature dynamic extraction network (PDE-Net) and introducing a hybrid attention mechanism (HA) for multi-modal feature extraction, the recognition ability of runner flow surface defects is improved. For point cloud data, the K-nearest neighbor algorithm and dynamic adjacency graph construction of PDE-Net are used to fully extract defect features and adapt to different local structures. Through graph convolution and multi-layer feature extraction, the global features of the point cloud are obtained and mapped to specific defect categories.

[0041] Furthermore, for defect images, in the HA mechanism, GA calculates the similarity with all elements to obtain the global attention weight, and LA calculates within the local window to obtain the local attention weight. The two are fused to effectively utilize the global information and local details. The point cloud spatial structure feature vector and the defect image feature vector are fused to form a new feature matrix, which is used as the input for the subsequent detection network. Such multi-modal feature extraction helps to more accurately identify defect types and reduce the interference of the model by irrelevant factors. Using the fused features, the subsequent detection network can more accurately locate the defect position and classify it.

[0042] It should be noted that by appropriate processing and screening (such as setting confidence thresholds, etc.), the detection reliability can be improved. Visualizing the detection results and manually reviewing them can intuitively evaluate the accuracy and correct misjudgments. At the same time, the corrected data is fed back for model training optimization. Continuous training enables the model to adapt to new situations, improves the generalization ability and accuracy, and ensures the effectiveness and stability of the defect detection of the runner flow passage surface, providing strong support for the safe operation of the water turbine.

[0043] S4: Input the point cloud data into PDE-Net to realize the construction of the dynamic adjacency graph. After graph convolution and multi-layer feature extraction, it is mapped to specific defect categories.

[0044] Furthermore, aiming at the defect problems such as pores, inclusions, cracks, and depressions existing on the runner flow passage surface of the water turbine, a defect detection network with a hint mechanism (HintD-Net) is constructed. In order to effectively distinguish the gaps generated by the stitching of the runner point cloud data from the crack defects with similar structural features actually existing on the flow passage surface, a hint mechanism is introduced into the backbone network. By giving the defect detection network the features of the stitching cracks, the crack defects can be accurately identified.

[0045] Step 4a, construction of the hint mechanism. The hint mechanism, as a branch of the backbone detection network, inputs the image containing the stitching cracks and crack defects into this branch network. By designing the corresponding loss function, the learning of the crack defect and stitching crack features is realized. The calculation formula of the loss function TotalLOSS is: , where, and are the weights for the seam detection and crack detection tasks, and Loss is the cross-entropy loss function. The hint mechanism applies a Sigmoid activation function to generate a binary classification mask, representing the probability that each pixel point is a seam or a crack, and outputs the extracted seam feature vector; is the detection loss of the seam, is the detection loss of the real crack defect.

[0046] Step 4b, construction of the defect detection network. The overall network structure of the defect detection of the runner flow passage surface of the water turbine uses the output multi-modal fusion features of the feature extraction network as the input of the detection network. At the same time, combined with the stitching crack information and features given by the hint mechanism, the detection network can not only effectively identify the defect types, but also distinguish the seams generated by point cloud stitching from the crack defects actually existing on the flow passage surface.

[0047] Furthermore, the loss function Loss of the HintD-Net defect detection network consists of the constructed confidence loss function , the bounding box loss function and the classification loss function Obtained by weighted summation, and the calculation formula is as follows: , Furthermore, by calculating the loss function, the rationality of each defect annotation box is evaluated, enabling the detection network to adaptively distinguish the quality of annotations, thereby continuously optimizing the training process.

[0048] Construct a hint mechanism in the defect detection steps. Input images containing spliced cracks and crack defects into the branch network. Learn crack and spliced crack features by designing a loss function. The hint mechanism uses the Sigmoid activation function to generate a binary classification mask representing the probability that a pixel point is a seam or a crack, and outputs a seam feature vector. Using the multi-modal fusion features output by the feature extraction network as input, combining the spliced crack information and features of the hint mechanism, enables the detection network to effectively identify defect types and distinguish between seam and crack defects. Its loss function is obtained by weighted summation of the confidence, bounding box, and classification loss functions. By calculating this loss function, the rationality of the defect annotation box is evaluated, enabling the detection network to adaptively distinguish the quality of annotations and optimize the training process.

[0049] It should be noted that for model deployment: Build a defect detection network (HintD-Net), including a backbone network and a hint mechanism branch. Determine the structure and parameters of the backbone network, as well as the weight settings of the loss function in the hint mechanism branch, etc. Set the weighted parameters of the confidence loss function, bounding box loss function, and classification loss function in the HintD-Net defect detection network. Training process: Initialization: Set the detection network architecture. Initialize network parameters and provide a dataset. Training loop: Input the processed images, point clouds, and their hierarchical features into the network. The network outputs predicted values. Calculate the loss function value and update the network weights through backpropagation. Validation and adjustment: Regularly test the model performance on the validation set and monitor overfitting and other potential problems. Adjust the learning rate and other hyperparameters according to the performance results. Final evaluation: Use the test dataset to perform a final evaluation on the trained model. For the defect detection network, based on the confusion matrix results, use two key indicators, Precision and Recall, to evaluate the detection results of the network. Calculate TP (True Positive), FP (False Positive), FN (False Negative), calculate Precision and Recall, and comprehensively evaluate the model's ability to locate and identify defects and distinguish between spliced cracks and crack defects.

[0050] Example 2 is the second example of the present invention. This example provides an over-flow surface defect detection and recognition system with multi-modal features and a hint mechanism, including: An acquisition unit that acquires image data and point cloud data of the over-flow surface of a water turbine runner, and converts the point cloud data into a two-dimensional image of surface defects.

[0051] Reconstruction unit, through an image super-resolution reconstruction method based on a deep recursive network, the obtained high-resolution defect image improves the defect recognition accuracy.

[0052] Data augmentation unit, input the high-resolution defect image into a generative adversarial network, and judge the data source according to the output probability value. Optimize the generative model through the objective function to generate high-quality defect images, and expand the dataset for use by the defect recognition and detection model.

[0053] Fusion unit, construct a PDE-Net for the point cloud spatial structure feature to realize the construction of a dynamic adjacency graph, and map it to specific defect categories after graph convolution and multi-layer feature extraction.

[0054] Recognition unit, combine the information and features of the hint mechanism, so that the detection network can not only effectively identify the defect type, but also distinguish the point cloud stitching seam from the actual defect of the flow surface.

[0055] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not restrictive. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered by the scope of the claims of the present invention.

[0056] Embodiment 3, which is the third embodiment of the present invention, is different from the previous two embodiments in that: If the above functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical solution of the present invention, or the part that contributes to the prior art, or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing storage medium includes: USB flash drive, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other various media that can store program codes.

[0057] The logic and / or steps represented in the flowchart or otherwise described herein can, for example, be considered as a definitional sequence of executable instructions for implementing logical functions, and can be embodied specifically in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with the instruction execution system, apparatus, or device.

[0058] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection portion having one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, as the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then storing it in a computer memory.

[0059] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one of the following techniques known in the art and their combinations can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), and the like.

[0060] Example 4, the following is an embodiment of the present invention, which provides an overcurrent surface defect detection and recognition method for multimodal features and a prompt mechanism. To verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0061] The experiment aims to verify the effect of constructing a defect detection network for the flow-through surface of a hydraulic turbine runner by fusing multi-modal features and a hint mechanism for detecting surface defects on the flow-through surface of the hydraulic turbine runner. The purpose of the experiment is to demonstrate the advantages of the defect detection method for the flow-through surface of the hydraulic turbine runner that fuses multi-modal features and a hint mechanism in terms of detection accuracy, adaptability to complex environments, and processing speed compared with traditional defect detection techniques. In the experiment setup, data preparation uses a defect image dataset of the flow-through surface of the hydraulic turbine runner, which contains defective (such as cracks, corrosion, depressions, etc.) and defect-free images.

[0062] The experiment first performs preprocessing of the images, adjusting the images to a fixed size and normalizing them to meet the model input requirements.

[0063] In the experiment, feature extraction and model operation are carried out for extracting point cloud spatial structure features and defect image features. The HintD-Net is used to detect defects in the adjusted images. The model output includes the position of the bounding box and the confidence of the defect category. Effective detections are screened by setting a confidence threshold (such as 0.5). For the comparative experiment, a common traditional defect detection technique for the flow-through surface of the hydraulic turbine runner, such as a method based on single-feature image processing, is selected, and its parameters are configured and the model is prepared according to its conventional settings.

[0064] Data recording: Record data such as defect detection accuracy, processing time, average confidence, and detection rates of different types of defects (cracks, rust, depressions), etc. Specifically, as shown in Table 1.

[0065] Table 1 Data recording table , The data shows the performance of different detection methods in defect detection of the flow-through surface of the hydraulic turbine runner. It can be seen from the data that the defect detection accuracy of the present invention reaches the highest (88.1%), which has significant advantages compared with the methods of only single image features (75.0%) and only single point cloud features (78.0%), proving that the present invention's fusion of multi-modal features and hint mechanism effectively improves the defect detection performance of the model under complex working conditions. In addition, the processing time of the present invention is 0.45 seconds, between that of only single image features (0.33 seconds) and only single point cloud features (0.40 seconds). Although it is not the shortest, considering other advantages such as its high accuracy, it shows a good balance. It also indicates that the present invention can optimize and control the processing time to a certain extent while ensuring the detection effect in algorithm design.

[0066] In terms of the average confidence level, the present invention reaches 0.80, which is significantly higher than 0.55 of only single image features and also higher than 0.71 of only single point cloud features. This further confirms the high reliability and accuracy of the present invention in the detection process, and can output detection results with more confidence under various working conditions. Specifically for the defect types, the crack detection rate of the present invention is 87.0%, which is a significant improvement compared to 64.0% of only single image features and 71.0% of only single point cloud features; the rust detection rate of the present invention is 79.0%, which is higher than 65.0% of only single image features and 70.0% of only single point cloud features; the dent detection rate of the present invention is 88.0%, which is also far higher than 75.0% of only single image features and 82.0% of only single point cloud features. This shows that the present invention can more effectively detect various types of defects. Regardless of the shape and characteristics of the defects, they can be more accurately identified and detected with the application of the method of the present invention, comprehensively improving the ability and level of defect detection on the flow-through surface of the water turbine runner.

Claims

1. An overcurrent surface defect detection and recognition method for multimodal features and prompt mechanisms, characterized in that, Including: Collecting the image data and point cloud data of the flow-through surface of the water turbine runner, and converting the point cloud data into a two-dimensional surface defect image; Based on the image super-resolution reconstruction method of the deep recursive network, obtaining a high-resolution defect image; Inputting the defect image into the generative adversarial network, judging the data source, and optimizing the generative model through the objective function to generate a high-quality defect image, and expanding the data set for use by the defect recognition and detection model; Inputting the point cloud data into PDE-Net for dynamic adjacency graph construction, and mapping the category of the defect after graph convolution and multi-layer feature extraction.

2. The overcurrent surface defect detection and recognition method of multimodal features and prompt mechanism according to claim 1, characterized in that: The image data and point cloud data include, after completing image acquisition, preprocessing the image, and recording the image specification data and image content data; Among them, the image specification data includes the resolution of each surface image of the water turbine runner's flow-through surface; the image content data includes, according to the preset image acquisition frequency, obtaining the surface images of the water turbine runner's flow-through surface at each sampling node; The point cloud data includes point cloud specification data and point cloud content data. After the point cloud acquisition of the flow-through surface of the hydraulic turbine runner is completed, point cloud filtering, point cloud downsampling, and point cloud feature extraction are performed on the point cloud data. The point cloud specification data includes the file format of the surface point cloud data of each flow-through surface of the hydraulic turbine runner. The point cloud content data includes point cloud position information ; The points in each point cloud are converted into image coordinates by using the orthogonal projection method , and the coordinates of each point in the point cloud can be mapped to the depth value of the corresponding image pixel; ​ Reconstructing the two-dimensional defect image from the point cloud data of the runner and performing super-resolution reconstruction of the defect image.

3. The overcurrent surface defect detection and recognition method of the multimodal feature and prompt mechanism according to claim 2, characterized in that: The image super-resolution reconstruction method includes using the outputs of the current three subnets of the embedding network, the inference network, and the reconstruction network to reconstruct the high-resolution defect image as the input of the next model; The embedding network extracts feature information from the original defect image and transmits it to the inference network in a fixed form, setting the input defect image , the function , , represent the subnet outputs of the embedding network, the inference network, and the reconstruction network respectively. The entire network model is expressed as: ， Inference network, the function is an embedding network The output is the input, and after recursive convolution calculation, the output is obtained , which is expressed by the formula as: ， Among them, represents the number of recursion times; represents the weight parameter; represents the bias term, is the output after the -th recursive convolution calculation; The reconstruction network converts the feature map representing the high-resolution image output by the inference network back to the original image space. It consists of two convolutional layers and is regarded as the inverse operation of the embedding network. The reconstruction network takes the output of the reconstruction network as input, and its calculation formula is expressed as: ， ， ， Among them, represents the high-resolution defect image of the final output, is the output of the inference network after the -th recursion, is the weight parameter after the -th recursion, is the bias term after the -th recursion, is the weight parameter after the -th recursion, is the bias term after the -th recursion; f() is the subnet function chain of the embedding network, inference network, and reconstruction network.

4. The overcurrent surface defect detection and recognition method of the multimodal feature and prompt mechanism according to claim 3, characterized in that: The generative adversarial network is to augment the data set of the runner's flow-through surface defect images and construct a generative adversarial network model; Input the obtained high-resolution defect image into the current network, and at the same time input the real dataset and the new data generated by the generation model , and judge the source of the input data according to the output probability value. If the probability value is greater than 0.5, it is determined to be from the real dataset; if it is less than 0.5, it is the new data generated by the generation model. The objective function of the defect image generation network is as follows: ， Among them, represents the expected value of the distribution function, represents the distribution of real samples, represents the noise distribution defined on the input of the generator, and through the parameter maps the input noise z data to a high-dimensional data space to obtain ; keeping the discriminator parameters fixed, optimize the generation model , that is, minimize , where O(G(Z)) is the output probability of the discriminator for the newly generated data , O(X) is the output probability of the discriminator for the input defective image X, is the expected value of the sample x sampled from the real data distribution , is the noise z sampled from the noise distribution, and O is the output probability of the discriminator.

5. The overcurrent surface defect detection and recognition method of the multimodal feature and prompt mechanism according to claim 4, characterized in that: The generation of high-quality defect images includes, during the processing of cloud data in multi-modal feature extraction, the construction of a dynamic adjacency graph. In the PDE-Net, the K-nearest neighbor algorithm is used to calculate the Euclidean distance between points for the input point cloud data to determine the nearest neighbors, and the adjacent points are recalculated during each layer of feature extraction to realize the construction of the dynamic adjacency graph. At the same time, for each pair of adjacent points and the variable features are calculated, which are obtained by splicing the features of the points and the corresponding position features. Among them, the features and positions of the corresponding points are represented by the feature mapping values: ， Among them, represents the feature mapping value between points and ; and represent the feature representations of points and ; represents the relative positions of the two points.

6. The overcurrent surface defect detection and recognition method of multimodal features and prompt mechanism according to claim 5, characterized in that: The expanded data set includes using the hybrid attention mechanism to extract the global features of the contour, shape, and texture of the image, and enhancing the local learning ability of the model for small-scale defects; The hybrid attention mechanism is composed of weighted fusion of the global attention mechanism and the local attention mechanism, and the formula is expressed as: ， Among them, is the global attention weight, and score represents the cosine similarity function, represents the length of the input sequence; , represent the elements of the input sequence , , k; , , k are variable indices; The local attention mechanism sets the local context window of each element to 3, calculates the similarity within the current window, and applies the softmax function to obtain the local attention weight, and the formula is expressed as: ， The hybrid attention mechanism uses the weights of the global attention mechanism and the local attention mechanism to perform weighted summation on the input to form a context representation as: ， ， And adopts a weighted method to fuse the context representations of the GA and LA mechanisms to form a new comprehensive feature vector, and the calculation formula is expressed as: ， Among them, is the local attention weight, denotes a hyperparameter that controls the weights of global and local information; represents the local context window; represents the weighted sum over the entire sequence, represents the weighted sum using the local context window.

7. The overcurrent surface defect detection and recognition method of the multimodal feature and prompt mechanism according to claim 6, characterized in that: The defect recognition and detection model includes constructing a hint mechanism in defect detection, inputting the image containing splicing cracks and crack defects into the branch network, learning the crack and splicing crack features through designing a loss function, and the hint mechanism uses the Sigmoid activation function to generate a binary classification mask representing the probability that a pixel point is a seam or a crack, and outputs a seam feature vector; Taking the multi-modal fusion feature output by the feature extraction network as the input, combining the splicing crack information and features of the hint mechanism, enabling the detection network to identify the defect type and distinguish between seam and crack defects, and the corresponding loss function is obtained by weighted summation of the confidence, bounding box, and classification loss functions, and evaluating the rationality of the defect annotation box by calculating the current loss function.

8. An overcurrent surface defect detection and recognition system with multi-modal features and a prompting mechanism, which applies the overcurrent surface defect detection and recognition method with multi-modal features and a prompting mechanism as described in any one of claims 1-7, is characterized in that Including: An acquisition unit, a reconstruction unit, a data enhancement unit, a fusion unit, and an identification unit; The acquisition unit collects the image data and point cloud data of the flow-through surface of the water turbine runner, and converts the point cloud data into a two-dimensional surface defect image; The reconstruction unit obtains a high-resolution defect image through an image super-resolution reconstruction method based on a deep recursive network; The data enhancement unit inputs the high-resolution defect image into a generative adversarial network, determines the data source according to the output probability value, and optimizes the generative model through an objective function to generate high-quality defect images to expand the dataset for use by the defect recognition and detection model; The fusion unit constructs a dynamic adjacency graph for the point cloud spatial structure feature by building a PDE-Net, and maps out the category of the defect after graph convolution and multi-layer feature extraction; The recognition unit combines the information and features of the hint mechanism, enabling the detection network to not only identify the defect type but also distinguish the point cloud stitching seam from the actual defect on the flow surface.

9. A computer device, comprising: A memory and a processor; the memory stores a computer program, characterized in that: when the processor executes the computer program, the steps of the multi-modal feature and hint mechanism-based flow surface defect detection and recognition method according to any one of claims 1-7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the steps of the multi-modal feature and hint mechanism-based flow surface defect detection and recognition method according to any one of claims 1-7 are implemented.

Images