Method and system for detecting and identifying flow surface defects based on multimodal features and prompting mechanism

Abstract

The present invention relates to the field of industrial defect detection technology, and discloses a flow surface defect detection and identification method and system with multimodal features and prompting mechanisms, including: collecting image data and point cloud data of the flow surface of a turbine runner, converting the point cloud data into a two-dimensional image of the surface defect; an image super-resolution reconstruction method based on a deep recursive network to obtain a high-resolution defect image; inputting the defect image into a generative adversarial network to determine the data source, generating a high-quality defect image through an objective function optimization model, and expanding the data set for use by a defect recognition and detection model; inputting the point cloud data into a PDE-Net to construct a dynamic adjacency graph, and mapping it to a specific defect category after graph convolution and multi-layer feature extraction. The beneficial effects of the present invention are: a hybrid attention mechanism is introduced in the multimodal feature extraction stage, which fully extracts the global information and local details in the defect image, thereby improving the detection capability of complex defects.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial defect detection, and in particular to a method and system for detecting and identifying flow surface defects using multimodal features and a prompt mechanism. Background Art

[0002] Turbine runner blades are often exposed to the impact and abrasion of high-speed water flow. Long-term operation can lead to various surface defects such as cracks, corrosion, wear, and inclusions. If these surface defects are not detected and repaired in a timely manner, they can cause unstable operation or even failure of the turbine, seriously affecting the safety and economic benefits of the power plant. Traditional visual inspection and simple physical measurements often fail to effectively detect tiny surface defects or internal hidden dangers. Modern turbines are mostly made of complex alloy materials, with complex surface morphology and high precision. Therefore, more accurate and efficient defect detection technologies are urgently needed to ensure their safe operation. Summary of the Invention

[0003] In view of the above-mentioned problems, the present invention is proposed.

[0004] Therefore, the technical problem solved by the present invention is that it is difficult to collect original samples. The turbine is large in size and complex in structure. The runner blades are complex inclined surfaces with different curvatures. In addition, the space between the blades is narrow and the light is dim. It is difficult to collect images of turbine blade defects, and the sample data is very limited. Defect acquisition is difficult. The defect only occupies a small area of ​​the entire blade surface. The defect area needs to be segmented from a large number of irrelevant background areas to improve the accuracy and speed of the subsequent detection model. It is difficult to identify tiny defects. The specific details of small and fine defects are easily lost during the detection process, which makes it difficult to locate and identify the defects.

[0005] To solve the above technical problems, the present invention provides the following technical solution: a method for detecting and identifying flow surface defects based on multimodal features and prompting mechanisms, comprising:

[0006] Collect image data and point cloud data of the flow surface of the turbine runner, and convert the point cloud data into a two-dimensional image of surface defects;

[0007] Image super-resolution reconstruction method based on deep recurrent network to obtain high-resolution defect images;

[0008] The defect image is input into the generative adversarial network to determine the data source. The model is optimized through the objective function to generate high-quality defect images, and the data set is expanded for use by the defect recognition and detection model.

[0009] The point cloud data is input into PDE-Net (Point Cloud Feature Dynamic Extraction Network) to construct a dynamic adjacency graph, and the defect category is mapped after graph convolution and multi-layer feature extraction.

[0010] As a preferred solution of the method for detecting and identifying flow surface defects with a multimodal feature and prompt mechanism of the present invention, wherein: the image data and point cloud data include, after completing image acquisition, pre-processing the image and recording image specification data and image content data;

[0011] The image specification data includes the resolution of the surface image of the flow surface of each turbine runner; the image content data includes the surface image of the flow surface of the turbine runner of each sampling node obtained according to the preset image acquisition frequency;

[0012] The point cloud data includes point cloud specification data and point cloud content data. After completing the point cloud collection of the turbine runner flow surface, the point cloud data is subjected to point cloud filtering, point cloud downsampling, and point cloud feature extraction. The point cloud specification data includes the point cloud data file format of each turbine runner flow surface; the point cloud content data includes point cloud position information. ;

[0013] The orthogonal projection method is used to transform each point in the point cloud , converted to image coordinates , the k-coordinate of each point in the point cloud can be mapped to the depth value of the corresponding image pixel;

[0014] The point cloud data of the runner flow surface is used to reconstruct a two-dimensional defect image and perform super-resolution reconstruction of the defect image.

[0015] As a preferred embodiment of the method for detecting and identifying flow surface defects with a multimodal feature and prompt mechanism of the present invention, the image super-resolution reconstruction method includes using the outputs of the current three sub-networks of the embedding network, the inference network, and the reconstruction network to obtain a high-resolution defect image as the input of the next model;

[0016] Embedding network, the embedding network extracts feature information from the original defect image and transmits it to the inference network in a fixed form, setting the input defect image ,function 、 、 Represents the subnetwork outputs of the embedding network, inference network, and reconstruction network respectively, and the entire network model Expressed as:

[0017] ,

[0018] The inference network is located in the middle of the entire architecture and after the embedding network. It is used to deduce the pixel values ​​of the high-resolution image. The function is embedded in the network. The output is the input, and the output is calculated after recursive convolution , the formula is:

[0019] ,

[0020] in, Indicates the number of recursions; represents the weight parameter; represents the bias term, For the The output after recursive convolution calculation;

[0021] Reconstruct the network and output the feature map of the high-resolution image from the inference network Converted back to the original image space, it consists of two convolutional layers, which can be regarded as the inverse operation of the embedding network, and the reconstruction network is obtained. The output of is taken as input, and the calculation formula is expressed as:

[0022] ,

[0023] ,

[0024] ,

[0025] in, represents the final output high-resolution defect image, For the The inference network output after recursion is: For the The weight parameter after recursion, For the The bias term after the recursion, For the The weight parameter after recursion, For the The bias term after the recursion; f() is the subnetwork function chain of the embedding network, inference network, and reconstruction network.

[0026] As a preferred solution of the flow surface defect detection and identification method with multimodal features and prompting mechanism described in the present invention, the generative adversarial network includes, for augmenting the runner flow surface defect image dataset, constructing a generative adversarial network model that generates a large-scale reliable dataset based on the original small-scale dataset;

[0027] The obtained high-resolution defect image is input into the current network and the real dataset at the same time. and new data generated by the generative model , the source of the input data is determined based on the output probability value. If the probability value is greater than 0.5, it is determined to be from the real data set. If it is less than 0.5, it is the data newly generated by the generative model. The objective function of the defect image generation network is for:

[0028] ,

[0029] in, represents the expected value of the distribution function, represents the distribution of real samples, Represents the noise distribution defined at the generator input, via the parameter Map the input noise z data to the high-dimensional data space to obtain ; Fixed discriminator parameters remain unchanged and optimized generation model , that is, minimize , O(G(Z)) is the new data generated by the discriminator The output probability of O(X) is the output probability of the discriminator for the input defect image X. From the real data distribution The expected value of the sample x sampled in, is the noise z sampled from the noise distribution, and O is the output probability of the discriminator.

[0030] As a preferred solution of the method for detecting and identifying flow surface defects with multimodal features and prompting mechanism of the present invention, the generation of high-quality defect images includes: constructing a dynamic adjacency graph when processing cloud data in multimodal feature extraction, PDE-Net uses the K nearest neighbor algorithm to calculate the Euclidean distance between points for the input point cloud data to determine the nearest neighbor, and recalculates the adjacent points when extracting each layer of features to realize the construction of the dynamic adjacency graph, and at the same time, for each pair of adjacent points and Calculate the variable features, which are obtained by the features of the splicing points and the corresponding position features, where the features and positions of the corresponding points are represented by the feature mapping values:

[0031] ,

[0032] in, To indicate a point and The feature map values ​​between and Indicates a point and The feature representation of Indicates the relative position of two points.

[0033] As a preferred solution of the flow surface defect detection and identification method with multimodal features and prompting mechanism described in the present invention, the expanded dataset includes extracting global features of image contour, shape, and texture using a hybrid attention mechanism, and enhancing the model's local learning ability for small-scale defects;

[0034] The hybrid attention mechanism consists of a weighted fusion of the global attention mechanism and the local attention mechanism, and the formula is expressed as:

[0035] ,

[0036] in, is the global attention weight, score represents the cosine similarity function, Indicates the length of the input sequence; 、 Represents an element of the input sequence 、 , k; i, j, k are variable indexes;

[0037] The local attention mechanism sets the local context window of each element to 3, calculates the similarity within the current window, and applies the softmax function to obtain the local attention weight. The formula is expressed as:

[0038] ,

[0039] The hybrid attention mechanism uses the weights of the global attention mechanism and the local attention mechanism to perform a weighted summation of the input to form a context representation:

[0040] ,

[0041] ,

[0042] The context representations of GA and LA mechanisms are fused in a weighted manner to form a new comprehensive feature vector. The calculation formula is expressed as:

[0043] ,

[0044] in, is the local attention weight, Represents a hyperparameter that controls the weight of global and local information; Represents a local context window; represents the weighted sum of the entire sequence, Indicates the use of weighted summation within the local context window.

[0045] As a preferred embodiment of the method for detecting and identifying flow surface defects using a multimodal feature and prompt mechanism according to the present invention, the defect recognition and detection model includes: constructing a prompt mechanism for defect detection, inputting images containing splicing cracks and crack defects into a branch network, learning crack and splicing crack features by designing a loss function, and the prompt mechanism using a Sigmoid activation function to generate a binary classification mask representing the probability of a pixel being a seam or crack, and outputting a seam feature vector;

[0046] Taking the multimodal fusion features output by the feature extraction network as input and combining the splicing crack information and features of the prompt mechanism, the detection network can effectively identify the defect type and distinguish between seam and crack defects. The corresponding loss function is obtained by the weighted sum of confidence, bounding box and classification loss function. By calculating the current loss function to evaluate the rationality of the defect annotation box, the detection network can adaptively distinguish the quality of the annotation to optimize the training process.

[0047] Another object of the present invention is to provide a flow surface defect detection and identification system with multi-modal features and prompt mechanisms.

[0048] To solve the above technical problems, the present invention provides the following technical solutions: a flow surface defect detection and identification system with multimodal features and prompting mechanisms, comprising: an acquisition unit, a reconstruction unit, a data enhancement unit, a fusion unit, and an identification unit;

[0049] The acquisition unit acquires image data and point cloud data of the flow surface of the turbine runner, and converts the point cloud data into a two-dimensional image of the surface defects;

[0050] The reconstruction unit obtains a high-resolution defect image by an image super-resolution reconstruction method based on a deep recurrent network;

[0051] The data enhancement unit inputs the high-resolution defect image into the generative adversarial network, determines the data source based on the output probability value, generates high-quality defect images by optimizing the generative model through the objective function, and expands the data set for use by the defect recognition and detection model;

[0052] The fusion unit constructs a dynamic adjacency graph for point cloud spatial structure features using PDE-Net, and maps defect categories after graph convolution and multi-layer feature extraction.

[0053] The identification unit combines the information and features of the prompt mechanism to enable the detection network to not only identify the defect type, but also distinguish between the point cloud splicing seams and the actual defects of the flow surface.

[0054] A computer device comprises: a memory and a processor; the memory stores a computer program, comprising: the steps of implementing any one of the methods of the present invention when the processor executes the computer program.

[0055] A computer-readable storage medium stores a computer program thereon, comprising: steps of implementing any one of the methods of the present invention when the computer program is executed by a processor.

[0056] Beneficial effects of the present invention: The present invention integrates multimodal features and prompting mechanisms to construct a turbine runner flow surface defect detection network. In order to extract the global features such as the contour, shape, and texture of the runner flow surface image, and enhance the model's local learning ability for small-scale defects, the present invention introduces a hybrid attention mechanism in the multimodal feature extraction stage, which fully extracts the global information and local details in the defect image, and improves the detection ability of complex defects. Since the gaps generated by the splicing of point cloud data have highly similar structural features to the actual crack defects on the flow surface, in order to accurately identify the actual crack defects on the flow surface, a prompting mechanism is embedded in the original defect detection backbone network. The prompting mechanism prompts the defect detection network with seam information by learning the characteristics of seams and cracks, so that the network can accurately identify and locate crack defects during the defect detection process. This method overcomes the problems of traditional defect detection networks missing small-scale defects and inaccurate defect type detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 An overall flow chart of a method for detecting and identifying flow surface defects using multimodal features and a prompt mechanism provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in this field without creative work should fall within the scope of protection of the present invention.

[0060] Example 1, reference Figure 1 , as one embodiment of the present invention, provides a method for detecting and identifying flow surface defects using multimodal features and a prompt mechanism, comprising:

[0061] S1: Collect image data and point cloud data of the flow surface of the turbine runner, and convert the point cloud data into a two-dimensional image of surface defects.

[0062] Furthermore, the image data includes, after image acquisition is completed, image preprocessing is performed on the image, and image specification data and image content data are recorded. The image data includes, after image acquisition is completed, image preprocessing is performed on the image, and image specification data and image content data are recorded; the image specification data includes the resolution of the surface image of each turbine runner flow surface.

[0063] Furthermore, the image content data includes, according to the preset image acquisition frequency, obtaining the surface image of the turbine runner flow surface at each sampling node; the point cloud data includes, after completing the point cloud acquisition, pre-processing the point cloud and recording the point cloud specification data and point cloud content data. The point cloud data includes, after completing the point cloud acquisition of the turbine runner flow surface, point cloud filtering, point cloud downsampling, and point cloud feature extraction; the point cloud specification data includes, the point cloud data file format of each turbine runner flow surface; the point cloud content data includes, point cloud location information .

[0064] Furthermore, the orthogonal projection method is used to convert the point cloud data into a two-dimensional defect image, and the original defect image is super-resolution reconstructed based on a deep recursive network, including the settings of an embedding network, an inference network, and a reconstruction network.

[0065] Furthermore, to ensure the accuracy of the two-dimensional surface defect image converted from point cloud data, enhance the useful information of the image, restore the detailed texture, and thus improve the accuracy of defect recognition, an image super-resolution reconstruction method based on a deep recurrent network is used. The reconstructed high-resolution defect image is used as the input of the next model. The specific steps are as follows:

[0066] Step 1, embedding network: extract feature information from the original defect image and pass the feature information to the inference network in a fixed form. Define the input defect image as ,function 、 、 Represent the outputs of the three sub-networks: embedding network, inference network and reconstruction network. Finally, the entire network model Expressed as a function:

[0067] ,

[0068] Step 2, Inference Network: The inference network is located in the middle of the entire architecture, after the embedding network. In the reconstructed image, the inference network is used to infer the pixel values ​​of the high-resolution image. Inference network function Embedded Network The output of is taken as input, and the output after recursive convolution is , and its calculation formula is:

[0069] ,

[0070] in, Indicates the number of recursions; represents the weight parameter; represents the bias term, For the The output after recursive convolution calculation.

[0071] Step 3, reconstruction network: The main task of the reconstruction network is to convert the feature map representing the high-resolution image output by the inference network back to the original image space. The reconstruction network is mainly composed of two layers of convolutional layers. It can be roughly understood that the reconstruction network is the inverse operation of the embedding network. Get the output of the inference network As input, the calculation formula is as follows:

[0072] ,

[0073] ,

[0074] ,

[0075] in, represents the final output high-resolution defect image, For the The inference network output after recursion is: For the The weight parameter after recursion, For the The bias term after the recursion, For the The weight parameter after recursion, For the The bias term after the recursion; f() is the subnetwork function chain of the embedding network, inference network, and reconstruction network.

[0076] It should be noted that model evaluation and optimization: in the generative adversarial network, the similarity distribution and diversity of the generated data and the real data are analyzed, and the parameters of the generator and discriminator are adjusted to improve the quality and authenticity of the generated data; for multimodal feature extraction, the accuracy and stability of the feature vector are calculated, and the parameters of the PDE-Net and hybrid attention mechanism are adjusted to ensure accurate representation of defect information; for the defect detection network, the detection accuracy, recall rate and other indicators of the verification data set are adjusted, and the parameters of the HintD-Net (defect detection network) backbone network and the hint mechanism branch are adjusted to optimize the defect type recognition and the ability to distinguish splicing cracks and crack defects; according to the verification results, the model hyperparameters (such as learning rate, batch size, number of network layers, etc.) are adjusted to improve performance and convergence speed.

[0077] S2: Image super-resolution reconstruction method based on deep recurrent network to obtain high-resolution defect images.

[0078] Furthermore, the dataset of impeller flow surface defect images is expanded. Since it is difficult to collect images of turbine impeller flow surface defects and the sample data is very limited, a generative adversarial network model is constructed to generate a large-scale reliable dataset based on the original small-scale dataset.

[0079] Furthermore, the obtained high-resolution defect image is input into the network and the real dataset is also input. and new data generated by the generative model , the source of the input data is determined based on the output probability value. If the probability value is greater than 0.5, it is determined to be from the real data set. If it is less than 0.5, it is the data newly generated by the generative model. The objective function of the defect image generation network is for:

[0080] ,

[0081] in, represents the expected value of the distribution function, represents the distribution of real samples, Represents the noise distribution defined at the generator input, via the parameter Map the input noise z data to the high-dimensional data space to obtain ; Fixed discriminator parameters remain unchanged and optimized generation model , that is, minimize , O(G(Z)) is the new data generated by the discriminator The output probability of O(X) is the output probability of the discriminator for the input defect image X. From the real data distribution The expected value of the sample x sampled in, is the noise z sampled from the noise distribution, and O is the output probability of the discriminator.

[0082] It should be noted that collecting data on turbine blade defects is extremely difficult, and the number of samples is extremely limited. This is especially true in actual turbine operation and maintenance scenarios, where obtaining a large number of accurate defect images is extremely challenging. By constructing a generative adversarial network model, it is possible to generate a large-scale, reliable dataset based on the original small-scale dataset. High-resolution defect images are input into the network, and the data source is determined based on the output probability value. The objective function involves the distribution of real samples and the noise distribution of the generator input. By optimizing the generative model, high-quality defect images can be generated to expand the original dataset, providing richer input data for defect recognition and detection models. This enables the model to better adapt to limited data, improves the ability to identify and detect defects on the flow surface of the turbine runner, ensures the safe and stable operation of the turbine, and reduces the risk of failures caused by undetected defects. This is of great significance to safe production in the hydropower industry.

[0083] S3: Defect images are input into the generative adversarial network to determine the data source. The generative model is optimized through the objective function to generate high-quality defect images, and the data set is expanded for use by the defect recognition and detection model.

[0084] Furthermore, multimodal feature extraction is performed on the surface defect data of the runner flow surface. The runner flow surface is large, and any defects occupy only a small area within it. Some tiny defects are difficult to distinguish, making them small targets. To more accurately identify defect types, multimodal feature extraction is performed on the surface defect data.

[0085] Furthermore, multimodal features mainly include two parts: one is the point cloud spatial structure feature; the other is the defect image feature. Steps 3a-3b are for point cloud spatial structure feature extraction, and step 3c is for defect image feature extraction.

[0086] Step 3a, point cloud dynamic adjacency graph construction. In order to fully extract the defect feature information in the point cloud data, a point cloud feature dynamic extraction network (PDE-Net) is constructed. For the input point cloud data, PDE-Net uses the K nearest neighbor algorithm to calculate the Euclidean distance between points to determine the nearest neighbors. In the feature extraction process of each layer, the adjacent points are recalculated to realize the construction of a dynamic adjacency graph. This dynamicity enables the model to better adapt to different local structures in the point cloud. For each pair of adjacent points and , calculate its variable features, and obtain them through the features of the splicing points and the relative position features between them:

[0087] ,

[0088] in, To indicate a point and The feature map values ​​between and Indicates a point and The feature representation of Indicates the relative position of two points.

[0089] Step 3b, graph convolution and multi-layer feature extraction: PDE-Net uses graph convolution operations to fuse the neighborhood information obtained in step (3a) and uses edge features to update the features of the center point:

[0090] ,

[0091] in, for point The updated feature representation, is a nonlinear activation function, Yes The neighbor set of the PDE-Net is composed of multiple graph convolution layers, each of which dynamically calculates the adjacency graph and extracts features. By stacking multiple layers, the model is able to capture features from local to global. After feature extraction, a global pooling operation is used to obtain the global features of the entire point cloud, compressing the features of different points into a fixed-length feature vector that represents the state of the entire runner flow surface. The global feature vector is input into the fully connected layer of the PDE-Net network, and the fully connected layer is used to map the global features to specific defect categories, such as cracks, corrosion, and wear.

[0092] Step 3c, defect image feature extraction. In order to extract the global features of the runner flow surface image, such as contour, shape, and texture, and enhance the model's local learning ability for small-scale defects, a hybrid attention mechanism (HA) is introduced. The HA mechanism consists of a weighted fusion of the global attention mechanism (GA) and the local attention mechanism (LA). For each element in the input sequence, the GA , calculate it and all other elements Similarity, apply the softmax function to obtain the global attention weight:

[0093] ,

[0094] in, is the global attention weight, score represents the cosine similarity function, Indicates the length of the input sequence; 、 Represents an element of the input sequence 、 , k; i, j, k are variable indexes.

[0095] When calculating LA, we first set the local context window of each element to 3, calculate the similarity within the window, and apply the softmax function to obtain the local attention weight:

[0096] ,

[0097] Furthermore, HA uses the weights of GA and LA mechanisms to perform a weighted summation of the input to form a contextual representation:

[0098] ,

[0099] ,

[0100] Furthermore, the context representations of the GA and LA mechanisms are fused in a weighted manner to form a new comprehensive feature vector, which is calculated as follows:

[0101] ,

[0102] in, is the local attention weight, Represents a hyperparameter that controls the weight of global and local information; Represents a local context window; represents the weighted sum of the entire sequence, Indicates the use of weighted summation within the local context window.

[0103] The hybrid mechanism can effectively utilize the global information and local details in the defect image, improve the performance of the model in defect image feature extraction, enhance the robustness of the model, and improve its detection ability for complex defects.

[0104] Step 3d: Multimodal feature fusion. The point cloud spatial structure feature vector obtained in step 3b is fused with the defect image feature vector obtained in step 3c to form a new feature matrix, which serves as the input information for the subsequent runner flow surface defect detection network.

[0105] Furthermore, by constructing a point cloud feature dynamic extraction network (PDE-Net) and introducing a hybrid attention mechanism (HA) for multimodal feature extraction, the ability to identify defects on the runner flow surface is enhanced. For point cloud data, the PDE-Net's K-nearest neighbor algorithm and dynamic adjacency graph construction are used to fully extract defect features and adapt to different local structures. Through graph convolution and multi-layer feature extraction, global point cloud features are obtained and mapped to specific defect categories.

[0106] Furthermore, for defect images, the GA in the HA mechanism calculates similarity with all elements to obtain global attention weights, while the LA calculates local attention weights within a local window. The fusion of these two effectively utilizes both global information and local details. The spatial structure feature vector of the point cloud is fused with the feature vector of the defect image to form a new feature matrix, which serves as input to the subsequent detection network. This multimodal feature extraction helps more accurately identify defect types and reduces the model's interference from irrelevant factors. Using these fused features, the subsequent detection network can more accurately locate and classify defects.

[0107] It should be noted that appropriate processing and screening (such as setting confidence thresholds) can improve detection reliability. Visualizing and manually reviewing test results allows for intuitive accuracy assessment and correction of misjudgments. Corrected data can also be used as feedback for model training and optimization. Continuous training enables the model to adapt to new situations, improving generalization and accuracy, ensuring the effectiveness and stability of runner flow surface defect detection and providing strong support for the safe operation of turbines.

[0108] S4: Input point cloud data into PDE-Net to construct a dynamic adjacency graph, which is then mapped to specific defect categories after graph convolution and multi-layer feature extraction.

[0109] Furthermore, a defect detection network (HintD-Net) with a hinting mechanism was constructed to address defects such as pores, inclusions, cracks, and dents on the flow surface of the turbine runner. To effectively distinguish between gaps created by splicing point cloud data of the runner flow surface and actual crack defects with similar structural characteristics, a hinting mechanism was introduced into the backbone network. This mechanism provides the defect detection network with hints of the spliced ​​crack characteristics, enabling accurate identification of crack defects.

[0110] Step 4a: Construct the prompt mechanism. The prompt mechanism is a branch of the main detection network. Images containing splicing cracks and crack defects are input into the branch network. The corresponding loss function is designed to learn the characteristics of crack defects and splicing cracks. The loss function TotalLOSS calculation formula is:

[0111] ,

[0112] in, and is the weight of the seam detection and crack detection tasks, and Loss is the cross entropy loss function. The prompt mechanism applies a Sigmoid activation function to generate a binary classification mask, indicating the probability of each pixel being a seam or a crack, and outputs the extracted seam feature vector; is the detection loss of the seam, is the detection loss of real crack defects.

[0113] Step 4b: Defect detection network construction. The overall network structure for turbine runner flow surface defect detection uses the multimodal fusion features output by the feature extraction network as input to the detection network. Combined with the information and features of the splicing cracks provided by the prompt mechanism, the detection network not only effectively identifies defect types, but also distinguishes between seams created by point cloud splicing and actual cracks on the flow surface.

[0114] Furthermore, the loss function Loss of HintD-Net defect detection network is constructed by constructing the confidence loss function , bounding box loss function And the classification loss function The weighted sum is obtained, and the calculation formula is as follows:

[0115] ,

[0116] Furthermore, by calculating the loss function and evaluating the rationality of each defect annotation box, the detection network can adaptively distinguish the quality of annotations, thereby continuously optimizing the training process.

[0117] A prompting mechanism is constructed for the defect detection step. Images containing splicing cracks and crack defects are fed into a branch network. A loss function is designed to learn the crack and splicing crack features. The prompting mechanism uses a sigmoid activation function to generate a binary classification mask representing the probability of a pixel being a seam or crack, and outputs a seam feature vector. The multimodal fusion features output by the feature extraction network are used as input, combined with the splicing crack information and features from the prompting mechanism. This enables the detection network to effectively identify defect types and distinguish between seams and cracks. The loss function is a weighted sum of confidence, bounding box, and classification losses. This loss function is used to evaluate the rationality of defect annotation boxes, enabling the detection network to adaptively distinguish between high and low annotation quality to optimize the training process.

[0118] It should be noted that model deployment involves building a defect detection network (HintD-Net), including a backbone network and a hinting mechanism branch. The backbone network structure and parameters, as well as the weight settings for the loss function in the hinting mechanism branch, are determined. The weighted parameters for the confidence loss function, bounding box loss function, and classification loss function in the HintD-Net defect detection network are set. Training process: Initialization: Sets the detection network architecture. Initializes network parameters and provides a dataset. Training loop: Inputs processed images and point clouds, along with their hierarchical features, into the network. The network outputs predicted values. Calculates the loss function value and updates the network weights through backpropagation. Validation and tuning: Regularly test model performance on the validation set to monitor overfitting and other potential issues. Adjust the learning rate and other hyperparameters based on performance results. Final evaluation: Perform a final evaluation of the trained model using the test dataset. For the defect detection network, accuracy and recall are used as key metrics to evaluate the network's detection results based on the confusion matrix results. TP (true positive), FP (false positive), FN (false negative), precision and recall are calculated to comprehensively evaluate the model's ability to locate and identify defects and distinguish between splicing cracks and crack defects.

[0119] Example 2 is the second embodiment of the present invention, which provides a flow surface defect detection and identification system with multi-modal features and prompt mechanisms, including:

[0120] The acquisition unit collects image data and point cloud data of the flow surface of the turbine runner, and converts the point cloud data into a two-dimensional image of surface defects.

[0121] The reconstruction unit uses an image super-resolution reconstruction method based on a deep recursive network to obtain a high-resolution defect image to improve the defect recognition accuracy.

[0122] The data augmentation unit feeds high-resolution defect images into the generative adversarial network and determines the data source based on the output probability values. By optimizing the generative model through the objective function, high-quality defect images are generated, expanding the dataset for use by the defect recognition and detection model.

[0123] The fusion unit constructs PDE-Net for the spatial structure features of the point cloud to realize dynamic adjacency graph construction, which is mapped to specific defect categories after graph convolution and multi-layer feature extraction.

[0124] The recognition unit, combined with the information and features of the prompt mechanism, enables the detection network to not only effectively identify the defect type, but also distinguish between the point cloud splicing seams and the actual defects on the flow surface.

[0125] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

[0126] Example 3 is the third embodiment of the present invention, which differs from the first two embodiments in that:

[0127] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0128] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0129] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, and then editing, interpreting, or processing in another suitable manner as necessary, and then storing it in a computer memory.

[0130] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any of the following technologies known in the art, or a combination thereof, may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0131] Example 4: The following is an embodiment of the present invention, which provides a method for detecting and identifying flow surface defects with multimodal features and prompt mechanisms. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0132] This experiment aims to verify the effectiveness of constructing a turbine runner surface defect detection network by fusing multimodal features and a prompting mechanism. The goal is to demonstrate the advantages of this method over traditional defect detection techniques in terms of accuracy, adaptability to complex environments, and processing speed. The experimental setup and data preparation utilize a dataset of turbine runner surface defect images, which includes both images with defects (such as cracks, corrosion, and dents) and images without defects.

[0133] The experiment first preprocesses the image, adjusts the image to a fixed size and normalizes it to meet the model input requirements.

[0134] In the experiment, feature extraction and model execution were performed to extract spatial structural features from point clouds and defect image features. HintD-Net was used to detect defects in the adjusted images. The model output included the location of the bounding box and the confidence level of the defect category. A confidence threshold (e.g., 0.5) was set to screen for valid detections. For the comparative experiment, a common conventional turbine runner flow surface defect detection technology, such as one based on single feature image processing, was used, with parameter configuration and model preparation following its standard settings.

[0135] Data recording includes defect detection accuracy, processing time, average confidence level, and detection rates for different types of defects (cracks, rust, and dents). See Table 1 for details.

[0136] Table 1 Data record table

[0137] ,

[0138] The data demonstrates the performance of different detection methods in detecting defects on the flow surface of a turbine runner. As can be seen from the data, the present invention achieves the highest defect detection accuracy (88.1%), significantly outperforming methods that use only a single image feature (75.0%) and only a single point cloud feature (78.0%). This demonstrates that the present invention's fusion of multimodal features and a prompting mechanism effectively improves the model's defect detection performance under complex working conditions. Furthermore, the present invention's processing time is 0.45 seconds, which, while not the shortest between methods using only a single image feature (0.33 seconds) and only a single point cloud feature (0.40 seconds), demonstrates a good balance when considering its other advantages, such as high accuracy. This also demonstrates that the present invention's algorithm design is able to optimize and control processing time while ensuring detection effectiveness.

[0139] In terms of the average confidence level, the present invention reached 0.80, which is significantly higher than the 0.55 of a single image feature alone and the 0.71 of a single point cloud feature alone. This further confirms the high reliability and accuracy of the present invention in the detection process, and can output detection results with greater confidence under various working conditions. Specifically for defect types, the crack detection rate of the present invention is 87.0%, which is a significant improvement compared to the 64.0% of a single image feature alone and the 71.0% of a single point cloud feature alone; the rust detection rate of the present invention is 79.0%, which is higher than the 65.0% of a single image feature alone and the 70.0% of a single point cloud feature alone; the dent detection rate of the present invention is 88.0%, which is also far higher than the 75.0% of a single image feature alone and the 82.0% of a single point cloud feature alone. This shows that the present invention can more effectively detect various types of defects. Regardless of the shape and characteristics of the defects, they can be more accurately identified and detected with the application of the method of the present invention, comprehensively improving the ability and level of defect detection on the flow surface of the turbine runner.

Claims

1. A method for detecting and identifying flow surface defects based on multimodal features and prompting mechanisms, characterized in that: include: Collect image data and point cloud data of the flow surface of the turbine runner, and convert the point cloud data into a two-dimensional image of surface defects; Image super-resolution reconstruction method based on deep recurrent network to obtain high-resolution defect images; The defect image is input into the generative adversarial network to determine the data source. The model is optimized through the objective function to generate high-quality defect images, and the data set is expanded for use by the defect recognition and detection model. The point cloud data is input into PDE-Net to construct a dynamic adjacency graph, and the defect categories are mapped after graph convolution and multi-layer feature extraction; The generation of high-quality defect images includes the following steps: constructing a dynamic adjacency graph when processing cloud data in multimodal feature extraction; PDE-Net uses the K-nearest neighbor algorithm to calculate the Euclidean distance between points for the input point cloud data to determine the nearest neighbor; and recalculates the adjacent points when extracting each layer of features to realize the dynamic adjacency graph construction; and for each pair of adjacent points p i and p j Calculate the variable features, which are obtained by the features of the splicing points and the corresponding position features, where the features and positions of the corresponding points are represented by the feature mapping values: e ij =f(p i ,p j )=[h i ,h j ,p j -p i ] Among them, e ij To represent point p i and p j The feature map value between i and h j Represents point p i and p j The feature representation, p j -p i Indicates the relative position of two points; The expanded dataset includes extracting global features of image contour, shape, and texture using a hybrid attention mechanism, and enhancing the model's local learning ability for small-scale defects; The hybrid attention mechanism consists of a weighted fusion of the global attention mechanism and the local attention mechanism, and the formula is expressed as: Among them, a ij is the global attention weight, score represents the cosine similarity function, and n represents the length of the input sequence; x i 、x j 、x k Represents the elements i, j, k of the input sequence; i, j, k are variable indices; The local attention mechanism sets the local context window of each element to 3, calculates the similarity within the current window, and applies the softmax function to obtain the local attention weight. The formula is expressed as: The hybrid attention mechanism uses the weights of the global attention mechanism and the local attention mechanism to perform a weighted summation of the input to form a context representation: The context representations of GA and LA mechanisms are fused in a weighted manner to form a new comprehensive feature vector. The calculation formula is expressed as: Among them, a ij ' is the local attention weight, Represents a hyperparameter that controls the weight of global and local information; local_window represents the local context window; represents the weighted sum of the entire sequence, Indicates the use of weighted summation in the local context window; The defect recognition and detection model includes building a prompt mechanism for defect detection, inputting images containing splicing cracks and crack defects into a branch network, learning crack and splicing crack features by designing a loss function, and the prompt mechanism using a Sigmoid activation function to generate a binary classification mask to represent the probability of a pixel being a seam or crack, and outputting a seam feature vector; Taking the multimodal fusion features output by the feature extraction network as input and combining the splicing crack information and features of the prompt mechanism, the detection network can identify the defect type and distinguish between seam and crack defects. The corresponding loss function is obtained by the weighted sum of confidence, bounding box and classification loss function. The rationality of the defect annotation box is evaluated by calculating the current loss function.

2. The method for detecting and identifying flow surface defects using a multimodal feature and prompt mechanism according to claim 1, wherein: The image data and point cloud data include, after completing image acquisition, pre-processing the image and recording image specification data and image content data; The image specification data includes the resolution of the surface image of the flow surface of each turbine runner; the image content data includes the surface image of the flow surface of the turbine runner of each sampling node obtained according to the preset image acquisition frequency; The point cloud data includes point cloud specification data and point cloud content data. After completing the point cloud acquisition of the turbine runner flow surface, the point cloud data is subjected to point cloud filtering, point cloud downsampling, and point cloud feature extraction. The point cloud specification data includes the point cloud data file format of each turbine runner flow surface; the point cloud content data includes point cloud position information (x, y, k). Orthogonal projection is used to convert each point (x, y, k) in the point cloud into image coordinates (u, v). The k coordinate of each point in the point cloud can be mapped to the depth value of the corresponding image pixel. The point cloud data of the runner flow surface is used to reconstruct a two-dimensional defect image and perform super-resolution reconstruction of the defect image.

3. The method for detecting and identifying flow surface defects using a multimodal feature and prompt mechanism according to claim 2, wherein: The image super-resolution reconstruction method includes using the outputs of the current three sub-networks of the embedding network, the inference network and the reconstruction network to obtain a high-resolution defect image as input to the next model; Embedding network, the embedding network extracts feature information from the original defect image and transmits it to the inference network in a fixed form. Suppose the input defect image X, the functions f1, f2, and f3 represent the subnetwork outputs of the embedding network, inference network, and reconstruction network respectively. The entire network model f(X) is expressed as: f(X)=f3(f2(f1(X))) The inference network function takes the output of the embedding network f1 as input and outputs H after recursive convolution calculation. d , the formula is: H d =max(0,W*H d-1 +b) Where d=1,...,D represents the number of recursions; W represents the weight parameter; b represents the bias term, H d-1 is the output after the d-1th recursive convolution calculation; Reconstruct the network and convert the feature map H output by the inference network to represent the high-resolution image D Converting back to the original image space, it consists of two convolutional layers, which is regarded as the inverse operation of the embedding network. The output of the reconstruction network f3 is taken as input. The calculation formula is expressed as: H D+1 =max(0,W D+1 *H D +b D+1 ) in, Represents the final output high-resolution defect image, H D+1 is the inference network output after the D+1th recursion, W D+1 is the weight parameter after the D+1th recursion, b D+1 is the bias term after the D+1th recursion, W D+2 is the weight parameter after the D+2th recursion, b D+2 is the bias term after the D+2th recursion; f() is the subnetwork function chain of the embedding network, inference network, and reconstruction network.

4. The method for detecting and identifying flow surface defects using a multimodal feature and prompt mechanism according to claim 3, wherein: The generative adversarial network is used to augment the runner flow surface defect image dataset and construct a generative adversarial network model; The obtained high-resolution defect image is input into the current network, and the real data set X and the new data G(Z) generated by the generative model are input at the same time. The source of the input data is determined based on the output probability value. If the probability value is greater than 0.5, it is determined to be from the real data set. If it is less than 0.5, it is determined to be the data newly generated by the generative model. The objective function of the defect image generation network is for: Among them, E represents the expected value of the distribution function, P data (x) represents the distribution of real samples, P noise (z) represents the noise distribution defined at the generator input, which is expressed by the parameter θ g Map the input noise z data to the high-dimensional data space to obtain P g =G(z,θ g ); fix the discriminator parameters and optimize the generative model G, that is, minimize minV(O,G), O(G(Z)) is the output probability of the discriminator for the generated new data G(Z), O(X) is the output probability of the discriminator for the input defect image X, is the distribution of x~P from real data data The expected value of the sample x in (x), z~P noise (z) is the noise z sampled from the noise distribution, and O is the output probability of the discriminator.

5. A flow surface defect detection and identification system with multimodal features and a prompt mechanism, applying the flow surface defect detection and identification method with multimodal features and a prompt mechanism as described in any one of claims 1 to 4, characterized in that: include: Acquisition unit, reconstruction unit, data enhancement unit, fusion unit and recognition unit; The acquisition unit acquires image data and point cloud data of the flow surface of the turbine runner, and converts the point cloud data into a two-dimensional image of the surface defects; The reconstruction unit obtains a high-resolution defect image by an image super-resolution reconstruction method based on a deep recurrent network; The data enhancement unit inputs the high-resolution defect image into the generative adversarial network, determines the data source based on the output probability value, generates high-quality defect images by optimizing the generative model through the objective function, and expands the data set for use by the defect recognition and detection model; The fusion unit constructs a dynamic adjacency graph for point cloud spatial structure features using PDE-Net, and maps defect categories after graph convolution and multi-layer feature extraction. The identification unit combines the information and features of the prompt mechanism to enable the detection network to not only identify the defect type, but also distinguish between the point cloud splicing seams and the actual defects of the flow surface.

6. A computer device comprising: A memory and a processor; the memory stores a computer program, characterized in that: when the processor executes the computer program, the steps of the method for detecting and identifying flow surface defects with a multimodal feature and prompt mechanism as described in any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for detecting and identifying flow surface defects with a multimodal feature and prompt mechanism as described in any one of claims 1 to 4 are implemented.

Images