A main propeller hub key component two-dimensional and three-dimensional fusion defect detection method

Abstract

This invention discloses a two-dimensional and three-dimensional fusion defect detection method for key components of a main propeller hub, comprising: acquiring two-dimensional surface images and three-dimensional topographic data of key components of the main propeller hub under controlled illumination; determining key detection areas based on a structural prior model; establishing a correspondence between two-dimensional pixels and three-dimensional points and generating geometric attribute vectors; extracting two-dimensional features and pixel-aligned three-dimensional features; generating two-dimensional confidence scores and three-dimensional confidence scores; constructing and training a cross-modal prediction model for cross-modal prediction of three-dimensional and two-dimensional features; calculating the inconsistency between two-dimensional and three-dimensional features during detection and fusing them to generate a defect score map; combining key region weights, observation confidence suppression, and structural geodesic constraints for post-processing; outputting the defect connectivity region and defect severity or level, and associating it with structural facets or hole systems. This invention can improve defect detection stability and reduce false alarms in areas such as hole boundaries under complex conditions such as high reflectivity, structural abrupt changes, and three-dimensional defects.

Description

Technical Field

[0001] This invention relates to the fields of industrial visual inspection, three-dimensional measurement and intelligent manufacturing quality control, specifically to a two-dimensional and three-dimensional fusion defect detection method for key components of the main propeller hub. Background Technology

[0002] The helicopter main rotor hub is one of the key load-bearing and force-transmitting components of the rotor system. During service, it is subjected to complex alternating loads, vibrations, and environmental corrosion. Surface defects in the main rotor hub and its key components (such as orifice edges, mating reference surfaces, transition fillets, bolt hole rings, grooves / oil holes, etc.) can cause stress concentration, crack initiation and propagation, or decreased fit accuracy, thereby affecting the reliability and safety of the rotor system. Therefore, highly reliable surface defect detection of key components of the main rotor hub is of great significance during manufacturing, assembly, maintenance, and overhaul.

[0003] However, the main rotor hub components exhibit significant complexity in terms of geometry and material surface properties:

[0004] Firstly, the components are mostly complex curved surface structures and contain a large number of holes, rounded transitions, ribs and steps, etc., which are areas of structural abrupt changes. Defects often occur in "high-risk areas" such as the edge ring of the hole, rounded transitions or the contact surface. These areas are not only stress-sensitive areas, but also areas of drastic geometric changes.

[0005] Secondly, the surface of the main rotor hub is mostly a machined metal surface or a coated surface, which is prone to specular reflection or local highlights, resulting in saturation, high contrast and texture distortion in two-dimensional imaging.

[0006] Third, the defects exhibit diversity in form and scale: they may include scratches, pitting, corrosion spots, coating peeling, pits, burrs, chipped edges at orifices, and micro-cracks, etc. Some of these defects have obvious texture / color abnormalities, while others are mainly manifested as geometric morphological changes (such as pits, chipped edges, and burr protrusions). There are also cases where they are easily detected only in two dimensions or only in three dimensions.

[0007] Fourth, the detection scenarios are often accompanied by occlusion, changes in viewing angle, and surface contamination, making it difficult for single-modal detection methods to simultaneously achieve accuracy, robustness, and field deployability.

[0008] Therefore, the following challenges need to be addressed simultaneously when inspecting the surface defects of key components of the main propeller hub:

[0009] (1) Achieve reliable positioning and consistency constraints at complex curved surfaces, orifice edges and structural abrupt changes;

[0010] (2) Stable extraction of two-dimensional features under conditions of high reflectivity and uneven illumination;

[0011] (3) Achieve effective utilization of 3D information under conditions of sparse, noisy, and missing 3D topography acquisition;

[0012] (4) Establish an accurate correspondence between two-dimensional and three-dimensional information to avoid unreasonable propagation across structural boundaries;

[0013] (5) Achieve trainable, generalizable and interpretable defect scores and severity outputs under conditions of diverse defect types and scarce samples.

[0014] Existing main rotor hub inspection methods typically fall into four categories: those based on manual visual inspection and traditional non-destructive testing, those based solely on two-dimensional images for surface defect detection, those based solely on three-dimensional topography for defect detection, and those combining two-dimensional and three-dimensional methods. However, each of these methods has its limitations, as detailed below:

[0015] Manual inspection relies on the experience of inspectors, which has problems such as strong subjectivity, poor repeatability, low efficiency, and difficulty in quantifying the severity of defects. Traditional non-destructive testing has advantages in local areas, but it is sensitive to the testing environment, operating process and material condition, and it is difficult to directly output pixel-level defect location results consistent with the prior structure, making it difficult to meet the quality closed-loop requirements of digitalization, automation and traceability.

[0016] Surface defect detection based on 2D images typically involves acquiring 2D surface images under controlled lighting conditions, extracting 2D features using thresholding, texture analysis, traditional features, or deep learning networks, and then outputting the defect region. While this approach is effective for texture or color anomalies such as scratches, stains, and corrosion spots, it still has significant limitations.

[0017] (1) For defects that are mainly characterized by changes in geometric shape (such as pits, burrs, and chipped edges at the opening), two-dimensional images may only produce weak texture differences, which may easily lead to missed detection.

[0018] (2) The high reflectivity of the main rotor hub metal surface, resulting in high gloss saturation, shadow and brightness gradient changes, will introduce false anomalies similar to defects, causing false alarms;

[0019] (3) At abrupt structural changes such as the edge of the aperture and the rounded corner, changes in projection and viewing angle can lead to texture distortion and a decrease in the stability of two-dimensional features.

[0020] (4) Two-dimensional methods often lack strong coupling constraints with the main hub structure, which can easily lead to false detections spreading across structural boundaries, thus affecting the accuracy and interpretability of defect location.

[0021] 3D topography-based defect detection schemes use 3D topography data (point clouds or depth maps) to detect defects, such as by using geometric quantities like curvature, normal, roughness, and local fitting residuals. These schemes are more sensitive to geometric anomalies like pits, bumps, and chipped edges, but they also have limitations.

[0022] (1) Three-dimensional acquisition may be sparse, occluded, noisy or missing, especially inside the borehole or in the deep borehole area. Incomplete three-dimensional data may lead to missed detection.

[0023] (2) For defects that are mainly manifested as color / texture changes (such as early corrosion spots, slight coating peeling, etc.), the three-dimensional morphology changes may not be obvious, and the three-dimensional method has insufficient detection capability.

[0024] (3) Pure 3D methods often require high-precision sensors and stable tooling, resulting in high on-site deployment costs;

[0025] (4) When the three-dimensional detection results do not directly correspond to the two-dimensional images, it is difficult to form an intuitive evidence area display and verification for process personnel.

[0026] Two-dimensional and three-dimensional fusion detection involves simultaneously acquiring two-dimensional surface images and three-dimensional topographic data. Through calibration and registration, a correspondence between two-dimensional pixels and three-dimensional points is established. The three-dimensional information is then projected onto a two-dimensional pixel grid, enabling fusion inference within the same pixel domain. While this approach has theoretical advantages, it still faces the following limitations in main propeller hub applications:

[0027] (1) Fusion error caused by correspondence error: The curvature of the main rotor hub surface changes greatly, and occlusion and projection ambiguity are likely to occur at the edge of the orifice and the step. If the correspondence between two-dimensional pixels and three-dimensional points is inaccurate, the fusion result will be misaligned, resulting in defect positioning offset.

[0028] (2) The fusion strategy lacks structural constraints: At the boundary of structural change, if simple neighborhood smoothing or pixel domain propagation is used, abnormal response is easily propagated across the aperture boundary, causing false alarms to spread.

[0029] (3) Lack of differentiated processing for key areas: The risk level and detection difficulty of different areas of the main rotor hub are different. If the key detection areas are not identified and weighted in conjunction with the structural prior model, the fusion algorithm is often either too conservative and misses key defects, or too sensitive and causes false alarms.

[0030] (4) Sample scarcity and generalization problem: It is difficult to obtain main rotor hub defect samples and the labeling cost is high. If the fusion model relies heavily on a large number of defect samples for supervised training, it will be difficult to meet the training data requirements in actual implementation; if unsupervised or weakly supervised methods are used, it is necessary to solve the error accumulation problem caused by changes in normal sample distribution, reflection changes and morphological noise.

[0031] (5) Resource and real-time issues: Some fusion solutions require maintaining a large-scale feature library or recalculating, and the inference latency and resource consumption may be difficult to meet the needs of production line deployment or on-site testing.

[0032] The aforementioned deficiencies became even more pronounced during actual testing of key components of the helicopter main rotor hub:

[0033] The annular area around the edge of the orifice is often subject to chamfering, burrs, and edge chipping. However, this area is prone to appearing as a highlight ring and edge shadow in 2D imaging, leading to false alarms in 2D detection. In 3D detection, the orifice may be obstructed, resulting in missing information and missed detection.

[0034] The area that mates with the reference surface is sensitive to assembly contact and stress. Slight pits, indentations or corrosion may lead to assembly quality risks. However, the defect size is small and the surface texture changes are weak, making it difficult to take into account a single mode.

[0035] The transition fillet area is a stress-sensitive area. Microcracks or pitting may not be obvious on the two-dimensional texture, and geometrically it may be difficult to distinguish between the machining texture and the abnormal morphology.

[0036] The structure of areas such as bolt hole rings and grooves / oil holes is complex and has many boundaries. Defect responses can easily spread across the boundaries, and it is difficult to form a stable evidence region output when there are no prior structural constraints. Summary of the Invention

[0037] To address the shortcomings of existing technologies, this invention proposes a two-dimensional and three-dimensional fusion defect detection method for key components of the main propeller hub. Based on the structural features of the main propeller hub, a reliable correspondence is established between two-dimensional surface images and three-dimensional topographic data. A key detection area mask and geometric attribute vector are generated by combining a prior structural model. Robust fusion is performed within the key detection area, and the defect region and severity are output through a fused defect score map. This method addresses the actual detection needs of key components of the main propeller hub, achieving complementary two-dimensional and three-dimensional information without relying on a large number of defect-annotated samples. This improves the detection reliability in complex structures and highly reflective scenes, and provides structurally consistent defect evidence regions and severity scores for quality closure.

[0038] The objective of this invention is achieved through the following technical solution:

[0039] A method for detecting defects in key components of the main propeller hub using a fusion of two-dimensional and three-dimensional methods, comprising:

[0040] S1: Acquire two-dimensional surface images of key components of the helicopter main rotor hub under controlled lighting conditions, as well as three-dimensional topographic data of the field of view corresponding to the two-dimensional surface images; determine key detection areas based on the structural prior model, and generate key detection area masks;

[0041] S2: Establish the correspondence between two-dimensional pixels and three-dimensional points and achieve feature alignment. Generate geometric attribute vectors for two-dimensional pixels based on the structural prior model, extract two-dimensional features and pixel-aligned three-dimensional features, and generate two-dimensional confidence and three-dimensional confidence.

[0042] S3: Construct and train a cross-modal prediction model containing a first prediction sub-model and a second prediction sub-model, learn the interpretable consistency relationship between two-dimensional and three-dimensional features, and use it to predict three-dimensional features and two-dimensional features across modalities, respectively.

[0043] S4: Using the same operations as S1 and S2, extract the two-dimensional features corresponding to the two-dimensional surface image to be detected, and the three-dimensional features corresponding to the three-dimensional topography data. Output the two-dimensional and three-dimensional features predicted by the cross-modal prediction model after training. Calculate the two-dimensional inconsistency map and the three-dimensional inconsistency map, fuse the two, and then combine the key detection region weights and confidence suppression to generate a fused defect score map.

[0044] S5: Perform surface geodesic neighborhood constraint post-processing based on the structural prior model on the fused defect score map within the mask of the key detection area to obtain the defect connected region, calculate the defect severity, and output the defect region and defect severity or defect level.

[0045] A two-dimensional and three-dimensional fusion defect detection device for key components of the main propeller hub is used to realize a two-dimensional and three-dimensional fusion defect detection method for key components of the main propeller hub; the device includes:

[0046] The image acquisition module is used to acquire two-dimensional surface images under controlled lighting conditions;

[0047] The topography acquisition module is used to acquire three-dimensional topography data corresponding to the field of view of the two-dimensional surface image;

[0048] The key region generation module is used to determine key detection regions based on the structural prior model and generate key detection region masks.

[0049] The correspondence establishment module is used to establish the correspondence between two-dimensional pixels and three-dimensional points, and to generate the geometric attribute vector corresponding to the pixel;

[0050] The feature extraction module is used to extract pixel-level two-dimensional features and pixel-aligned three-dimensional features, and generate two-dimensional confidence scores and three-dimensional confidence scores.

[0051] The cross-modal prediction module has a built-in cross-modal prediction model containing a first prediction sub-model and a second prediction sub-model. The first prediction sub-model takes the two-dimensional features of the current pixel position and the geometric attribute vector corresponding to the pixel position as input, and outputs the cross-modal predicted three-dimensional features corresponding to the pixel position. The second prediction sub-model takes the three-dimensional features of the current pixel position and the geometric attribute vector corresponding to the pixel position as input, and outputs the cross-modal predicted two-dimensional features corresponding to the pixel position.

[0052] The fusion detection module is used to calculate two-dimensional inconsistency maps and three-dimensional inconsistency maps based on cross-modal predicted three-dimensional features and cross-modal predicted two-dimensional features, and generate a fusion defect score map.

[0053] The post-processing and output module is used to perform surface geodesic neighborhood constraint post-processing based on the structure prior model on the fused defect score map in the key detection area, extract defect candidate areas, calculate severity scores, determine defect areas according to preset judgment thresholds, output defect levels according to preset grading thresholds, and associate with patch numbers or hole system numbers to achieve traceable output.

[0054] The beneficial effects of this invention are as follows:

[0055] 1) Key area focus driven by structural prior model: This invention generates a key detection area mask through a structural prior model and introduces weights in loss and fusion, so that the detection focus is on high-risk areas such as the edge ring of the aperture, the reference surface, and the transition fillet, reducing noise interference in non-critical areas and improving detection effectiveness and engineering usability.

[0056] 2) Complementary fusion of two-dimensional and three-dimensional data to improve robustness: This invention utilizes two-dimensional surface images and three-dimensional topography data simultaneously, which can take into account both texture / color anomalies and geometric topography anomalies, reducing the missed detection caused by a single modality.

[0057] 3) Training with defect-free samples reduces data dependence: During the training phase, this invention minimizes the weighted consistency loss based on defect-free samples, enabling the cross-modal prediction model to learn normal consistency relationships. Even in the main propeller hub scenario where defective samples are scarce or difficult to label, it still has trainability and transferability.

[0058] 4) Structural consistency regularization and surface geodesic constraints suppress false propagation: This invention uses structural consistency regularization terms and surface geodesic constraints to post-process the defect response to propagate along the continuous direction of the structure, while maintaining segmentation at the orifice boundary and structural abrupt boundary, thus suppressing false propagation across boundaries and improving defect positioning accuracy and structural consistency.

[0059] 5) Confidence constraints improve adaptability to reflections and missing data: This invention introduces two-dimensional and three-dimensional confidence levels and uses the suppression of anomalous responses in unreliable observation areas in the fused score map, effectively reducing false alarms caused by specular anomalies and three-dimensional missing data.

[0060] 6) Traceable severity output facilitates quality closed-loop: This invention outputs the defect connectivity area and severity score in the key detection area, and associates it with the facet number or hole system number of the structural prior model, so as to realize the structured expression and traceable output of the defect evidence area, which facilitates re-inspection and assembly quality closed-loop management. Attached Figure Description

[0061] Figure 1 This is a flowchart of a two-dimensional and three-dimensional fusion defect detection method for key components of the main propeller hub according to an embodiment of the present invention.

[0062] Figure 2 A schematic diagram illustrating the determination of key detection regions and the generation of key region masks guided by the structural prior model.

[0063] Figure 3 A schematic diagram is created to illustrate the correspondence between two-dimensional pixels and three-dimensional points.

[0064] Figure 4 A schematic diagram illustrating the process of constructing and training a cross-modal prediction model and performing cross-modal consistency learning and structural consistency constraints.

[0065] Figure 5 To obtain the defect connectivity region by performing surface geodesic neighborhood constraint post-processing on the fused defect score map within the mask of the key detection area, and to calculate the defect severity, a schematic diagram of the defect region and defect severity or defect level is output. Detailed Implementation

[0066] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. The purpose and effects of the present invention will become clearer. It should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0067] On the one hand, such as Figure 1 As shown, the two-dimensional and three-dimensional fusion defect detection method for key components of the main propeller hub in this embodiment includes the following steps:

[0068] Step 1: Acquire two-dimensional surface images of key components of the helicopter main rotor hub under controlled lighting conditions, as well as three-dimensional topographic data of the field of view corresponding to the two-dimensional surface images; determine the key detection areas based on the structural prior model, and generate a key detection area mask R to focus the detection on the key detection areas and avoid noise interference from non-critical areas.

[0069] like Figure 2As shown, step one specifically includes the following sub-steps:

[0070] S1.1: Acquire two-dimensional surface images I of key components of the helicopter main rotor hub under controlled lighting conditions.

[0071] In some embodiments, to suppress specular reflections from the main rotor hub's metal surface, controlled illumination may employ one or more of the following: ring diffused light, coaxial light, polarized illumination, or multi-exposure imaging.

[0072] To improve the stability of two-dimensional features, after acquiring the two-dimensional surface image I, a high reflectivity suppression preprocessing is used to perform specular highlight detection and intensity normalization on the two-dimensional surface image, so as to weaken the specular reflection component and reduce the false anomaly response caused by high reflectivity.

[0073] S1.2: Obtain the three-dimensional topography data P corresponding to the field of view of the two-dimensional surface image.

[0074] In some embodiments, the three-dimensional topography data P is a point cloud, a depth map, or a combination thereof, and the three-dimensional topography data P needs to cover the key detection area to be detected in the two-dimensional surface image I.

[0075] In engineering implementation, two-dimensional surface images and three-dimensional topography data are organized in pairs within the same field of view as input for a single detection task, facilitating subsequent registration, projection alignment, and verification and traceability. For example, the resolution of the two-dimensional surface image can be 2048×2048, and the number of points in the three-dimensional point cloud can be hundreds of thousands to millions; this embodiment is not limited to specific values.

[0076] S1.3: Determine the key detection areas based on the structural prior model.

[0077] The key inspection areas include at least one or more high-risk areas such as the orifice edge ring area, the contact reference surface area, the transition fillet area, the bolt hole ring area, the groove area, and the oil hole area.

[0078] S1.4: Map the corresponding facets, hole systems, or topological identifiers in the structural prior model to the coordinates of the two-dimensional surface image to generate a key detection area mask R, and extend the boundary of the key detection area with a preset safety margin to offset the deviations caused by registration errors, projection errors, and local occlusion.

[0079] In engineering, multi-value masking can be used to distinguish the types of key detection areas. For example, non-key detection areas can be marked with 0, and different markings can be used to distinguish the edge ring area of ​​the orifice, the area that fits the reference surface, etc., so that the weight and threshold can be determined in subsequent layering.

[0080] Step 2: Establish the correspondence between 2D pixels and 3D points and achieve feature alignment. Generate geometric attribute vectors for 2D pixels based on the structural prior model. Extract 2D features and pixel-aligned 3D features, and generate 2D confidence and 3D confidence to improve robustness under missing, occluded and noisy conditions.

[0081] like Figure 3 As shown, step two includes the following sub-steps:

[0082] S2.1: Register the 3D topography data P to the prior structural model to obtain the pose relationship, and based on the calibration parameters of the camera and 3D sensor, project the 3D points to the 2D pixel coordinates to establish the correspondence between the 2D pixels and the 3D points.

[0083] When projecting a 3D point onto a 2D pixel coordinate, if multiple 3D points are mapped to the same 2D pixel, the 3D information corresponding to that 2D pixel is generated by aggregation based on the minimum line-of-sight depth, consistent normal, or distance weighting. If a 2D pixel is missing a corresponding 3D point, the 3D information corresponding to that 2D pixel is completed based on neighborhood interpolation, and the 3D confidence of that 2D pixel is calculated for subsequent fusion suppression.

[0084] Establishing the correspondence between two-dimensional pixels and three-dimensional points lays the foundation for subsequent processing of three-dimensional features E. 3D (i) Projection to a two-dimensional pixel grid provides a basis for 2D and 3D alignment reasoning under a unified pixel index i.

[0085] S2.2: Generate geometric attribute vector g(i) for two-dimensional pixels based on the structural prior model.

[0086] The geometric attribute vector represents at least one or more of the geodesic distance, curvature, or normal deviation from the corresponding pixel location to the edge of the aperture, and may further include the distance to the reference plane or the deviation of the aperture axis. The geometric attribute vector is used to characterize the local geometric semantics of the main rotor hub structure, enabling the cross-modal prediction model to distinguish between normal geometric changes and abnormal defect morphologies at structural abrupt changes.

[0087] In this embodiment, geometric attributes are organized as multi-channel data aligned with the two-dimensional surface image to participate in cross-modal prediction and weight determination. For example, when the geodesic distance from pixel (820, 1040) to the orifice edge is small and the curvature is large, it can be given a higher weight in the subsequent weighting strategy to improve the sensitivity to small defects in the orifice edge ring region.

[0088] Through the above steps, this invention does not treat all pixels equally in defect judgment. Instead, it uses a priori structural model to identify key detection areas such as the edge ring of the orifice, the bonding reference surface, the transition fillet, the bolt hole ring, and the groove / oil hole. The key detection area mask R and weight w(i) are used to enhance the detection sensitivity of the key detection area, thereby avoiding false alarms caused by noise in non-key areas.

[0089] S2.3: Perform feature extraction on the two-dimensional surface image and the three-dimensional topography data respectively to obtain pixel-level two-dimensional features E. 2D (i) Pixel-aligned 3D features E 3D (i) Three-dimensional features are obtained by projecting three-dimensional topography data onto a two-dimensional pixel grid through the correspondence between two-dimensional pixels and three-dimensional points.

[0090] In some embodiments, two-dimensional feature extraction can employ networks or operators sensitive to information such as surface texture, edges, and local contrast; three-dimensional features can employ geometric quantities such as curvature, normal, local fitting residuals, and roughness, or their learned features. For pixel locations where three-dimensional points are missing after projection, three-dimensional features can be completed through neighborhood interpolation, and a lower three-dimensional confidence level C can be recorded. 3D (i) thus avoiding the negative impact of missing positions on detection.

[0091] S2.4: Generate two-dimensional and three-dimensional confidence scores to reflect the reliability of two-dimensional and three-dimensional observations at the corresponding pixels.

[0092] Two-dimensional confidence level C 2D (i) It can be used to characterize the observational unreliability caused by high light saturation, shadow occlusion, etc. in two-dimensional images; three-dimensional confidence C 3D (i) It can be used to characterize the unreliability of observations caused by sparse point clouds, occlusion, or high noise. The introduction of confidence enables the fused score map S(i) to suppress anomalous responses in areas of unreliable observations, thereby improving overall robustness.

[0093] In some embodiments, the 2D confidence score can be reduced based on factors such as specular highlight saturation, strong shadow occlusion, and image blurring; the 3D confidence score can be reduced based on factors such as sparse point cloud, occlusion defects, and interpolation completion ratio. For example, if a pixel's 2D observation is affected by highlights, resulting in a 2D confidence score of 0.6, while the 3D point is complete, resulting in a 3D confidence score of 0.9, then the subsequent fusion will use the smaller value of 0.6 to suppress the defect score of that pixel, thereby reducing false alarms caused by highlights.

[0094] Step 3: Construct and train a cross-modal prediction model to learn the interpretable consistency relationship between 2D and 3D and generate cross-modal prediction features. The cross-modal prediction model includes a first prediction sub-model and a second prediction sub-model, which are used to predict 3D features and 2D features across modalities, respectively.

[0095] During the training phase, a weighted consistency loss is minimized based on defect-free samples, and a structural consistency regularization term is introduced, thus enabling the acquisition of a usable cross-modal consistency baseline even under conditions where defective samples are scarce. Specifically, as follows... Figure 4 As shown.

[0096] The expression for the loss function L during the training phase is as follows:

[0097]

[0098]

[0099] in, This represents the first predictive sub-model. This represents the second predictive sub-model; Two-dimensional features representing cross-modal prediction; Three-dimensional features representing cross-modal prediction; The feature difference measure is Euclidean distance, cosine distance, Huber distance, or a combination thereof; w(i) is the weight determined by the key detection region type, where the weight of the orifice edge ring region and the region conforming to the reference surface can be higher than that of non-key regions to guide the model to prioritize learning the normal consistency law of the key structural region of the main rotor hub; λ is the regularization weight coefficient; R geo The structural consistency regularization term for the geodesic neighborhood of the surface based on the structural prior model is used to constrain the continuity of predicted features or fusion scores within the geodesic neighborhood of the main rotor hub surface, suppress unreasonable propagation across the aperture boundary or structural abrupt boundary, and make the final result more consistent with the real structural boundary of the main rotor hub.

[0100] The design of the first and second prediction sub-models enables 2D and 3D features to predict each other under the condition of geometric attribute vectors. The first prediction sub-model takes the 2D features of the current pixel position and the corresponding geometric attribute vector as input, and outputs the cross-modal predicted 3D features corresponding to the pixel position. The second prediction sub-model takes the 3D features of the current pixel position and the corresponding geometric attribute vector as input, and outputs the cross-modal predicted 2D features corresponding to the pixel position. The 2D features of the current pixel position are derived from visual information such as texture, edge, and local contrast of the 2D surface image within the pixel's neighborhood. The geometric attribute vector of the current pixel position is not directly generated from the 2D surface image, but is first located in the structural prior model by establishing the correspondence between 2D pixels and 3D points. Then, it is calculated based on the geometric relationships of this corresponding position in the structural prior model, including one or more of the following: geodesic distance to the aperture edge, curvature, normal deviation, and distance to the contact reference surface.

[0101] As one implementation method, the first and second prediction sub-models can be implemented using two independent feedforward neural networks. Each feedforward neural network can include an input layer, several hidden layers, and an output layer. The input layer is used to receive the "feature vector concatenated with the single-modal feature and the geometric attribute vector", the hidden layer is used to learn the nonlinear mapping relationship between the single-modal feature and the structural geometric attribute, and the output layer is used to output the prediction feature with the same dimension as the target modal feature.

[0102] During the training phase, defect-free samples are used as training samples. First, pixel-level 2D features, pixel-aligned 3D features, and geometric attribute vectors are obtained through steps two and three. Then, the 2D features and geometric attribute vectors are input into the first prediction sub-model to obtain predicted 3D features, and the 3D features and geometric attribute vectors are input into the second prediction sub-model to obtain predicted 2D features. Subsequently, the differences between the predicted features and the actual observed features are calculated, and the two prediction branches are jointly optimized based on weighted consistency loss and structural consistency regularization. After training is complete, during the detection phase, the two trained prediction sub-models are invoked, outputting predicted 3D features and predicted 2D features respectively, for subsequent inconsistency calculation and fusion detection.

[0103] In some embodiments, the structural consistency regularization term determines the geodesic neighborhood of the surface through the structural prior model and cuts off the neighborhood connection at the orifice boundary and the structural abrupt boundary, thereby avoiding the spread of predicted features or defect responses across boundaries and causing continuous false detections.

[0104] In some embodiments, the geodesic neighborhood of a curved surface can be represented as N. geo (i), that is, the set of geodesic neighborhood points corresponding to pixel position i on the surface of the structural prior model; R geo Can be or In N geo (i) imposes a penalty on neighborhood differences to maintain consistency in the structural continuity direction while preserving segmentation at the structural boundary.

[0105] Through the above training method, this invention can learn the cross-modal consistency relationship of two-dimensional / three-dimensional systems under normal conditions using defect-free samples, provided that defective samples are scarce.

[0106] Step 4: In the detection phase, the same operations as in Steps 1 and 2 are used to extract the two-dimensional features corresponding to the two-dimensional surface image to be detected, as well as the three-dimensional features corresponding to the three-dimensional topography data. The two-dimensional and three-dimensional features predicted by the cross-modal prediction model are output through the trained cross-modal prediction model. The two-dimensional inconsistency map and the three-dimensional inconsistency map are calculated and fused together. Then, the fused defect score map is generated by combining the key detection region weights and confidence suppression.

[0107] The inconsistency degree is obtained by comparing observed features with cross-modal predicted features using a difference metric, which characterizes the degree of deviation between observations and predictions. Therefore, the two-dimensional inconsistency graph d... 2D (i) Inconsistency map with 3D 3D The formula for calculating (i) is as follows:

[0108]

[0109]

[0110] The formula for calculating the fusion defect score map S(i) is as follows:

[0111]

[0112] in, Here, t is the Sigmoid function, a and b are the fusion coefficients, and t is the fusion coefficient. 2D t 3D For threshold parameters, a hierarchical adaptive thresholding strategy is adopted to determine the threshold parameters for different key detection region types. "Hierarchical" refers to grouping the pixels to be detected according to the key detection region type, including at least one or more of the following: orifice edge ring region, datum surface region, transition rounded corner region, bolt hole ring region, groove region, and oil hole region. The two-dimensional and three-dimensional inconsistency distributions of the defect-free samples in each group are statistically analyzed, and then the corresponding two-dimensional and three-dimensional threshold parameters are determined. The adaptive thresholding strategy is determined based on the quantiles, mean, variance, or robust statistics of the inconsistency distribution of defect-free samples within the corresponding key detection regions.

[0113] In this embodiment, the weight w(i) in the loss function and the weight w(i) in the fusion defect score map are the same positional weight coefficient, both representing the weight of the key detection region corresponding to pixel position i, and are jointly determined by the type of key detection region to which the pixel position belongs and its corresponding structural position. During the training phase, the weight w(i) is used to enhance the contribution of key detection region samples to the weighted consistency loss, and during the detection phase, it is used to enhance the abnormal response of pixels in the key detection region in the fusion defect score map. For the aperture edge ring region and the mating reference surface region, the corresponding weight w(i) is higher than that for non-key regions.

[0114] The calculation of the fusion defect score map S(i) must consider at least three types of factors simultaneously:

[0115] (1) Weight w(i) of key detection areas: strengthen high-risk areas such as the edge of the orifice and the reference surface;

[0116] (2) Two-dimensional inconsistency diagram d 2D (i) Inconsistency map with 3D3D (i): Characterizes the inconsistency between observed features and cross-modal predicted features;

[0117] (3) Confidence level min(C) 2D (i), C 3D (i)): Suppress anomalous responses in areas where observations are unreliable.

[0118] Through the above fusion, the present invention can complement two-dimensional texture anomalies and three-dimensional shape anomalies, enhance the significance of defects, and reduce false alarms and false negatives caused by reflective highlights or three-dimensional defects.

[0119] The following is an engineering calculation example from step four, used to illustrate the calculation method for the inconsistency and fusion defect score map, and does not constitute a limitation on the numerical range:

[0120] For example, for a pixel within the annular region of the aperture edge, the two-dimensional observation features are [0.20, 0.50, 0.10], and the two-dimensional prediction features obtained from the three-dimensional prediction are [0.18, 0.47, 0.16]. Using Euclidean distance as the difference measure, the difference between the two is [0.02, 0.03, -0.06], thus the two-dimensional inconsistency is approximately 0.07. The three-dimensional side observation features are [1.20, 0.80], and the three-dimensional prediction features obtained from the two-dimensional prediction are [1.05, 0.75]. The Euclidean distance is approximately 0.158, thus the three-dimensional inconsistency is approximately 0.158.

[0121] Furthermore, the distribution of inconsistency of defect-free samples in the annular region of the orifice edge is statistically analyzed. If the 95th percentile of the two-dimensional inconsistency is 0.05 and the 95th percentile of the three-dimensional inconsistency is 0.10, then the threshold for this region can be set to 0.05 and 0.10 respectively, so that normal fluctuations caused by structural abrupt changes do not trigger alarms.

[0122] During the fusion stage, when the pixel belongs to the aperture edge ring area, the key region weight can be set higher. The 2D confidence is 0.9 and the 3D confidence is 0.7. Then the confidence suppression takes the smaller value of the two, 0.7. When the inconsistency of both 2D and 3D is higher than the threshold of the key detection area, the fusion defect score will be increased accordingly. When the confidence of any modality is low, the fusion defect score is suppressed, thereby reducing false alarms caused by highlights or 3D defects.

[0123] Step 5: Perform surface geodesic neighborhood constraint post-processing based on the structural prior model on the fused defect score map within the mask of the key detection area to obtain the defect connected region, calculate the defect severity, and output the defect region and defect severity or defect level, thereby reducing false alarms and false negatives and enhancing the interpretability and traceability of defect evidence.

[0124] like Figure 5As shown, step five includes the following sub-steps:

[0125] S5.1: Perform surface geodesic neighborhood constraint post-processing based on the structural prior model on the fused defect score map within the mask of the key detection area to make the defect response more consistent along the continuous direction of the structure and suppress unreasonable propagation across the orifice boundary or across the structural abrupt boundary, so as to obtain a response distribution consistent with the structural boundary.

[0126] In some embodiments, post-processing may include:

[0127] Based on the geodesic neighborhood N of the curved surface geo (i) score propagation or anisotropic smoothing ensures that the scores remain consistent in the direction of structural continuity while maintaining segmentation at orifice boundaries or structural abrupt changes.

[0128] Specifically, the score propagation refers to weighting and transferring the scores of adjacent pixels according to structural continuity within the geodesic neighborhood determined by the structural prior model, based on the current pixel score in the fused defect score map. When a neighboring pixel and the current pixel are located in the same continuous surface region, the neighboring pixel's score is allowed to participate in the current pixel's score update. When a neighboring pixel and the current pixel cross an aperture boundary or a structural abrupt boundary, the score propagation is weakened or blocked. The anisotropic smoothing refers to different smoothing intensities in different directions, with a larger smoothing intensity along the structural continuity direction and a smaller or no smoothing intensity along the aperture boundary normal or structural abrupt direction, thereby preventing the defect response from spreading across structural boundaries.

[0129] By using the above-mentioned score propagation or anisotropic smoothing process, the response integrity of continuous defect regions can be enhanced, while isolated noise points can be suppressed, and the boundary of the defect region can be made more consistent with the actual structural boundary of the main rotor hub.

[0130] S5.2: Thresholding and connected component extraction are performed on the post-processed fusion defect score map within the key detection area to obtain defect candidate regions.

[0131] In some embodiments, minimum area filtering can be configured to remove isolated noise while preserving small-scale defect responses in the annular region at the orifice edge.

[0132] S5.3: For each defect candidate region, calculate the area (Area), the maximum score (max(S)) within the region, and the location weight (Wpos), and calculate the severity score accordingly. Then, compare the severity score with the preset judgment threshold and the preset grading threshold. When the severity score is lower than the preset judgment threshold, the corresponding defect candidate region is removed or marked as a non-defect response. When the severity score reaches or exceeds the preset judgment threshold, the corresponding defect candidate region is determined as a defect region, and the defect level is further output according to the correspondence between the severity score and the preset grading threshold.

[0133] The severity score is calculated using the following formula:

[0134]

[0135] Among them, W pos The location weight of the orifice edge ring area and the mating reference surface area is determined by the type of critical detection area to which the defect candidate area belongs, giving them higher weights than non-critical areas. This location weight reflects the impact of the defect location on the main propeller hub assembly quality and stress safety, ensuring that the severity score reflects not only the defect area and local anomaly intensity but also the importance of the structural area where the defect is located. u1, u2, and u3 represent the maximum score max(S), area Area, and location weight W, respectively. pos The weighting coefficients.

[0136] For example, for a certain defect candidate region, the statistical area is 120 (pixels), the maximum score within the region is 0.92, and the candidate region belongs to the region that fits the reference plane, so the corresponding position weight is higher; if the coefficients of the severity score are 0.6, 0.002, and 0.2 respectively, then the severity score is approximately 0.6×0.92 + 0.002×120 + 0.2×1.4 = 1.072, and the defect level is output by corresponding to the preset grading threshold.

[0137] S5.4: Associate the defect area with the facet number or hole system number of the structural prior model to achieve traceable output of the inspection results.

[0138] Specifically, when the defect area mainly falls within the annular area of ​​the orifice edge, the hole system number is output; when it mainly falls within the area of ​​the reference surface, the surface patch number is output, thereby supporting the closed loop of rework positioning and re-inspection.

[0139] On the other hand, embodiments of the present invention also provide a two-dimensional and three-dimensional fusion defect detection device for key components of the main propeller hub, the device comprising:

[0140] The image acquisition module is used to acquire two-dimensional surface images under controlled lighting conditions;

[0141] The topography acquisition module is used to acquire three-dimensional topography data corresponding to the field of view of the two-dimensional surface image;

[0142] The key region generation module is used to determine key detection regions based on the structural prior model and generate key detection region masks.

[0143] The correspondence establishment module is used to establish the correspondence between two-dimensional pixels and three-dimensional points, and to generate the geometric attribute vector corresponding to the pixel;

[0144] The feature extraction module is used to extract pixel-level two-dimensional features and pixel-aligned three-dimensional features, and generate two-dimensional confidence scores and three-dimensional confidence scores.

[0145] The cross-modal prediction module has a built-in cross-modal prediction model containing a first prediction sub-model and a second prediction sub-model. The first prediction sub-model takes the two-dimensional features of the current pixel position and the geometric attribute vector corresponding to the pixel position as input, and outputs the cross-modal predicted three-dimensional features corresponding to the pixel position. The second prediction sub-model takes the three-dimensional features of the current pixel position and the geometric attribute vector corresponding to the pixel position as input, and outputs the cross-modal predicted two-dimensional features corresponding to the pixel position.

[0146] The fusion detection module is used to calculate two-dimensional inconsistency maps and three-dimensional inconsistency maps based on cross-modal predicted three-dimensional features and cross-modal predicted two-dimensional features, and generate a fusion defect score map.

[0147] The post-processing and output module is used to perform surface geodesic neighborhood constraint post-processing based on the structure prior model on the fused defect score map in the key detection area, extract defect candidate areas, calculate severity scores, determine defect areas according to preset judgment thresholds, output defect levels according to preset grading thresholds, and associate with patch numbers or hole system numbers to achieve traceable output.

[0148] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A method for detecting defects in key components of a main propeller hub using a two-dimensional and three-dimensional fusion process, characterized in that, include: S1: Acquire two-dimensional surface images of key components of the helicopter main rotor hub under controlled lighting conditions, as well as three-dimensional topographic data of the field of view corresponding to the two-dimensional surface images. The key detection regions are determined based on the structural prior model, and a mask for the key detection regions is generated. S2: Establish the correspondence between two-dimensional pixels and three-dimensional points and achieve feature alignment. Generate geometric attribute vectors for two-dimensional pixels based on the structural prior model, extract two-dimensional features and pixel-aligned three-dimensional features, and generate two-dimensional confidence and three-dimensional confidence. S3: Construct and train a cross-modal prediction model containing a first prediction sub-model and a second prediction sub-model, learn the interpretable consistency relationship between two-dimensional and three-dimensional features, and use it to predict three-dimensional features and two-dimensional features across modalities, respectively. S4: Using the same operations as S1 and S2, extract the two-dimensional features corresponding to the two-dimensional surface image to be detected, as well as the three-dimensional features corresponding to the three-dimensional topography data, and output the two-dimensional and three-dimensional features of cross-modal prediction through the trained cross-modal prediction model. Calculate the two-dimensional inconsistency map and the three-dimensional inconsistency map, fuse the two, and then combine the key detection area weights and confidence suppression to generate a fused defect score map; S5: Perform surface geodesic neighborhood constraint post-processing based on the structural prior model on the fused defect score map within the mask of the key detection area to obtain the defect connected region, calculate the defect severity, and output the defect region and defect severity or defect level.

2. The method for detecting defects in key components of the main propeller hub using a two-dimensional and three-dimensional fusion process according to claim 1, characterized in that, The controlled illumination employs one or more of the following: ring diffused light, coaxial light, polarized illumination, or multi-exposure imaging.

3. The method for detecting defects in key components of the main propeller hub using a two-dimensional and three-dimensional fusion process according to claim 1, characterized in that, After acquiring two-dimensional surface images of key components of the helicopter main rotor hub under controlled lighting conditions, high reflectivity suppression preprocessing is used to perform specular highlight detection and intensity normalization on the two-dimensional surface images to weaken specular reflection components and reduce pseudo-anomaly responses caused by high reflectivity.

4. The method for detecting defects in key components of the main propeller hub using a two-dimensional and three-dimensional fusion process according to claim 1, characterized in that, The three-dimensional topography data is point cloud, depth map, or a combination of point cloud and depth map.

5. The method for detecting defects in key components of the main propeller hub using a two-dimensional and three-dimensional fusion process according to claim 1, characterized in that, The key detection areas include at least one or more of the following: the orifice edge ring area, the contact reference surface area, the transition rounded corner area, the bolt hole ring area, the groove area, and the oil hole area.

6. The method for detecting defects in key components of the main propeller hub using a two-dimensional and three-dimensional fusion process according to claim 1, characterized in that, The geometric attribute vector at least represents one or more of the geodesic distance, curvature, or normal deviation from the edge of the aperture at the corresponding pixel location.

7. The method for detecting defects in key components of the main propeller hub using a two-dimensional and three-dimensional fusion process according to claim 1, characterized in that, The first prediction sub-model takes the two-dimensional features of the current pixel position and the geometric attribute vector corresponding to the pixel position as input, and outputs the cross-modal predicted three-dimensional features corresponding to the pixel position. The second prediction sub-model takes the three-dimensional features of the current pixel location and the geometric attribute vector corresponding to that pixel location as input, and outputs the cross-modal predicted two-dimensional features corresponding to that pixel location.

8. The method for detecting defects in key components of the main propeller hub using a two-dimensional and three-dimensional fusion process according to claim 7, characterized in that, The geometric attribute vector of the current pixel position is first located in the corresponding position in the structural prior model by the correspondence between two-dimensional pixels and three-dimensional points, and then calculated based on the geometric relationship of the corresponding position in the structural prior model. It includes one or more of the following: geodesic distance to the edge of the aperture, curvature, normal deviation, and distance to the reference surface.

9. The method for detecting defects in key components of the main propeller hub using a two-dimensional and three-dimensional fusion process according to claim 1, characterized in that, S5 includes the following sub-steps: S5.1: Perform surface geodesic neighborhood constraint post-processing based on the structural prior model on the fused defect score map within the mask of the key detection area to make the defect response more consistent along the continuous direction of the structure and suppress unreasonable propagation across the orifice boundary or across the structural abrupt boundary, so as to obtain a response distribution consistent with the structural boundary. S5.2: Thresholding and connected component extraction are performed on the post-processed fused defect score map within the key detection area to obtain defect candidate regions; S5.3: For each defect candidate region, calculate the area, maximum score within the region, and location weight, and then calculate the severity score accordingly; The severity score is then compared with the preset judgment threshold and the preset grading threshold. When the severity score is lower than the preset judgment threshold, the corresponding defect candidate region is removed or marked as a non-defect response; otherwise, the corresponding defect candidate region is determined as a defect region, and the defect level is further output according to the correspondence between the severity score and the preset grading threshold. S5.4: Associate the defect area with the facet number or hole system number of the structural prior model to achieve traceable output of the inspection results.

10. A two-dimensional and three-dimensional fusion defect detection device for key components of a main propeller hub, characterized in that, The apparatus is used to implement the two-dimensional and three-dimensional fusion defect detection method for key components of the main propeller hub as described in any one of claims 1 to 9; the apparatus includes: The image acquisition module is used to acquire two-dimensional surface images under controlled lighting conditions; The topography acquisition module is used to acquire three-dimensional topography data corresponding to the field of view of the two-dimensional surface image; The key region generation module is used to determine key detection regions based on the structural prior model and generate key detection region masks. The correspondence establishment module is used to establish the correspondence between two-dimensional pixels and three-dimensional points, and to generate the geometric attribute vector corresponding to the pixel; The feature extraction module is used to extract pixel-level two-dimensional features and pixel-aligned three-dimensional features, and generate two-dimensional confidence scores and three-dimensional confidence scores. The cross-modal prediction module has a built-in cross-modal prediction model containing a first prediction sub-model and a second prediction sub-model. The first prediction sub-model takes the two-dimensional features of the current pixel position and the geometric attribute vector corresponding to the pixel position as input, and outputs the cross-modal predicted three-dimensional features corresponding to the pixel position. The second prediction sub-model takes the three-dimensional features of the current pixel position and the geometric attribute vector corresponding to the pixel position as input, and outputs the cross-modal predicted two-dimensional features corresponding to the pixel position. The fusion detection module is used to calculate two-dimensional inconsistency maps and three-dimensional inconsistency maps based on cross-modal predicted three-dimensional features and cross-modal predicted two-dimensional features, and generate a fusion defect score map. The post-processing and output module is used to perform surface geodesic neighborhood constraint post-processing based on the structure prior model on the fused defect score map in the key detection area, extract defect candidate areas, calculate severity scores, determine defect areas according to preset judgment thresholds, output defect levels according to preset grading thresholds, and associate with patch numbers or hole system numbers to achieve traceable output.

Images