Intelligent identification and inspection method for defects of port crane based on unmanned aerial vehicle vision

Abstract

The present application relates to the technical field of intelligent inspection and computer vision, and provides a port crane defect intelligent identification and inspection method based on unmanned aerial vehicle vision, comprising: controlling the unmanned aerial vehicle to collect a multi-view image sequence and pre-processing; detecting and outputting a defect category, a position and a confidence level through a defect identification model; converting image coordinates to a global space coordinate system based on detection results and pose information to construct three-dimensional defect distribution data; combining a basic three-dimensional model to construct a dynamic digital twin model containing a timestamp label; performing spatiotemporal correlation on multi-period defect labels to generate an evolution sequence, predicting the remaining time for the defect to reach a preset threshold, and displaying the prediction in the model. The present application realizes accurate positioning of defects and dynamic twin mapping, supports defect evolution prediction, and improves the intelligent level of inspection and the scientific nature of maintenance decisions.

Description

Technical Field

[0001] This invention relates to the fields of intelligent inspection and computer vision technology, specifically to a method for intelligent identification and inspection of defects in port cranes based on UAV vision. Background Technology

[0002] Currently, in the field of operation and maintenance inspection of metal structures of port cranes, manual inspection or automatic inspection based on visible light images acquired by drones is commonly used. However, existing technical solutions still have many shortcomings in practical applications: On the one hand, traditional drone inspections mostly rely on single-view static image acquisition, which is affected by changes in lighting, shooting distance, and occlusion, making it difficult to reliably capture minute defects such as micro-cracks and early corrosion. Moreover, single inspection data lacks spatiotemporal correlation analysis, making it impossible to predict defect evolution trends and assess remaining lifespan. On the other hand, existing digital twin models are mostly static geometric models based on design drawings, which cannot dynamically integrate real-time visual perception data from inspections, resulting in a disconnect between the model and the physical entity's state. Summary of the Invention

[0003] To address the problems of low defect detection rate, lack of spatiotemporal evolution prediction capability, disconnect between digital twin models and actual conditions, and insufficient cross-scenario generalization capability in existing technologies, this application proposes a port crane defect intelligent identification and inspection method based on UAV vision, which achieves high-precision identification of crane defects, three-dimensional spatial positioning, dynamic twin mapping, and remaining life prediction.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: The system controls a drone to fly along a preset route to collect multi-view image sequences of a target port crane, and preprocesses the multi-view image sequences to generate standardized image data. A defect identification model is used to detect defects in the standardized image data, outputting defect detection results including defect category, image location coordinates, and confidence level. Based on the defect detection results and the drone's pose information, the image location coordinates are transformed to a global spatial coordinate system to construct three-dimensional defect distribution data. A dynamic digital twin model is generated based on the basic three-dimensional model of the target port crane and the three-dimensional defect distribution data. The dynamic digital twin model includes defect annotation information with associated timestamps. The defect annotation information of the same physical defect obtained from multiple inspections is spatiotemporally correlated to generate a defect evolution sequence. Based on the defect evolution sequence, defect growth is predicted to determine the remaining time for the physical defect to reach a preset threshold, and this is displayed in the dynamic digital twin model.

[0005] As one implementation, the defect recognition model employs a multi-scale feature extraction structure based on an attention mechanism. The step of detecting the standardized image data using the defect recognition model and outputting a defect detection result containing the defect category, image location coordinates, and confidence level includes: extracting multi-layer convolutional feature maps from the standardized image data using a backbone network; performing positional encoding processing on the multi-layer convolutional feature maps using a coordinate attention module to generate spatially enhanced feature maps; performing multi-scale fusion of the spatially enhanced feature maps using a feature pyramid network and a path aggregation network to generate multi-scale defect detection feature maps; and processing the multi-scale defect detection feature maps using a decoupled detection head to output the defect category, image location coordinates, and confidence level, respectively.

[0006] This preferred scheme introduces positional encoding through a coordinate attention module, enabling the network to focus on both channel features and spatial location information, effectively suppressing interference from complex backgrounds. By combining a feature pyramid and a path aggregation network, it enhances the fusion and transmission of shallow high-resolution details and deep semantic features, significantly improving the detection accuracy of multi-scale targets such as micro-cracks and loose bolts. The decoupled detection head design avoids mutual interference between classification and regression tasks, further optimizing the accuracy of bounding box regression.

[0007] As one implementation method, the step of converting the image position coordinates to a global spatial coordinate system based on the defect detection results and the UAV's pose information to construct three-dimensional defect distribution data includes: estimating the six-degree-of-freedom pose of the UAV relative to a preset starting point using a simultaneous localization and mapping algorithm and a continuous image sequence; converting the image position coordinates to a three-dimensional direction vector in the body coordinate system through inverse perspective projection transformation according to camera intrinsic parameters and the six-degree-of-freedom pose; converting the three-dimensional direction vector to the global spatial coordinate system using the six-degree-of-freedom pose; performing ray intersection between the three-dimensional direction vector and the basic three-dimensional model, and calculating the spatial coordinates of the intersection point in the global spatial coordinate system; filtering and fusing multiple localization results of the same defect in multiple frames of images, and outputting the optimized three-dimensional defect distribution data.

[0008] This preferred scheme establishes a precise mapping relationship from two-dimensional pixels to three-dimensional physical space by estimating the high-precision pose of the UAV in real time and combining the intersection of ray diffraction in the camera-involved three-dimensional model. The multi-frame filtering and fusion mechanism effectively eliminates the random error of single-frame positioning, ensuring the accuracy and stability of the spatial position of defects in the digital twin model, and laying a reliable spatial benchmark for subsequent temporal evolution analysis.

[0009] As one implementation method, generating a dynamic digital twin model based on the basic three-dimensional model of the target port crane and the three-dimensional defect distribution data includes: establishing an initial three-dimensional geometric model containing component attributes based on computer-aided design drawings as the basic three-dimensional model; converting the defect detection results into structured data and associating them with the corresponding spatial regions in the initial three-dimensional geometric model; generating an information card for each defect in the dynamic digital twin model, the information card containing defect type, size, discovery time, and historical evolution curve; and organizing multi-period defect information according to a time axis to support viewing defect distribution heatmaps for different inspection cycles by sliding a time bar.

[0010] This preferred solution transforms discrete visual inspection results into structured twin data elements, and realizes dynamic mapping and backtracking of physical entity status in digital space through information cards and timeline visualization interaction. This dynamic update mechanism overcomes the shortcomings of traditional static models that cannot reflect the real-time health status of equipment, enabling managers to intuitively grasp the spatiotemporal distribution patterns and evolution trends of defects.

[0011] As one implementation, before detecting the standardized image data using the defect identification model, a cross-port visual model collaborative update step is included: obtaining the model parameter update amount generated by each participating port after training an initial defect identification model based on its local image dataset; performing federated aggregation of the received model parameter update amounts from all participating ports through a central server to generate a global model parameter update and distributing it to each participating port; each participating port receiving the global model parameter update and applying it to its local model, while introducing an adversarial domain adaptation module to align the image feature distribution of different ports; iteratively executing the above parameter update acquisition, aggregation and distribution, and domain adaptation steps until the global model converges, and using the converged model as the defect identification model.

[0012] This preferred solution employs a federated learning framework, enabling collaborative training of multi-port models without requiring the original image data to leave the port. This not only safeguards the security of enterprise data assets but also leverages multi-party data to enhance the model's generalization capabilities. The introduction of the adversarial domain adaptation module further eliminates feature distribution shifts caused by differences in lighting, weather, and paint schemes among different ports, ensuring that the model maintains stable recognition performance when deployed across different scenarios.

[0013] As one implementation method, the introduction of the adversarial domain adaptation module to align the image feature distributions of different ports includes: connecting a domain discriminator after the feature extraction layer of the defect recognition network, the domain discriminator being used to determine the environmental source of the input features; setting a gradient inversion layer during training, so that the gradient of the domain discriminator is inverted during backpropagation, so that the feature extractor learns domain-invariant features; constructing a total loss function that is the defect classification loss minus the product of the domain discrimination loss and the dynamic weight coefficient, setting the dynamic weight coefficient to a preset initial value in the early stage of training, and gradually reducing the dynamic weight coefficient in the later stage of training.

[0014] This preferred approach forces the feature extractor to generate feature representations that cannot be distinguished from their source by the domain discriminator through a gradient inversion layer, thereby learning domain-invariant features that are robust to environmental changes. The dynamic weight strategy strengthens domain alignment in the early stage of training to promote feature space fusion, and reduces weights in the later stage to focus on defect classification tasks, effectively balancing the relationship between domain adaptation and task performance and avoiding negative transfer.

[0015] As one implementation, before detecting the standardized image data using the defect recognition model, a rare defect visual sample synthesis step is included: selecting rare defect categories with a sample count below a preset threshold from the local dataset and obtaining corresponding real images; constructing a conditional generative adversarial network (GAN), wherein the generator of the GAN takes random noise and the conditional encoding vector of the rare defect category as input and outputs a synthesized image; updating the parameters of the GAN based on the real images; using the parameter-updated generator to generate images of the rare defect categories in batches, and performing quality screening on the generated images, adding the screened generated images to the training set to balance the number of defect samples of each category.

[0016] This preferred approach addresses the long-tail distribution problem caused by the scarcity of rare defect samples in industrial scenarios. It utilizes conditional generative adversarial networks to synthesize high-quality defect samples in a targeted manner, effectively mitigating the adverse effects of class imbalance on model training. By introducing class conditional encoding, it ensures that the generated images are semantically consistent with the target defect category, thereby improving the model's recall rate for identifying low-frequency defects.

[0017] As one implementation method, the quality screening of the generated images includes: inputting the real image and the synthesized image into a pre-trained image feature extraction network respectively; calculating the initial Frescher distance between the synthesized image and the real image in the feature space; retaining synthesized images whose initial Frescher distance is lower than a first threshold; calculating the no-reference image sharpness index and structural similarity index for the retained synthesized images; and removing images whose sharpness is lower than a second threshold or whose structural similarity is lower than a third threshold; and mixing the screened synthesized images with the real images at a preset ratio to form an enhanced local dataset.

[0018] This preferred scheme establishes a dual screening mechanism based on feature distribution distance and perceptual quality indicators, eliminating low-quality or pattern-collapsed synthetic samples from both statistical distribution and visual perception dimensions, ensuring the authenticity and effectiveness of the augmented data; the proportional mixing strategy prevents synthetic data from overdoing the training process, ensuring that the model does not forget the distribution characteristics of real data while learning new samples.

[0019] As one implementation method, the process of controlling the UAV to fly along a preset route to collect multi-view image sequences of the target port crane also includes a real-time edge detection and active re-inspection step: a lightweight defect recognition model is deployed on the UAV's onboard edge computing unit to perform defect detection frame by frame on the real-time video stream; when a potential defect is detected and the confidence level exceeds a preset re-inspection threshold, an active re-inspection mode is triggered; in the active re-inspection mode, the UAV is controlled to automatically hover, the gimbal angle is adjusted to center the defect in the image, the optical zoom is increased, and the UAV flies along a preset orbital trajectory to acquire multi-view detailed images; the multi-view detailed images are transmitted back to the ground station, where the full version of the defect recognition model performs detailed confirmation, and the confirmation result is updated to the dynamic digital twin model.

[0020] This preferred solution constructs an edge-cloud collaborative mechanism of "edge initial screening - cloud fine inspection". It uses an airborne lightweight model to filter invalid areas in real time and triggers active fine acquisition only when a suspected defect is found. This solves the contradiction between the low efficiency of global high-resolution acquisition and the easy omission of global low-resolution acquisition. The hovering, zooming and circling action sequence in the active re-inspection mode ensures the sufficiency and discriminativeness of the acquired information, and significantly improves the final confirmation accuracy of minor defects.

[0021] As one implementation method, the step of acquiring multi-view detailed images by flying along a preset orbital trajectory includes: uniformly sampling multiple candidate observation points on the hemisphere accessible to the UAV, centered on the initially detected defect location; calculating the expected information gain for each candidate observation point based on the recognition accuracy of various defects at that observation angle in historical data; selecting the two candidate observation points with the largest expected information gain for sequential re-inspection; calculating the defect classification confidence in real time after acquiring the image at the first candidate observation point; if the defect classification confidence exceeds a preset confirmation threshold, skipping the image acquisition of the second candidate observation point; otherwise, continuing to acquire the image at the second candidate observation point.

[0022] This preferred scheme plans the re-inspection trajectory based on expected information gain rather than fixed rules, and prioritizes the observation perspective that has historically achieved the best identification effect for this type of defect, maximizing the information value of a single acquisition; the dynamic skip mechanism allows the re-inspection to be terminated in advance when the first high-gain perspective is confirmed, saving the drone's endurance and computing resources to the greatest extent while ensuring identification accuracy, and achieving the optimal balance between inspection efficiency and detection quality.

[0023] The intelligent defect identification and inspection method for port cranes based on UAV vision provided by this invention has the following beneficial effects: First, by controlling the UAV to collect multi-view image sequences along a preset route and preprocessing them to generate standardized image data, it overcomes the problem of missed detection caused by changes in lighting, occlusion, and shooting distance in traditional single-view inspections. This enriches the visual information of defects from the data source and improves the detection coverage of defects in key parts of the crane. Second, by using a defect identification model to detect the standardized image data and outputting the defect category, image position coordinates, and confidence level, it achieves automated identification and quantitative evaluation of multiple types of defects such as visible cracks, surface corrosion, structural deformation, and loose bolts, replacing traditional manual visual inspection and significantly improving detection efficiency and result consistency. Third, based on the defect detection results and UAV pose information, the image position coordinates are transformed to a global spatial coordinate system to construct three-dimensional defect distribution data. This establishes a precise positional mapping of defects in the physical space of the crane, elevating defect localization from a two-dimensional image plane to three-dimensional physical space, providing a reliable spatial benchmark for subsequent defect management and analysis. Fourth, a dynamic digital twin model containing associated timestamps is constructed based on the basic 3D model and 3D defect distribution data. This enables the digital twin model to dynamically integrate visual perception data from each inspection, overcoming the problem of the disconnect between traditional static geometric models and the actual health status of the physical entity, and achieving synchronous mapping between the model and the actual state of the crane. Fifth, the defect annotation information of the same physical defect obtained from multiple inspections is spatiotemporally correlated to generate a defect evolution sequence. Based on the defect evolution sequence, defect growth is predicted to determine the remaining time, enabling maintenance personnel to grasp the development trend of defects and formulate maintenance plans in advance. This realizes the transformation from passive detection to proactive predictive maintenance and provides a scientific basis for the full life cycle management of equipment. Attached Figure Description

[0024] Figure 1 This is a flowchart of a method for intelligent identification and inspection of port crane defects based on UAV vision, according to an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. Example 1

[0027] like Figure 1 As shown, this embodiment provides a method for intelligent defect identification and inspection of port cranes based on UAV vision. This method constructs a complete technical closed loop from data acquisition to lifespan prediction. Specifically, the method includes the following steps: Step S100 involves controlling the UAV to fly along a preset route to collect multi-view image sequences of the target port crane, and preprocessing the multi-view image sequences to generate standardized image data. Specifically, the preset route is pre-planned based on the metal structural topology of the port crane, covering key load-bearing components such as main beams, tie rods, door frames, and spreaders, as well as easily corroded areas, ensuring that the collected image sequences can reflect the surface condition of the equipment without blind spots. The acquisition of multi-view image sequences aims to overcome the problem of missing subtle defects caused by changes in lighting, structural obstruction, or excessive shooting distance under a single viewpoint, and improves the completeness of information through redundant observation from multiple angles. The preprocessing process includes, but is not limited to, image denoising, contrast enhancement, and lens distortion correction. Its purpose is to eliminate imaging differences caused by different weather conditions, time of day, and camera posture, transforming the original images into standardized image data with consistent feature distribution, providing a stable input benchmark for the subsequent defect identification model, thereby significantly improving the robustness of the algorithm in complex outdoor environments.

[0028] Step S200 involves detecting defects in standardized image data using a defect recognition model, outputting a defect detection result including defect category, image location coordinates, and confidence level. In this embodiment, the defect recognition model is a trained deep neural network functional module, and its internal structure is not limited, as long as it can achieve the mapping from standardized images to structured defect information. The output defect category at least covers typical damage types such as visible cracks, surface corrosion, structural deformation, and bolt loosening; the image location coordinates are usually represented by a bounding box or segmentation mask to indicate the precise region of the defect in the two-dimensional image; and the confidence level reflects the model's certainty regarding the current detection result. This step achieves the crucial transformation from unstructured visual data to structured semantic data, providing basic data elements for subsequent 3D localization and temporal analysis.

[0029] Step S300: Based on the defect detection results and the UAV's pose information, the image position coordinates are transformed to a global spatial coordinate system to construct three-dimensional defect distribution data. Since two-dimensional image coordinates lack depth information and are affected by the shooting angle, they cannot be directly used for engineering evaluation; therefore, they must be mapped to a unified physical space. This transformation process relies on the high-precision pose information of the UAV at the time of acquisition (including three-dimensional position and attitude angles), using a coordinate transformation matrix to restore pixel coordinates to three-dimensional points or regions in the global spatial coordinate system. The constructed three-dimensional defect distribution data not only includes the spatial location of the defects but also preserves their relative relationship with the crane's main structure, enabling the aggregation and management of discrete detection results within a unified geographic reference framework. This solves the technical problems of fragmented and spatially difficult-to-align traditional inspection data.

[0030] Step S400: A dynamic digital twin model is generated based on the basic 3D model and 3D defect distribution data of the target port crane. The dynamic digital twin model includes defect annotation information with associated timestamps. The basic 3D model is typically a static geometric model built based on design drawings or as-built documentation, describing only the ideal initial state of the equipment. The dynamic digital twin model constructed in this embodiment integrates and maps the real-time sensing data (i.e., 3D defect distribution data) generated in step S300 onto the basic model, enabling the virtual entity in the digital space to synchronously reflect the real health status of the physical entity. Specifically, each defect annotation is bound to a precise timestamp, which not only records the specific time the defect was discovered but also provides an indispensable time dimension index for subsequent tracking of defect changes over time. This distinguishes it from traditional static BIM models and realizes the "dynamic" attribute of the digital twin.

[0031] Step S500 involves spatiotemporally correlating the defect annotation information of the same physical defect obtained from multiple inspection cycles to generate a defect evolution sequence. Based on this sequence, defect growth is predicted to determine the remaining time for the physical defect to reach a preset threshold, which is then displayed in a dynamic digital twin model. Specifically, spatiotemporal correlation refers to matching defect records belonging to the same physical entity in data from different inspection cycles based on spatial proximity and feature similarity, thereby forming a time-ordered sequence of defect size or state changes (i.e., a defect evolution sequence). Based on this sequence, prediction algorithms such as linear regression, exponential smoothing, or machine learning can be used to fit the defect growth trend and calculate the remaining time required for the defect to expand to a safe limit (i.e., a preset threshold, where the safe limit for size or area is set based on factors such as the defect's location). The prediction results are directly visualized in the dynamic digital twin model using color coding, progress bars, or pop-up warnings. This mechanism elevates inspection operations from simply "identifying problems" to "predicting risks," enabling maintenance personnel to develop preventative maintenance plans based on quantitative data. This effectively avoids over-maintenance or neglect, significantly improving the efficiency and safety of port cranes throughout their entire lifecycle management.

[0032] It should be understood that although this embodiment uses a port crane as an example for detailed explanation, the overall process of this method is also applicable to intelligent drone inspection scenarios for other large infrastructure such as bridges, transmission towers, and wind turbines. Furthermore, the specific implementation algorithms for each step (such as the network architecture of the defect identification model, the specific mathematical formulas for coordinate transformation, and the selection of the prediction model) can be adjusted according to actual needs. As long as the core logical closed loop of "collection-detection-location-twin-prediction" is maintained, it should all be covered within the scope of protection of this invention. Example 2

[0033] Building upon Example 1, this example further defines the specific network architecture of the defect recognition model. As one implementation method, the defect recognition model employs a multi-scale feature extraction structure based on an attention mechanism. This structure aims to address the technical challenges of large morphological differences in defects, strong background interference, and the ease with which small targets are missed in port crane scenarios. Specifically, the process of detecting standardized image data through the defect recognition model and outputting defect detection results containing defect category, image location coordinates, and confidence level can be refined into the following steps S201 to S204.

[0034] Step S201 involves extracting multi-layer convolutional feature maps from the standardized image data using a backbone network. Specifically, the backbone network, as the cornerstone of feature extraction, transforms the input standardized image data into a feature representation rich in semantic information. In this embodiment, the backbone network preferably employs a lightweight backbone architecture such as a deep residual network or CSPDarknet, generating a series of feature maps from shallow to deep layers through layer-by-layer convolutional operations. The shallow feature maps have higher resolution, preserving fine geometric texture information such as crack edges and bolt outlines; the deep feature maps have lower resolution but contain abstract semantic context information such as corrosion areas and structural deformation. This multi-layered feature representation is the physical basis for subsequent multi-scale detection, ensuring that the model can both "see" minute details and "understand" macroscopic damage patterns.

[0035] Step S202 involves using a coordinate attention module to perform positional encoding on the multi-layer convolutional feature map, generating a spatially enhanced feature map. This step is a key improvement in this embodiment that distinguishes it from traditional detection algorithms. In port crane inspection scenarios, many defects manifest as elongated metal fatigue cracks with extremely high aspect ratios. Traditional Squeeze-and-Excitation (SE) attention mechanisms only model the channel dimension, completely ignoring spatial positional information, making it difficult for the model to distinguish cracks from linear interference objects in the background (such as welds and water stains). While hybrid attention mechanisms like CBAM introduce a spatial dimension, their global pooling operation easily blurs the boundary features of elongated targets and incurs significant computational overhead. In contrast, the coordinate attention module used in this embodiment decomposes the positional encoding into two independent one-dimensional feature vectors in the horizontal and vertical directions, capturing long-range dependencies respectively. This decomposition strategy enables the network to accurately perceive the extension trend of cracks in the horizontal and vertical directions, significantly enhancing the model's sensitivity and response strength to elongated, irregularly oriented crack features while suppressing complex background noise.

[0036] Step S203 involves multi-scale fusion of the spatially enhanced feature map using a feature pyramid network and a path aggregation network to generate a multi-scale defect detection feature map. Because crane defects range in size from millimeter-level cracks to meter-level deformations, features at a single scale are insufficient. The feature pyramid network (FPN) constructs a top-down feature propagation path, upsampling deep, high-semantic information and fusing it with shallow features, giving lower-level features stronger semantic discriminative power. The path aggregation network (PAN) supplements the bottom-up feature propagation path, transmitting precise positioning information from shallow layers back to deeper layers, compensating for spatial details lost during downsampling. This bidirectional fusion mechanism forms a strengthened multi-scale feature representation system, particularly beneficial for preserving the edge and texture details of small target defects such as bolt loosening and early pitting corrosion. It effectively solves the problem of small target features being submerged in traditional unidirectional fusion structures, significantly improving the detection recall rate across the entire scale range.

[0037] Step S204 involves processing the multi-scale defect detection feature map using a decoupled detection head, outputting the defect category, image location coordinates, and confidence score. In conventional detection networks, classification and regression tasks typically share the same detection head, but this presents an inherent contradiction in industrial defect detection: classification tasks focus on the semantic invariance of features, aiming for consistent features for the same type of defect across different locations and angles; while regression tasks focus on the positional sensitivity of features, requiring precise perception of boundary offsets. This coupling leads to conflicting gradient optimization directions, affecting convergence speed and final accuracy. This embodiment employs a decoupled detection head design, physically separating the classification and regression branches, giving each independent convolutional layers and loss functions. The classification branch focuses on optimizing the accuracy of defect type discrimination, while the regression branch focuses on optimizing the localization accuracy and confidence score of bounding boxes. This parallel processing mechanism not only eliminates mutual interference between tasks but also allows for customized loss functions for different task characteristics (e.g., using Focal Loss to alleviate sample imbalance in classification and CIoU Loss to accelerate bounding box regression in regression), thereby achieving a comprehensive improvement in detection performance while ensuring real-time performance.

[0038] It should be understood that although this embodiment has been described in detail using coordinate attention, FPN+PAN and decoupling head as examples, in other embodiments, other attention mechanisms with position awareness, other bidirectional feature fusion structures or other task decoupling strategies can be selected according to actual computing power and accuracy requirements. As long as they can achieve the core functions of multi-scale feature enhancement and task conflict mitigation, they should be covered within the protection scope of this invention. Example 3

[0039] Based on Example 1, this example further elaborates on the construction process of the three-dimensional defect distribution data in step S300 and the generation mechanism of the dynamic digital twin model in step S400. As one implementation method, based on the defect detection results and the pose information of the UAV, the image position coordinates are transformed to the global spatial coordinate system to construct the three-dimensional defect distribution data, specifically including the following steps S301 to S304.

[0040] Step S301 involves estimating the six-DOF pose of the UAV relative to a preset starting point using a simultaneous localization and mapping (SLAM) algorithm based on a continuous image sequence. Specifically, due to the complex structure and high repetition of metal surface textures in port cranes, relying solely on the Global Navigation Satellite System (GNSS) often results in positioning drift due to signal blockage or multipath effects, failing to meet the requirements for millimeter-level defect localization. This embodiment employs a visual simultaneous localization and mapping (SLAM) algorithm. By extracting feature points from continuous frame images and performing matching tracking, combined with inertial measurement unit (INS) data, it calculates in real-time the high-precision six-DOF pose of the UAV relative to the inspection starting point when acquiring each frame image, including three-dimensional translation vectors and three-dimensional rotation quaternions. This step provides a precise spatial reference for subsequent coordinate transformation, ensuring that image data acquired at different times are unified in the same local coordinate system, effectively eliminating accumulated errors caused by UAV jitter or flight path deviations.

[0041] Step S302: Based on the camera intrinsic parameters and the six-DOF pose, the image position coordinates are transformed into a three-dimensional direction vector in the body coordinate system through inverse perspective projection transformation. Specifically, the image position coordinates output by the defect recognition model are only two-dimensional pixel values, lacking depth information. In this embodiment, using a pre-calibrated camera intrinsic parameter matrix, the pixel coordinates are first distorted and normalized to the unit plane in the camera coordinate system to obtain normalized coordinates; then, combined with the current frame pose obtained in step S301, the normalized coordinates are transformed to the global spatial coordinate system through rotation matrices and translation vectors. It should be noted that, since monocular vision cannot directly determine the actual distance of the defect, the result calculated in this step is not the absolute three-dimensional coordinates of the defect, but a three-dimensional ray direction vector originating from the camera optical center and passing through the defect pixel. This processing method preserves the geometric constraints of the observation, laying the foundation for subsequent determination of the exact location using a priori models.

[0042] Step S303: The three-dimensional direction vector is transformed to the global spatial coordinate system using the six-degree-of-freedom pose. The three-dimensional direction vector is then intersected with the basic three-dimensional model using rays, and the spatial coordinates of the intersection point in the global spatial coordinate system are calculated. Specifically, the basic three-dimensional model, as a digital geometric representation of the port crane, contains precise surface mesh information of the metal structure. In this embodiment, the ray generated in step S302 is considered as a probe beam, and an intersection test algorithm between the ray and the triangular mesh is executed in three-dimensional space. When the ray intersects the model surface, the effective intersection point closest to the camera's optical center is taken as the three-dimensional spatial coordinates of the defect from the current observation perspective. This mechanism cleverly utilizes the known geometric priors of the crane to solve the depth ambiguity problem of monocular vision, accurately "anchoring" two-dimensional visual perception to the physical entity surface. It should be understood that if the ray does not intersect the model or intersects multiple surfaces, anomaly rejection or optimization can be performed based on strategies such as normal direction consistency or distance thresholds (the distance threshold is set based on a reasonable tolerance range under the crane's structural scale) to ensure the physical rationality of the positioning results.

[0043] Step S304 involves filtering and fusing the multiple localization results of the same defect in multiple frames of images to output optimized 3D defect distribution data. Specifically, due to limitations in image noise, feature matching errors, and model simplification, the coordinates obtained from a single ray intersection inevitably exhibit random fluctuations. This embodiment introduces a Kalman filter or extended Kalman filter algorithm, treating multiple observations of the same physical defect in different frames and from different angles as noisy measurements of the same state variable. By establishing a state transition model and an observation model, the algorithm can recursively fuse historical estimates with current observations, automatically reducing the weight of high-variance observations, thereby outputting the optimal position estimate with minimized variance. This not only significantly improves the accuracy and stability of defect localization but also gives the system robustness to instantaneous localization jumps, ensuring that the final generated 3D defect distribution data is accurate and reliable.

[0044] After completing the construction of the three-dimensional defect distribution data, as one implementation method, a dynamic digital twin model is generated based on the basic three-dimensional model of the target port crane and the three-dimensional defect distribution data, specifically including the following steps S401 to S404.

[0045] Step S401 involves establishing an initial 3D geometric model containing component attributes based on computer-aided design drawings, serving as the foundation 3D model. Specifically, this foundation 3D model is not merely a visual mesh, but a structured data volume carrying rich semantic information. During the modeling process, based on the original design drawings or as-built documentation, not only are the geometric shapes of each crane component reconstructed, but they are also assigned unique component identifiers, material specifications, design loads, and topological connection relationships. This semantic modeling approach provides the necessary indexing foundation for the accurate loading and management of subsequent defect data, enabling the digital twin model to truly reflect the engineering essence of the equipment, rather than simply a digital replica of its appearance.

[0046] Step S402 converts the defect detection results into structured data and associates them with the corresponding spatial regions in the initial 3D geometric model. Specifically, the 3D defect distribution data processed in step S300 includes information such as the spatial coordinates, category, size, and confidence level of the defects. This embodiment automatically matches each defect instance to the closest component or panel in the basic 3D model through spatial proximity search or bounding box collision detection algorithms, and establishes a bidirectional reference relationship. This means that clicking on a component in the model allows you to query all defect records on its surface, and vice versa. This association mechanism realizes the transformation of unstructured perceptual data into structured digital assets, breaks down the barriers between visual inspection data and engineering design data, and provides a unified data foundation for location-based maintenance decisions.

[0047] Step S403: An information card is generated for each defect in the dynamic digital twin model. The information card includes the defect type, size, discovery time, and historical evolution curve. Specifically, the information card serves as a digital carrier for the entire lifecycle management of defects. In addition to recording current static attributes, the card dynamically links to the defect's historical observation records. Each time a new round of inspection data is correlated, the system automatically updates the evolution curve within the card, visually displaying the crack length propagation rate or corrosion area change trend. Furthermore, the card can embed thumbnail links to the original inspection images, allowing users to trace on-site images with a single click. This design aggregates scattered time-series data into defect-centric knowledge units, significantly improving the efficiency of information acquisition and the accuracy of judgment for maintenance personnel.

[0048] Step S404: Organize multi-period defect information according to the timeline to support viewing defect distribution heatmaps for different inspection cycles by sliding the time bar. Specifically, the core value of the dynamic digital twin model lies in its "dynamic" characteristic, that is, it can continuously absorb new data and present state evolution over time. This embodiment introduces a time dimension controller in the model rendering layer to index the defect data found in each inspection by timestamp. Users can slide the time bar through the interactive interface to replay the change process of equipment health status, or overlay the high-incidence areas and density distribution of defects in a specific time period in the form of a heatmap. This spatiotemporal integrated visualization interaction method enables managers to transcend the limitations of a single time slice and gain insight into the degradation patterns of equipment from a macro-evolutionary perspective, thereby realizing the transformation from passive response to proactive predictive maintenance. It should be understood that although this embodiment uses a sliding time bar and heatmap as examples, other implementations may also use animation playback, difference comparison views, or other time-series visualization methods, as long as they can realize the dynamic retrospective and trend presentation function of multi-period defect data, they should all be covered within the protection scope of this invention. Example 4

[0049] This embodiment mainly describes the training and optimization process of the defect recognition model, which is independent of the online inspection inference process described in Embodiment 1. As one implementation method, before detecting standardized image data using the defect recognition model, a cross-port visual model collaborative update step is included. This step aims to solve the data distribution offset problem caused by differences in ambient lighting, equipment coating, and background between different ports, while ensuring data privacy and security for each port throughout the training process. Specifically, the cross-port visual model collaborative update includes the following steps S501 to S504.

[0050] Step S501 involves obtaining the model parameter update amounts generated by each participating port after training the initial defect recognition model based on its local image dataset. Specifically, each participating port trains its initial defect recognition model on its local private server using its own collected crane visible light image dataset. During this process, the original image data and annotation information are strictly retained locally and are never uploaded to the central server. Only the gradient or difference between the local model parameters in the current round and the global model parameters in the previous round is calculated as the model parameter update amount. This "data remains stationary, model moves" mechanism fundamentally eliminates the risk of leakage of sensitive industrial images, meets the stringent requirements of port enterprises for data asset security, and also avoids the bandwidth bottleneck caused by the transmission of massive amounts of high-definition images.

[0051] In step S502, the central server performs federated aggregation of the received model parameter updates from all participating ports to generate a global model parameter update, which is then distributed to each participating port. Specifically, the central server uses a federated averaging algorithm to weight the received model parameter updates based on the number of samples in each port's local dataset, thereby generating a global model parameter update that incorporates knowledge from multiple sources. Subsequently, this global update is sent back to each participating port. This process allows each port's local model to indirectly learn diverse defect features from other ports, effectively alleviating the overfitting problem caused by the limited sample types of a single port, and significantly improving the generalization ability of the defect identification model in unknown scenarios.

[0052] In step S503, each participating port receives the global model parameter update and applies it to its local model. Simultaneously, an adversarial domain adaptation module is introduced to align the image feature distributions of different ports. Specifically, after synchronizing its local model using the global update, each port does not directly enter the next round of regular training but instead performs additional domain adaptation optimization. This is because while simple parameter averaging incorporates knowledge, it cannot eliminate the distribution differences in the feature space between different ports (e.g., Port A mainly produces cloudy blue-toned images, while Port B mainly produces sunny warm-toned images). Without aligning these distributions, the model's performance will drastically degrade when processing cross-domain data. The adversarial domain adaptation module is designed to force the model to learn domain-invariant features independent of the environment; its specific implementation mechanism will be detailed below in conjunction with steps S601 to S603.

[0053] Step S504 involves iteratively executing the parameter update acquisition, aggregation and distribution, and domain adaptation steps described above until the global model converges. The converged model is then used as the defect identification model. Specifically, S501 to S503 constitute a complete federated training round. The system continuously monitors the performance metrics (such as mAP or Loss value) of the global model on the validation set. Convergence is determined when the metrics stabilize or reach the preset maximum number of rounds. The final output model not only includes common defect knowledge from multiple ports but also possesses robustness to environmental changes. It can be directly deployed in the UAV edge computing unit or ground station described in Example 1 to perform high-precision detection tasks.

[0054] Regarding the adversarial domain adaptation module mentioned in step S503, as one implementation method, the adversarial domain adaptation module is introduced to align the image feature distribution of different ports, specifically including the following steps S601 to S603.

[0055] Step S601: A domain discriminator is connected after the feature extraction layer of the defect recognition network. The domain discriminator is used to determine the environmental source of the input features. Specifically, the domain discriminator is a branch of a binary or multi-class neural network. Its input is the high-level semantic features output by the feature extraction layer, and its output is the probability that the feature belongs to a source domain (i.e., which port). In terms of training objectives, the domain discriminator strives to accurately distinguish the source of features, while the feature extractor strives to generate confusing features that the domain discriminator cannot distinguish. This adversarial game prompts the feature extractor to strip away domain-specific information related to the environment, such as lighting, color, and texture, and retain only domain-invariant semantic information related to the essence of defects, such as cracks and corrosion, thereby achieving deep alignment at the feature level.

[0056] Step S602 involves setting a gradient inversion layer during training to invert the gradient of the domain discriminator during backpropagation, enabling the feature extractor to learn domain-invariant features. Specifically, the gradient inversion layer is embedded between the feature extractor and the domain discriminator. During forward propagation, the gradient inversion layer performs an identity transformation, not affecting feature transfer; however, during backpropagation, this layer multiplies the gradient from the domain discriminator by -1 before passing it to the feature extractor. From an optimization perspective, this means the domain discriminator attempts to maximize the domain classification loss to distinguish sources, while the feature extractor, receiving the inverted gradient, is actually minimizing the domain classification loss to deceive the discriminator. This ingenious mathematical construction enables end-to-end adversarial training, eliminating the need for alternating optimization of the two networks, significantly simplifying the training process and ensuring convergence stability.

[0057] Step S603: Construct the total loss function as the product of the defect classification loss and the domain discrimination loss, and the dynamic weight coefficients. Set preset initial values ​​for the dynamic weight coefficients during the initial training phase, and gradually decrease them during the later training phase. Specifically, the mathematical expression of the total loss function is as follows: ,in Loss for defect classification, To determine the loss for the domain, These are dynamic weight coefficients. In the early stages of training, when the model features are not yet aligned, setting larger coefficients is appropriate. A value (e.g., 1.0) forces the model to prioritize domain alignment, quickly eliminating distributional differences. As training progresses and the feature spaces gradually merge, maintaining high weights and excessive domain alignment may impair the discriminative power of defective features, leading to negative transfer. Therefore, this embodiment employs a monotonically decreasing strategy (such as cosine annealing or linear decay) to gradually reduce the weights. This dynamic balancing mechanism effectively solves the problem of balancing "alignment" and "classification" in traditional adversarial domain adaptation, ensuring that the model can adapt to new environments when deployed across ports without sacrificing the detection accuracy of minor defects.

[0058] It should be understood that although this embodiment uses the federated averaging algorithm and gradient reversal layer as examples, other federated aggregation strategies (such as FedProx) or other domain adaptation methods (such as maximum mean difference MMD) can also be used in other implementations. As long as the core functions of multi-party collaborative training, data privacy protection, and alignment of cross-domain feature distribution can be achieved, they should all be covered within the scope of protection of this invention. Example 5

[0059] This embodiment mainly describes the data augmentation process before training the defect recognition model. This process is independent of the online inspection inference process described in Embodiment 1 and the federated collaborative training process described in Embodiment 4. As one implementation method, before detecting standardized image data through the defect recognition model, a rare defect visual sample synthesis step is also included. This step aims to solve the problem that defect samples in port crane operation and maintenance scenarios exhibit an extremely long-tail distribution, that is, there are sufficient samples of common defects (such as surface rust), while samples of critical but rare defects (such as fatigue cracks with specific orientations, lack of fusion at the weld root, etc.) are extremely scarce. Direct training will result in a very low recall rate for rare defects. Specifically, the rare defect visual sample synthesis includes the following steps S701 to S704.

[0060] Step S701 involves filtering out rare defect categories from the local dataset whose sample count is below a preset threshold and acquiring the corresponding real images. Specifically, the system first counts the number of samples for each defect category in the local dataset and marks categories with fewer than a preset threshold (e.g., 50 or 100 images) as rare defect categories. For these categories, all available real visible light images are collected as seed samples for subsequent generation. This step ensures that synthetic resources are accurately directed to the weak areas of model learning, avoiding wasting computational power on common categories that have already been fully learned, and also providing necessary prior distribution references for the conditional generative adversarial network.

[0061] Step S702: Construct a Conditional Generative Adversarial Network (CGAN). The generator of the CGAN takes random noise and conditional encoding vectors of rare defect categories as input and outputs a synthesized image. Specifically, unlike ordinary generative adversarial networks that can only generate images with a single distribution, the CGAN used in this embodiment embeds category conditional information in both the generator and the discriminator. The generator receives a random noise vector following a Gaussian distribution and a conditional encoding vector (usually a one-hot vector or embedding vector) representing the target defect category, and injects it into each layer of the network through methods such as concatenation or adaptive instance normalization. This design allows the generation process to be explicitly semantically controlled, enabling the generation of defect images of specified categories on demand, rather than randomly generating uncontrollable textures. At the same time, in order to ensure training stability and prevent pattern collapse, the convolutional layers of both the generator and the discriminator use spectral normalization techniques to constrain the spectral norm of the weight matrix, and a gradient penalty term is introduced into the discriminator loss to force the Lipschitz continuity constraint, thereby ensuring that it can still converge to a reasonable manifold distribution with a small number of real samples.

[0062] Step S703 updates the parameters of the conditional generative adversarial network (GAN) based on real images. Specifically, using the small number of real images and their corresponding labels obtained in step S701, the network parameters are iteratively updated by alternately optimizing the generator and discriminator. The discriminator aims to distinguish between real and generated images and determine whether an image matches the given conditional encoding; the generator aims to deceive the discriminator and generate high-quality images that meet the conditions. After multiple rounds of game-like training, the generator gradually learns the inherent rules of rare defects in texture, shape, and background fusion, and acquires the ability to generate realistic samples.

[0063] Step S704 involves using the updated generator to generate images of rare defect categories in batches, and then performing quality screening on the generated images. The screened images are then added to the training set to balance the number of defect samples across different categories. Specifically, although the trained generator can theoretically produce target samples, the actual generated results may still contain low-quality images with semantic errors, structural distortions, or blurriness. If these noisy images are used directly for training without discrimination, they will seriously mislead the learning direction of the defect recognition model. Therefore, a strict quality screening mechanism must be established to retain only synthetic samples that are highly similar to real data in both statistical distribution and visual perception. The specific implementation of this screening mechanism will be detailed below in conjunction with steps S801 to S803.

[0064] Regarding the quality screening mentioned in step S704, as one implementation method, the generated image is subjected to quality screening, specifically including the following steps S801 to S803.

[0065] Step S801: Input the real image and the synthetic image into the pre-trained image feature extraction network, respectively, and calculate the initial Frescher distance between the synthetic image and the real image in the feature space. Synthetic images with an initial Frescher distance lower than a first threshold are retained. Specifically, the initial Frescher distance is an indicator that measures the similarity between two multidimensional Gaussian distributions. This embodiment utilizes a feature extraction network (such as Inception-V3 or ResNet) pre-trained on a large-scale image dataset to extract high-level semantic features from the real image set and a single synthetic image, respectively, and calculates the distance between their feature distributions. This indicator can assess whether the synthetic image "resembles" a real crane defect at the semantic level, rather than just at the pixel level. Setting a first threshold (e.g., initial Frescher distance < 15) as an initial screening threshold can effectively eliminate failed samples that, although clear in texture, are semantically completely deviated (e.g., generating irrelevant objects or pure noise), ensuring that the data entering the next round of screening is credible in terms of the essential features of the defect.

[0066] In step S802, a no-reference image sharpness index and a structural similarity index are calculated for the retained synthetic images. Images with sharpness below a second threshold (e.g., Brenner gradient value below 50) or structural similarity below a third threshold (e.g., SSIM below 0.7) are discarded. Specifically, the initial Fraser distance alone cannot completely eliminate visually flawed samples, as some semantically correct images may exhibit blurring, ghosting, or unnatural block effects in local areas. The no-reference image sharpness index (such as Brenner gradient or Laplacian variance) is used to quantify the focus and edge sharpness of the image, ensuring that the synthetic image has sufficient detail for the model to learn minute crack features. The structural similarity index is used to measure the perceptual consistency between the synthetic image and the natural image in terms of brightness, contrast, and structural information, preventing the introduction of artifacts that violate the laws of physical imaging. Only images that pass both perceptual quality checks are considered qualified samples. This dual "semantic + perceptual" screening mechanism constitutes a key defense point that distinguishes it from conventional generative adversarial network applications, fundamentally ensuring the usability of augmented data.

[0067] Step S803 involves mixing the selected synthetic images with real images at a preset ratio to form an enhanced local dataset. Specifically, even if the synthetic images are of extremely high quality, they cannot completely replace real data; otherwise, the model may overfit to the inherent bias of the generator. This embodiment employs a mixing strategy, for example, maintaining the proportion of real images at no less than 60%, or controlling the number of synthetic images to within 1 to 2 times that of real images. This ratio significantly alleviates the class imbalance problem, giving the model sufficient opportunities to learn rare defect features, while preserving the dominance of real data and anchoring the model's decision boundaries. The resulting enhanced dataset can be directly used for training the defect recognition model in Example 1 or for federated local training in Example 4, significantly improving the recall and robustness for long-tail defects without increasing manual annotation costs.

[0068] It should be understood that although this embodiment uses conditional generative adversarial networks and the Fraser initial distance + clarity + SSIM triple screening as examples, other generative architectures such as diffusion models can be used in other embodiments, or other distribution metrics such as maximum mean difference and kernel density estimation can be used. As long as high-quality targeted synthesis and effectiveness verification for rare defects can be achieved, they should be covered within the protection scope of this invention. Example 6

[0069] Based on Example 1, this example further refines the process of the UAV acquiring multi-view image sequences in step S100. As one implementation, the process of controlling the UAV to fly along a preset route to acquire multi-view image sequences of the target port crane also includes a real-time edge detection and active re-inspection step. This step aims to resolve the contradiction between "low efficiency of global high-resolution acquisition" and "easy to miss detections" in traditional inspection modes. By constructing an edge-cloud collaborative mechanism of "initial edge screening - cloud-based fine inspection," optimal allocation of inspection resources can be achieved. Specifically, the real-time edge detection and active re-inspection include the following steps S101 to S104.

[0070] Step S101 involves deploying a lightweight defect detection model on the UAV's onboard edge computing unit to perform defect detection frame-by-frame on the real-time video stream. Specifically, the onboard edge computing unit refers to an embedded AI computing module (such as the NVIDIA Jetson series or Huawei Atlas series) integrated into the UAV itself or a mounted pod. While its computing power is lower than that of a ground station server, it is sufficient to support real-time inference. The lightweight defect detection model is obtained by compressing the full defect detection model described in Example 2. Compression methods include, but are not limited to, channel pruning, knowledge distillation, and INT8 quantization. This deployment method allows the UAV to complete preliminary screening locally without transmitting massive amounts of high-definition video back, greatly reducing dependence on communication link bandwidth and eliminating response lag caused by data transmission delays. It should be understood that although this embodiment uses INT8 quantization as an example, in other implementations, FP16 mixed precision or other model acceleration engines can be used depending on hardware characteristics, as long as an acceptable detection rate is maintained while meeting real-time requirements.

[0071] Step S102: When a potential defect is detected and the confidence level exceeds the preset re-inspection threshold, an active re-inspection mode is triggered. Specifically, the setting logic of the preset re-inspection threshold differs from the final confirmation threshold. Since the accuracy of the lightweight model at the edge is inevitably lower than that of the full model, directly using a high confidence standard would lead to a large number of real defects being filtered out; conversely, if the threshold is too low, it would cause frequent false triggers. Therefore, this embodiment sets the re-inspection threshold to a relatively loose range (e.g., 0.4 to 0.6), the core purpose of which is "better to catch the wrong one than miss one," including all suspected areas in the scope of fine-grained investigation. This design ensures the lower limit of the recall rate of the system for difficult-to-detect targets such as micro-cracks, while keeping the computational power consumption for fine-grained processing at a reasonable level.

[0072] Step S103: In active re-inspection mode, the drone is controlled to automatically hover, adjust the gimbal angle to center the defect in the image, increase the optical zoom, and fly along a preset orbital trajectory to acquire multi-view detailed images. Specifically, this is a strict sequence of physical actions. First, automatic hovering is to eliminate motion blur and provide a stable imaging reference for subsequent high-magnification zoom. Second, adjusting the gimbal to center the defect and increasing the optical zoom is to maximize the resolution of the defect area with limited sensor pixels, capturing texture details that are difficult to discern with the naked eye. Finally, flying along the orbital trajectory is to acquire multi-dimensional observation data of the defect under different lighting conditions and occlusion relationships, overcoming the perception blind spot under a single viewpoint. This series of actions is achieved through the tight coupling of the flight control system, gimbal controller, and vision algorithm, reflecting the intelligent inspection essence of "perception-driven control" rather than simple fixed-point photography.

[0073] Step S104 involves transmitting multi-view detailed images back to the ground station for further refinement by the full-version defect identification model, and updating the dynamic digital twin model with the confirmation results. Specifically, the ground station runs the uncompressed full-version defect identification model (i.e., the model described in Example 2), which has higher feature extraction capabilities and classification accuracy. By transmitting a small number of high-value images selected from the edge back to the ground station for secondary confirmation, the bandwidth bottleneck of full video transmission is avoided, and the accuracy of the final data stored is ensured. The confirmed defect information (including precise category, size, and three-dimensional coordinates) will be synchronized in real time to the dynamic digital twin model described in Example 3, completing the business loop from discovery to recording. It should be understood that although this embodiment uses the ground station as the execution subject for detailed confirmation, in other embodiments, if the UAV is equipped with a high-performance computing module or connected to a cloud server via a 5G / 6G link, the detailed confirmation step can also be completed by an airborne high-performance unit or a cloud cluster. As long as the principle of "edge coarse screening + back-end fine inspection" is followed, it should be covered within the scope of protection of this invention.

[0074] Regarding the orbital planning mentioned in step S103, as one implementation method, flying along a preset orbital trajectory to obtain multi-view detailed images specifically includes the following steps S105 to S108.

[0075] Step S105 involves uniformly sampling multiple candidate observation points on a hemisphere accessible to the UAV, centered on the initially detected defect location. Specifically, the "drone-accessible hemisphere" refers to the effective observation space centered on the defect point, with a radius equal to the UAV's minimum safe operating distance, excluding areas obstructed by the crane itself and no-fly zones. Uniform sampling can employ a Fibonacci spherical grid or an icosahedral subdivision algorithm to ensure isotropic spatial distribution of candidate points. This step provides a complete candidate solution space for subsequent intelligent decision-making, avoiding the problem of missing the optimal observation angle due to sparse sampling.

[0076] Step S106: For each candidate observation point, calculate the expected information gain based on the identification accuracy of various defects at that observation angle in historical data. Specifically, the expected information gain is a quantitative indicator that measures the contribution of a certain observation perspective to defect identification. Unlike traditional evaluation methods based on geometric visibility or information entropy, this embodiment directly utilizes historical inspection big data statistics to show that for specific types of defects (such as vertically oriented cracks), the identification accuracy at certain specific angles (such as a 30° side view) is significantly higher than at other angles. The calculation formula can be expressed as follows: ,in For the first The expected information gain of each candidate point Let h be the historical accuracy function, h be the category hypothesis of the current suspected defect, and c be the environmental conditions. This is the observation angle. This gain assessment based on empirical data allows trajectory planning to adaptively focus on the most discriminative perspective, rather than blindly traversing all possible locations.

[0077] Step S107: Select the two candidate observation points with the highest expected information gain and perform re-inspection sequentially. Specifically, selecting Top-2 instead of Top-1 is to introduce necessary redundancy verification to prevent confirmation failure due to momentary interference (such as bird obstruction or strong light reflection) during a single observation; limiting it to two rather than more is to strictly control the re-inspection time and power consumption. This "selective redundancy" strategy maximizes inspection efficiency while ensuring robustness. It should be understood that although this embodiment uses two observation points as an example, in practical applications, the number of selected points can be dynamically adjusted according to the remaining battery power or the severity of the defect (e.g., three for urgent defects and one for general defects). Such variations are all within the protection scope of this invention.

[0078] In step S108, after acquiring images at the first candidate observation point, the defect classification confidence score is calculated in real time. If the defect classification confidence score exceeds a preset confirmation threshold, the image acquisition for the second candidate observation point is skipped; otherwise, the image acquisition for the second candidate observation point continues. Specifically, this is the key logic for achieving "dynamic skipping" in this embodiment. The preset confirmation threshold is usually higher than the re-inspection threshold in step S102 (e.g., 0.85 or 0.9), indicating that the model has reached a state of confidence. Once the first high-gain perspective has provided sufficient evidence, the system immediately terminates subsequent acquisition tasks and directly returns to cruise mode. This conditional triggering mechanism allows the system to complete confirmation with only one re-inspection in most simple cases, consuming additional resources only in difficult cases, thereby significantly reducing the average inspection time and energy consumption in a statistical sense. Compared to fixed-track inspection or unconditional multi-angle shooting, this strategy demonstrates true intelligent decision-making capability, which is an important feature that distinguishes this invention from the prior art.

[0079] It should be understood that although this embodiment has been described in detail using the expected information gain based on historical accuracy and the Top-2 dynamic skip strategy as examples, in other embodiments, trajectory planning can also be driven by Bayesian uncertainty estimation, reinforcement learning reward function or other perspective quality evaluation indicators. As long as the core idea is to dynamically adjust the acquisition behavior according to the difference in observation value, rather than mechanically executing a preset path, it should be covered within the protection scope of this invention. Example 7

[0080] This embodiment takes the regular inspection of a quay crane at a container terminal as an example to further illustrate the application effect and technical advantages of the intelligent defect identification and inspection method for port cranes based on UAV vision provided by this invention in a real engineering scenario. In this application scenario, the system comprehensively utilizes the edge-end active re-inspection, dynamic digital twin construction, and cross-domain collaborative training technologies described in the aforementioned embodiments to achieve a closed-loop intelligent management system covering the entire lifecycle from defect discovery, precise location, trend prediction to maintenance decision-making.

[0081] Specifically, during routine inspections of the quay crane's metal structure, the drone first automatically collects data along a pre-planned flight path covering key load-bearing components such as the main beam, tie rods, and door frames. When the drone flies over the seaside upper beam area, a lightweight defect identification model deployed on the airborne edge computing unit analyzes the real-time video stream frame by frame, detecting a suspected fatigue crack with a confidence level of 0.52, exceeding the preset re-inspection threshold of 0.45 but not reaching the final confirmation standard. At this point, instead of blindly moving forward or simply hovering to take pictures as in traditional fixed-route inspections, the system immediately triggers an active re-inspection mode. The drone automatically hovers and adjusts the gimbal to center the suspected area, while simultaneously sampling candidate observation points on the reachable hemisphere based on the expected information gain algorithm. Since historical data shows that the identification accuracy of this type of crack at a 30-degree side-view angle is significantly higher than at front-view or top-view angles, the system calculates that this angle has the highest expected information gain, and therefore prioritizes controlling the drone to fly to this location for high-magnification zoom photography. After the collected detailed images were transmitted back to the ground station, they were confirmed by the complete defect identification model to be an early fatigue crack approximately 12mm in length, with a confidence level increased to 0.96. This process fully demonstrates the synergistic value of edge screening and intelligent re-inspection: if traditional full-coverage high-resolution imaging were used, the inspection time for this area would increase significantly; if only a single edge inspection were relied upon, it would be highly likely to result in missed detections or false alarms due to excessive distance or poor angle. However, dynamic planning based on expected information gain allows the UAV to acquire key evidence only when necessary from the optimal perspective, maximizing the conservation of battery life and computing resources while ensuring detection accuracy.

[0082] After the defect was accurately identified, the system automatically executed a 3D coordinate transformation and twin mapping process. Through pose information calculated by synchronous positioning and mapping algorithms, and ray intersection calculations, the crack was precisely anchored to the surface of the specific component on the seaside upper beam of the quay crane's digital twin model, with an error controlled within 5mm. Subsequently, the system generated an independent information card for the defect in the dynamic digital twin model and automatically retrieved historical inspection records. Through spatiotemporal correlation matching, the system discovered that an 8mm long micro-crack had been recorded at this location during the last inspection six months prior, thus generating a defect evolution sequence and fitting a linear growth trend. Based on this trend, the prediction model calculated that under the current load conditions, the remaining time for the crack to extend to the safe limit of 50mm is approximately 14 months. The system then automatically generated a maintenance work order containing the defect location, current size, predicted lifespan, and recommended maintenance level, and pushed it to the equipment management system. This shift from "passive discovery" to "proactive prediction" allows the maintenance team to plan maintenance windows in advance, avoiding the risk of sudden downtime and significantly improving equipment availability.

[0083] Furthermore, this application scenario also verified the practical benefits of the auxiliary enhancement technology. When the system of this invention was first deployed at the terminal, thanks to the cross-port federated learning mechanism, the local model did not need to be trained from scratch. Instead, it directly inherited the global model parameters from the other three participating ports. Combined with the adversarial domain adaptation module, it quickly aligned with the local high-salt-fog illumination characteristics, improving the defect identification accuracy during the cold start phase and shortening the data accumulation cycle compared to traditional independent training methods. Simultaneously, for the rare "weld root fusion failure" defect at this terminal, the system used a conditional generative adversarial network to synthesize 200 high-quality samples and added them to the training set. The synthesized data, after triple quality screening, improved the recall rate of this type of defect, effectively solving the identification blind spot problem caused by long-tail samples.

[0084] To quantify the technical effectiveness of this invention, a three-month comparative test was conducted at the wharf. The test results showed that, compared to traditional manual suspended platform inspection, the method of this invention improved the detection rate of minor defects, shortened the time for a single full-machine inspection, and completely eliminated the safety risks to personnel working at heights. Compared to conventional UAV automatic inspection schemes without an active re-inspection mechanism, this invention increased the number of effective defect confirmations under the same endurance conditions and reduced the false alarm rate. These data strongly demonstrate that this invention, through the technical combination of "edge intelligent triggering + information gain planning + dynamic twin prediction," effectively solves the industry pain point of balancing efficiency and accuracy in the inspection of large port cranes. It should be understood that the above specific values ​​are only exemplary results under specific test conditions. Performance indicators in actual applications will vary depending on the equipment model, environmental conditions, and parameter configurations, but as long as the core technical concept described in this invention is adopted, it should be covered within the protection scope of this invention.

[0085] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of variations or substitutions within the technical scope disclosed in this invention, such as adjustments to the network structure of the defect identification model, equivalent transformations of the three-dimensional coordinate transformation algorithm, changes to the federated aggregation strategy, replacement of the generated sample quality evaluation index, and adaptive modifications to the edge-end re-inspection trigger logic and trajectory planning method, all of which should be covered within the scope of protection of this invention. Therefore, the scope of protection of this invention should be determined by the scope of the claims.

Claims

1. A method for intelligent identification and inspection of defects of a port crane based on unmanned aerial vehicle vision, characterized in that, include: The drone is controlled to fly along a preset route to collect multi-view image sequences of cranes at the target port, and the multi-view image sequences are preprocessed to generate standardized image data. The standardized image data is detected using a defect recognition model, and the defect detection result is output, which includes the defect category, image location coordinates, and confidence level. Based on the defect detection results and the UAV pose information, the image position coordinates are transformed to the global spatial coordinate system to construct three-dimensional defect distribution data; A dynamic digital twin model is generated based on the basic 3D model of the target port crane and the 3D defect distribution data. The dynamic digital twin model includes defect annotation information with associated timestamps. The defect annotation information of the same physical defect obtained from multiple inspections is spatiotemporally correlated to generate a defect evolution sequence. Based on the defect evolution sequence, defect growth is predicted, the remaining time for the physical defect to reach a preset threshold is determined, and the result is displayed in the dynamic digital twin model.

2. The method for intelligent identification and inspection of port crane defects based on UAV vision according to claim 1, characterized in that, The defect recognition model employs a multi-scale feature extraction structure based on an attention mechanism. The model detects defects in the standardized image data and outputs a defect detection result containing the defect category, image location coordinates, and confidence level. Multi-layer convolutional feature maps of the standardized image data are extracted using the backbone network; The coordinate attention module performs positional encoding on the multi-layer convolutional feature map to generate a spatially enhanced feature map. The spatial augmentation feature map is fused at multiple scales using a feature pyramid network and a path aggregation network to generate a multi-scale defect detection feature map. The multi-scale defect detection feature map is processed by the decoupled detection head, and the defect category, the image location coordinates, and the confidence level are output respectively.

3. The method for intelligent identification and inspection of port crane defects based on UAV vision according to claim 1, characterized in that, The step of transforming the image position coordinates to a global spatial coordinate system based on the defect detection results and the pose information of the UAV to construct three-dimensional defect distribution data includes: Based on the simultaneous localization and mapping algorithm, the six-degree-of-freedom pose of the UAV relative to a preset starting point is estimated using a continuous image sequence; Based on the camera intrinsic parameters and the six-degree-of-freedom pose, the image position coordinates are transformed into a three-dimensional direction vector in the body coordinate system through inverse perspective projection transformation; The three-dimensional direction vector is transformed to the global spatial coordinate system using the six-degree-of-freedom pose. The three-dimensional direction vector is then intersected with the basic three-dimensional model by ray calculation, and the spatial coordinates of the intersection point in the global spatial coordinate system are calculated. The multiple localization results of the same defect in multiple frames of images are filtered and fused to output the optimized three-dimensional defect distribution data.

4. The method for intelligent identification and inspection of port crane defects based on UAV vision according to claim 1, characterized in that, The generation of a dynamic digital twin model based on the basic 3D model of the target port crane and the 3D defect distribution data includes: An initial three-dimensional geometric model containing component attributes is established based on computer-aided design drawings as the basic three-dimensional model; The defect detection results are converted into structured data and associated with the corresponding spatial region in the initial three-dimensional geometric model; In the dynamic digital twin model, an information card is generated for each defect, and the information card includes the defect type, size, discovery time, and historical evolution curve; The defect information is organized according to the timeline to support viewing the defect distribution heatmap of different inspection cycles by sliding the time bar.

5. The method for intelligent identification and inspection of port crane defects based on UAV vision according to claim 1, characterized in that, Before the standardized image data is detected by the defect identification model, a cross-port visual model collaborative update step is also included: Obtain the update amount of model parameters generated by each participating port after training the initial defect recognition model based on the local image dataset; The central server performs federated aggregation of the model parameter updates received from all participating ports, generates global model parameter updates, and distributes them to each participating port. Each participating port receives the global model parameter update and applies it to its local model. At the same time, an adversarial domain adaptation module is introduced to align the image feature distribution of different ports. The above steps of obtaining parameter update amounts, aggregation and distribution, and domain adaptation are executed iteratively until the global model converges. The converged model is then used as the defect identification model.

6. The method for intelligent identification and inspection of port crane defects based on UAV vision according to claim 5, characterized in that, The introduction of the adversarial domain adaptation module to align the image feature distributions of different ports includes: A domain discriminator is connected after the feature extraction layer of the defect identification network. The domain discriminator is used to determine the environmental source of the input features. During training, a gradient inversion layer is set up so that the gradient of the domain discriminator is inverted during backpropagation, so that the feature extractor learns domain-invariant features. The total loss function is constructed as the defect classification loss minus the domain discrimination loss and the dynamic weight coefficient. The dynamic weight coefficient is set to a preset initial value in the early stage of training and gradually reduced in the later stage of training.

7. The method for intelligent identification and inspection of port crane defects based on UAV vision according to claim 1, characterized in that, Before detecting the standardized image data using the defect recognition model, a rare defect visual sample synthesis step is also included: The system filters out rare defect categories with fewer than a preset threshold of samples from the local dataset and obtains the corresponding real images. A conditional generative adversarial network is constructed, wherein the generator of the conditional generative adversarial network takes random noise and the conditional encoding vector of the rare defect category as input and outputs a synthesized image; The parameters of the conditional generative adversarial network are updated based on the real image; The generator with updated parameters is used to generate images of the rare defect categories in batches. The generated images are then quality-screened, and the screened generated images are added to the training set to balance the number of defect samples of each category.

8. The method for intelligent identification and inspection of port crane defects based on UAV vision according to claim 7, characterized in that, The quality screening of the generated images includes: The real image and the synthetic image are respectively input into a pre-trained image feature extraction network. The initial Frescher distance between the synthetic image and the real image in the feature space is calculated. Synthetic images whose initial Frescher distance is lower than a first threshold are retained. For the retained synthetic images, calculate the no-reference image sharpness index and structural similarity index, and remove images with sharpness below the second threshold or structural similarity below the third threshold; The selected synthetic images are mixed with the real images at a preset ratio to form an enhanced local dataset.

9. The method for intelligent identification and inspection of port crane defects based on UAV vision according to claim 1, characterized in that, The process of controlling the drone to fly along a preset route to collect multi-view image sequences of cranes at the target port also includes real-time edge detection and active re-inspection steps: A lightweight defect recognition model is deployed on the airborne edge computing unit of a drone to perform defect detection frame by frame on a real-time video stream; When a potential defect is detected and the confidence level exceeds the preset re-inspection threshold, the active re-inspection mode is triggered. In the active re-inspection mode, the drone is controlled to automatically hover, adjust the gimbal angle to center the defect in the image, increase the optical zoom, and fly along a preset orbital trajectory to acquire multi-view fine images. The multi-view detailed images are transmitted back to the ground station, where they are further refined by the full version of the defect identification model, and the confirmation results are updated to the dynamic digital twin model.

10. The method for intelligent identification and inspection of port crane defects based on UAV vision according to claim 9, characterized in that, The process of acquiring multi-view detailed images by flying along a preset orbital trajectory includes: Using the initially detected defect location as the center, multiple candidate observation points are uniformly sampled on the hemisphere accessible by the UAV. For each candidate observation point, the expected information gain is calculated based on the accuracy of identifying various defects at that observation angle in historical data. The two candidate observation points with the largest expected information gain are selected and re-examined sequentially. After acquiring images at the first candidate observation point, the defect classification confidence score is calculated in real time. If the defect classification confidence score exceeds a preset confirmation threshold, the image acquisition of the second candidate observation point is skipped; otherwise, the image acquisition of the second candidate observation point continues.

Images