An unmanned intelligent inspection system based on computer vision

Abstract

This invention discloses an unmanned intelligent inspection system based on computer vision. The invention relates to the fields of computer vision, artificial intelligence, and industrial internet technology. The cross-domain small-sample defect detection module based on feature decoupling adopts a dual-branch feature extraction architecture with parallel content and style branches. Orthogonal decoupling of content and style features is forced through mutual information minimization loss. Combined with a dynamically updated defect prototype memory network, small-sample cross-domain defect classification is achieved, enabling the system to maintain high-precision detection capability for unseen scenes with only a small number of labeled samples. The cross-period spatiotemporal evolution graph neural network early warning module constructs a spatiotemporal evolution graph that integrates temporal and spatial edges through two-stage cross-period image registration. After modeling by a multi-head spatiotemporal attention encoder, it outputs a quantitative fault prediction time based on Weibull degradation curve parameter estimation, achieving proactive predictive early warning for progressive defects.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, artificial intelligence and industrial internet technology, specifically to an unmanned intelligent inspection system based on computer vision. Background Technology

[0002] In the daily operation and maintenance of industrial facilities, the use of unmanned inspection platforms (drones or wheeled inspection robots) equipped with visual sensors to perform automated defect detection of equipment has become an important technical means. Convolutional Neural Networks (CNNs) have become the mainstream framework for image feature extraction since AlexNet. Object detection networks based on Faster R-CNN and YOLO series are widely used in industrial quality inspection scenarios. The current mainstream unmanned inspection platforms are equipped with multimodal sensors such as RGB industrial cameras, thermal imagers, and LiDAR, as well as inertial measurement units and RTK-GPS positioning modules. With the help of edge computing platforms, real-time deep learning inference can be realized on the device side.

[0003] Existing industrial inspection visual inspection systems mainly fall into two representative categories. The first category is a static defect detection system based on supervised learning: a large number of equipment images are collected in a specific factory area and manually labeled, a scene-specific target detection network is trained, and the inspection images are reasoned frame by frame and the detection results are reported. The core limitations of this approach are: the model requires a large amount of high-quality labeled data for training, resulting in extremely high labeling costs; the model is strongly bound to the training scene, and the detection accuracy is severely lost after cross-domain transfer; the detection logic is based only on a single frame of static image and lacks temporal analysis capabilities. The second category is a change detection system based on simple difference comparison: the current inspection image is subjected to pixel-level difference operation with historical baseline images, and the changed areas are filtered by threshold segmentation. This approach has extremely high requirements for the repeatability accuracy of the inspection platform's pose. Even a small deviation in pose will generate a large amount of false change noise, and it is extremely insensitive to small gradual changes, making it unable to complete fault trend prediction.

[0004] The existing solutions mentioned above have two major technical flaws: First, insufficient generalization ability leads to extremely high deployment costs across scenarios. Existing supervised learning models couple defect features with environmental features such as background texture and lighting conditions in the same feature space for joint learning. When the lighting conditions or background texture of the target scene change, the feature distribution activated by the model shifts from the training distribution, and the detection confidence drops sharply. Each time a new factory area is connected, labeled data needs to be collected again and a new model needs to be trained, making it impossible to achieve large-scale rapid deployment. Second, the lack of temporal modeling capability makes it impossible to predict faults. Existing systems use frames as the smallest processing unit and do not maintain historical state information across inspection cycles. In addition, the pose error of the unmanned platform makes it difficult to accurately register images from different cycles. The system can only passively judge whether there are defects in the current frame that have reached the detection threshold. It cannot perceive minute anomalies in their infancy until the defects develop into obvious damage before they are discovered, posing a significant safety hazard. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide an unmanned intelligent inspection system based on computer vision. By combining a cross-domain small sample defect detection method based on feature decoupling with a cross-period spatiotemporal evolution graph neural network early warning method, the two core problems of insufficient accuracy in small sample cross-domain defect detection and the inability to predictively perceive progressive anomalies are fundamentally solved.

[0006] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows: An unmanned intelligent inspection system based on computer vision includes a data acquisition and preprocessing module, a cross-domain small sample defect detection module based on feature decoupling (FD-FSDD), a cross-period spatiotemporal evolution graph neural network early warning module (CC-STGT), and a cloud-based continuous learning module. The system adopts a distributed architecture that combines edge inference and cloud modeling. The FD-FSDD and CC-STGT modules are deployed on the edge computing unit carried by the unmanned inspection platform. The cloud-based continuous learning module is responsible for periodically receiving all inspection data and updating the model parameters of the FD-FSDD and CC-STGT modules before sending them to the edge, forming a closed-loop self-learning system.

[0007] The data acquisition and preprocessing module is responsible for image acquisition, synchronous pose recording, and image quality filtering. The unmanned inspection platform is equipped with an RGB industrial camera, an inertial measurement unit, and an RTK-GPS positioning module. It synchronously records a timestamp and a six-DOF pose tuple containing three-dimensional position coordinates and roll, pitch, and yaw angles for each frame of image. It uses a sharpness score based on the Laplacian operator to evaluate the quality of each frame of image. Blurry or overexposed frames with a sharpness score below a preset threshold are directly discarded and do not enter the subsequent processing flow.

[0008] The core idea of ​​the FD-FSDD module is to forcibly decompose the image feature space into a content feature space related to the semantics of defects and a style feature space related to the environmental background, so that defect recognition relies only on content features, thus naturally possessing robustness to environmental changes. The FD-FSDD module includes a dual-branch feature extractor and a defect prototype memory network.

[0009] The dual-branch feature extractor contains two parallel feature extraction paths: a content branch and a style branch. The content branch uses a ResNet-50 network as the backbone feature extractor, removes the final fully connected classification layer, and retains the output up to the fourth residual block to obtain a semantic feature map with a spatial resolution of 1 / 32 of the input image. This feature map retains sufficient location information in the spatial dimension to capture the geometry, edge orientation, and topological distribution of defects. For an input image with a height of H pixels and a width of W pixels, the content branch outputs a content feature tensor with a spatial size of H / 32 multiplied by W / 32 and a channel dimension of 256. The style branch uses the same ResNet-50 backbone network structure as the content branch, but inserts an adaptive instance normalization (AdaIN) layer after each residual block. This layer calculates the mean and standard deviation of the current feature map in each channel as a style descriptor. The mean and standard deviation vectors extracted from each residual block are concatenated to obtain a style feature vector with a dimension of 128. This vector compactly encodes the global illumination intensity, material reflectivity, and background texture statistical characteristics of the image, without containing any defect geometric information.

[0010] To force the output features of the two branches to be completely orthogonal at the information level, a mutual information minimization loss (MIN Loss) based on the mutual information neural estimation (MINE) framework is introduced.

[0011] Let the content features be denoted as Style characteristics are denoted as The introduced parameters are Three-layer fully connected mutual information estimation network The mutual information estimates of the two features The total loss is included as a minimization objective in the total loss function. Its composition is as follows: ; in, The standard object detection loss consists of the sum of the classification cross-entropy loss and the bounding box regression IoU loss. For mutual information estimation network The estimated and Mutual information between them, measured in nats. The hyperparameter for controlling the strength of the decoupling constraint is set to 0.1, and is minimized during training. , making content features It approximates the representation of pure defect geometry semantics, fundamentally eliminating the interference of environmental factors.

[0012] The defect prototype memory network uses the mean vector of content features for each type of defect as a prototype and maintains it dynamically to achieve small-sample classification. During the initial system deployment, using 5 to 10 labeled support set samples for each defect type, the mean vector of content features for all support samples corresponding to each defect type is calculated and used as the initial prototype vector for that defect type. Let the first... The number of supporting samples for this type of defect is , No. The content feature vector obtained after content branching and global average pooling of each supporting sample is: Then the initial prototype The calculation is as follows: ; During each inspection and reasoning process, if there are defective samples with a classification confidence score greater than 0.9 in the detection results, the corresponding prototype is updated using the exponential moving average (EMA) method. The prototype update rules are as follows: ; in, For the first The prototype vector of the class defect at the current moment. For the newly detected Content feature vector of defect sample The momentum coefficient, with a value of 0.99, is used to control the stability of prototype updates and avoid excessive perturbation of the prototype by a single new sample. During inference, the content features extracted from the query image through the content branch (after global average pooling) are calculated, and their cosine similarity to all prototype vectors in the prototype library is used. The category corresponding to the highest similarity is taken as the predicted defect category, and the highest cosine similarity value is normalized by Softmax and used as the classification confidence output. With the Class prototype Cosine similarity between The calculation is as follows: ; in, Let be the inner product of two vectors. and These are the L2 norms of the corresponding vectors. Furthermore, in the content feature map... A lightweight fully convolutional detection head is superimposed on the defect region to perform regression prediction of the bounding box coordinates. Redundant boxes are filtered out by the non-maximum suppression (NMS) algorithm, and the localization result of each defect instance is output.

[0013] The CC-STGT module is the core solution for solving the problem of progressive anomaly prediction. Its workflow covers four stages: cross-cycle accurate registration, spatiotemporal graph construction, CC-STGT coding, and fault time prediction.

[0014] A two-stage approach is used for precise registration across cycles.

[0015] The first stage is coarse registration based on RTK pose: using the six-DOF pose tuple recorded in each frame, the relative pose transformation matrix of the historical periodic image relative to the current periodic image is calculated. Combined with the camera intrinsic parameter matrix, the historical image is projected to the current view coordinate system through homography transformation. After coarse registration, the remaining pixel deviation between the two images is usually within 10 pixels. The second stage is fine registration based on the improved Super Glue algorithm: Based on the coarse registration results, Super Point key points and their descriptors are extracted from the two images. An improved Super Glue graph neural network based on the coarse registration homography matrix space prior is introduced into the original Super Glue attention mechanism to establish the correspondence between key points across images. The fine homography matrix is ​​robustly estimated through the Random Sample Consensus (RANSAC) algorithm. A fine homography matrix transformation is applied to the historical images, and the remaining registration error can be controlled within 1 to 2 pixels.

[0016] The spatiotemporal graph is constructed based on an aligned set of multi-period images.

[0017] A region of interest of a fixed size is defined centered on the locations of all defect instances detected by the FD-FSDD module. Content feature vectors are extracted from each region of interest as the initial features of nodes in the spatiotemporal graph, where the superscript indicates the inspection cycle number and the subscript indicates the detection point number within that cycle.

[0018] Edges in the spatiotemporal graph are divided into two categories: temporal edges connect pairs of nodes corresponding to the same physical detection point in different periods, encoding the state change relationship of the same equipment part over time; spatial edges connect pairs of neighboring detection point nodes within the same period with a geographical distance of less than 0.5 meters, encoding the structural relationship between adjacent equipment parts. Each temporal edge is assigned a time interval position encoding vector, taking the actual physical time interval (unit: days) as input, and is mapped to a vector space of the same dimension as the node features through a sine-cosine position encoding function, enabling the model to perceive the actual time span between two observations.

[0019] The CC-STGT encoder consists of six stacked Transformer coding layers, each containing a multi-head spatiotemporal attention sublayer. It simultaneously models spatial neighborhood attention (message transmission between neighboring nodes within the same period) and temporal self-attention (message transmission between historical observations of different periods at the same detection point) for all nodes in the graph, and adaptively fuses the two types of attention information through a learnable attention head weight matrix.

[0020] Each sublayer employs residual connections and follows layer normalization operations to ensure the stability of deep network training. After 6 layers of stacked encoding, each node obtains a final representation vector that integrates complete temporal history information and spatial context information, with a dimension of 256.

[0021] The Time-of-Failure (TTF) prediction method employs the Weibull degradation curve fitting approach. The TTF prediction head takes the final node representation vector as input and outputs a scalar health status score through a two-layer fully connected network (128-dimensional intermediate layer with ReLU activation). A score of 0 indicates the device is fully healthy, while 1 indicates the device has reached the functional failure threshold. The health status score sequence corresponding to the historical time-series observations is input into a lightweight LSTM module. This module outputs the shape and scale parameters of the Weibull distribution. The shape parameter describes the degradation rate over time, and the scale parameter describes the feature lifetime, thus constructing an individualized degradation curve for the device. Based on the estimated Weibull distribution parameters and the current health status score, the expected time when the device reaches the failure threshold is calculated using the inverse function of the Weibull cumulative distribution function. The difference between this expected time and the current time is the Time-of-Failure (TTF), output in days.

[0022] The system performs risk level fusion judgment based on FD-FSDD detection results and CC-STGT prediction results, and divides them into three levels: low risk (fault prediction time greater than 30 days), medium risk (fault prediction time between 7 and 30 days) and high risk (fault prediction time less than 7 days). The system triggers different response actions respectively, such as normal recording and updating the historical database, generating defect reports and pushing alarms to operation and maintenance personnel, and triggering emergency warnings and recommending immediate shutdown and maintenance.

[0023] To meet the industrial deployment requirements of achieving real-time inference speeds of over 25 FPS on edge computing units, the FD-FSDD and CC-STGT modules have been lightweighted: The convolutional kernels of all convolutional layers in both modules are sorted by L1 norm, and the 30% of convolutional kernels with the lowest importance are pruned and then fine-tuned to restore accuracy. By using TensorRT to perform INT8 precision quantization calibration, the model weights and activation values ​​are compressed from FP32 to INT8 representation, reducing memory usage by about 75% and increasing inference throughput by about 3 times. The quantized model is compiled into a TensorRT dedicated inference engine, which makes full use of the TensorCore of the edge computing unit to accelerate matrix operations. On the NVIDIA Jetson AGX Orin platform, a stable inference frame rate of 28 to 32 FPS is achieved in actual tests, which meets the real-time requirements of industrial sites.

[0024] Compared with the prior art, the beneficial effects of the present invention by adopting the above technical solution are as follows: 1. This invention completely separates the geometric topological features of defects from the style features of the environmental background by decoupling dual-branch features and minimizing mutual information. The defect representation learned by the model is not affected by environmental factors such as lighting and material. Theoretically, it can be applied to any unseen new scene, greatly reducing the data annotation cost of new scene access and solving the problem of the sharp decline in cross-domain transfer accuracy of existing solutions. 2. Through a dynamically updated defect prototype memory network, the system only needs 5 to 10 labeled samples per type of defect to complete the deployment, and it continuously adapts during the inference process without the need for manual re-labeling; 3. Through two-stage cross-cycle precise registration and spatiotemporal evolution modeling, the system can perform time-series modeling of the evolution trend of minor defects, output quantitative fault prediction time, upgrade the inspection system from passive post-event discovery to proactive predictive early warning, and provide maintenance personnel with ample time window for intervention. 4. By employing a lightweight edge deployment scheme with structured pruning and INT8 quantization, real-time inference under limited computing resources is achieved, meeting the deployment requirements of industrial sites. Attached Figure Description

[0025] Figure 1 This is a diagram showing the complete system architecture of the present invention; Figure 2 This is a schematic diagram of the internal data flow of the FD-FSDD module; Figure 3 This is a schematic diagram of the complete processing flow of the CC-STGT module; Figure 4 This is a detailed end-to-end flowchart of the entire system of the present invention. Detailed Implementation

[0026] The following is in conjunction with the appendix Figure 1 -Appendix Figure 4 The specific embodiments of the present invention will be further described below. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0027] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0028] A computer vision-based unmanned intelligent inspection system consists of four functional modules: a data acquisition and preprocessing module, a feature-decoupled cross-domain small sample defect detection module (FD-FSDD), a cross-period spatiotemporal evolution graph neural network early warning module (CC-STGT), and a cloud-based continuous learning module. The system adopts a distributed architecture that combines edge inference and cloud modeling to achieve collaborative work between real-time detection and long-term trend analysis.

[0029] Specific implementation of the data acquisition and preprocessing module The unmanned inspection platform is equipped with the following sensor suite: an RGB industrial camera with a resolution of no less than 1920×1080 and a frame rate of no less than 30FPS; an inertial measurement unit with a sampling frequency of no less than 200Hz; and an RTK-GPS positioning module with a positioning accuracy better than 2cm.

[0030] The edge computing unit uses NVIDIA Jetson AGX Orin and is responsible for all real-time inference tasks.

[0031] The system records the corresponding timestamp and pose tuple for each frame of image. The definition is as follows: ; in, , , These are the three-dimensional position coordinates of the platform in the geodetic coordinate system, in meters. This is the roll angle. The pitch angle, The yaw angle is expressed in radians.

[0032] The above six degrees of freedom fully describe the spatial state of the platform at the time of acquisition and serve as the basic input for subsequent cross-period image registration.

[0033] The system uses a sharpness score based on the Laplacian operator to evaluate the quality of each image frame. Blurry or overexposed frames with a sharpness score below a preset threshold (default 100) are discarded directly and not included in subsequent processing to avoid low-quality images contaminating the detection results. Images that pass the quality filter are uniformly scaled to 640×640 pixels before being sent to subsequent modules.

[0034] Specific implementation of the FD-FSDD module The FD-FSDD module consists of two parts: a dual-branch feature extractor and a defect prototype memory network.

[0035] The content branch of the dual-branch feature extractor uses a standard ResNet-50 network as the backbone feature extractor, removes the final fully connected classification layer, and retains the output up to the fourth residual block (Layer 4) to obtain a semantic feature map with a spatial resolution of 1 / 32 of the input image. For a height of pixels, width Pixel input image The content branch outputs a content feature tensor. ,in The channel dimension for content features is set to 256 by default. This feature map retains sufficient location information in the spatial dimension to capture the geometry, edge orientation, and topological distribution of defects, such as the linear extension structure of cracks and the regional clustering morphology of corrosion.

[0036] The style branch uses the same ResNet-50 backbone network structure as the content branch, but inserts an adaptive instance normalization (AdaIN) layer after each residual block. The AdaIN layer calculates the mean of the current feature map in each channel. and standard deviation These two statistics are the style descriptors extracted by this layer, representing the brightness bias and texture contrast of the corresponding layer in the current image, respectively. The mean and standard deviation vector extracted from each residual block are concatenated to obtain the style feature vector. ,in This is the dimension for style features, set to 128 by default. It compactly encodes the global illumination intensity, material reflectivity, and statistical properties of background texture of the image, without containing any spatial location information, and therefore does not carry geometric information about defects.

[0037] The specific implementation of the mutual information minimization constraint (MIN Loss) is as follows: A parameter is introduced. Mutual information estimation network This network is a three-layer fully connected network that estimates the mutual information of the two features. Incorporate the minimization objective into the total loss function: ; in, The standard object detection loss consists of the sum of the classification cross-entropy loss and the bounding box regression IoU loss. For the network The estimated and Mutual information between them, measured in nats. The hyperparameter for controlling the strength of the decoupling constraint is set to 0.1.

[0038] Initialization phase of the defect prototype memory network: When the system is first deployed on a certain type of device, using 5 to 10 labeled support set samples for each type of defect, the mean vector of the content features of all support samples corresponding to each type of defect is calculated as the initial prototype vector of that type of defect.

[0039] Let the first The number of supporting samples for this type of defect is , No. Each supporting sample, after content branching, undergoes global average pooling for dimensionality reduction. The content feature vector is Then the initial prototype The calculation is as follows: ; Prototype dynamic update phase: During each inspection and reasoning process, if there are defective samples with a classification confidence score greater than 0.9 in the detection results, the corresponding prototype is updated using an exponential moving average method. ; in, This is the momentum coefficient, with a value of 0.99. For the newly detected Content feature vector of a defect sample.

[0040] During inference, the content features of the query image are extracted through content branching and then subjected to global average pooling. Calculate its cosine similarity with all prototype vectors in the prototype library: ; in, Let be the inner product of two vectors. and These are the L2 norms of the corresponding vectors. The category corresponding to the highest cosine similarity is taken as the predicted defect category, and the highest cosine similarity value is normalized by Softmax and used as the classification confidence output in the content feature map. A lightweight fully convolutional detection head consisting of 3×3 and 1×1 convolutional layers is superimposed on the defect region to perform regression prediction of the bounding box coordinates (center point x-coordinate, center point y-coordinate, width, and height), and redundant boxes are filtered out by non-maximum suppression algorithm.

[0041] Taking power inspection scenario as an example: The system completes initialization with only 5 labeled samples for each type of defect. It detects test sets containing three types of defects: insulator flashover, conductor wear, and hardware corrosion. After migrating to a new plant area with different lighting conditions, the detection accuracy is significantly improved compared to the traditional full-supervision scheme, verifying the cross-domain generalization ability of this scheme.

[0042] Specific implementation of the CC-STGT module The two-stage implementation of the cross-cycle precise registration submodule is as follows.

[0043] Coarse registration stage: using pose tuples recorded in each frame of image Calculate the relative pose transformation matrix of the historical periodic image with respect to the current periodic image. This matrix is ​​a 4×4 homogeneous transformation matrix containing a 3×3 rotation matrix. With 3×1 translation vector Using the camera intrinsic parameter matrix (Obtained from factory calibration, including focal length) , and principal point coordinates , The historical image is projected onto the current viewpoint coordinate system through homography transformation to complete coarse registration. After coarse registration, the remaining pixel deviation between the two images is usually within 10 pixels.

[0044] Fine registration stage: Based on the coarse registration results, SuperPoint keypoints and their descriptors are extracted from the two images. Then, an improved SuperGlue graph neural network is used to establish the keypoint correspondence across images (introducing a spatial prior based on the coarse registration homography matrix into the original SuperGlue attention mechanism to accelerate convergence and improve the matching robustness of small texture regions). The fine homography matrix is ​​robustly estimated using the Random Sample Consensus (RANSAC) algorithm. Ultimately, this applies to historical images. The transformation achieves pixel-level precise alignment, and the remaining registration error can be controlled within 1 to 2 pixels.

[0045] The spatiotemporal graph construction submodule: On the aligned multi-period image set, a fixed-size region of interest (ROI) (default 64×64 pixels) is defined centered on the locations of all defect instances detected by the FD-FSDD module. For each ROI, RoI Pooling is used to map regions of different sizes into fixed-dimensional feature vectors, which serve as nodes in the spatiotemporal graph. The initial features, where the superscript Indicates the inspection cycle number, subscript This indicates the detection point number within that period.

[0046] Timing edges connect different periods and ( In the context of nodes whose geographical locations correspond to the same physical detection point, Temporal edges encode the state changes of the same device part over time, while spatial edges connect the same period. Neighborhood detection point pairs with an internal geographic distance of less than 0.5 meters Spatial edge encoding determines the structural relationships between adjacent equipment parts (such as the spatial propagation direction of cracks).

[0047] Assign a time interval position encoding vector to each temporal edge This vector is in the form of actual physical time intervals. (Unit: days) is used as input, and mapped to a vector space of the same dimension as the node features through a sine-cosine positional encoding function: ; in, For node feature dimensions, For encoding dimension index, and These are the positions of the encoded vectors. peacekeeping The addition of dimensional components enables the model to perceive the actual time span between two observations.

[0048] The CC-STGT encoder is made of The encoding is composed of stacked Transformer layers, each containing a multi-head spatiotemporal attention sublayer. It models both spatial neighborhood attention (message propagation between neighboring nodes within the same period) and temporal self-attention (message propagation between historical observations from different periods at the same detection point), and adaptively fuses these two types of attention information through a learnable attention head weight matrix. Each sublayer employs residual connections and follows layer normalization. After six stacked encoding layers, each node obtains a final representation vector that fuses complete temporal history and spatial context information. ,in This is the dimension of the encoder's hidden layer, which defaults to 256.

[0049] Time-of-Failure Prediction Submodule: TTF Prediction Header As input, a scalar health status score is output through a two-layer fully connected network (the middle layer has a dimension of 128 and the activation function is ReLU). Where 0 indicates that the equipment is completely healthy, and 1 indicates that the equipment has reached the functional failure threshold, the health status score sequence corresponding to the time series historical observation sequence is used. (in The actual timestamps for each historical period (in days) are input into a lightweight LSTM module, which outputs the shape parameters of the Weibull distribution. (Describing the trend of degradation rate over time) and scale parameters (Describes the characteristic lifetime, i.e., the time corresponding to when the cumulative failure probability reaches 63.2%), Weibull cumulative distribution function. The definition is as follows: ; in, For time variables (unit: days). It is a natural exponential function, based on the estimated parameters. and and the current health status score. The failure threshold of the equipment is calculated by using the inverse function of the Weibull cumulative distribution function. The expected time corresponding to ) and the current time The difference is the Fault Prediction Time (TTF), and the output value is in days.

[0050] The system determines the risk level based on the TTF value: a TTF greater than 30 days is considered low risk, and the system records and updates the historical database normally; a TTF between 7 and 30 days is considered medium risk, and a defect report is generated and an operation and maintenance alarm is pushed; a TTF less than 7 days is considered high risk, triggering an emergency warning and recommending immediate shutdown for maintenance.

[0051] Taking the corrosion defect of a steel structure in a substation as an example: the system performed cross-cycle registration and spatiotemporal modeling on images of the detection point for 12 consecutive inspection cycles (spanning about 6 months), successfully identified the slow expansion trend of the corrosion area in the first 8 cycles, and output a TTF warning about 21 days before the corrosion area exceeded the critical value, triggering a medium-risk operation and maintenance response, thus verifying the predictive perception capability of this solution for progressive defects.

[0052] Specific implementation of lightweight deployment of edge models To meet the industrial deployment requirements of achieving real-time inference speeds of over 25 FPS on the NVIDIA Jetson AGX Orin, the FD-FSDD and CC-STGT modules were lightweighted as follows: Structured pruning: Sort the convolutional kernels of all convolutional layers in both modules according to their L1 norm, prune the 30% of convolutional kernels with the lowest importance, retain 70% of the convolutional kernels and then fine-tune them to restore accuracy loss.

[0053] INT8 quantization: The pruned model is calibrated with INT8 precision quantization using the TensorRT tool, compressing the model weights and activation values ​​from FP32 (32-bit floating point) to INT8 (8-bit integer) representation, reducing memory usage by about 75% and increasing inference throughput by about 3 times.

[0054] TensorRT acceleration engine construction: The quantized model is compiled into a TensorRT dedicated inference engine, which makes full use of the Tensor Core units on Jetson AGX Orin for matrix operation acceleration. Finally, a stable inference frame rate of 28 to 32 FPS was achieved in actual tests on the platform, which meets the real-time requirements of industrial field.

[0055] Specific implementation of the cloud-based continuous learning module The cloud-based continuous learning module periodically receives full inspection data, performs full fine-tuning updates on the model parameters of the FD-FSDD and CC-STGT modules, and then distributes the updated model parameters to each edge device. The fine-tuning process uses the same total loss function as the training phase. This ensures that the feature decoupling constraint is maintained during the continuous learning process. Newly acquired high-confidence detection samples are simultaneously added to the update queue of the prototype memory, further enhancing the prototype memory's ability to represent the defect distribution in the current deployment environment.

[0056] The complete end-to-end workflow of the system described in this invention is as follows: Figure 4 As shown, it covers the entire process from the start of the inspection to continuous learning in the cloud, with complete logic and closed-loop self-adaptation.

[0057] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, any modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope defined by the claims of the present invention.

Claims

1. A computer vision-based unmanned intelligent inspection system, characterized in that, It includes a data acquisition and preprocessing module, a cross-domain small sample defect detection module based on feature decoupling, a cross-period spatiotemporal evolution graph neural network early warning module, and a cloud-based continuous learning module; The data acquisition and preprocessing module is mounted on the unmanned inspection platform and includes an RGB industrial camera, an inertial measurement unit and an RTK-GPS positioning module. It synchronously records the timestamp and six-degree-of-freedom pose tuple for each frame of image and uses a sharpness score based on the Laplacian operator to filter low-quality image frames. The cross-domain small sample defect detection module based on feature decoupling includes a dual-branch feature extractor and a defect prototype memory network. The dual-branch feature extractor includes a parallel content branch and a style branch. The content branch extracts content features containing defect geometric topology information, and the style branch extracts style features containing ambient lighting and background texture statistics. The module applies orthogonal constraints to the content features and style features by minimizing mutual information with a weight coefficient of 0.1, so that the content features approach the pure geometric semantic representation of defects. The defect prototype memory network uses the mean vector of the content features of various defects as the initial prototype, updates it dynamically with an exponential moving average, and achieves small sample defect classification through cosine similarity matching. The cross-period spatiotemporal evolution graph neural network early warning module performs two-stage cross-period accurate registration on historical multi-period inspection images, constructs a spatiotemporal evolution graph that simultaneously contains temporal edges and spatial edges on the aligned image, and after being encoded by the cross-period spatiotemporal graph neural network encoder, outputs a quantitative fault prediction time through Weibull degradation curve parameter estimation. The cloud-based continuous learning module periodically receives all inspection data, updates the model parameters of the feature-decoupling-based cross-domain small sample defect detection module and the cross-period spatiotemporal evolution graph neural network early warning module, and then sends the updated data to the edge.

2. The system according to claim 1, characterized in that, In the dual-branch feature extractor, the content branch uses a ResNet-50 network as the backbone feature extractor, removes the final fully connected classification layer, and retains the output up to the fourth residual block to obtain a content feature tensor with a spatial resolution of 1 / 32 of the input image. The style branch adopts the same backbone network structure as the content branch, and an adaptive instance normalization layer is inserted after each residual block. The mean and standard deviation of each channel are calculated as style descriptors, and the mean and standard deviation vectors extracted from each residual block are concatenated to form a style feature vector.

3. The system according to claim 1, characterized in that, The mutual information minimization loss introduces a three-layer fully connected mutual information estimation network based on the mutual information neural estimation framework, incorporating the mutual information estimate between content features and style features as an additional constraint term into the total loss function; The total loss consists of the target detection loss and the weighted mutual information estimate, where the weight coefficient of the weighted mutual information estimate is 0.

1.

4. The system according to claim 1, characterized in that, The prototype dynamic update strategy of the defect prototype memory network is as follows: during each inspection inference process, when the detection classification confidence of a certain type of defect is greater than 0.9, the feature vector of the newly detected defect sample of that type is integrated into the corresponding prototype vector by an exponential moving average with a momentum coefficient of 0.99, so as to realize the continuous adaptation of the prototype library to the defect distribution of the current deployment environment.

5. The system according to claim 1, characterized in that, The two-stage cross-cycle precise registration includes: The first stage is coarse registration based on RTK pose: the relative pose transformation matrix of the historical periodic image relative to the current periodic image is calculated based on the six-degree-of-freedom pose tuple recorded in each frame image. Combined with the camera intrinsic parameter matrix, the historical image is projected to the current view coordinate system through homography transformation, so that the remaining pixel deviation between the two images is controlled within 10 pixels. The second stage is fine registration based on the improved Super Glue algorithm: On the basis of the coarse registration result, local key points and their descriptors are extracted from the two images. The improved Super Glue graph neural network, which introduces the coarse registration homography matrix space prior in the attention mechanism, is used to establish the correspondence between key points across images. The fine homography matrix is ​​estimated by the random sampling consensus algorithm and the historical images are transformed to keep the remaining registration error within 1 to 2 pixels.

6. The system according to claim 1, characterized in that, The edges of the spatiotemporal evolution graph are divided into two categories: temporal edges and spatial edges. The temporal edge connects the node pairs that correspond to the same physical detection point in different inspection cycles, and assigns a sine-cosine time interval position encoding vector with the actual physical time interval as input to each temporal edge; The spatial edge connects neighboring detection point nodes that are geographically less than 0.5 meters apart within the same inspection cycle; The cross-period spatiotemporal graph neural network encoder consists of six stacked Transformer encoding layers. Each layer contains a multi-head spatiotemporal attention sublayer that simultaneously models spatial neighborhood attention and temporal self-attention. The two types of attention information are adaptively fused through a learnable attention head weight matrix. Each sublayer uses residual connections and follows the layer normalization operation.

7. The system according to claim 1, characterized in that, The specific implementation of the Weibull degradation curve parameter estimation is as follows: the final representation vector of the node is mapped to a health status score with a value range of 0 to 1 through a two-layer fully connected network, where 0 indicates that the device is completely healthy and 1 indicates that the device has reached the functional failure threshold. The historical multi-period health status score sequence is input into a lightweight LSTM module, and the shape and scale parameters of the Weibull distribution are output to construct the individualized degradation curve of the device. The expected time when the device will reach the fault threshold is calculated by using the inverse function of the Weibull cumulative distribution function. The difference between this expected time and the current time is the fault prediction time output value in days.

8. The system according to claim 1, characterized in that, The system performs a three-level risk fusion judgment based on defect detection results and fault prediction time: If the fault prediction time is greater than 30 days, it is judged as low risk, and normal recording is performed and the historical database is updated. When the predicted fault time is between 7 and 30 days, it is classified as medium risk, a defect report is generated, and an alarm is pushed to the operation and maintenance personnel. If the fault prediction time is less than 7 days, it is considered high risk, triggering an emergency warning and recommending immediate shutdown for maintenance.

9. The system according to claim 1, characterized in that, Before being deployed to the edge computing unit, the cross-domain small sample defect detection module based on feature decoupling and the cross-period spatiotemporal evolution graph neural network early warning module undergo the following lightweight processing: After sorting the kernels of all convolutional layers by L1 norm and pruning the 30% of kernels with the lowest importance, fine-tuning was performed. The TensorRT tool was used to perform INT8 precision quantization calibration, compressing the model weights and activation values ​​from 32-bit floating-point to 8-bit integer representation. The quantized model is compiled into a TensorRT-specific inference engine, achieving a stable inference frame rate of no less than 25 FPS on edge computing units.

10. The system according to claim 1, characterized in that, The edge computing unit of the unmanned inspection platform is responsible for all real-time inference tasks of the cross-domain small sample defect detection module based on feature decoupling and the cross-period spatiotemporal evolution graph neural network early warning module. The cloud-based continuous learning module receives all inspection data uploaded from the edge devices, performs full fine-tuning and updates to the model parameters of both modules, and distributes the updated model parameters to each edge device. Newly acquired high-confidence detection samples are simultaneously added to the prototype memory update queue to enhance the prototype memory's ability to characterize the defect distribution in the current deployment environment.

Images