Visual anomaly detection method and system in industrial field, terminal and storage medium

Abstract

The invention discloses a visual anomaly detection method and system in the industrial field, a terminal and a storage medium, and the method comprises the steps: obtaining a normal sample image, and carrying out the preprocessing of the normal sample image, and obtaining an enhanced depth image; performing background region elimination processing on the enhanced depth map to obtain a foreground image and a foreground depth map; performing feature extraction processing and fusion processing on the foreground image and the foreground depth map to obtain fusion features, and constructing a memory bank according to the fusion features; and obtaining a to-be-detected image, and performing nearest neighbor Euclidean distance calculation and image anomaly judgment according to the to-be-detected image and the memory bank to obtain an image anomaly detection result. According to the method, the background region of the image of the detection equipment is removed, the interference of background noise is avoided, then the foreground image and the foreground depth image are fused, and finally judgment is performed based on the fusion features, so that the accuracy of abnormal state detection of the equipment can be effectively improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a visual anomaly detection method, system, terminal, and computer-readable storage medium for industrial applications. Background Technology

[0002] Industrial visual anomaly detection is a key technology for ensuring the safe and efficient operation of modern manufacturing and large-scale infrastructure (such as wind power plants). Its goal is to automatically identify defects or anomalies on the surface of objects using machine vision. In the automated inspection of wind turbine generators, drones equipped with high-definition cameras typically photograph the inspection equipment, and algorithms detect defects such as cracks and wear. This is crucial for ensuring the safety of energy facilities and reducing operation and maintenance costs. Because anomaly samples are scarce and diverse in industrial settings, unsupervised learning (i.e., learning only from normal samples) has become the mainstream technological paradigm in this field.

[0003] However, existing unsupervised learning detection methods rely on the color and texture information of RGB images, but are easily affected by complex backgrounds in the images, resulting in a high false alarm rate for device anomaly detection. This fails to meet users' needs for accurate device status and may even affect the normal operation of the device.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a visual anomaly detection method, system, terminal, and computer-readable storage medium for the industrial field. It aims to solve the problem that existing unsupervised learning detection methods rely on the color and texture information of RGB images, but are easily interfered with by complex backgrounds in the images, resulting in a high false alarm rate in anomaly detection.

[0006] To achieve the above objectives, the present invention provides a visual anomaly detection method for the industrial field, the method comprising the following steps: Obtain normal sample images and preprocess them to obtain enhanced depth maps; The enhanced depth map is subjected to background region culling to obtain a foreground image and a foreground depth map; Feature extraction and fusion processing are performed on the foreground image and the foreground depth map to obtain fused features, and a memory bank is constructed based on the fused features; The image to be detected is acquired, and the nearest neighbor Euclidean distance is calculated and the image anomaly is determined based on the image to be detected and the memory bank to obtain the image anomaly detection result.

[0007] Optionally, the aforementioned visual anomaly detection method in the industrial field, wherein acquiring normal sample images and preprocessing the normal sample images to obtain enhanced depth maps specifically includes: Obtain normal sample images and input them into a pre-trained monocular depth estimation network to output a pseudo depth map; The pseudo-depth map is subjected to image-level quality enhancement processing to obtain an enhanced depth map; The image-level quality enhancement processing includes dark area adjustment and smoothing filtering.

[0008] Optionally, in the aforementioned visual anomaly detection method for the industrial field, the step of performing background region removal processing on the enhanced depth map to obtain a foreground image and a foreground depth map specifically includes: The enhanced depth map is input into the adaptive foreground segmentation module, which performs linear mapping and quantization processing on the enhanced depth map to obtain a grayscale image. The grayscale image is subjected to mask extraction and mask fusion processing to obtain the final mask; The final mask is subjected to morphological closing operation and maximum connected component analysis to obtain the target mask; The target mask is multiplied pixel by pixel to obtain the foreground image and the foreground depth map.

[0009] Optionally, in the aforementioned visual anomaly detection method for the industrial field, the step of performing mask extraction and mask fusion processing on the grayscale image to obtain the final mask specifically includes: Find the optimal global threshold in the grayscale image, and obtain the global mask based on the optimal global threshold; Obtain the pixel neighborhood of the grayscale image, find the optimal local threshold in the pixel neighborhood, and obtain the local mask based on the optimal local threshold; The global mask and the local mask are fused together to obtain the final mask.

[0010] Optionally, the aforementioned visual anomaly detection method in the industrial field, wherein the step of performing feature extraction and fusion processing on the foreground image and the foreground depth map to obtain fused features, and constructing a memory bank based on the fused features, specifically includes: The RGB backbone network and the deep backbone network are determined, and the foreground image is input into the RGB backbone network to obtain the first multimodal feature; The foreground depth map is input into the deep backbone network to obtain the second multimodal feature; Determine the enhanced attention fusion module, and input the first multimodal feature and the second multimodal feature into the enhanced attention fusion module; The enhanced attention fusion module performs importance weight allocation and aggregation processing on the first multimodal feature and the second multimodal feature to obtain fused features, and constructs a memory bank based on the fused features.

[0011] Optionally, the aforementioned visual anomaly detection method in the industrial field, wherein the step of assigning importance weights and aggregating the first multimodal features and the second multimodal features through the enhanced attention fusion module to obtain fused features specifically includes: The enhanced attention fusion module concatenates the first multimodal feature and the second multimodal feature along the channel dimension to obtain the concatenated feature. The spliced ​​features are subjected to lightweight MLP mapping to obtain importance weights; The first multimodal feature and the second multimodal feature are weighted according to the importance weight to obtain the first weighted feature and the second weighted feature; Asymmetric multi-head cross attention is used to perform information aggregation processing on the first weighted feature and the second weighted feature to obtain attention weights, and attention output is obtained based on the attention weights; The enhanced features are obtained by fusing and optimizing the first weighted features and the attention output. The enhanced features and the second weighted features are then spliced ​​and projected to obtain the fused features.

[0012] Optionally, the aforementioned visual anomaly detection method in the industrial field, wherein acquiring the image to be detected and performing nearest neighbor Euclidean distance calculation and image anomaly determination based on the image to be detected and the memory bank to obtain the image anomaly detection result specifically includes: Acquire the image to be detected, and acquire the fusion features to be detected for each small block in the image to be detected; Each of the fused features to be detected is compared with each normal feature in the memory bank using the nearest neighbor Euclidean distance to obtain multiple feature distances; Select the minimum value among multiple feature distances, and use the minimum value as the anomaly score corresponding to each small block; Calculate the maximum anomaly score corresponding to each small block, and use the maximum value as the final anomaly score; A preset anomaly threshold is determined. When the final anomaly score is greater than the preset anomaly threshold, the image to be detected is determined to be abnormal, and the image anomaly detection result is output.

[0013] Furthermore, to achieve the above objectives, the present invention also provides a visual anomaly detection system for the industrial field, wherein the visual anomaly detection system for the industrial field includes: An image preprocessing module is used to acquire normal sample images and preprocess the normal sample images to obtain enhanced depth maps; The background region removal module is used to perform background region removal processing on the enhanced depth map to obtain a foreground image and a foreground depth map; The memory bank construction module is used to perform feature extraction and fusion processing on the foreground image and the foreground depth map to obtain fused features, and to construct a memory bank based on the fused features; The image anomaly detection module is used to acquire the image to be detected, and to perform nearest neighbor Euclidean distance calculation and image anomaly detection based on the image to be detected and the memory bank to obtain the image anomaly detection result.

[0014] In this invention, a normal sample image is acquired and preprocessed to obtain an enhanced depth map. The enhanced depth map undergoes background region removal to obtain a foreground image and a foreground depth map. Feature extraction and fusion processing are performed on the foreground image and the foreground depth map to obtain fused features, and a memory is constructed based on these fused features. An image to be detected is acquired, and nearest neighbor Euclidean distance calculation and image anomaly determination are performed based on the image to be detected and the memory to obtain an image anomaly detection result. This invention effectively improves the accuracy of detecting abnormal device states by removing background regions from the image of the detection device to avoid interference from background noise, fusing the foreground image and the foreground depth map, and finally making a determination based on the fused features. Simultaneously, it enhances robustness to changes in illumination and viewing angle. Attached Figure Description

[0015] Figure 1 This is the detection result corresponding to wind turbine images against complex backgrounds in existing technologies; Figure 2 This is a flowchart of a preferred embodiment of the visual anomaly detection method in the industrial field of the present invention; Figure 3 This is a schematic diagram of the overall structure and implementation process of a preferred embodiment of the visual anomaly detection method in the industrial field of the present invention; Figure 4 This is a schematic diagram of comparative experimental results of a preferred embodiment of the visual anomaly detection method in the industrial field of the present invention; Figure 5 This is a qualitative analysis schematic diagram of a preferred embodiment of the visual anomaly detection method in the industrial field of the present invention; Figure 6 This is a schematic diagram of the ablation experiment results of a preferred embodiment of the visual anomaly detection method in the industrial field of the present invention; Figure 7 This is a structural diagram of a preferred embodiment of the visual anomaly detection system for the industrial field of the present invention; Figure 8 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0017] Industrial Visual Anomaly Detection (IVAD) is a key technology for ensuring the safe and efficient operation of modern manufacturing and large-scale infrastructure (such as wind power plants). Its goal is to automatically identify defects or anomalies on the surface of objects using machine vision. In the automated inspection of wind turbine generators, drones equipped with high-definition cameras photograph the blades and algorithms detect defects such as cracks and wear, which is crucial for ensuring the safety of energy facilities and reducing operation and maintenance costs. Because anomalous samples are scarce and diverse in industrial settings, unsupervised learning (i.e., learning only from normal samples) has become the mainstream technological paradigm in this field.

[0018] Currently, mainstream unsupervised visual anomaly detection methods are primarily based on deep learning, such as the "memory bank" approach represented by PatchCore and PaDiM. This approach extracts features from pre-trained deep neural networks (such as WideResNet) on normal samples and constructs a "normal feature" database. During inference, anomalies are determined by calculating the distance between the test sample features and this database. In addition, there are various other technical approaches based on flow models (such as MSFlow and CFlow) or knowledge distillation (such as RD4AD).

[0019] In summary, the shortcomings of existing technologies include: 1. Susceptible to complex background interference, leading to a high false alarm rate: Existing methods rely almost entirely on the color and texture information of RGB images, such as... Figure 1 As shown, in scenarios such as drone inspection of wind turbine blades, the blade body only occupies a small part of the image. Features of complex backgrounds such as large areas of sky, mountains and ground can "contaminate" the detection model, causing the detection model to incorrectly identify normal background textures (such as ground vegetation and cloud edges) as anomalies, resulting in an extremely high false alarm rate.

[0020] 2. Insufficient robustness: Relying solely on RGB information makes the detection model highly sensitive to changes in lighting, shadows, reflections, and shooting angle. When these environmental factors cause changes in the appearance of the leaf, the detection model may misjudge it as an anomaly because it lacks an understanding of the three-dimensional geometry of the leaf itself.

[0021] To address the aforementioned issues, this invention proposes a depth-guided multimodal industrial anomaly detection method. The basic idea is to introduce a low-cost monocular depth estimation technique as a supplementary modality to RGB vision, achieving a two-stage processing framework of "segmentation first, fusion later." First, depth information is used to accurately segment the foreground (wind turbine blades) of the image, eliminating most background noise interference from the input. Then, at the feature level, an enhanced attention fusion module is designed to dynamically and efficiently fuse the texture information of the foreground RGB image with the geometric structure information of the foreground depth map, generating a feature representation that is more robust to environmental changes and has stronger discriminative capabilities. Finally, anomaly detection is performed based on this feature representation.

[0022] This invention can significantly reduce the false alarm rate of anomaly detection in complex backgrounds and greatly improve the detection precision and overall discriminative power (ROC-AUC), while enhancing the robustness of the detection model to changes in illumination and viewing angle.

[0023] The preferred embodiment of the visual anomaly detection method in the industrial field described in this invention, such as... Figure 2 As shown, the visual anomaly detection method in the industrial field includes the following steps: Step S10: Obtain a normal sample image and preprocess the normal sample image to obtain an enhanced depth map.

[0024] Normal sample images refer to images in the normal state of the detection device. This invention uses normal sample images to train the model and build a memory library.

[0025] Preprocessing includes the generation of pseudo-depth maps and image-level quality enhancement.

[0026] As shown in Figure 3, to address the problem of high false detection rates caused by complex background interference in traditional unsupervised methods when detecting anomalies in wind turbine images, this invention proposes a depth-guided masked patchcore framework. For any image (i.e., the normal sample image in this invention)... (in, Represents the set of real numbers. Represents the height of the image. (3 represents the width of the image, and 3 represents the number of channels in the image). First, it is passed through a pre-trained monocular depth estimation network. Generate its corresponding pseudo-depth map .

[0027] Specifically, a normal sample image is acquired and input into a pre-trained monocular depth estimation network to output a pseudo-depth map; the pseudo-depth map is then subjected to image-level quality enhancement processing to obtain an enhanced depth map; wherein, the image-level quality enhancement processing includes dark area adjustment and smoothing filtering processing.

[0028] The process of depth information generation and enhancement is as follows: Taking the anomaly detection of wind turbine blades as an example, in the actual anomaly detection of wind turbine blades, it is both expensive and inconvenient to acquire precisely aligned depth sensors and synchronize them to a large-scale UAV inspection line. Therefore, this invention uses monocular depth estimation to obtain pseudo-depth as a geometric prior at low cost. DepthAnything-v2 (i.e., the pre-trained monocular depth estimation network in this invention) does not require any additional training or fine-tuning on the wind turbine dataset and can directly generate high-precision pseudo-depth maps with good structural information for outdoor blade images captured by UAVs.

[0029] The specific process of generating a pseudo-depth map is as follows: Given an image Using a pre-trained DepthAnything-v2 network Dense depth prediction is generated, resulting in a pseudo-depth map. : , .

[0030] Nevertheless, depth estimation may still contain problems such as scale error, noise, and insufficient local contrast. Therefore, this invention addresses the original depth (i.e., pseudo-depth map). Several image-level quality enhancement operations were performed. First, gamma correction was applied to adjust the contrast between mid-gray and dark areas; when γ < 1 (…),… This is an adjustable parameter (e.g., γ=0.5). The function curve will bulge upwards, meaning that dark area pixels with lower input values ​​(such as 0.1, 0.2) will be mapped to a higher, wider output range. Subsequently, an edge-preserving smoothing filter is applied, denoted as... The processing involves considering two weights simultaneously when calculating the new value for each pixel. In flat areas (such as the surface of a leaf), the brightness values ​​of neighboring pixels are very similar, resulting in high weights for both. The filter performs strong smoothing, effectively removing noise. In edge areas (such as the boundary between the leaf and the sky), the brightness difference between pixels on the sky side and the center pixel on the leaf side is extremely large. Therefore, even if the spatial distance is very close, the "pixel value similarity weight" becomes extremely low, almost zero. This allows the filter to almost completely ignore pixels on the other side of the edge when calculating the new value, thus avoiding "blurring" the sky color onto the leaf and perfectly protecting the edge sharpness. This suppresses noise while preserving geometric boundaries, resulting in an enhanced depth map (i.e., an enhanced depth map). The above processing can both enhance the separability of the blades and the background in the depth domain and reduce the risk of missegmentation caused by estimation noise, providing a more robust input for the next step of mask generation.

[0031] Step S20: Perform background region removal processing on the enhanced depth map to obtain a foreground image and a foreground depth map.

[0032] This invention designs an adaptive foreground segmentation module (DTF, Depth-based Threshold Fusion), which uses this depth information to obtain a wind turbine foreground image and a foreground depth map after removing the background region. The process expression is as follows: The above processing aims to reduce the interference of irrelevant features on subsequent modeling from the source.

[0033] Specifically, the enhanced depth map is input to the adaptive foreground segmentation module, which performs linear mapping and quantization on the enhanced depth map to obtain a grayscale image. The optimal global threshold in the grayscale image is found, and a global mask is obtained based on the optimal global threshold. The pixel neighborhood of the grayscale image is obtained, and the optimal local threshold in the pixel neighborhood is found, and a local mask is obtained based on the optimal local threshold. The global mask and the local mask are fused to obtain a final mask. Morphological closing operations and maximum connected component analysis are performed on the final mask to obtain a target mask. The target mask is multiplied pixel-by-pixel to obtain the foreground image and the foreground depth map.

[0034] Understandably, this invention designs an unsupervised foreground segmentation strategy (DTF) that integrates global and local statistics in the depth domain to achieve stable extraction of the leaf body under different lighting and local noise conditions. Firstly, the depth-enhanced... (i.e., enhancing the depth map) linearly mapped and quantized into a grayscale image. The linear mapping process involves finding the minimum and maximum values ​​in the enhanced depth map and proportionally "stretching" or "compressing" all pixel values ​​to the standard [0, 1] range. Quantization converts continuous floating-point values ​​within the [0, 1] range into integer values ​​within the [0, 255] range. Specifically, each pixel value after linear mapping is multiplied by 255, rounded, and then the threshold is estimated and fused from both global and local neighborhood perspectives.

[0035] From a global perspective, the Otsu method is employed (the Otsu algorithm is an adaptive thresholding method, also known as the Otsu thresholding method, which is the optimal segmentation in the least squares sense; it divides the image into background and foreground based on the image's grayscale characteristics). This is achieved by traversing all possible grayscale thresholds. To find an optimal threshold that maximizes the variance between foreground and background pixel classes. Its mathematical expression is:

[0036] in, These are the current grayscale thresholds. The pixel ratios of the following two categories, For the corresponding within-class mean, further, based on The global mask can then be obtained. .

[0037] A local perspective is within the pixel neighborhood. Calculate the mean and subtract the bias term. To obtain the local threshold Its expression is:

[0038] This forms a local mask. (by local threshold) (i, v) is obtained, where (u, v) represents the coordinates of a pixel. The two coordinates are then fused to obtain the final mask. .

[0039] Then the final mask Morphological closing and maximum connected component analysis are applied. The morphological closing operation, specifically designed to fill small black holes inside foreground objects, consists of two basic operations: dilation followed by erosion. Dilation expands the boundary of the foreground region outwards, thus "swallowing" and filling the internal holes. The subsequent erosion shrinks the boundary of the foreground region back to its original size, but the filled holes do not reappear. After closing, isolated black spots or holes caused by noise inside the leaf mask are automatically filled in, making the foreground region more complete.

[0040] Maximum connected component analysis is used to remove isolated small "white dots" (noise) outside the foreground. The maximum connected component analysis algorithm scans the entire mask image, finds all the white pixel blocks that are connected together (called "connected components"), and calculates the area of ​​each block. Then, it only keeps the connected component with the largest area (which is the main body of the leaf required by this invention), while deleting all other small and isolated white noise blocks (setting them to black).

[0041] After this step, all small areas separated from the main leaf body and those misjudged as foreground by background noise in the mask will be completely removed, and finally, pixel-by-pixel multiplication (using...) will be performed. (This indicates that) the target mask is applied to the input by comparing the target mask Mi with the original RGB image (normal sample image) x. i Each color channel (R, G, B channels) is multiplied pixel by pixel. For pixels in the foreground region, the mask value is 1, so the original pixel value * 1 = the original pixel value, and the color of this part of the image is fully preserved. For pixels in the background region, the mask value is 0, so the original pixel value * 0 = 0 (black), and the color of this part of the image is directly set to black, resulting in the foreground image. and foreground depth map This fusion strategy ensures the stability and adaptability of the segmentation, reducing background contamination at the source.

[0042] Step S30: Perform feature extraction and fusion processing on the foreground image and the foreground depth map to obtain fused features, and construct a memory bank based on the fused features.

[0043] This invention sets up two parallel pre-trained encoders, respectively from... and Extracting patch-level multimodal features To achieve effective information fusion, this invention further proposes an Enhanced Attention Fusion Module (EAFM). The EAFM first dynamically assigns importance weights to the RGB and depth modalities of each image patch to highlight more reliable information sources. Subsequently, it utilizes a cross-modal attention mechanism to aggregate complementary texture and geometric cues, ultimately generating a more robust fused feature representation (i.e., the final mask in this invention). Finally, this fusion feature was used to build the PatchCore's memory.

[0044] Specifically, an RGB backbone network and a deep backbone network are determined. The foreground image is input into the RGB backbone network to obtain a first multimodal feature. The foreground depth map is input into the deep backbone network to obtain a second multimodal feature. An enhanced attention fusion module is determined, and the first and second multimodal features are input into the enhanced attention fusion module. The enhanced attention fusion module concatenates the first and second multimodal features along the channel dimension to obtain a concatenated feature. The concatenated feature is then subjected to lightweight MLP mapping processing to obtain importance weights.

[0045] The specific process for enhancing attention fusion is as follows: After obtaining the input with mask constraints... and Then, they are transmitted through independent RGB backbone networks. With deep backbone network After a series of convolutional and pooling layers, its patch-level representation, i.e., the first multimodal feature, is extracted. Second multimodal features ,in, Where B represents the batch size, L represents the sequence length, and D represents the feature dimension. Considering the varying importance of RGB and depth information at different locations, this invention designs an enhanced attention fusion module. This module extracts complementary information through learnable weight allocation and asymmetric cross-modal attention. Specifically, the enhanced attention fusion module first concatenates the two modalities along the channel dimension and obtains importance weights through a lightweight MLP mapping:

[0046] ; in, This is an MLP mapping.

[0047] The first multimodal feature and the second multimodal feature are weighted according to the importance weights to obtain a first weighted feature and a second weighted feature; asymmetric multi-head cross attention is used to perform information aggregation processing on the first weighted feature and the second weighted feature to obtain attention weights, and attention output is obtained according to the attention weights; fusion optimization processing is performed on the first weighted feature and the attention output to obtain enhanced features; the enhanced features and the second weighted feature are concatenated and projected to obtain fused features, and a memory bank is constructed according to the fused features.

[0048] Furthermore, this invention utilizes importance weights to weight the original features (i.e., the first multimodal feature and the second multimodal feature) element by element to obtain the first weighted feature. With the second weighted feature To suppress unreliable modes and enhance effective modes, asymmetric multi-head cross-attention is then used for information aggregation to obtain attention weights. : ;

[0049] in, It is a linear projection. Represents the attention weight matrix. Representing the attention output, h is the dimension of the attention head. This design allows RGB features to actively "query" geometric cues in the depth. For stable fusion, this invention uses residual connections and layer normalization to obtain enhanced features. :

[0050] Then the features will be enhanced. With the second weighted feature The data is then stitched together and projected to obtain the final fused representation (i.e., fused features). The expression is: .

[0051] Step S40: Obtain the image to be detected, and perform nearest neighbor Euclidean distance calculation and image anomaly determination based on the image to be detected and the memory bank to obtain the image anomaly detection result.

[0052] After the model is trained and the memory is built, the image to be detected can be input into the trained model to obtain the fusion feature corresponding to the image to be detected. The nearest neighbor Euclidean distance is then calculated between this fusion feature and the normal features in the memory, and the image anomaly is determined. Finally, the image anomaly detection result of the image to be detected is obtained.

[0053] Specifically, the process involves acquiring an image to be detected and obtaining the fusion features to be detected for each small block in the image; calculating the nearest neighbor Euclidean distance between each fusion feature to be detected and each normal feature in the memory bank to obtain multiple feature distances; selecting the minimum value among the multiple feature distances and using the minimum value as the anomaly score corresponding to each small block; calculating the maximum value of the anomaly score corresponding to each small block and using the maximum value as the final anomaly score; determining a preset anomaly threshold; and when the final anomaly score is greater than the preset anomaly threshold, determining that the image to be detected is abnormal and outputting the image anomaly detection result.

[0054] This invention first constructs a "normal pattern" memory M using only the fusion features of normal (defect-free) samples during the training phase. During detection, this invention extracts the fusion features of each patch in the image to be detected and calculates the Euclidean distance between this feature (i.e., the fusion feature to be detected) and all "normal" features in memory M. The shortest Euclidean distance is defined as the anomaly score of this patch, quantifying the degree to which the region deviates from the normal pattern. The final anomaly score for the entire image is then calculated. Then, the maximum value among all the small block scores is taken. In practical applications, the threshold for judging anomaly scores is to select the score point on the validation set that maximizes the F1 score. When the anomaly score of an image to be detected is higher than this threshold, it is judged as an anomaly.

[0055] It is understandable that this invention uses a two-stage process based on "normal pattern learning" to ultimately determine whether there are anomalies in the wind turbine blades. This process is the core of unsupervised anomaly detection, and the training phase only uses fused patch representations of normal samples. Building a memory bank During the inference phase, the Euclidean distance to the nearest neighbor is calculated for each test patch. The expression is:

[0056] Image-level anomaly scores are obtained using max pooling. This fusion module works in concert with dynamic weight allocation, cross-modal attention, and residual enhancement to improve the discriminativeness of the fused representation while maintaining compatibility with the PatchCore memory structure.

[0057] The specific process of experimental verification is as follows: 1. Experimental setup: For the datasets: This invention validated the effectiveness of the method on two datasets: 1) Blade0: This is a self-built UAV wind farm inspection dataset. After preprocessing and image enhancement, the dataset contains 2600 high-resolution (5184*3888) images, covering 7 anomaly types in real-world scenarios, including cracks and wear. This invention divides it into 2400 normal images for training and 200 images (100 normal and 100 abnormal) for evaluation.

[0058] 2) DTU: A publicly available drone inspection dataset for wind turbine blades, containing multiple types of defects. Following the same data processing flow, it constructs a training set of 1160 images and an evaluation set of 140 images (70 normal and 70 abnormal).

[0059] For evaluation metrics: This invention uses a variety of standard metrics for comprehensive evaluation, including: ROC-AUC (overall discrimination capability), F1-score (precision and recall balance), Recall (anomaly coverage), Accuracy (overall accuracy), and Precision (false positive control capability).

[0060] For experimental details: The experiment was conducted on a device equipped with an NVIDIA GeForce RTX 4060 Ti graphics card. The model configuration of this invention is as follows: The RGB branch uses WideResNet-50 as the backbone network, extracting features from its second and third layers; the depth branch uses a single-channel EfficientNet-B3, whose first-layer convolutional weights are initialized with the mean of pre-trained three-channel weights to transfer knowledge, and features from its third and fifth layers are extracted. For feature storage, this invention uses approx_greedy_coreset as the sampler, with a sampling ratio set to 0.1, and the anomaly scorer uses one nearest neighbor.

[0061] 2. Comparative Experiments: To verify the effectiveness of this invention, it was further compared with other state-of-the-art algorithms, including PaDiM and PatchCore based on memory banks, CFlow and MSFlow based on flow models, RD4AD based on knowledge distillation, and the lightweight SimpleNet. All methods were fairly evaluated on the same Blade0 and DTU datasets.

[0062] Experimental results are as follows Figure 4 As shown: For the Blade0 dataset: This invention significantly outperforms other methods in key comprehensive metrics, with ROC-AUC (0.929) and F1-score (0.888) far exceeding all comparison methods. This demonstrates the significant advantage of this invention's model in discriminative ability and precision-recall balance. It is noteworthy that while RD4AD achieves the highest recall, its precision is only 0.631, meaning it trades high recall for a large number of background-driven false positives. In contrast, this invention achieves the highest precision while maintaining high recall, achieving a superior balance.

[0063] For the DTU dataset: DG-PatchCore's leading position is even more solid, ranking first or tied for first place in all five metrics, especially in ROC-AUC (0.907), F1-score (0.919), and Precision (0.872), where its advantage is particularly significant. Although CFlow and MSFlow also achieved the highest recall rate (0.971) comparable to this invention, their precision is extremely low (0.581 and 0.619, respectively). This clearly shows that the competing methods of CFlow and MSFlow sacrifice precision for high recall, while this invention, with its foreground perception capability, successfully avoids misclassifying background as anomalies while accurately identifying all real anomalies, achieving the best performance balance.

[0064] 3. Qualitative Analysis: For an inspection image containing a complex background, such as... Figure 5 As shown ( Figure 5 In this context, 'a' represents the input image containing a complex background. Figure 5 In this context, 'b' represents the anomaly heatmap generated by the PatchCore method, which is the benchmark in the prior art. Figure 5 In this paper, 'c' represents the anomaly heatmap generated by this invention. The benchmark PatchCore method generates a large number of false positives, and its anomaly score is mainly driven by the background, failing to focus on the blade itself. In stark contrast, the DG-PatchCore of this invention, thanks to depth-guided foreground segmentation, almost completely ignores the interference of the background region, generating a clean and accurate anomaly heatmap that accurately focuses on the surface of the blade to be detected.

[0065] 4. Ablation Experiments: To verify the effectiveness of foreground extraction and depth information introduction in the proposed method, multiple ablation experiments were conducted on the Blade0 and DTU datasets. Specific results are as follows: Figure 6 As shown.

[0066] For the Blade0 dataset: the simple Adaptive method offers limited improvement, Otsu's global thresholding method effectively suppresses some background interference, and the fusion method (DTF) further improves detection performance. After introducing deep modality (DTF+EAFM), the ROC, F1, and Accuracy metrics all reach their highest levels, indicating that the proposed method can effectively reduce background interference and improve the discriminative power of anomaly detection. Furthermore, improvements in related self-supervised and representation methods also have a positive effect on improving intra-class compactness.

[0067] For the DTU dataset: the overall trend is consistent with the Blade0 dataset, especially in terms of the Recall metric, DTF+EAFM reaches 0.971, which is significantly higher than other methods, indicating that this method has stronger coverage and robustness in detecting anomalies.

[0068] In summary, the experimental results verify the effectiveness and versatility of the foreground extraction algorithm and depth information introduction strategy in this invention. It not only performs well on self-built UAV-collected datasets, but also achieves significant advantages on public datasets.

[0069] 5. Development of an intelligent inspection and maintenance system for wind turbine blades: This part of the work transformed the technical model into practical applications, embedding the improved anomaly detection algorithm into the complete system, forming a closed-loop structure from front-end data acquisition, back-end processing, result display to feedback. The system has high deployment efficiency, is easily expandable, and possesses industrial-grade usability and practical value, providing strong technical support for intelligent inspection in the wind power field. This invention has been applied in the wind turbine blade inspection and lightning protection testing project of seven wind farms belonging to a certain company, completing the inspection and maintenance of 265 wind power facilities and generating professional inspection reports, effectively reducing the cost of wind farm inspection and improving work efficiency and safety.

[0070] The key innovations of this invention include: Key innovation 1: The invention proposes to apply monocular depth estimation technology to unsupervised anomaly detection tasks with complex outdoor backgrounds, such as wind turbine blades, and designs a complete "depth guidance-foreground segmentation-multimodal fusion" technical process, which fundamentally solves the pain point of existing methods being easily affected by background interference.

[0071] Key innovation 2 (DTF module included): A foreground segmentation method specifically designed for depth maps, incorporating both global (Otsu) and local (adaptive) thresholding, was implemented. Compared to single-threshold methods, it exhibits stronger robustness to illumination variations and noise, provides more accurate segmentation, and effectively avoids background contamination.

[0072] Key innovation point 3 (set up EAFM module): An enhanced attention fusion module for anomaly detection. Its innovation lies in the combination of dynamic weight allocation and asymmetric cross attention. The former adaptively balances the contributions of different modalities at different spatial locations, while the latter allows texture information to actively query geometric information, achieving efficient feature complementarity.

[0073] Furthermore, variations of the present invention include the following: 1. Variations in the source of depth information: This invention uses monocular depth estimation, but in some scenarios, it can be replaced by real depth data directly obtained from binocular cameras, structured light, or lidar (LiDAR).

[0074] 2. Variations of the segmentation model: The DTF module of this invention is unsupervised, but it can also be replaced with a lightweight, supervised semantic segmentation model pre-trained on relevant datasets (such as lightweight U-Net), as long as it can effectively segment the foreground.

[0075] 3. Encoder variants: RGB and depth feature extractors (such as WideResNet, EfficientNet) can be replaced with any other effective deep learning backbone network, such as Vision Transformer (ViT) or other CNN variants.

[0076] 4. Variations of the fusion mechanism: The fusion strategy in the EAFM module can have a variety of variations, such as using bidirectional cross attention, decoupling dynamic weights from the attention mechanism, or replacing it with simpler methods such as splicing or addition (but the effect may be reduced).

[0077] Furthermore, such as Figure 7 As shown, based on the above-mentioned visual anomaly detection method in the industrial field, the present invention also provides a visual anomaly detection system in the industrial field, wherein the visual anomaly detection system in the industrial field includes: Image preprocessing module 51 is used to acquire normal sample images and preprocess the normal sample images to obtain enhanced depth maps; Background region removal module 52 is used to perform background region removal processing on the enhanced depth map to obtain a foreground image and a foreground depth map; The memory bank construction module 53 is used to perform feature extraction and fusion processing on the foreground image and the foreground depth map to obtain fused features, and to construct a memory bank based on the fused features; The image anomaly detection module 54 is used to acquire the image to be detected, and to perform nearest neighbor Euclidean distance calculation and image anomaly detection based on the image to be detected and the memory bank to obtain the image anomaly detection result.

[0078] Furthermore, such as Figure 8 As shown, based on the above-mentioned visual anomaly detection method and system in the industrial field, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 8 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0079] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores an industrial visual anomaly detection program 40, which can be executed by the processor 10 to implement the industrial visual anomaly detection method of this application.

[0080] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the visual anomaly detection method in the industrial field.

[0081] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface.

[0082] In one embodiment, the steps of the industrial field visual anomaly detection method are implemented when the processor 10 executes the industrial field visual anomaly detection program 40 in the memory 20.

[0083] In summary, this invention provides a visual anomaly detection method, system, and terminal for the industrial field. The method includes: acquiring a normal sample image and preprocessing the normal sample image to obtain an enhanced depth map; performing background region removal processing on the enhanced depth map to obtain a foreground image and a foreground depth map; performing feature extraction and fusion processing on the foreground image and the foreground depth map to obtain fused features, and constructing a memory bank based on the fused features; acquiring an image to be detected, and performing nearest neighbor Euclidean distance calculation and image anomaly determination based on the image to be detected and the memory bank to obtain an image anomaly detection result. This invention effectively improves the accuracy of equipment anomaly detection by removing background regions from the image of the detection device to avoid interference from background noise, fusing the foreground image and the foreground depth map, and finally determining the anomaly based on the fused features. Simultaneously, it enhances robustness to changes in illumination and viewing angle.

[0084] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0085] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0086] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A visual anomaly detection method for industrial applications, characterized in that, The visual anomaly detection method in the industrial field includes: Obtain normal sample images and preprocess them to obtain enhanced depth maps; The enhanced depth map is subjected to background region culling to obtain a foreground image and a foreground depth map; Feature extraction and fusion processing are performed on the foreground image and the foreground depth map to obtain fused features, and a memory bank is constructed based on the fused features; The image to be detected is acquired, and the nearest neighbor Euclidean distance is calculated and the image anomaly is determined based on the image to be detected and the memory bank to obtain the image anomaly detection result.

2. The visual anomaly detection method in the industrial field according to claim 1, characterized in that, The process of acquiring normal sample images and preprocessing them to obtain enhanced depth maps specifically includes: Obtain normal sample images and input them into a pre-trained monocular depth estimation network to output a pseudo depth map; The pseudo-depth map is subjected to image-level quality enhancement processing to obtain an enhanced depth map; The image-level quality enhancement processing includes dark area adjustment and smoothing filtering.

3. The visual anomaly detection method in the industrial field according to claim 1, characterized in that, The step of performing background region removal processing on the enhanced depth map to obtain a foreground image and a foreground depth map specifically includes: The enhanced depth map is input into the adaptive foreground segmentation module, which performs linear mapping and quantization processing on the enhanced depth map to obtain a grayscale image. The grayscale image is subjected to mask extraction and mask fusion processing to obtain the final mask; The final mask is subjected to morphological closing operation and maximum connected component analysis to obtain the target mask; The target mask is multiplied pixel by pixel to obtain the foreground image and the foreground depth map.

4. The visual anomaly detection method in the industrial field according to claim 3, characterized in that, The process of extracting and fusing the mask from the grayscale image to obtain the final mask specifically includes: Find the optimal global threshold in the grayscale image, and obtain the global mask based on the optimal global threshold; Obtain the pixel neighborhood of the grayscale image, find the optimal local threshold in the pixel neighborhood, and obtain the local mask based on the optimal local threshold; The global mask and the local mask are fused together to obtain the final mask.

5. The visual anomaly detection method in the industrial field according to claim 1, characterized in that, The process of extracting and fusing features from the foreground image and the foreground depth map to obtain fused features, and constructing a memory bank based on the fused features, specifically includes: The RGB backbone network and the deep backbone network are determined, and the foreground image is input into the RGB backbone network to obtain the first multimodal feature; The foreground depth map is input into the deep backbone network to obtain the second multimodal feature; Determine the enhanced attention fusion module, and input the first multimodal feature and the second multimodal feature into the enhanced attention fusion module; The enhanced attention fusion module performs importance weight allocation and aggregation processing on the first multimodal feature and the second multimodal feature to obtain fused features, and constructs a memory bank based on the fused features.

6. The visual anomaly detection method in the industrial field according to claim 5, characterized in that, The enhanced attention fusion module performs importance weight allocation and aggregation on the first multimodal features and the second multimodal features to obtain fused features, specifically including: The enhanced attention fusion module concatenates the first multimodal feature and the second multimodal feature along the channel dimension to obtain the concatenated feature. The spliced ​​features are subjected to lightweight MLP mapping to obtain importance weights; The first multimodal feature and the second multimodal feature are weighted according to the importance weight to obtain the first weighted feature and the second weighted feature; Asymmetric multi-head cross attention is used to perform information aggregation processing on the first weighted feature and the second weighted feature to obtain attention weights, and attention output is obtained based on the attention weights; The enhanced features are obtained by fusing and optimizing the first weighted features and the attention output. The enhanced features and the second weighted features are then spliced ​​and projected to obtain the fused features.

7. The visual anomaly detection method in the industrial field according to claim 1, characterized in that, The process of acquiring the image to be detected, and calculating the nearest neighbor Euclidean distance and determining image anomalies based on the image to be detected and the memory bank to obtain the image anomaly detection result, specifically includes: Acquire the image to be detected, and acquire the fusion features to be detected for each small block in the image to be detected; Each of the fused features to be detected is compared with each normal feature in the memory bank using the nearest neighbor Euclidean distance to obtain multiple feature distances; Select the minimum value among multiple feature distances, and use the minimum value as the anomaly score corresponding to each small block; Calculate the maximum anomaly score corresponding to each small block, and use the maximum value as the final anomaly score; A preset anomaly threshold is determined. When the final anomaly score is greater than the preset anomaly threshold, the image to be detected is determined to be abnormal, and the image anomaly detection result is output.

8. A visual anomaly detection system for industrial applications, characterized in that, The visual anomaly detection system for the industrial field includes: An image preprocessing module is used to acquire normal sample images and preprocess the normal sample images to obtain enhanced depth maps; The background region removal module is used to perform background region removal processing on the enhanced depth map to obtain a foreground image and a foreground depth map; The memory bank construction module is used to perform feature extraction and fusion processing on the foreground image and the foreground depth map to obtain fused features, and to construct a memory bank based on the fused features; The image anomaly detection module is used to acquire the image to be detected, and to perform nearest neighbor Euclidean distance calculation and image anomaly detection based on the image to be detected and the memory bank to obtain the image anomaly detection result.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and an industrial visual anomaly detection program stored in the memory and executable on the processor. When executed by the processor, the industrial visual anomaly detection program implements the steps of the industrial visual anomaly detection method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an industrial visual anomaly detection program, which, when executed by a processor, implements the steps of the industrial visual anomaly detection method as described in any one of claims 1-7.

Images