A multi-modal industrial anomaly detection and segmentation method and system based on polynomial aggregation

Abstract

The application provides a kind of based on polynomial aggregation multimodal industrial anomaly detection and segmentation method and system, method includes: the two-dimensional image and three-dimensional point cloud of the object to be measured are collected and preprocessed;Based on the two-dimensional image and three-dimensional point cloud after preprocessing, using visual encoder and point cloud encoder, two-dimensional texture features and three-dimensional structure features are extracted respectively;Based on the two-dimensional texture features and the three-dimensional structure features, combined with cross-modal learning network, cross-modal anomaly map is obtained;Based on the two-dimensional texture features and the three-dimensional structure features, combined with single-modal learning network, single-modal anomaly map is obtained;The cross-modal anomaly map and the single-modal anomaly map are weighted and fused through multi-step aggregation strategy, and the final anomaly detection result map is generated.The application solves the problem of insufficient structure defect detection of traditional methods, significantly reduces the micro-defect miss rate, and shows stable detection performance in different material and different defect type scenes.

Description

Technical Field

[0001] This invention belongs to the field of industrial quality inspection and computer vision technology, specifically relating to a multimodal industrial anomaly detection and segmentation method and system based on polynomial aggregation. Background Technology

[0002] Industrial anomaly detection is a core task of quality assurance in manufacturing, aiming to identify structural or surface defects in products without human intervention. Due to the scarcity of defect samples in real-world scenarios, unsupervised learning has become the mainstream paradigm. Models learn the data distribution solely from normal samples, thus significantly deviating from identifying anomalous samples.

[0003] Early detection methods relied on single two-dimensional image data, which struggled to capture the three-dimensional structural information of complex products. Therefore, the industry gradually shifted towards multimodal anomaly detection, achieving more comprehensive defect identification by fusing two-dimensional image data with three-dimensional point cloud data. Current mainstream methods follow a paradigm of "feature extraction - cross-modal mapping - defect localization," but key unresolved issues remain: First, three-dimensional structural defect detection is insufficient. Cross-modal mapping manifests as a two-dimensional low-pass filter and a three-dimensional full-pass filter in the frequency domain. While effectively capturing two-dimensional texture defects, it cannot filter out three-dimensional high-frequency noise, resulting in an unsmooth three-dimensional representation and difficulty in thoroughly detecting structural defects. Second, anomaly map aggregation suffers from a "dilution effect." Existing methods often employ linear aggregation strategies such as summation and averaging, which can exacerbate the risk of missing small defects. Third, current approaches lack lightweight design; most models require significant GPU memory and computational resources.

[0004] Therefore, existing multimodal industrial anomaly detection methods have problems in 3D defect capture, micro-defect identification, and deployment efficiency. How to propose a technical solution that can accurately detect micro-defects in images and point clouds, requires fewer samples, and is lightweight has become an urgent technical problem to be solved in the field of industrial anomaly detection. Summary of the Invention

[0005] To address the problems of insufficient point cloud structural defect detection, missed detection of minute defects, high resource consumption, and difficulty in obtaining training samples in existing multimodal industrial anomaly detection methods, this invention provides a multimodal industrial anomaly detection and segmentation method and system based on polynomial aggregation. This method can simultaneously and accurately detect image texture defects and point cloud structural defects, enhance the detection capability of minute defects, and take into account the characteristics of lightweight and few samples, thus adapting to the current situation of scarce defect samples in industrial scenarios.

[0006] To achieve the above objectives, the present invention provides the following solution: A multimodal industrial anomaly detection and segmentation method based on polynomial aggregation, the method comprising: Acquire two-dimensional images and three-dimensional point clouds of the object under test and perform preprocessing; Based on the preprocessed 2D image and 3D point cloud, a visual encoder and a point cloud encoder are used to extract 2D texture features and 3D structural features, respectively. Based on the two-dimensional texture features and the three-dimensional structural features, and combined with a cross-modal learning network, a cross-modal anomaly map is obtained; Based on the two-dimensional texture features and the three-dimensional structural features, and combined with a single-modal learning network, a single-modal anomaly map is obtained. The cross-modal anomaly map and the single-modal anomaly map are weighted and fused using a multi-step aggregation strategy to generate the final anomaly detection result map.

[0007] Preferably, the method for obtaining a cross-modal anomaly map based on the two-dimensional texture features and the three-dimensional structural features, combined with a cross-modal learning network, includes: Construct a cross-modal learning network from 2D to 3D to integrate the 2D texture features. Projecting onto a 3D feature space yields cross-modal predicted 3D features. ,calculate and Feature distance to obtain anomalies Figure 1 ; Construct a 3D-to-2D cross-modal learning network to integrate the 3D structural features. Projecting onto a two-dimensional feature space yields cross-modal prediction two-dimensional features. ,calculate and Feature distance to obtain anomalies Figure 2 ; The abnormality Figure 1 and the aforementioned anomaly Figure 2 This forms a cross-modal anomaly graph.

[0008] Preferably, the method for obtaining a single-modal anomaly map based on the two-dimensional texture features and the three-dimensional structural features, combined with a single-modal learning network, includes: Construct a 2D-to-2D unimodal learning network to process the 2D texture features in a 2D feature space. Reconstruction is performed to obtain intramodal reconstructed two-dimensional features. ,calculate and Feature distance to obtain anomalies Figure 3 ; Construct a 3D-to-3D single-modal learning network to learn the 3D structural features in a 3D feature space. Reconstruction is performed to obtain intramodal reconstructed 3D features. ,calculate and Based on the feature distance, anomaly map 4 is obtained; The abnormality Figure 3Together with the aforementioned anomaly diagram four, they constitute a single-modal anomaly diagram.

[0009] Preferably, the method for weighted fusion of the cross-modal anomaly map and the single-modal anomaly map using a multi-step aggregation strategy to generate the final anomaly detection result map includes: The abnormal Figure 1 The aforementioned anomaly Figure 2 The aforementioned anomaly Figure 3 The anomaly map and the anomaly map 4 are fused together by summation or multiplication to obtain a two-dimensional anomaly map and a three-dimensional anomaly map; Based on the two-dimensional anomaly map and the three-dimensional anomaly map, a nonlinear aggregation function is used to fuse cross-modal anomaly information to generate the final anomaly detection result map and anomaly detection value.

[0010] The present invention also provides a multimodal industrial anomaly detection and segmentation system based on polynomial aggregation. The system is used to implement the aforementioned method and includes: an acquisition and processing module, an extraction module, a cross-modal module, a single-modal module, and an aggregation module. The acquisition and processing module is used to acquire two-dimensional images and three-dimensional point clouds of the object under test and perform preprocessing. The extraction module is used to extract two-dimensional texture features and three-dimensional structural features based on the preprocessed two-dimensional image and three-dimensional point cloud, using a visual encoder and a point cloud encoder, respectively. The cross-modal module is used to obtain a cross-modal anomaly map based on the two-dimensional texture features and the three-dimensional structural features, combined with a cross-modal learning network. The single-modal module is used to obtain a single-modal anomaly map based on the two-dimensional texture features and the three-dimensional structural features, combined with a single-modal learning network. The aggregation module is used to perform weighted fusion of the cross-modal anomaly map and the single-modal anomaly map through a multi-step aggregation strategy to generate the final anomaly detection result map.

[0011] Preferably, the cross-modal module includes: a first mapping unit, a second mapping unit, and a first constituent unit; The first mapping unit is used to construct a two-dimensional to three-dimensional cross-modal learning network to map the two-dimensional texture features. Projecting onto a 3D feature space yields cross-modal predicted 3D features. ,calculate and Feature distance to obtain anomalies Figure 1 ; The second mapping unit is used to construct a three-dimensional to two-dimensional cross-modal learning network, and to map the three-dimensional structural features. Projecting onto a two-dimensional feature space yields cross-modal prediction two-dimensional features. ,calculate and Feature distance to obtain anomalies Figure 2 ; The first constituent unit is used for the abnormality Figure 1 and the aforementioned anomaly Figure 2 This forms a cross-modal anomaly graph.

[0012] Preferably, the single-modal module includes: a first reconstruction unit, a second reconstruction unit, and a second constituent unit; The first reconstruction unit is used to construct a two-dimensional to two-dimensional unimodal learning network to process the two-dimensional texture features in a two-dimensional feature space. Reconstruction is performed to obtain intramodal reconstructed two-dimensional features. ,calculate and Feature distance to obtain anomalies Figure 3 ; The second reconstruction unit is used to construct a 3D-to-3D single-modal learning network to reconstruct the 3D structural features in a 3D feature space. Reconstruction is performed to obtain intramodal reconstructed 3D features. ,calculate and Based on the feature distance, anomaly map 4 is obtained; The second constituent unit is used for the abnormality. Figure 3 Together with the aforementioned anomaly diagram four, they constitute a single-modal anomaly diagram.

[0013] Preferably, the aggregation module includes: a first aggregation unit and a second aggregation unit; The first aggregation unit is used to aggregate the anomaly Figure 1 The aforementioned anomaly Figure 2 The aforementioned anomaly Figure 3 The anomaly map and the anomaly map 4 are fused together by summation or multiplication to obtain a two-dimensional anomaly map and a three-dimensional anomaly map; The second aggregation unit is used to fuse cross-modal anomaly information based on the two-dimensional anomaly map and the three-dimensional anomaly map using a nonlinear aggregation function to generate the final anomaly detection result map and anomaly detection value.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention achieves simultaneous and accurate detection of texture and structural defects through a dual-branch network and multinomial aggregation design, solving the problem of insufficient structural defect detection in traditional methods. The multinomial aggregation strategy effectively avoids the "dilution effect" of linear aggregation, significantly reducing the false negative rate of minute defects. Lightweight design through model structure optimization greatly reduces resource consumption, meeting the needs of real-time industrial deployment. It also exhibits strong adaptability to small sample sizes, requiring only a few normal samples for training, adapting to the scarcity of defect samples in industrial scenarios, and demonstrating stable detection performance in scenarios with different materials and defect types. Attached Figure Description

[0015] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the industrial anomaly detection and segmentation method based on polynomial aggregation according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the implementation method of the multimodal industrial anomaly detection and segmentation model according to an embodiment of the present invention; Figure 3 These are different types of abnormal images and their abnormal detection maps implemented in embodiments of the present invention. Detailed Implementation

[0017] The specific implementation of this invention is based on a cross-modal feature mapping and single-modal feature reconstruction framework. It achieves multimodal industrial anomaly detection and segmentation through a complete process of data preprocessing, feature extraction, dual-branch feature processing, training optimization, and anomaly graph aggregation. The anomaly detection and segmentation model consists of a visual encoder, a point cloud encoder, a 2D-to-2D single-modal learning network, a 3D-to-3D single-modal learning network, a 2D-to-3D cross-modal learning network, a 3D-to-2D cross-modal learning network, and an aggregation module. The following detailed description, including specific parameters, operational details, technical principles, and model evaluation metrics, ensures that those skilled in the art can repeat the implementation without creative effort. The technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] Example 1 This invention provides a multimodal industrial anomaly detection and segmentation method based on polynomial aggregation, comprising: Acquire two-dimensional images and three-dimensional point clouds of the object under test and perform preprocessing; Based on the preprocessed 2D image and 3D point cloud, a visual encoder and a point cloud encoder are used to extract 2D texture features and 3D structural features, respectively. Based on the two-dimensional texture features and the three-dimensional structural features, and combined with a cross-modal learning network, a cross-modal anomaly map is obtained; Based on the two-dimensional texture features and the three-dimensional structural features, and combined with a single-modal learning network, a single-modal anomaly map is obtained. The cross-modal anomaly map and the single-modal anomaly map are weighted and fused using a multi-step aggregation strategy to generate the final anomaly detection result map.

[0020] like Figures 1-3 As shown, the specific implementation process of the present invention is as follows: S1. Acquire 2D images and 3D point clouds of the object under test and perform preprocessing. The data preprocessing stage involves performing size unification processing on the 2D images and 3D point cloud data of the object under test in the industrial scene, including: S11. 3D point cloud preprocessing: The background plane is removed by random sampling consensus algorithm, isolated noise points are removed by connected component analysis, and the spatial dimension is matched to the preset input dimension by filling operation. S12, 2D image preprocessing: Adjust the size of the 2D image to the preset input dimension to ensure consistency with the point cloud dimension.

[0021] First, data preprocessing is performed by collecting paired 2D images and 3D point cloud data of the object to be tested in an industrial scene (any part-type product is acceptable, such as screws, carrots, bagels, etc.). The 2D images are stored in formats such as JPG, PNG, and JPEG2000, and the 3D point clouds are stored in formats such as TIFF.

[0022] For 3D point cloud data, the random sampling consensus algorithm is first used to remove background planes to ensure accurate separation of target objects from backgrounds such as desktops; then, connected component analysis is used to remove isolated noise points, and a connected component area threshold is set to filter discrete noise generated during point cloud acquisition; finally, a filling operation is performed to uniformly crop 3D point clouds of different sizes into spatial dimensions with specified length and width, ensuring the consistency of subsequent feature extraction.

[0023] For two-dimensional images, bilinear interpolation is used to crop their length and width to a specified size, so that the pixel dimension of the image is completely matched with the spatial dimension of the preprocessed three-dimensional point cloud, maintaining the spatial correspondence between the two. If the point cloud data size is not in the range of [0,1], an additional coordinate transformation step is added. Through rotation and scaling operations, the orientation of the point cloud is normalized to a unified direction, and the scale is normalized to the range of [0,1], eliminating the interference caused by the inherent differences in the dataset.

[0024] S2. Based on the preprocessed 2D image and 3D point cloud, a visual encoder and a point cloud encoder are used to extract 2D texture features and 3D structural features, respectively, including: S21. Two-dimensional feature extraction: A pre-trained visual encoder, including but not limited to ViT-S8-384 and ViT-B8-768, is used; the visual encoder parameters are frozen, and the pre-processed two-dimensional image is input to extract two-dimensional texture features.

[0025] S22. 3D Feature Extraction: A pre-trained point cloud encoder, including but not limited to the PointMAE model, is used; the point cloud encoder parameters are frozen, the pre-processed 3D point cloud is input, and the 3D structural features are output by merging multiple intermediate layer features.

[0026] S23. Upsampling algorithms are used to achieve dimensional consistency between two-dimensional texture features and three-dimensional structural features, providing adaptability for subsequent cross-modal processing.

[0027] In this embodiment, after data preprocessing is completed, the feature extraction stage begins. The core is to utilize the feature representation capabilities of the pre-trained model to capture two-dimensional texture information and three-dimensional structural information respectively.

[0028] Two-dimensional texture feature extraction employs a pre-trained visual encoder. First, all visual encoder parameters are frozen to avoid overfitting caused by fine-tuning. Second, the pre-processed two-dimensional image is input into the model, and the two-dimensional feature blocks output from the last layer are extracted as the two-dimensional texture features. For example, using the DINO ViT-B8-768 model, with a 224×224 two-dimensional image as input, the output of the last layer is extracted, resulting in 784 two-dimensional feature blocks with a dimension of 768. These two-dimensional feature blocks can accurately represent the high-level texture information of the image, such as surface scratches, color differences, and other defect-related features.

[0029] The extraction of 3D structural features employs a pre-trained point cloud encoder. First, all point cloud encoder parameters are frozen to avoid overfitting. Second, the pre-processed structured point cloud is input into the model, and 3D feature blocks output from multiple intermediate layers of the point cloud encoder are selected as the 3D structural features. For example, for a PointMAE model with 12 layers, inputting 224×224 point cloud data, selecting the output features from layers 4, 7, and 12, and finally stitching them together to form 784 three-dimensional feature blocks with a dimension of 768. This is completely consistent with the number and dimension of two-dimensional feature blocks, providing good adaptability for subsequent cross-modal processing. These three-dimensional feature blocks can effectively characterize the three-dimensional structural information of objects, such as structural defects such as depressions, protrusions, and missing parts.

[0030] S3. Based on two-dimensional texture features and three-dimensional structural features, combined with a cross-modal learning network, a cross-modal anomaly map is obtained, including: S31. Construct a cross-modal learning network from 2D to 3D to integrate 2D texture features. Projecting onto a 3D feature space yields cross-modal predicted 3D features. ,calculate and Feature distance to obtain anomalies Figure 1 ; S32. Construct a 3D-to-2D cross-modal learning network to integrate 3D structural features. Projecting onto a two-dimensional feature space yields cross-modal prediction two-dimensional features. ,calculate and Feature distance to obtain anomalies Figure 2 ; S33, Abnormal Figure 1 and abnormal Figure 2 This forms a cross-modal anomaly graph.

[0031] After feature extraction is completed, cross-modal mapping and single-modal reconstruction are performed.

[0032] Cross-modal feature mapping (i.e., cross-modal feature mapping branch) uses two multilayer perceptrons as learnable mapping functions, and includes two mapping directions: First, a 2D-to-3D mapping is used. A cross-modal learning network from 2D to 3D projects 2D texture features onto a 3D feature space, outputting cross-modal predicted 3D features. This mapping, in the frequency domain, behaves as an approximate three-dimensional all-pass filter, capable of identifying minute defect information in three-dimensional structures. Secondly, it involves a three-dimensional to two-dimensional mapping, projecting three-dimensional structural features onto a two-dimensional feature space through a 3D-to-2D cross-modal learning network, outputting cross-modal predicted two-dimensional features. This mapping manifests as a two-dimensional low-pass filter in the frequency domain, which can smooth high-frequency noise in two-dimensional textures and achieve accurate capture of 2D texture defects.

[0033] The calculation process can be represented as follows: In this embodiment, the core objective of the training optimization phase is to optimize the model parameters through a joint loss function, enabling the cross-modal learning network to learn the feature distribution of normal samples and providing a clear reference benchmark for subsequent anomaly detection. The cross-modal training data uses only normal industrial samples (defect-free samples), without relying on defective sample annotations, thus adapting to the scarcity of defective samples in industrial scenarios. The loss function employs cross-modal consistency loss. The training is as follows: in, Output features by minimizing the cross-modal mapping , With the original features of the corresponding modality , The cosine distance is used to optimize the cross-modal mapping relationship. Based on the trained cross-modal learning network, the cosine distance is calculated... and The Euclidean distance is anomaly. Figure 1 : ; through calculation and The Euclidean distance is anomaly. Figure 2 : .

[0034] The cross-modal learning network architecture can employ a 7-layer structure, including one input layer, five hidden layers, and one output layer. The hidden layer dimension is set to 768, which is the mean of the input and output feature dimensions. A Gaussian error linear unit (Gaussian Linear Unit) activation function is used, which offers advantages such as smoothing gradients and mitigating gradient vanishing, thereby enhancing the model's feature learning ability. Training can utilize the Adam optimizer. Due to the relatively simple and low-dimensional network architecture, minimal resource consumption is required.

[0035] S4. Based on two-dimensional texture features and three-dimensional structural features, combined with a single-modal learning network, a single-modal anomaly map is obtained, including: S41. Construct a 2D to 2D single-modal learning network to learn 2D texture features in a 2D feature space. Reconstruction is performed to obtain intramodal reconstructed two-dimensional features. ,calculate and Feature distance to obtain anomalies Figure 3 ; S42. Construct a 3D-to-3D single-modal learning network to learn 3D structural features in a 3D feature space. Reconstruction is performed to obtain intramodal reconstructed 3D features. ,calculate and Based on the feature distance, anomaly map 4 is obtained; S43, Abnormal Figure 3 Together with anomaly diagram 4, they form a single-mode anomaly diagram.

[0036] Single-modal feature reconstruction (i.e., the single-modal feature mapping branch) employs a multilayer perceptron and includes two reconstruction directions: The first method is 2D-to-2D reconstruction. This involves using a 2D-to-2D single-modal learning network to reconstruct the original 2D texture features in a 2D feature space, outputting the reconstructed 2D features within the modality. The reconstruction process, in the frequency domain, behaves as an approximate two-dimensional all-pass filter, capable of identifying minute two-dimensional defects. Secondly, there is three-dimensional to three-dimensional reconstruction, which uses a 3D-to-3D single-modal learning network to reconstruct the original three-dimensional structural features in the three-dimensional feature space, outputting the reconstructed three-dimensional features within the modality. The reconstruction process is represented by a three-dimensional low-pass filter in the frequency domain, which can effectively filter high-frequency noise in three-dimensional structures, identify three-dimensional defects, and solve the problem of insufficient detection of 3D structural defects in traditional cross-modal methods.

[0037] The calculation process can be represented as follows: The single-modal training data uses only normal industrial samples (defect-free samples) and does not rely on defective sample annotations, thus adapting to the scarcity of defective samples in industrial scenarios. The loss function uses single-modal consistency loss. as follows: in, By minimizing the intramodal reconstruction output features , The cosine distance from the original features is used to optimize intra-modal reconstruction accuracy. Based on a pre-trained single-modal learning network, the accuracy is optimized by calculating... and The Euclidean distance is anomaly. Figure 3 : ; through calculation and The Euclidean distance yields anomaly diagram four: .

[0038] A unimodal learning network can employ a 7-layer structure, including one input layer, five hidden layers, and one output layer. The hidden layer dimension is set to 768, which is the mean of the input and output feature dimensions. The activation function is a Gaussian error linear unit. Training can utilize the Adam optimizer. Due to the relatively simple and low-dimensional network structure, minimal resource consumption is required.

[0039] Compared with existing technologies, this invention directly achieves cross-modal prediction and single-modal reconstruction within two-dimensional texture feature spaces and three-dimensional structural feature spaces, eliminating the need to construct an additional latent feature space as an intermediate representation layer. This reduces feature compression, information loss, and error accumulation during intermediate mapping, and helps preserve original texture and structural information, thereby improving the accuracy of anomaly detection and localization. Simultaneously, this invention omits the additional modeling process of the latent feature space, simplifying the model structure and processing flow, reducing parameter scale and computational complexity, and improving training efficiency and inference speed, making it more suitable for real-time deployment applications in industrial scenarios. Furthermore, directly completing cross-modal prediction and intra-modal reconstruction within the corresponding feature spaces helps enhance the stability and interpretability of anomaly responses, providing better identification capabilities for minute defects, weak anomalies, and structural anomalies.

[0040] S5. The cross-modal anomaly map and the single-modal anomaly map are weighted and fused using a multi-step aggregation strategy to generate the final anomaly detection result map, including: S51, exception Figure 1 ,abnormal Figure 2 ,abnormal Figure 3 The anomaly map and the anomaly map are fused together by summation or multiplication to obtain a two-dimensional anomaly map and a three-dimensional anomaly map; S52. Based on the two-dimensional anomaly map and the three-dimensional anomaly map, a nonlinear aggregation function is used to fuse cross-modal anomaly information to generate the final anomaly detection result map and anomaly detection value.

[0041] The four anomaly images are then aggregated according to the designed aggregation method to accurately suppress background noise and output the final defect detection and location results: First, intermediate anomaly maps of the same modality are merged into a unified two-dimensional and three-dimensional anomaly map using linear aggregation. The fusion method can be either summation or product, and the calculation is as follows: Two-dimensional anomaly diagram : or ; 3D anomaly diagram : or ; Finally, cross-modal fusion is achieved through polynomial aggregation, calculated as follows: in, This is a graph showing the results of anomaly detection.

[0042] This polynomial function solves the "dilution effect" of traditional linear aggregation by using a nonlinear combination of "product terms to filter noise and sum terms to amplify consensus signals". Since the chosen aggregation method has a clear mathematical expression and low complexity, it does not require much resource consumption.

[0043] Compared to existing methods that use multiplicative logical AND operations to aggregate anomaly scores, this invention does not simply multiply the anomaly scores of two modalities. Instead, it first fuses the four anomaly maps obtained from S3 and S4 in stages to form a two-dimensional and a three-dimensional anomaly map, and then generates the final anomaly detection result map through a nonlinear polynomial aggregation function. This aggregation method can simultaneously utilize cross-modal and single-modal anomaly information to generate a stronger response to the real defect region, thereby improving the detection capability of minute defects, weak anomalies, and structural anomalies.

[0044] Furthermore, in this embodiment, the final generated anomaly map is normalized by dividing by the square root of the mean of its non-zero regions. The normalized elements are the pixel-level detection values ​​s(p), where p is the pixel position. The maximum value of s(p) is the image-level detection value. The above calculations yield anomaly detection values ​​in two dimensions. One is the final image-level anomaly detection value used to determine whether the entire image has defects. The first is used to calculate I-AUROC; the second is the pixel-level anomaly detection value s(p) for anomaly segmentation, which is used to calculate FPR, PRO and AUROC@L. AUROC@L specifies the normalized result of the area under the PRO curve within the false positive rate range [0, L]. L means that the model performance is only counted under the strict condition that the proportion of normal pixels being misclassified does not exceed L%.

[0045] The three performance evaluation metrics—I-AUROC, AUPRO@30%, and AUPRO@1%—are calculated as follows: This formula is used to calculate the false positive rate of the model detection at a specified anomaly detection threshold t, that is, the proportion of normal pixels that are mistakenly identified as anomalies. curve and The indicator provides the horizontal axis calculation baseline. Among them, This represents the number of pixels correctly identified as normal at threshold t, with +1 being a smoothing correction term used in the engineering implementation to avoid numerical jumps in boundary value calculations. This represents the total number of all pixels that were originally normal. The true negative rate is the percentage of normal pixels; subtracting the true negative rate from 1 gives the false positive rate (FPR) at the current threshold.

[0046] This formula is used to calculate the average detection overlap rate (PRO) of the model for defective regions at a specified anomaly detection threshold (t), quantifying the model's overall recall capability for defective connected regions and providing a vertical axis calculation benchmark for the PRO curve and the AUPRO metric. Among these, For the first A defective connected domain The number of pixels that were misjudged as normal. The false negative rate is 1; subtracting the false negative rate from 1 gives the detection rate of abnormal pixels within the connected component; the arithmetic mean of the detection rates of the K defect connected components is the PRO value at the current threshold. Compared to pixel-level metrics, PRO can more fairly evaluate the model's ability to detect defects of different sizes, avoiding the dominant influence of large-sized defects on the metric results.

[0047] This formula quantifies the model's overall detection performance of defect areas within a preset low false positive rate range, and is a core evaluation indicator for industrial anomaly detection scenarios. Among other things, For For the horizontal axis, Plotted for the vertical axis The curve is integrated over an interval of [0, L], meaning only the area under the curve within the interval where the false positive rate does not exceed L is counted. Dividing by L is used for normalization to eliminate the influence of different L values ​​on the index value, enabling horizontal performance comparison between different models. In engineering implementation, this formula uses the trapezoidal rule to perform numerical integration on the PRO curve, adapting to the strict constraints on false positive rates in industrial scenarios, and can accurately evaluate the model's defect detection capability within an acceptable false positive range.

[0048] This formula quantifies the model's anomaly detection capability in the image dimension and is a core metric for evaluating the model's binary classification accuracy in determining whether the entire image contains defects. This formula uses the true labels of each image in the test set. As the classification benchmark, the maximum anomaly score of all pixels within a single image is used. To calculate the anomaly prediction score for this image, plot the image-level... curve, For the first test set The image to be tested is AUC, which is the area under the ROC curve. The value ranges from 0 to 1. The closer the value is to 1, the higher the accuracy of the model in judging whether there are defects in the whole image. It is suitable for the detection process in industrial scenarios where the whole image is first screened for defects and then the defects are located.

[0049] The following comparative experiments on the MVTec-3D-AD dataset illustrate the effectiveness of this invention: Table 1 Table 2 Table 3 This experiment uses the MVTec-3D-AD industrial anomaly detection standard dataset and selects 10 mainstream 3D anomaly detection methods for comparison. The performance is evaluated using three core indicators: I-AUROC (image-level anomaly recognition capability), AUPRO@30% (defect detection performance under normal false alarm tolerance), and AUPRO@1% (core performance indicator under stringent industrial requirements for low false alarms). The closer the indicator value is to 1, the better the detection performance. In the table, bold indicates the best performance for the same indicator, and underline indicates the second best performance.

[0050] The core results of the experiment are as follows: Table 1 I-AUROC: The average index of the method of this invention reaches 0.978, which is the best among all the comparison schemes. It is 0.007 higher than the existing mainstream best scheme, and has better image-level anomaly recognition capability.

[0051] Table 2 AUPRO@30%: The average index of the method of this invention reaches 0.980, which is the best among all comparison schemes. It has stable detection performance across all categories and excellent adaptability to conventional scenarios.

[0052] Table 3 AUPRO@1%: The average index of the method of this invention reaches 0.480, which is the best among all the comparison schemes. It is 0.012 higher than the existing mainstream best scheme, and its performance advantage is significant in the extremely low false alarm range of the core industrial application.

[0053] In summary, the method of this invention outperforms existing mainstream methods across all three core evaluation dimensions, with particularly outstanding performance improvement under the core low false alarm constraint in industrial scenarios, fully verifying the technical advancement and practical applicability of this invention.

[0054] In summary, this invention provides a multimodal industrial anomaly detection and segmentation method based on polynomial aggregation. Through a dual-branch network consisting of a cross-modal learning network and a single-modal learning network, and a polynomial aggregation design, it achieves simultaneous and accurate detection of texture and structural defects, solving the problem of insufficient structural defect detection in traditional methods. The polynomial aggregation strategy effectively avoids the "dilution effect" of linear aggregation, significantly reducing the false negative rate of minute defects. Lightweight design through model structure optimization greatly reduces resource consumption, meeting the needs of real-time industrial deployment. It also possesses strong adaptability to small sample sizes, requiring only a small number of normal samples for training, adapting to the scarcity of defect samples in industrial scenarios, and exhibiting stable detection performance in scenarios with different materials and defect types.

[0055] Example 2 Based on the same inventive concept, the present invention also provides a multimodal industrial anomaly detection and segmentation system based on polynomial aggregation, for implementing the methods described in the foregoing embodiments. The system includes: an acquisition and processing module, an extraction module, a cross-modal module, a single-modal module, and an aggregation module. The acquisition and processing module is used to acquire two-dimensional images and three-dimensional point clouds of the object under test and perform preprocessing. The extraction module is used to extract two-dimensional texture features and three-dimensional structural features based on the preprocessed two-dimensional image and three-dimensional point cloud, using a visual encoder and a point cloud encoder, respectively. The cross-modal module is used to obtain a cross-modal anomaly map based on the two-dimensional texture features and the three-dimensional structural features, combined with a cross-modal learning network. The single-modal module is used to obtain a single-modal anomaly map based on the two-dimensional texture features and the three-dimensional structural features, combined with a single-modal learning network. The aggregation module is used to perform weighted fusion of the cross-modal anomaly map and the single-modal anomaly map through a multi-step aggregation strategy to generate the final anomaly detection result map.

[0056] Furthermore, in this embodiment, the cross-modal module includes: a first mapping unit, a second mapping unit, and a first constitutive unit; The first mapping unit is used to construct a two-dimensional to three-dimensional cross-modal learning network to map the two-dimensional texture features. Projecting onto a 3D feature space yields cross-modal predicted 3D features. ,calculate and Feature distance to obtain anomalies Figure 1 ; The second mapping unit is used to construct a three-dimensional to two-dimensional cross-modal learning network, and to map the three-dimensional structural features. Projecting onto a two-dimensional feature space yields cross-modal prediction two-dimensional features. ,calculate and Feature distance to obtain anomalies Figure 2 ; The first constituent unit is used for the abnormality Figure 1 and the aforementioned anomaly Figure 2 This forms a cross-modal anomaly graph.

[0057] Furthermore, in this embodiment, the single-modal module includes: a first reconstruction unit, a second reconstruction unit, and a second constituent unit; The first reconstruction unit is used to construct a two-dimensional to two-dimensional unimodal learning network to process the two-dimensional texture features in a two-dimensional feature space. Reconstruction is performed to obtain intramodal reconstructed two-dimensional features. ,calculate and Feature distance to obtain anomalies Figure 3 ; The second reconstruction unit is used to construct a 3D-to-3D single-modal learning network to reconstruct the 3D structural features in a 3D feature space. Reconstruction is performed to obtain intramodal reconstructed 3D features. ,calculate and Based on the feature distance, anomaly map 4 is obtained; The second constituent unit is used for the abnormality. Figure 3 Together with the aforementioned anomaly diagram four, they constitute a single-modal anomaly diagram.

[0058] Furthermore, in this embodiment, the aggregation module includes: a first aggregation unit and a second aggregation unit; The first aggregation unit is used to aggregate the anomaly Figure 1 The aforementioned anomaly Figure 2 The aforementioned anomaly Figure 3 The anomaly map and the anomaly map 4 are fused together by summation or multiplication to obtain a two-dimensional anomaly map and a three-dimensional anomaly map; The second aggregation unit is used to fuse cross-modal anomaly information based on the two-dimensional anomaly map and the three-dimensional anomaly map using a nonlinear aggregation function to generate the final anomaly detection result map and anomaly detection value.

[0059] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A multimodal industrial anomaly detection and segmentation method based on polynomial aggregation, characterized in that, The method includes: Acquire two-dimensional images and three-dimensional point clouds of the object under test and perform preprocessing; Based on the preprocessed 2D image and 3D point cloud, a visual encoder and a point cloud encoder are used to extract 2D texture features and 3D structural features, respectively. Based on the two-dimensional texture features and the three-dimensional structural features, and combined with a cross-modal learning network, a cross-modal anomaly map is obtained; Based on the two-dimensional texture features and the three-dimensional structural features, and combined with a single-modal learning network, a single-modal anomaly map is obtained. The cross-modal anomaly map and the single-modal anomaly map are weighted and fused using a multi-step aggregation strategy to generate the final anomaly detection result map; A method for obtaining cross-modal anomaly maps based on the aforementioned two-dimensional texture features and three-dimensional structural features, combined with a cross-modal learning network, includes: Construct a cross-modal learning network from 2D to 3D to integrate the 2D texture features. Projecting onto a 3D feature space yields cross-modal predicted 3D features. ,calculate and Based on the feature distance, an anomaly map 1 is obtained; Construct a 3D-to-2D cross-modal learning network to integrate the 3D structural features. Projecting onto a two-dimensional feature space yields cross-modal prediction two-dimensional features. ,calculate and Based on the feature distance, anomaly map 2 is obtained; The anomaly diagram 1 and the anomaly diagram 2 constitute a cross-modal anomaly diagram; Based on the aforementioned two-dimensional texture features and three-dimensional structural features, and combined with a single-modal learning network, a method for obtaining a single-modal anomaly map includes: Construct a 2D-to-2D unimodal learning network to process the 2D texture features in a 2D feature space. Reconstruction is performed to obtain intramodal reconstructed two-dimensional features. ,calculate and Based on the feature distance, anomaly map 3 is obtained; Construct a 3D-to-3D single-modal learning network to learn the 3D structural features in a 3D feature space. Reconstruction is performed to obtain intramodal reconstructed 3D features. ,calculate and Based on the feature distance, anomaly map 4 is obtained; The anomaly diagram 3 and the anomaly diagram 4 constitute a single-modal anomaly diagram; The method for weighted fusion of the cross-modal anomaly map and the single-modal anomaly map using a multi-step aggregation strategy to generate the final anomaly detection result map includes: The anomaly diagrams 1, 2, 3, and 4 are fused together by summation or multiplication to obtain a two-dimensional anomaly diagram and a three-dimensional anomaly diagram. Based on the two-dimensional anomaly map and the three-dimensional anomaly map, a nonlinear aggregation function is used to fuse cross-modal anomaly information to generate the final anomaly detection result map and anomaly detection value.

2. A multimodal industrial anomaly detection and segmentation system based on polynomial aggregation, the system being used to implement the method of claim 1, characterized in that, The system includes: an acquisition and processing module, an extraction module, a cross-modal module, a single-modal module, and an aggregation module; The acquisition and processing module is used to acquire two-dimensional images and three-dimensional point clouds of the object under test and perform preprocessing. The extraction module is used to extract two-dimensional texture features and three-dimensional structural features based on the preprocessed two-dimensional image and three-dimensional point cloud, using a visual encoder and a point cloud encoder, respectively. The cross-modal module is used to obtain a cross-modal anomaly map based on the two-dimensional texture features and the three-dimensional structural features, combined with a cross-modal learning network. The single-modal module is used to obtain a single-modal anomaly map based on the two-dimensional texture features and the three-dimensional structural features, combined with a single-modal learning network. The aggregation module is used to perform weighted fusion of the cross-modal anomaly map and the single-modal anomaly map through a multi-step aggregation strategy to generate the final anomaly detection result map.

3. The system according to claim 2, characterized in that, The cross-modal module includes: a first mapping unit, a second mapping unit, and a first constituent unit; The first mapping unit is used to construct a two-dimensional to three-dimensional cross-modal learning network to map the two-dimensional texture features. Projecting onto a 3D feature space yields cross-modal predicted 3D features. ,calculate and Based on the feature distance, an anomaly map 1 is obtained; The second mapping unit is used to construct a three-dimensional to two-dimensional cross-modal learning network, and to map the three-dimensional structural features. Projecting onto a two-dimensional feature space yields cross-modal prediction two-dimensional features. ,calculate and Based on the feature distance, anomaly map 2 is obtained; The first constituent unit is used to form a cross-modal anomaly diagram from the anomaly diagram one and the anomaly diagram two.

4. The system according to claim 3, characterized in that, The single-modal module includes: a first reconstruction unit, a second reconstruction unit, and a second constituent unit; The first reconstruction unit is used to construct a two-dimensional to two-dimensional unimodal learning network to process the two-dimensional texture features in a two-dimensional feature space. Reconstruction is performed to obtain intramodal reconstructed two-dimensional features. ,calculate and Based on the feature distance, anomaly map 3 is obtained; The second reconstruction unit is used to construct a 3D-to-3D single-modal learning network to reconstruct the 3D structural features in a 3D feature space. Reconstruction is performed to obtain intramodal reconstructed 3D features. ,calculate and Based on the feature distance, anomaly map 4 is obtained; The second constituent unit is used to form a single-modal anomaly diagram from the anomaly diagram three and the anomaly diagram four.

5. The system according to claim 4, characterized in that, The aggregation module includes: a first aggregation unit and a second aggregation unit; The first aggregation unit is used to fuse the first anomaly image, the second anomaly image, the third anomaly image, and the fourth anomaly image by summation or product to obtain a two-dimensional anomaly image and a three-dimensional anomaly image. The second aggregation unit is used to fuse cross-modal anomaly information based on the two-dimensional anomaly map and the three-dimensional anomaly map using a nonlinear aggregation function to generate the final anomaly detection result map and anomaly detection value.

Images