Multi-modal anomaly detection and class discovery method based on stable abnormal regions

Abstract

The application discloses a kind of multi-modal anomaly detection and class discovery method based on stable abnormal area.It includes the following steps: obtaining the two-dimensional appearance image data of the industrial product to be detected and the corresponding three-dimensional point cloud data;Extract two-dimensional appearance feature map;Based on two-dimensional appearance feature map, extract stable abnormal area;Construct the fusion appearance feature representation containing local detail information and global context semantic information;Extract the three-dimensional geometric feature representation corresponding to the stable abnormal area;Construct and train the bidirectional cross-modal feature mapping model between two-dimensional appearance features and aligned geometric features;Calculate the cross-modal structure deviation metric corresponding to the stable abnormal area;Construct area-level anomaly description vector, and automatically divide the clustering result according to the similarity relationship between anomaly description vectors.The application improves the stability and robustness of abnormal area positioning, and simultaneously realizes the automatic classification of anomalies under unsupervised conditions.

Description

Technical Field

[0001] This invention relates to the field of industrial visual inspection technology, specifically to a method for multimodal anomaly detection and category discovery based on stable anomaly regions. Background Technology

[0002] With the continuous improvement of intelligent manufacturing and industrial automation, machine vision-based industrial inspection technology has been widely used in product quality control, defect detection, and production process monitoring. By automating the inspection of the surface or structure of industrial products, the cost of manual inspection can be effectively reduced, and the efficiency and consistency of inspection can be improved.

[0003] In recent years, with the development of deep learning technology, anomaly detection using image data has gradually become an important research direction in industrial visual inspection. Unsupervised or weakly supervised anomaly detection methods based on deep learning learn the distribution characteristics of normal samples and determine the presence of anomalies during the detection phase based on the difference between the input sample and the normal distribution. Although this method improves the automation level of anomaly detection to some extent, the following technical problems still exist: First, most of them rely on single-modal image information, such as two-dimensional image information, for modeling, which makes it difficult to fully depict the spatial structural features of the product; Second, these methods typically perform anomaly analysis at the entire image level, making them susceptible to interference from background areas. Furthermore, the extraction results of anomaly areas are highly sensitive to parameter or threshold settings, often resulting in scattered and unstable distribution of anomaly areas, which affects the reliability of subsequent anomaly analysis.

[0004] Furthermore, the maturity of 3D sensing technology enables the introduction of point cloud data into industrial inspection scenarios, thus providing a richer information foundation for the analysis of product structure and geometry. Existing multimodal industrial anomaly detection methods mainly improve detection accuracy by jointly modeling or simply fusing 2D images and 3D point cloud features, such as directly stitching multimodal features or applying consistency constraints at the whole image level. However, such methods often suffer from the following technical problems: First, the stability of abnormal regions under different threshold conditions is ignored; Second, it usually focuses on anomaly localization or anomaly scoring, and it is difficult to further automatically classify or discover anomalies under unsupervised conditions. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a multimodal anomaly detection and category discovery method based on stable anomaly regions. This method can reliably identify key anomaly regions in industrial scenarios and perform multimodal consistency analysis, thereby achieving anomaly detection and automatic anomaly category discovery and improving the robustness and efficiency of multimodal industrial anomaly detection.

[0006] To achieve the above objectives, the present invention provides a multimodal anomaly detection and category discovery method based on stable anomaly regions, characterized by the following steps: S1) Obtain two-dimensional appearance image data of the industrial product to be inspected, and three-dimensional point cloud data corresponding to the space of the industrial product to be inspected; S2) Input the two-dimensional appearance image data into the two-dimensional appearance feature extractor to extract the two-dimensional appearance feature map, which includes global appearance features and local appearance features. S3) Based on the two-dimensional appearance feature map, calculate the anomaly significance distribution map to characterize the anomaly intensity at each spatial location; S4) Perform multi-threshold analysis on the anomaly significance distribution map to extract stable anomaly regions; S5) Based on the spatial location information of stable anomaly regions in the original two-dimensional appearance feature map, extract the regional-level global features corresponding to the stable anomaly regions from the global appearance features, and encode and fuse them with the local appearance features of the stable anomaly regions to construct a fused appearance feature representation containing local detail information and global contextual semantic information. S6) Using the two-dimensional region mask and the two-dimensional to three-dimensional mapping relationship, a subset of point clouds corresponding to the stable and abnormal region space is selected in the three-dimensional point cloud data, and the point cloud subset is input into the three-dimensional geometric feature extractor to extract the three-dimensional geometric feature representation corresponding to the stable and abnormal region. S7) Establish the spatial correspondence between the two-dimensional appearance image and the three-dimensional point cloud, and map the three-dimensional geometric features in the stable abnormal region to the two-dimensional space. At the same time, construct and train a bidirectional cross-modal feature mapping model that fuses appearance features and three-dimensional geometric features to achieve consistent alignment of multimodal features. S8) During the detection phase, the cross-modal structure deviation metric corresponding to the stable abnormal region is obtained by calculating the difference between the cross-modal predicted features and the actual features. S9) Construct a region-level anomaly description vector based on the fusion of appearance features, three-dimensional geometric features, and cross-modal structural deviation measures of stable anomaly regions; Then, unsupervised clustering analysis is performed on the anomaly description vectors corresponding to multiple stable anomaly regions. Based on the similarity relationship between the anomaly description vectors, they are automatically divided into several clustering results. Each clustering result corresponds to an anomaly self-discovery mode, thereby realizing the differentiation and summarization of different anomaly types.

[0007] Furthermore, in S4), the specific steps for extracting stable anomaly regions include: S41) Set a threshold set and binarize the anomaly significance distribution map to obtain candidate anomaly regions under different threshold conditions; S42) Calculate the stability of the candidate anomaly region under different threshold conditions. When the stability is ≥ the stability threshold, the candidate anomaly region is determined to be a stable anomaly region. S43) Using the stable abnormal region as the center, the original two-dimensional appearance feature map is cropped to obtain the abnormal center region image, and the corresponding abnormal region mask is generated.

[0008] Furthermore, in S41), the binarization process is performed using the following formula. In the formula, B τk (x,y) represents the candidate anomaly region under the set threshold condition. S(x,y) represents the distribution map of the significance of the anomalies. τ k This indicates that a threshold value is set.

[0009] Furthermore, in S42), the stability is calculated using the following formula. In the formula, D(R) represents the stability of the candidate anomaly region under different threshold conditions. B τk This indicates candidate anomaly regions under a set threshold condition. R represents the set of region indices in a two-dimensional pixel space. Ⅱ(⋅) denotes the binarization indicator function. M represents the total number of candidate anomaly regions that meet the anomaly threshold condition.

[0010] Furthermore, in S7), the specific implementation steps include: S71) The spatial points in the 3D point cloud are mapped to the pixel coordinate system of the 2D appearance image through the projection function; S72) Maps three-dimensional geometric features to two-dimensional space using a projection operator; S73) Construct a bidirectional cross-modal feature mapping model that integrates appearance features and three-dimensional geometric features. The bidirectional cross-modal feature mapping model includes a mapping model from integrated appearance features to three-dimensional geometric features, and a mapping model from three-dimensional geometric features to integrated appearance features.

[0011] Furthermore, in S71), the mapping relationship between spatial points in the 3D point cloud and the 2D appearance image is expressed by the following formula. In the formula, (X i ,Yi Z i () represents the spatial coordinates of the i-th point in a 3D point cloud. (u i ,v i () represents the two-dimensional image pixel coordinates corresponding to the spatial coordinates of the i-th point in the 3D point cloud. ∏(▪) represents the projection function.

[0012] Furthermore, in S72), the mapping process by which the projection operator maps three-dimensional geometric features to two-dimensional space is performed using the following formula. In the formula, H R This represents a two-dimensional geometric feature aligned with the two-dimensional appearance feature in terms of spatial resolution. G R Represents the three-dimensional geometric features within a stable anomaly region. ψ(▪) represents the geometric feature mapping operator from three dimensions to two dimensions.

[0013] Furthermore, in S8), the cross-modal structure deviation metric is expressed by the following formula: In the formula, E R This represents a measure of cross-modal structural deviation. Ω R Indicates a stable anomaly region. R represents the set of region indices in a two-dimensional pixel space. This represents the three-dimensional geometric features predicted from two-dimensional appearance features using a cross-modal prediction network. Represents three-dimensional geometric features aligned with two-dimensional appearance features in spatial resolution. This represents the fused features obtained by back-predicting geometric features through a cross-modal prediction network. This represents the fused appearance feature obtained by the feature fusion module.

[0014] Further, in S9), the anomaly description vector is represented by the following formula. In the formula, Q R Represents an anomaly description vector. ZR This indicates the fused appearance features corresponding to stable anomaly regions. G R This represents the three-dimensional geometric features corresponding to stable anomaly regions. E R This represents the cross-modal structural deviation metric corresponding to stable anomalous regions.

[0015] The advantages of this invention are: 1. This invention introduces a stable anomaly region extraction mechanism, and performs cross-modal structural deviation measurement analysis on anomaly regions under multiple threshold conditions. This can effectively reduce the sensitivity of anomaly detection results to threshold parameter settings, reduce the impact of noise interference on anomaly localization results, and improve the stability and robustness of anomaly region localization. 2. This invention constructs an anomaly center sub-region centered on a stable anomaly region and establishes a regional spatial correspondence between the two-dimensional appearance image and the three-dimensional point cloud data, which can reduce the interference of the background region on feature analysis and improve the accuracy of multimodal feature modeling. 3. This invention constructs a cross-modal feature mapping model between two-dimensional appearance features and three-dimensional geometric features, and performs anomaly discrimination based on cross-modal consistency violation, which can effectively improve the detection capability of complex structural anomalies and cross-modal anomalies; 4. This invention constructs a region-level anomaly description vector that integrates appearance features, geometric features, and cross-modal structural deviation metrics, enabling refined characterization of anomaly regions. By performing unsupervised clustering analysis on the region-level anomaly description vector, it achieves automatic summarization and differentiation of different anomaly patterns, avoiding reliance on predefined anomaly categories or extensive manual annotation. This approach can adapt to situations where anomaly types are unknown or constantly changing in industrial applications. Compared with existing methods based solely on a single modality or a single anomaly score, this invention fully utilizes the complementary relationships between multimodal information, improving the stability and interpretability of anomaly identification results while ensuring the accuracy of anomaly detection. 5. It can be widely applied to various industrial detection and monitoring scenarios. Through joint analysis of multi-source sensing data, it can realize automatic detection of abnormal areas and discovery of abnormal types, which helps to improve the intelligent detection level of industrial systems, reduce reliance on manual labor, and has good engineering application value and promotion prospects. This invention provides a multimodal anomaly detection and category discovery method based on stable anomaly regions. This method not only effectively reduces the sensitivity of anomaly detection results to threshold parameter settings and improves the stability and robustness of anomaly region localization, but also reduces the interference of background regions on feature analysis, improves the accuracy of multimodal feature modeling, and enables automatic anomaly classification under unsupervised conditions. Attached Figure Description

[0016] Figure 1 This is a flowchart of the present invention; Figure 2 This is a flowchart of the extraction process for stable abnormal regions in this invention; Figure 3 This is a flowchart illustrating the construction process of integrating appearance features in this invention; Figure 4 This is a flowchart of the extraction process for three-dimensional geometric features in this invention. Detailed Implementation

[0017] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] like Figure 1 As shown, the present invention provides a multimodal anomaly detection and category discovery method based on stable anomaly regions, comprising the following steps: S1) Obtain two-dimensional appearance image data of the industrial product to be inspected. I and the three-dimensional point cloud data corresponding to the space of the industrial product to be inspected. .

[0019] The two-dimensional appearance image and the three-dimensional point cloud data are then mapped using pre-calibrated imaging parameters. This mapping relationship is used for spatial alignment of the two-dimensional and three-dimensional data.

[0020] S2) Transfer two-dimensional appearance image data I Input a 2D appearance feature extractor to extract 2D appearance feature maps. A R The two-dimensional appearance feature map A R This includes global appearance features and local appearance features.

[0021] The two-dimensional appearance feature extractor includes a global appearance feature encoder branch and a local appearance feature encoder branch, and achieves joint modeling between global and local appearance features through a region-level feature alignment and feature combination module.

[0022] The global appearance feature encoder branch takes the complete RGB image as input. It divides the input image into spatially overlapping image feature sequences through an overlapping image feature sequence extraction layer, mapping each image block to a 768-dimensional feature vector, resulting in an initial feature sequence of size N×768. This overlapping image feature sequence extraction layer, located at the beginning of the global appearance feature encoder branch, is used to construct the global receptive field while maintaining local continuity. The feature sequence is then input into 12 consecutively stacked global self-attention modules for feature modeling. Each global self-attention module has 12 attention heads and a feature dimension of 768 to capture long-range dependencies between different spatial locations in the two-dimensional appearance image, thereby obtaining a global appearance feature representation.

[0023] S3) Based on two-dimensional appearance feature map A R Calculate the anomaly significance distribution map to characterize the anomaly intensity at each spatial location.

[0024] The formula for calculating the anomaly significance distribution map is as follows: In the formula, S (x,y) represents the distribution of the significance of the anomalies. A R (x,y) represents a two-dimensional appearance feature map. W This represents the mapping function for the significance of anomalies.

[0025] S4) Perform multi-threshold analysis on the anomaly significance distribution map to extract stable anomaly regions. For example... Figure 2 The diagram shown is a flowchart of the extraction process for stable anomaly regions in this invention. Specifically, it includes the following steps: S41) Define a threshold set T = {τ1, τ2, ..., τ} M The anomaly significance distribution map was binarized to obtain candidate anomaly regions under different threshold conditions.

[0026] The binarization process is performed using the following formula: In the formula, B τk (x,y) represents the candidate anomaly region under the set threshold condition. S(x,y) represents the distribution map of the significance of the anomalies. τ k This indicates that a threshold value is set.

[0027] S42) Determine the regional stability of candidate abnormal regions obtained under different threshold conditions. Specifically, calculate the stability D(R) of the candidate abnormal region under different threshold conditions. When D(R) ≥ stability threshold γ, the candidate abnormal region is determined to be a stable abnormal region.

[0028] The stability is calculated using the following formula. In the formula, D(R) represents the stability of the candidate anomaly region under different threshold conditions. B τk This indicates candidate anomaly regions under a set threshold condition. R represents the set of region indices in a two-dimensional pixel space. Ⅱ(⋅) denotes the binarization indicator function. M represents the total number of candidate anomaly regions that meet the anomaly threshold condition.

[0029] Specifically, based on the spatial overlap of candidate anomaly regions under different threshold conditions, the region intersection-union ratio (IUU) is used as the subsequent cross-modal structural deviation metric to define a stability function for candidate anomaly regions. When the average spatial overlap of candidate anomaly regions under adjacent threshold conditions is not lower than the stability threshold, the region is considered insensitive to changes in the stability threshold and possesses stable anomaly characteristics. The stability threshold is set according to the spatial consistency principle and the false detection rate requirements of industrial anomaly detection; in this embodiment, the value ranges from 0.6 to 0.85.

[0030] S43) Using the stable abnormal region as the center, the original two-dimensional appearance feature map is cropped to obtain the abnormal center region image, and the corresponding abnormal region mask is generated.

[0031] S5) Based on the spatial location information of stable anomaly regions in the original two-dimensional appearance feature map, extract the region-level global features corresponding to the stable anomaly regions from the global appearance features, and encode and fuse them with the local appearance features of the stable anomaly regions to construct a fused appearance feature representation containing local detail information and global contextual semantic information. Z R .like Figure 3 The diagram shown is a flowchart illustrating the construction process of integrating appearance features in this invention.

[0032] The specific steps are as follows: S51) After obtaining the global appearance features and stable abnormal regions, based on the spatial position of the stable abnormal regions in the original two-dimensional appearance feature map, perform region feature alignment and clipping operations from the global appearance features to extract the region-level global features corresponding to the stable abnormal regions. Then, transform the feature dimension of the region-level global features through the linear mapping module to make them consistent with the local appearance features in terms of feature dimension, thereby obtaining the region-level global feature representation.

[0033] The local appearance feature encoder branch takes the stable anomalous region image as input. It divides the stable anomalous region into local image patches through a local feature embedding layer and maps them to 384-dimensional feature vectors, forming a local feature sequence of size M×384. This local feature sequence is then input into six consecutively stacked local self-attention modules for feature encoding. Each local self-attention module has six attention heads and a feature dimension of 384, focusing on modeling the local texture variations and detailed structural features within the stable anomalous region, thereby obtaining a local appearance feature representation.

[0034] S52) After completing the encoding of global and local appearance features, the regional-level global and local appearance features are input into the feature combination module for fusion to obtain the fused appearance feature representation. Z R .

[0035] The fused appearance features integrate local detail information of stable anomaly regions and their contextual semantic information in the overall appearance structure, and are used for subsequent cross-modal feature prediction, cross-modal structural deviation measurement analysis, and construction of regional anomaly description vectors.

[0036] S6) Using the two-dimensional region mask and the two-dimensional to three-dimensional mapping relationship, a subset of point clouds corresponding to the stable anomalous region space is selected in the three-dimensional point cloud data, and the point cloud subset is input into the three-dimensional geometric feature extractor to extract the three-dimensional geometric feature representation corresponding to the stable anomalous region.

[0037] The three-dimensional geometric feature extractor includes a cross-modal correspondence module, a mask constraint module, and a geometric feature encoding module.

[0038] The cross-modal correspondence module receives 3D point cloud data input. Based on a pre-calibrated 2D-to-3D projection mapping relationship, this module projects spatial points in the 3D point cloud onto the pixel coordinate system of the 2D appearance image, thereby establishing a spatial correspondence between 3D points and 2D pixels. The mask constraint module, combined with the 2D anomaly region mask corresponding to the stable anomaly region, performs spatial constraint filtering on the 3D point cloud data, retaining only the 3D points projected onto the stable anomaly region, thus obtaining a subset of point clouds spatially corresponding to the stable anomaly region. The filtering relationship is expressed by the following formula. In the formula, P R This represents a subset of the point cloud corresponding to a stable anomaly region. p i This represents the i-th spatial point in a 3D point cloud. M R (▪) represents the two-dimensional anomaly mask corresponding to the stable anomaly region. (u i ,v i ) represents a spatial point p i The pixel coordinates are mapped to the two-dimensional appearance image through a two-dimensional to three-dimensional projection relationship.

[0039] The mask constraint module enables precise selection of points in 3D point cloud data that correspond to stable and anomalous regions, thereby preventing background point clouds from interfering with subsequent geometric feature modeling. Based on this, the selected point cloud subset P... R The input is fed into the geometric feature encoding module. In this embodiment, the geometric feature encoding module adopts a point cloud neural network structure based on dynamic graph construction. It uses the three-dimensional coordinates of each point in the point cloud subset as the network input and constructs a local neighborhood graph only within the point cloud subset to characterize the local geometric relationships between points in stable abnormal regions.

[0040] Specifically, for each point in the point cloud subset, a K-nearest neighbor search is performed based on Euclidean distance to construct a local neighborhood, where the value of K is adaptively adjusted according to the point cloud density. The difference between the center point coordinates and the coordinates of its neighboring points is used as the edge feature input. A multilayer perceptron performs nonlinear mapping on the edge features, and max pooling is used to aggregate the neighborhood features to obtain a point-level geometric feature representation. By stacking multilayer graph convolutional structures, the receptive field of the point cloud features under the mask constraint of stable anomaly regions is expanded layer by layer, thereby achieving step-by-step modeling from local geometric details to region-level structural information, ultimately outputting a point-level or region-level 3D geometric feature representation corresponding to the stable anomaly region. For example... Figure 4 The diagram shown is a flowchart of the extraction process for three-dimensional geometric features in this invention.

[0041] S7) Establish the spatial correspondence between the two-dimensional appearance image and the three-dimensional point cloud, and map the three-dimensional geometric features in the stable abnormal region to the two-dimensional space. At the same time, construct and train a bidirectional cross-modal feature mapping model that integrates appearance features and three-dimensional geometric features to achieve consistent alignment of multimodal features.

[0042] The specific steps are as follows: S71) By using the projection function ∏(▪), spatial points in the 3D point cloud are mapped to the pixel coordinate system of the 2D appearance image, thereby establishing a one-to-one or many-to-one spatial correspondence between 2D pixels and 3D spatial points. The mapping relationship is expressed by the following formula. In the formula, (X i ,Y i Z i () represents the spatial coordinates of the i-th point in a 3D point cloud. (u i ,v i () represents the two-dimensional image pixel coordinates corresponding to the spatial coordinates of the i-th point in the 3D point cloud. ∏(▪) represents the projection function.

[0043] The input to the projection function ∏(▪) includes the spatial coordinates of the three-dimensional point cloud and the calibration parameters of the imaging system. The spatial mapping between the two-dimensional image and the three-dimensional point cloud adopts the pinhole camera projection model, and the three-dimensional spatial points are accurately mapped to the two-dimensional image coordinate system through the camera intrinsic and extrinsic parameters.

[0044] Based on the above mapping relationship, a corresponding subset of point cloud can be obtained by filtering the two-dimensional stable anomaly region in the three-dimensional point cloud, thereby achieving spatial alignment between the two-dimensional anomaly region and the three-dimensional geometric structure.

[0045] S72) The three-dimensional geometric features are mapped to two-dimensional space using a projection operator. The mapping process is performed by the following formula. In the formula, H R This represents a two-dimensional geometric feature aligned with the two-dimensional appearance feature in terms of spatial resolution. G R Represents the three-dimensional geometric features within a stable anomaly region. ψ(▪) represents the geometric feature mapping operator from three dimensions to two dimensions.

[0046] In this embodiment, firstly, based on the calibrated camera projection model, each spatial point in the 3D point cloud is projected onto the pixel coordinate system of the 2D appearance image, thereby establishing a spatial correspondence between 3D points and 2D pixels. Using 2D pixels as units, the geometric features corresponding to multiple 3D points projected to the same pixel location are aggregated to obtain a 2D geometric feature representation of that pixel location. The aggregation operation employs max pooling or weighted pooling to enhance the responsiveness to local geometric anomalies. For pixel locations that do not correspond to any 3D point, geometric features are completed using spatial neighborhood-based interpolation, thus forming a spatial correspondence and providing a foundation for subsequent multimodal feature fusion and cross-modal structural deviation measurement analysis.

[0047] (S73) Construct a bidirectional cross-modal feature mapping model between fused appearance features and 3D geometric features to characterize the correspondence between fused appearance features and 3D geometric features under normal conditions. The bidirectional cross-modal feature mapping model includes a mapping model from fused appearance features to 3D geometric features, and a mapping model from 3D geometric features to fused appearance features.

[0048] In this embodiment, the bidirectional cross-modal feature mapping models all adopt a feature regression network structure based on a multilayer perceptron. The network consists of several fully connected layers and nonlinear activation functions, used to achieve nonlinear mapping between different modal feature spaces. The mapping model Φ... A→G Fusion appearance features corresponding to stable anomaly regions Z R As input, the output is the predicted result of the three-dimensional geometric features corresponding to its space; mapping model Φ G→A Three-dimensional geometric features aligned with the fused appearance features in spatial resolution G R As input, the corresponding fused appearance feature prediction result is output.

[0049] The bidirectional cross-modal feature mapping model is trained based on normal samples. During the training phase, by minimizing the difference between predicted features and true features, the model ensures that the fusion of appearance features and 3D geometric features maintains consistency after cross-modal mapping under normal conditions. The mapping relationship is shown in the following formula. In the formula, Φ A→G This represents a mapping model from fused appearance features to 3D geometric features. Z R This indicates the fused appearance features corresponding to stable anomaly regions. This represents the prediction results of three-dimensional geometric features. Φ G→A This represents a mapping model from 3D geometric features to fused appearance features. G R Represents three-dimensional geometric features aligned with the merged appearance features in terms of spatial resolution. This indicates the prediction result of the fused appearance features.

[0050] S8) During the detection phase, the cross-modal structural deviation metric corresponding to the stable abnormal region is obtained by calculating the difference between the cross-modal predicted features and the actual features.

[0051] The cross-modal structure deviation metric is expressed by the following formula. In the formula, E R This represents a measure of cross-modal structural deviation. Ω R Indicates a stable anomaly region. R represents the set of region indices in a two-dimensional pixel space. This represents the three-dimensional geometric features predicted from two-dimensional appearance features using a cross-modal prediction network. Represents three-dimensional geometric features aligned with two-dimensional appearance features in spatial resolution. This represents the fused features obtained by back-predicting geometric features through a cross-modal prediction network. This represents the fused appearance feature obtained by the feature fusion module.

[0052] Specifically, when the consistency between the fused appearance features and the three-dimensional geometric features is disrupted, the difference increases significantly, thus indicating an anomaly in the region. Through the aforementioned bidirectional cross-modal feature mapping and cross-modal structural deviation metric modeling mechanism, the synergistic relationship between two-dimensional appearance information and three-dimensional geometric structure information can be effectively utilized to improve the detection and characterization capabilities of complex anomaly regions.

[0053] S9) Based on the fusion of appearance features, three-dimensional geometric features and cross-modal structural deviation measures of stable anomaly regions, construct a region-level anomaly description vector.

[0054] Then, unsupervised clustering analysis is performed on the anomaly description vectors corresponding to multiple stable anomaly regions. Based on the similarity relationship between the anomaly description vectors, they are automatically divided into several clustering results. Each clustering result corresponds to an anomaly self-discovery mode, thereby realizing the differentiation and summarization of different anomaly types.

[0055] The anomaly description vector is represented by the following formula: In the formula, Q R Represents an anomaly description vector. Z R This indicates the fused appearance features corresponding to stable anomaly regions. G R This represents the three-dimensional geometric features corresponding to stable anomaly regions. E R This represents the cross-modal structural deviation metric corresponding to stable anomalous regions.

[0056] This invention provides a multimodal anomaly detection and category discovery method based on stable anomaly regions. This method not only effectively reduces the sensitivity of anomaly detection results to threshold parameter settings and improves the stability and robustness of anomaly region localization, but also reduces the interference of background regions on feature analysis, improves the accuracy of multimodal feature modeling, and enables automatic anomaly classification under unsupervised conditions.

[0057] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for multimodal anomaly detection and category discovery based on stable anomaly regions, characterized in that, Includes the following steps: S1) Acquire two-dimensional appearance image data of the industrial product to be inspected, and three-dimensional point cloud data corresponding to the space of the industrial product to be inspected; S2) Input the two-dimensional appearance image data into the two-dimensional appearance feature extractor to extract the two-dimensional appearance feature map, which includes global appearance features and local appearance features. S3) Based on the two-dimensional appearance feature map, calculate the anomaly significance distribution map to characterize the anomaly intensity at each spatial location; S4) Perform multi-threshold analysis on the anomaly significance distribution map to extract stable anomaly regions; S5) Based on the spatial location information of stable anomaly regions in the original two-dimensional appearance feature map, extract the regional-level global features corresponding to the stable anomaly regions from the global appearance features, and encode and fuse them with the local appearance features of the stable anomaly regions to construct a fused appearance feature representation containing local detail information and global contextual semantic information. S6) Using the two-dimensional region mask and the two-dimensional to three-dimensional mapping relationship, a subset of point clouds corresponding to the stable and abnormal region space is selected in the three-dimensional point cloud data, and the point cloud subset is input into the three-dimensional geometric feature extractor to extract the three-dimensional geometric feature representation corresponding to the stable and abnormal region. S7) Establish the spatial correspondence between the two-dimensional appearance image and the three-dimensional point cloud, and map the three-dimensional geometric features in the stable abnormal region to the two-dimensional space. At the same time, construct and train a bidirectional cross-modal feature mapping model that fuses appearance features and three-dimensional geometric features to achieve consistent alignment of multimodal features. S8) During the detection phase, the cross-modal structure deviation metric corresponding to the stable abnormal region is obtained by calculating the difference between the cross-modal predicted features and the actual features. S9) Construct a region-level anomaly description vector based on the fusion of appearance features, three-dimensional geometric features, and cross-modal structural deviation measures of stable anomaly regions; Then, unsupervised clustering analysis is performed on the anomaly description vectors corresponding to multiple stable anomaly regions. Based on the similarity relationship between the anomaly description vectors, they are automatically divided into several clustering results. Each clustering result corresponds to an anomaly self-discovery mode, thereby realizing the differentiation and summarization of different anomaly types.

2. The method for multimodal anomaly detection and category discovery based on stable anomaly regions according to claim 1, characterized in that, In S4), the specific steps for extracting stable anomaly regions include: S41) Set a threshold set and binarize the anomaly significance distribution map to obtain candidate anomaly regions under different threshold conditions; S42) Calculate the stability of the candidate anomaly region under different threshold conditions. When the stability is ≥ the stability threshold, the candidate anomaly region is determined to be a stable anomaly region. S43) Using the stable abnormal region as the center, the original two-dimensional appearance feature map is cropped to obtain the abnormal center region image, and the corresponding abnormal region mask is generated.

3. The method for multimodal anomaly detection and category discovery based on stable anomaly regions according to claim 2, characterized in that: In S41), the binarization process is performed using the following formula. In the formula, B τk (x,y) represents the candidate anomaly region under the set threshold condition. S(x,y) represents the distribution map of the significance of the anomalies. τ k This indicates that a threshold value is set.

4. The method for multimodal anomaly detection and category discovery based on stable anomaly regions according to claim 3, characterized in that: In S42), the stability is calculated by the following formula. In the formula, D(R) represents the stability of the candidate anomaly region under different threshold conditions. B τk This indicates candidate anomaly regions under a set threshold condition. R represents the set of region indices in a two-dimensional pixel space. Ⅱ(⋅) denotes the binarization indicator function. M represents the total number of candidate anomaly regions that meet the anomaly threshold condition.

5. The method for multimodal anomaly detection and category discovery based on stable anomaly regions according to claim 1, characterized in that: In S7), the specific implementation steps include: S71) The spatial points in the 3D point cloud are mapped to the pixel coordinate system of the 2D appearance image through the projection function; S72) Maps three-dimensional geometric features to two-dimensional space using a projection operator; S73) Construct a bidirectional cross-modal feature mapping model that integrates appearance features and three-dimensional geometric features. The bidirectional cross-modal feature mapping model includes a mapping model from integrated appearance features to three-dimensional geometric features, and a mapping model from three-dimensional geometric features to integrated appearance features.

6. The multimodal anomaly detection and category discovery method based on stable anomaly regions according to claim 5, characterized in that: In S71), the mapping relationship between spatial points in a 3D point cloud and a 2D appearance image is expressed by the following formula. In the formula, (X i ,Y i Z i () represents the spatial coordinates of the i-th point in a 3D point cloud. (u i ,v i () represents the two-dimensional image pixel coordinates corresponding to the spatial coordinates of the i-th point in the 3D point cloud. ∏(▪) represents the projection function.

7. The multimodal anomaly detection and category discovery method based on stable anomaly regions according to claim 6, characterized in that: In S72), the mapping process by which the projection operator maps three-dimensional geometric features to two-dimensional space is performed by the following formula. In the formula, H R This represents a two-dimensional geometric feature aligned with the two-dimensional appearance feature in terms of spatial resolution. < i>G R Represents the three-dimensional geometric features within a stable anomaly region. ψ(▪) represents the geometric feature mapping operator from three dimensions to two dimensions.

8. The method for multimodal anomaly detection and category discovery based on stable anomaly regions according to claim 1, characterized in that: In S8), the cross-modal structure deviation metric is expressed by the following formula. In the formula, E R This represents a measure of cross-modal structural deviation. Ω R Indicates a stable anomaly region. R represents the set of region indices in a two-dimensional pixel space. This represents the three-dimensional geometric features predicted from two-dimensional appearance features using a cross-modal prediction network. Represents three-dimensional geometric features aligned with two-dimensional appearance features in spatial resolution. This represents the fused features obtained by back-predicting geometric features through a cross-modal prediction network. This represents the fused appearance feature obtained by the feature fusion module.

9. The method for multimodal anomaly detection and category discovery based on stable anomaly regions according to claim 1, characterized in that: In S9), the anomaly description vector is represented by the following formula. In the formula, Q R Represents an anomaly description vector. Z R This indicates the fused appearance features corresponding to stable anomaly regions. < i>G R This represents the three-dimensional geometric features corresponding to stable anomaly regions. < i>E R This represents the cross-modal structural deviation metric corresponding to stable anomalous regions.

Images