A 3D zero-shot anomaly detection method based on a 2D multi-modal large model

Abstract

The application belongs to the field of 3D anomaly detection, and discloses a 3D zero-shot anomaly detection method based on a 2D multi-modal large model, comprising the following steps: step 1, obtaining auxiliary multi-modal point cloud data set; step 2, obtaining 2D global and local information representation; step 3, obtaining 3D implicit representation; step 4, obtaining 3D explicit representation; step 5, performing hierarchical prompt learning; step 6, performing implicit point cloud representation learning; step 7, performing explicit point cloud representation learning; step 8, performing cross-level comparison alignment; step 9, constructing a zero-shot anomaly detection framework and training; step 10, performing 3D zero-shot anomaly detection; step 11, judging the anomaly according to the anomaly score, and segmenting the abnormal area according to the anomaly score map. The application first realizes 3D zero-shot anomaly detection across semantic categories, does not require target domain training data, and solves the problem of limited deployment caused by data loss in traditional methods.

Description

Technical Field

[0001] This invention relates to the field of 3D anomaly detection, specifically a 3D zero-sample anomaly detection method based on a 2D multimodal large model. Background Technology

[0002] In recent years, 2D anomaly detection technology has been widely used in fields such as industrial quality inspection. However, its reliance on RGB information limits its ability to detect subtle spatial anomalies (such as geometric distortions and structural misalignments). In contrast, 3D anomaly detection, by analyzing the spatial relationships of point clouds, can more accurately capture hidden defects. However, existing 3D anomaly detection methods typically require the pre-construction of a normal sample feature library of the target object, and detection is achieved by comparing the deviation between the test sample and the feature library. Such methods face two major bottlenecks: first, the acquisition of target domain data is limited by privacy protection or the inaccessibility of real-world scenarios; second, the model's generalization ability for unknown categories is severely insufficient. Although research on 2D zero-shot anomaly detection based on visual language models such as CLIP has shown initial success, the 3D domain still faces the challenge of representational heterogeneity—how to effectively transfer the generalization ability of 2D pre-trained models to 3D space, while simultaneously considering the multi-dimensional semantic expression of texture and geometric anomalies, has become a technical barrier that urgently needs to be overcome. Summary of the Invention

[0003] The purpose of this invention is to provide a 3D zero-shot anomaly detection method based on a 2D multimodal large model, so as to solve the problems mentioned in the background art.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] A 3D zero-shot anomaly detection method based on a 2D multimodal large model includes the following steps:

[0006] Step 1: Obtain the auxiliary multimodal point cloud dataset;

[0007] Step 2: Obtain representations of 2D global and local information;

[0008] Step 3: Obtain the 3D implicit representation;

[0009] Step 4: Obtain 3D explicit representation;

[0010] Step 5: Provide tiered learning prompts;

[0011] Step 6: Perform implicit point cloud representation learning;

[0012] Step 7: Perform explicit point cloud representation learning;

[0013] Step 8: Perform cross-level comparison and alignment;

[0014] Step 9: Construct and train a zero-shot anomaly detection framework;

[0015] Step 10: Perform 3D zero-shot anomaly detection;

[0016] Step 11: Determine anomalies based on anomaly scores and segment the abnormal regions based on the anomaly score map.

[0017] Further, step 1 includes:

[0018] Step 1.1: Obtain the auxiliary point cloud dataset for anomaly detection, and denote the dataset as follows: Represents the i-th point cloud. This represents the true value of the i-th point cloud number. There are N point clouds in the dataset.

[0019] Step 1.2: Then, offline high-precision rendering is used to acquire the image data of the point cloud. Each point cloud and its ground truth value are rendered into K two-dimensional images from the corresponding K viewpoints. The k-th rendered image of the i-th point cloud is denoted as [image 1]. The corresponding truth rendering result is denoted as

[0020] The rendering method is as follows:

[0021] Given an auxiliary point cloud dataset The rendering matrix for the k-th viewpoint is defined as R. (k) There are K views in total. Point clouds and their corresponding ground truth values ​​are rendered simultaneously from different perspectives to obtain the corresponding 2D rendered image. The expression is:

[0022]

[0023] Abnormal pixels are marked as 1, and normal pixels are marked as 0.

[0024] Step 2 includes:

[0025] For each point cloud, the original visual encoder of the multimodal large model CLIP is used to render the image from the point cloud. In the process, extract global information representation in 2D. and 2D local information representation This represents the global features of the i-th point cloud and the k-th rendered image. This represents the local features of the i-th point cloud and the k-th rendered image.

[0026] Furthermore, in step 3, for each point cloud, 3D implicit global and local representations are obtained through rendering relationships, specifically as follows:

[0027] 3.1 3D Implicit Global Representation: For the i-th point cloud, K 2D global representations are combined to form a 3D implicit global representation, denoted as p. i p i The expression is:

[0028]

[0029] 3.2, 3D Implicit Local Representation: For the i-th point cloud, a 3D implicit local representation is obtained through K 2D local representations and the occlusion relationships of the original points in each rendered view, denoted as . The occlusion relationship between points is denoted as This indicates whether the j-th point of the i-th point cloud is visible in the k-th rendered image; 1 indicates visibility, and 0 indicates invisibility. The expression is:

[0030]

[0031] Where n represents the number of points in the i-th point cloud. Let represent the 3D implicit local representation of the k-th view of the i-th point cloud. (u,v)=R' (k) (a i,j |b i,j ,c i,j )} represents the feature of the j-th point in the k-th view of the i-th point cloud, and the 3D coordinates of this point are (a i,j ,b i,j ,c i,j ), It is an intermediate variable, R' (k) It's the rendering conversion between points and pixels, derived from... h and H represent rendering parameters;

[0032] definition Let be the set of points in the i-th point cloud that are projected as pixels at (u,v) in the k-th view. Then:

[0033]

[0034]

[0035] in It is an indicator function, where argmin represents the value of the argument that the function takes to reach its minimum value;

[0036] Finally, the 3D implicit local representation of the k-th view of the i-th point cloud is calculated by the following formula:

[0037]

[0038] Further, step 4 includes:

[0039] In obtaining the implicit local representation of the i-th 3D point cloud Then, based on the positional relationships of the original point clouds, the implicit representations are aggregated at the point level to obtain the 3D explicit global representation of the i-th point cloud, denoted as . Specifically:

[0040] Obtain the 3D implicit local representation through step 3. Afterwards, The features of each point are summed and averaged to obtain the aggregated features. Right now:

[0041]

[0042] use Representing all points in the i-th point cloud, using The coordinates of the j-th point in the i-th point cloud are represented by the following formula. Select the k nearest neighbors (excluding the current point), and use each point as... This is represented as:

[0043]

[0044] in, As an intermediate quantity, This indicates the calculation of the Euclidean distance between two points, and TopK(·) represents the points whose values ​​are in the first k positions;

[0045] Then, use the Gaussian kernel. Calculate the Gaussian distance between two points, where exp represents the exp function, and σ is an adjustable parameter representing the normalized Gaussian distance. for:

[0046]

[0047] Then the neighborhood features of the j-th point in the i-th point cloud are:

[0048]

[0049] in Let represent the set of neighborhood features of the j-th point in the i-th point cloud.

[0050] Finally, the original features and the domain features are weighted to obtain the final explicit local representation of the j-th point in the i-th point cloud.

[0051]

[0052] Where α is an adjustable parameter.

[0053] Further, step 5 includes:

[0054] Step 5.1: Design dual semantic rendering prompts for point clouds. and geometric prompts Among them, rendering prompts The normal semantics are denoted as Abnormal semantics are denoted as Geometric prompts The normal semantics are denoted as Abnormal semantics are denoted as and The expression is:

[0055]

[0056] in, These are learnable embedding words, where E represents the number of learnable parameters, [object] represents the text embedding of an object, [damage] represents the embedding of an abnormal object, and [point][cloud] represents a point cloud.

[0057] Step 5.2, then encode the text of the design using the original CLIP text encoder. of and Encoded and recorded as and of and Encoded and recorded as and

[0058] Further, step 6 includes:

[0059] Step 6.1, use cross-entropy to calculate the 3D global loss of the implicit representation, denoted as . The calculated anomaly score map and normal score map of the point cloud are denoted as follows: and The local loss of the 3D implicit representation is calculated and denoted as . The expression is:

[0060]

[0061] in, This indicates that normal text features and abnormal text features are calculated and rendered separately. The cosine similarity is used to determine the probability that it belongs to an anomaly. max means taking the maximum value, and CrossEntropy() means calculating the cross-entropy.

[0062] and The expression is:

[0063]

[0064] The expression is:

[0065]

[0066] Where Dice() represents the calculation of the Dice loss function, and I represents the identity matrix.

[0067] Step 6.2, calculate the 2D global and local losses of the pixel, denoted as follows: and Will The summation yields the implicit representation fusion loss of the point cloud, denoted as .

[0068] The expression is:

[0069]

[0070] The expression is:

[0071]

[0072] Where Focal() represents the calculation of the focal loss function. This indicates a splicing operation.

[0073] in Let represent the normal score image of the k-th rendered image of the i-th point cloud. The portion of the image that indicates its abnormality.

[0074] The expression is:

[0075]

[0076] Further, step 7 includes:

[0077] The calculated anomaly score map and normal score map of the point cloud are denoted as follows: and The local loss of the 3D explicit representation is calculated and denoted as... The expression is:

[0078]

[0079] Step 8 includes:

[0080] The cross-loss of the cue word encoding is calculated using cross-entropy, denoted as L. cross L cross The expression is:

[0081]

[0082] Here, CE() represents calculating the cross-entropy, and cos represents the cosine function.

[0083] Further, step 9 includes:

[0084] The implicit representation mixing loss of the point cloud obtained in step 6, the local loss of the 3D explicit representation obtained in step 7, and the crossover loss of the cue word encoding obtained in step 8 are summed to obtain the final loss function, denoted as L. hierarchical During training, the original parameters of the CLIP model are frozen, and the loss function L is minimized. hierarchical To optimize learnable text prompts, the multimodal large model CLIP can effectively capture cross-category 3D generalization anomaly patterns.

[0085] Further, step 10 includes:

[0086] After training is complete, the zero-shot anomaly detection framework performs zero-shot inference based on point clouds, or multimodal 3D inference incorporating RGB color information, as follows:

[0087] Zero-shot inference based on point clouds: After inputting a point cloud, implicit and explicit representations of the point cloud are obtained through a zero-shot anomaly detection framework. Cosine similarity is then calculated between these representations and encoded bisemantic rendering and bisemantic geometric prompts, respectively, to obtain anomaly score maps. and By combining the two anomaly score maps, the final anomaly score map of the point cloud is obtained, denoted as . By combining the global features and text encoding of the rendered graph with the maximum value in the anomaly score graph, the global anomaly score is obtained, denoted as . The expression is:

[0088]

[0089] Among them G σ (·) represents a Gaussian filter. This represents the cosine similarity between the abnormal point cloud warning text and each point in the point cloud;

[0090] The expression is:

[0091]

[0092] P(·) represents the cosine similarity calculation operation, g c F represents the semantic text encoding of anomalies. i A set representing global features;

[0093] Multimodal 3D inference incorporating RGB color information: The zero-shot anomaly detection framework directly feeds the 2D RGB image into the 2D branch to obtain 2D features, thereby fusing color information; then, it projects the 2D features back into 3D space to calculate the anomaly score map and anomaly score of the color information, denoted as [reference needed]. and use The original color information anomaly score map is represented; the final multimodal anomaly score map and anomaly score are denoted as... and and The expression is:

[0094]

[0095] Furthermore, the abnormality judgment and abnormal region segmentation in step 11 specifically refer to judging whether the detected object has an abnormality based on the abnormality score, and segmenting the abnormal region of the detected object based on the abnormality score map.

[0096] Compared with the prior art, the beneficial effects of the present invention are:

[0097] 1) Breakthrough in zero-shot generalization capability: For the first time, zero-shot anomaly detection in 3D across semantic categories is achieved without the need for training data in the target domain, solving the deployment limitation problem caused by data shortage in traditional methods;

[0098] 2) Multimodal collaborative characterization innovation: Through an implicit-explicit dual-branch architecture, surface texture anomalies (such as scratches and corrosion) and geometric anomalies (such as deformation and assembly misalignment) are captured simultaneously;

[0099] 3) Dynamic scalability design: Supports plug-and-play fusion of RGB information;

[0100] 4) Training efficiency optimization: During training, the CLIP backbone network parameters are frozen, and only the lightweight prompt word embedding module is optimized. Attached Figure Description

[0101] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0102] Figure 2 This is a schematic diagram of the zero-shot detection inference process based on point clouds in this invention;

[0103] Figure 3This is a schematic diagram of the multimodal reasoning of the present invention. Detailed Implementation

[0104] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0105] Please see Figures 1-3 A method and system for detecting zero-shot anomalies in 3D, or based on a 2D multimodal large model, characterized by comprising the following steps:

[0106] Step 1, Obtaining the Auxiliary Multimodal Point Cloud Dataset: Based on experimental needs, obtain the auxiliary point cloud dataset for anomaly detection. Denote the dataset as... Represents the i-th point cloud. Let represent the ground truth value of the i-th point cloud. The dataset contains N point clouds. Then, offline high-precision rendering is used to acquire the image data of the point clouds. Each point cloud and its ground truth value are rendered into K 2D images from K corresponding viewpoints. The k-th rendered image of the i-th point cloud is denoted as . The corresponding truth rendering result is denoted as

[0107] The auxiliary multimodal point cloud dataset and rendering method are as follows:

[0108] Given an auxiliary point cloud dataset The rendering matrix for the k-th viewpoint is defined as R. (k) There are K views in total. Point clouds and their corresponding ground truth values ​​are rendered simultaneously from different perspectives to obtain the corresponding 2D rendered image. Specifically... and in, and This represents the ground truth value of the k-th rendered image corresponding to the i-th point cloud and the corresponding pixel level. Abnormal pixels are marked as 1, and normal pixels are marked as 0.

[0109] Step 2, Representation of 2D Global and Local Information: For each point cloud, the original visual encoder is used with a multimodal large model CLIP, from which the rendered image is derived. In the process, extract global information representation in 2D. and 2D local information representation This represents the global features of the i-th point cloud and the k-th rendered image. This represents the local features of the i-th point cloud and the k-th rendered image.

[0110] Among them, 2D global features reflect the global features of points in the rendering graph, while 2D local features can reflect the features of each point in the rendering graph.

[0111] Step 3, Obtaining 3D Implicit Representations: For each point cloud, after obtaining its 2D global and local representations, the 3D implicit global and local representations are obtained through rendering relationships, specifically as follows:

[0112] Step 3.1, 3D Implicit Global Representation: For the i-th point cloud, K 2D global representations are combined to form a 3D implicit global representation, denoted as p. i .

[0113] Step 3.2, 3D Implicit Local Representation: For the i-th point cloud, a 3D implicit local representation is obtained through K 2D local representations and the occlusion relationships of the original points in each rendered view, denoted as . The occlusion relationship between points is denoted as This indicates whether the j-th point of the i-th point cloud is visible in the k-th rendered image. Visible is represented by 1, and invisible is represented by 0.

[0114] The specific method for obtaining 3D implicit representation is as follows:

[0115] (1) Method for obtaining 3D implicit global representation:

[0116]

[0117] (2) Methods for obtaining 3D implicit local representations:

[0118]

[0119] Where n represents the number of points in the i-th point cloud. Let k be the 3D implicit local representation of the k-th view of the i-th point cloud. (u,v)=R' (k) (a i,j |b i,j ,c i,j )} represents the feature of the j-th point in the k-th view of the i-th point cloud, and the 3D coordinates of this point are (a i,j ,b i,j ,c i,j ). It is an intermediate variable, R' (k) It's the rendering conversion between points and pixels, derived from... h and H represent rendering parameters.

[0120] definition Let be the set of points in the i-th point cloud that are projected as pixels at (u,v) in the k-th view. Then:

[0121]

[0122]

[0123] Finally, the 3D implicit local representation of the k-th view of the i-th point cloud is calculated by the following formula:

[0124] Step 4, Obtaining the 3D explicit representation: This involves obtaining the implicit local representation of the i-th 3D point cloud. Then, based on the positional relationships of the original point clouds, the implicit representations are aggregated at the point level to obtain the 3D explicit global representation of the i-th point cloud, denoted as .

[0125] The specific method for obtaining 3D explicit representations is as follows:

[0126] The 3D implicit local representation is obtained through step 3. Afterwards, The features of each point are summed and averaged to obtain the aggregated features. Right now:

[0127]

[0128] use Representing all points in the i-th point cloud, using The coordinates of the j-th point in the i-th point cloud are represented by the following formula. Select the k nearest neighbors (excluding the current point), and use each point as... This is represented as:

[0129]

[0130] in, As an intermediate quantity, This indicates that the Euclidean distance between two points is calculated, and TopK(·) represents the points whose values ​​are in the first k positions.

[0131] Then, use the Gaussian kernel. Calculate the Gaussian distance between two points, where exp represents the exp function, and σ is an adjustable parameter representing the normalized Gaussian distance. for:

[0132]

[0133] Then the neighborhood features of the j-th point in the i-th point cloud are:

[0134]

[0135] in Let represent the set of neighborhood features of the j-th point in the i-th point cloud.

[0136] Finally, the original features and the domain features are weighted to obtain the final explicit local representation of the j-th point in the i-th point cloud.

[0137]

[0138] Step 5, Layered Cue Learning: Design dual semantic rendering cue words for point clouds separately. and geometric prompts Among them, rendering prompts The normal semantics are denoted as Abnormal semantics are denoted as Geometric prompts The normal semantics are denoted as Abnormal semantics are denoted as The design text is then encoded using the original CLIP text encoder. of and Encoded and recorded as and of and Encoded and recorded as and

[0139] The designed tiered learning prompts are as follows:

[0140]

[0141] in, These are learnable embeddings, where E represents the number of learnable parameters, [object] represents the text embedding of an object, [damage] represents the embedding of an abnormal object, and [point][cloud] represents a point cloud.

[0142] Step 6, Implicit Point Cloud Representation Learning: This involves obtaining the implicit global representation p of the point cloud. i and local representation and rendering prompts Then, the 3D global loss of the implicit representation is calculated using cross-entropy, denoted as . The calculated anomaly score map and normal score map of the point cloud are denoted as follows: and The local loss of the 3D implicit representation is calculated and denoted as . Finally, the 2D global and local losses of the pixel are calculated and denoted as follows: and Adding the above four losses together yields the implicit representation mixture loss of the point cloud, denoted as .

[0143] The loss function is calculated as follows:

[0144] (1) Global 3D implicit representation loss:

[0145]

[0146] in, This indicates that normal text features and abnormal text features are calculated and rendered separately. The cosine similarity is used to determine the probability that it belongs to an anomaly. max means taking the maximum value, and CrossEntropy() means calculating the cross-entropy.

[0147] (2) The abnormal and normal score charts are obtained as follows:

[0148]

[0149] (3) Local 3D implicit representation loss:

[0150]

[0151] Where Dice() represents the calculation of the Dice loss function, and I represents the identity matrix.

[0152] (4) The method for obtaining 2D global features is as follows:

[0153]

[0154] (5) The method for obtaining 2D local features is as follows:

[0155]

[0156] Where Focal() represents the calculation of the focal loss function. This indicates a splicing operation.

[0157] in Let represent the normal score image of the k-th rendered image of the i-th point cloud. The portion of the image that indicates its abnormality.

[0158] Finally, the loss function for the hybrid 3D implicit representation is:

[0159]

[0160] Step 7, Explicit Point Cloud Representation Learning: This involves acquiring explicit representations of the point cloud. and geometric prompts Then, the anomaly score map and normal score map of the point cloud are calculated and denoted as follows: and The local loss of the 3D explicit representation is calculated and denoted as...

[0161] The local loss of the 3D explicit representation is specifically calculated as follows:

[0162]

[0163] Step 8, Cross-layer contrast alignment: After obtaining the encoding of the rendering tooltip and geometric tooltip in Step 5... and and and Then, the cross-loss of the cue word encoding is calculated using cross-entropy, denoted as L. cross .

[0164] The cross-loss of the cue word encoding is specifically calculated as follows:

[0165]

[0166] Here, CE() represents calculating the cross-entropy, and cos represents the cosine function.

[0167] Step 9, construct and train the zero-shot anomaly detection framework: sum the three loss functions obtained in steps 6 to 8 to obtain the final loss function, denoted as L. hierarchical During training, the original parameters of the CLIP model are frozen, and the loss function L is minimized. hierarchical To optimize learnable text prompts, the CLIP model is able to effectively capture cross-category 3D generalization anomaly patterns.

[0168] The specific training process is as follows:

[0169] The original parameters of the CLIP model are frozen to preserve its strong generalization ability. By optimizing the two-layer learnable cue text through minimizing the mixture loss function, the CLIP model can effectively capture cross-class 3D generalized anomalous patterns. Ultimately, this enables the bounding box to understand anomalous semantics in point clouds and pixels.

[0170] Step 10, Detection Inference Process: After training is complete, this framework can perform zero-shot inference based on point clouds, or multimodal 3D inference incorporating RGB color information, as detailed below:

[0171] Step 10.1, Zero-shot inference based on point clouds: After inputting the point cloud, the implicit and explicit representations of the point cloud are obtained through the framework. Then, cosine similarity is calculated with the encoded bisemantic rendering prompts and bisemantic geometric prompts to obtain the anomaly score map. and By combining the two anomaly score maps, the final anomaly score map of the point cloud is obtained, denoted as . Furthermore, by combining the global features and text encoding of the rendered graph with the maximum value in the anomaly score graph, the global anomaly score is obtained, denoted as .

[0172] Step 10.2, Multimodal 3D Inference with RGB Color Information: This framework can directly feed 2D RGB images into the 2D branch to obtain 2D features, thereby fusing color information. Then, the 2D features are projected back into 3D space to calculate the anomaly score map and anomaly score of the color information, denoted as follows: and The final multimodal anomaly score plot and anomaly score are denoted as: and

[0173] The specific calculation process of the anomaly score graph and the anomaly score in the two inference processes is as follows:

[0174] (1) Zero-shot inference based on point clouds

[0175] The abnormal score graph is as follows:

[0176]

[0177] Among them G σ (·) represents a Gaussian filter. This represents the cosine similarity between the abnormal point cloud warning text and each point in the point cloud.

[0178] The global anomaly score integrates global and local anomaly semantics, and its calculation formula is as follows:

[0179]

[0180] P(·) represents the cosine similarity calculation operation, g c F represents the semantic text encoding of anomalies. i A set representing global features.

[0181] (2) Multimodal 3D inference with added RGB color information

[0182] When color information is available, a two-dimensional color image can be directly fed into a two-dimensional branch to obtain two-dimensional features, thereby fusing the color information. Then, the two-dimensional features are projected back into three-dimensional space to calculate the color anomaly map.

[0183]

[0184] Color anomaly score:

[0185]

[0186] use This represents the original color information anomaly score map.

[0187] The final multimodal anomaly score plot and anomaly scores are as follows:

[0188]

[0189] Step 11, Anomaly Judgment and Segmentation: Finally, anomalies are judged based on anomaly scores, and anomaly regions are segmented based on the anomaly score map.

[0190] Specifically, the judgment of anomalies and the segmentation of abnormal regions refer to judging whether the detected object has an anomaly based on the anomaly score, and segmenting the abnormal region of the detected object based on the anomaly score map.

[0191] Example

[0192] This embodiment conducts training and zero-shot detection experiments on the industrial anomaly point cloud dataset MVTec3D-AD. This dataset contains 4171 point cloud data points for 10 object types, along with RGB information and ground truth annotations. Details of the dataset are shown in Table 1.

[0193] Table 1. Detailed information about the MVTec 3D-AD dataset.

[0194]

[0195] In this embodiment, the 3D zero-shot anomaly detection method based on a 2D multimodal large model was tested on the aforementioned dataset. The results show that the method achieves zero-shot anomaly detection and localization of 3D point clouds and multimodal zero-shot anomaly detection and localization. The detailed implementation steps are as follows:

[0196] Step 1: Re-divide the MVTec3D-AD point cloud dataset shown in Table 1 to obtain an auxiliary dataset. Specifically, use the point cloud data from the original training, validation, and test sets in the cookie category as auxiliary point clouds, and use all other data for testing to evaluate the effectiveness of zero-shot detection. The auxiliary point cloud dataset used for training in this example has N=363, consisting of 260 normal point clouds and 103 abnormal point clouds. Then, Δ... 3dEach point cloud in the image is rotated along the X-axis by nine different angles using the Open3D library, and a 336x336 image is rendered for each angle. The nine angles are... The rendered point cloud image and its corresponding ground truth image are denoted as... and i = 1, 2...363 represents the nth point cloud, and k = 1, 2...9 represents the nth rendering image.

[0197] Step 2: After obtaining the rendered image in Step 2, the visual encoder of CLIP's original transformer architecture is used to encode the rendered image into feature vectors. Specifically, the... Encode as a global feature vector and local feature vectors The global feature vector reflects the common information of all points in the rendered image, while the local feature vector reflects the information of each point in the rendered image.

[0198] Step 3: After obtaining the 2D features of the rendered image in Step 3, combine the 9 2D global features of the 9 rendered images of each point cloud to obtain the 3D implicit global feature p of the 3D point cloud. i ,Right now:

[0199]

[0200] Simultaneously, by combining the nine 2D local features from the nine rendered images of each point cloud, the explicit 3D local features of the 3D point cloud are initially obtained. Right now:

[0201]

[0202] Where n represents the number of points in each point cloud. This represents the local feature of the j-th point in the i-th point cloud in the k-th rendered image. Next, the occlusion relationships of the point clouds in each rendered image are calculated:

[0203]

[0204] Where (·) is an indicator function. The calculated... When, it means that the j-th point of the i-th point cloud is not occluded in the k-th rendered image. This indicates that it is obscured.

[0205] The final 3D local features of the k-th view of the i-th point cloud are calculated by the following formula:

[0206] Step 4: Obtain the implicit local features of the point cloud. Then, the features are summed and averaged along the point dimension to obtain a feature for each point in the point cloud. Right now:

[0207]

[0208] Then, using The coordinates of the j-th point in the i-th point cloud are represented by the following formula. Select the k nearest neighbors other than the given point, denoted as:

[0209]

[0210] in This indicates the calculation of the Euclidean distance between two points.

[0211] Then, use the Gaussian kernel. Calculate the Gaussian distance between two points. The normalized Gaussian distance is:

[0212]

[0213] Then the final neighborhood feature of the j-th point in the i-th point cloud is:

[0214]

[0215] Finally, the original features and domain features are weighted to obtain the final explicit representation of the j-th point in the i-th point cloud.

[0216]

[0217] Step 5, the designed tiered learning prompts are as follows:

[0218]

[0219]

[0220] V and W are learnable embedded words, encoded using the original CLIP text encoder.

[0221] Step 6, calculate the loss function for the implicit representation as follows:

[0222]

[0223] in:

[0224]

[0225] Step 7: Calculate the explicit representation loss using the following formula:

[0226]

[0227] in:

[0228]

[0229] Step 8, calculate the cross-loss of the text encoding as follows:

[0230]

[0231] Step 9, construct and train a zero-shot anomaly detection framework, including:

[0232] The original parameters of the CLIP model are frozen to preserve its strong generalization ability. By optimizing the learnable textual cues through minimizing the mixture loss function, the CLIP model is enabled to effectively capture cross-class 3D generalized anomalous patterns. Ultimately, this allows the bounding box to understand anomalous semantics in point clouds and pixels.

[0233] Step 10: After training the learnable text prompts through Step 9, conduct two types of reasoning tests:

[0234] (1) Without including RGB information, point cloud data excluding the cookie category is used as test samples in the dataset shown in Table 1. The specific implementation method during testing is as follows: Figure 2 As shown:

[0235] After obtaining the two anomaly score maps using the methods described in steps 6 and 7, the overall anomaly score map is calculated as follows:

[0236]

[0237] Among them G σ (·) represents a Gaussian filter.

[0238] The global anomaly score integrates global and local anomaly semantics, and its calculation formula is as follows:

[0239]

[0240] (2) Incorporate RGB information into inference, as detailed in the following implementation method. Figure 3 As shown: Using the method described in step 2, global and local 2D visual features are extracted from the original RGB image, and 3D features are obtained using the methods shown in steps 3 and 4. Next, the trained prompt text is encoded using the methods described in steps 6 and 7. Finally, the color anomaly score map and color anomaly score are calculated using the following formula.

[0241] Specific color anomaly diagram:

[0242]

[0243] Color anomaly score:

[0244]

[0245] The final multimodal anomaly score plot and anomaly scores are as follows:

[0246]

[0247] Step 11: Statistical analysis of the inference results described in Step 9 yields the following final test results: I-AUROC index 82.5%, AP index 94.8%, P-AUROC index 96.9%, and AUPRO index 88.7%. After incorporating RGB information, the I-AUROC index becomes 87.7%, the AP index 96.6%, the P-AUROC index 98.2%, and the AUPRO index 93.1%.

[0248] Meanwhile, by drawing a heatmap using anomaly fractions, it was observed that the abnormal areas could be well segmented in the heatmap.

[0249] This invention presents a zero-shot anomaly detection method for 3D point clouds based on a 2D multimodal large model. It achieves cross-modal detection without requiring target domain data through multimodal representation alignment and hierarchical semantic adaptation mechanisms. The overall process is as follows: Figure 1 As shown, the specific process includes: performing multi-view offline rendering of 3D point clouds to generate 2D images, and constructing a cross-modal mapping relationship between point clouds and images; extracting global and local features from the 2D images, as well as the geometric structure and spatial distribution features of the 3D point clouds, and achieving dual-branch modeling of implicit and explicit anomalies through hybrid representation learning; designing learnable rendering cue words and geometric cue words, and combining them with a pre-trained text encoder to generate semantically adapted anomaly discrimination rules; optimizing cross-modal feature alignment based on contrastive loss; and after training, determining the abnormal state of the tested object by fusing multimodal anomaly scores and generating an anomaly region segmentation map. This method supports single 3D point cloud input or multimodal input combined with RGB data, and can simultaneously complete surface defect detection and spatial anomaly localization. Figure 2 , Figure 3 This document outlines the point cloud-based zero-shot detection inference process and the zero-shot multimodal zero-shot anomaly detection inference process of this method. After training, the method can comprehensively determine whether the tested object has anomalies based on anomaly scores and use anomaly score maps to segment abnormal regions, providing a guarantee for subsequent anomaly detection in industrial production processes.

[0250] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A 3D zero-shot anomaly detection method based on a 2D multimodal large model, characterized in that, Includes the following steps: Step 1: Obtain the auxiliary multimodal point cloud dataset; Step 2: Obtain representations of 2D global and local information; Step 3: Obtain the 3D implicit representation; Step 4, obtain the 3D explicit representation, including: In obtaining the implicit local representation of the i-th 3D point cloud Then, based on the positional relationships of the original point clouds, the implicit representations are aggregated at the point level to obtain the 3D explicit global representation of the i-th point cloud, denoted as . Specifically: Obtain the 3D implicit local representation through step 3. Afterwards, The features of each point are summed and averaged to obtain the aggregated features. ,Right now: ， use Representing all points in the i-th point cloud, using The coordinates of the j-th point in the i-th point cloud are represented by the following formula. Select the k nearest neighbors (excluding the current point), and use each point as... This is represented as: ， in, As an intermediate quantity, This indicates the calculation of the Euclidean distance between two points, and TopK(·) represents the points whose values ​​are in the first k positions; Then, use the Gaussian kernel. Calculate the Gaussian distance between two points, where exp represents the exp function, and σ is an adjustable parameter representing the normalized Gaussian distance. for: ， Then the neighborhood features of the j-th point in the i-th point cloud are: ， in This represents the set of neighborhood features for the j-th point in the i-th point cloud; Finally, the original features and the domain features are weighted to obtain the final explicit local representation of the j-th point in the i-th point cloud. : ， Where α is an adjustable parameter; Step 5: Provide tiered learning prompts; Step 6: Perform implicit point cloud representation learning; Step 7: Perform explicit point cloud representation learning; Step 8: Perform cross-level comparison and alignment; Step 9: Construct and train a zero-shot anomaly detection framework; Step 10: Perform 3D zero-shot anomaly detection; Step 11: Determine anomalies based on anomaly scores and segment the abnormal regions based on the anomaly score map.

2. The 3D zero-shot anomaly detection method based on a 2D multimodal large model according to claim 1, characterized in that, Step 1 includes: Step 1.1: Obtain the auxiliary point cloud dataset for anomaly detection, and denote the dataset as follows: , Represents the i-th point cloud. This represents the true value of the i-th point cloud number. There are N point clouds in the dataset. Step 1.2: Then, offline high-precision rendering is used to acquire the image data of the point cloud. Each point cloud and its ground truth value are rendered into K two-dimensional images from the corresponding K viewpoints. The k-th rendered image of the i-th point cloud is denoted as [image 1]. The corresponding truth rendering result is denoted as ; The rendering method is as follows: Given an auxiliary point cloud dataset The rendering matrix for the k-th viewpoint is defined as R. (k) There are K views in total. Point clouds and their corresponding ground truth values ​​are rendered simultaneously from different perspectives to obtain the corresponding 2D rendered image. The expression is: ， ， Abnormal pixels are marked as 1, and normal pixels are marked as 0; Step 2 includes: For each point cloud, the original visual encoder of the multimodal large model CLIP is used to render the image from the point cloud. In the process, extract global information representation in 2D. and 2D local information representation , This represents the global features of the i-th point cloud and the k-th rendered image. This represents the local features of the i-th point cloud and the k-th rendered image.

3. The 3D zero-shot anomaly detection method based on a 2D multimodal large model according to claim 1, characterized in that, In step 3, for each point cloud, 3D implicit global and local representations are obtained through rendering relationships, specifically as follows: Step 3.1, 3D Implicit Global Representation: For the i-th point cloud, K 2D global representations are combined to form a 3D implicit global representation, denoted as p. i p i The expression is: ， Step 3.2, 3D Implicit Local Representation: For the i-th point cloud, a 3D implicit local representation is obtained through K 2D local representations and the occlusion relationships of the original points in each rendered view, denoted as . The occlusion relationship between points is denoted as , indicating whether the j-th point of the i-th point cloud is visible in the k-th rendered image, with 1 indicating visibility and 0 indicating invisibility. The expression is: ， Where n represents the number of points in the i-th point cloud. Let represent the 3D implicit local representation of the k-th view of the i-th point cloud. This represents the feature of the j-th point in the k-th view of the i-th point cloud, where the 3D coordinates of this point are... , It is an intermediate variable. It's the rendering conversion between points and pixels, derived from... h and H represent rendering parameters; definition Let be the set of points in the i-th point cloud that are projected as pixels at (u,v) in the K-th view. Then: ， ， in It is an indicator function, where argmin represents the value of the argument that the function takes to reach its minimum value; Finally, the 3D implicit local representation of the k-th view of the i-th point cloud is calculated by the following formula: .

4. The 3D zero-shot anomaly detection method based on a 2D multimodal large model according to claim 1, characterized in that, Step 5 includes: Step 5.1: Design dual semantic rendering prompts for point clouds. and geometric prompts Among them, rendering prompts The normal semantics are denoted as Abnormal semantics are denoted as Geometric prompts The normal semantics are denoted as Abnormal semantics are denoted as , and The expression is: ， ， in, These are learnable embedding words, where E represents the number of learnable parameters, [object] represents the text embedding of an object, [damage] represents the embedding of an abnormal object, and [point][cloud] represents a point cloud. Step 5.2, then encode the text of the design using the original CLIP text encoder. of and Encoded and recorded as and ; of and Encoded and recorded as and .

5. The 3D zero-shot anomaly detection method based on a 2D multimodal large model according to claim 4, characterized in that, Step 6 includes: Step 6.1, use cross-entropy to calculate the 3D global loss of the implicit representation, denoted as . The abnormal score map and normal score map of the point cloud are calculated and denoted as follows: and And calculate the local loss of the 3D implicit representation, denoted as . , The expression is: ， in, This indicates that normal text features and abnormal text features are calculated and rendered separately. The cosine similarity is used to determine the probability that it belongs to an anomaly. max means taking the maximum value, and CrossEntropy() means calculating the cross-entropy. and The expression is: ， ， The expression is: ， Where Dice() represents the calculation of the Dice loss function, and I represents the identity matrix; Step 6.2, calculate the 2D global and local losses of the pixel, denoted as follows: and ,Will , , , The summation yields the implicit representation fusion loss of the point cloud, denoted as . ; The expression is: ， The expression is: ， Where Focal() represents the calculation of the focal loss function. Indicates a splicing operation; in Let represent the normal score image of the k-th rendered image of the i-th point cloud. The portion of the image indicating its abnormality; The expression is: 。 6. The 3D zero-shot anomaly detection method based on a 2D multimodal large model according to claim 5, characterized in that, Step 7 includes: The calculated anomaly score map and normal score map of the point cloud are denoted as follows: and And calculate the local loss of the 3D explicit representation, denoted as , The expression is: ， ， ； Step 8 includes: The cross-loss of the cue word encoding is calculated using cross-entropy, denoted as L. cross L cross The expression is: ， Here, CE() represents calculating the cross-entropy, and cos represents the cosine function.

7. The 3D zero-shot anomaly detection method based on a 2D multimodal large model according to claim 6, characterized in that, Step 9 includes: The implicit representation mixing loss of the point cloud obtained in step 6, the local loss of the 3D explicit representation obtained in step 7, and the crossover loss of the cue word encoding obtained in step 8 are summed to obtain the final loss function, denoted as L. hierarchical During training, the original parameters of the CLIP model are frozen, and the loss function L is minimized. hierarchical To optimize learnable text prompts, the multimodal large model CLIP can effectively capture cross-category 3D generalization anomaly patterns.

8. The 3D zero-shot anomaly detection method based on a 2D multimodal large model according to claim 7, characterized in that, Step 10 includes: After training is complete, the zero-shot anomaly detection framework performs zero-shot inference based on point clouds, or multimodal 3D inference incorporating RGB color information, as follows: Zero-shot inference based on point clouds: After inputting a point cloud, implicit and explicit representations of the point cloud are obtained through a zero-shot anomaly detection framework. Cosine similarity is then calculated between these representations and encoded bisemantic rendering and bisemantic geometric prompts, respectively, to obtain anomaly score maps. and By combining the two anomaly score maps, the final anomaly score map of the point cloud is obtained, denoted as... By combining the global features and text encoding of the rendered graph with the maximum value in the anomaly score graph, the global anomaly score is obtained, denoted as . , The expression is: ， Among them G σ (·) represents a Gaussian filter. This represents the cosine similarity between the abnormal point cloud warning text and each point in the point cloud; The expression is: ， P(·) represents the cosine similarity calculation operation, g c F represents the semantic text encoding of anomalies. i A set representing global features; Multimodal 3D inference incorporating RGB color information: The zero-shot anomaly detection framework directly feeds the 2D RGB image into the 2D branch to obtain 2D features, thereby fusing color information; then, it projects the 2D features back into 3D space to calculate the anomaly score map and anomaly score of the color information, denoted as [reference needed]. and ,use The original color information anomaly score map is represented; the final multimodal anomaly score map and anomaly score are denoted as... and , , , and The expression is: ， ， ， 。 9. The 3D zero-shot anomaly detection method based on a 2D multimodal large model according to claim 1, characterized in that, The abnormality judgment and abnormal region segmentation in step 11 specifically refer to judging whether the detected object has an abnormality based on the abnormality score, and segmenting the abnormal region of the detected object based on the abnormality score map.

Images