A zero-shot point cloud anomaly detection method and system considering prompt learning
Through multi-view projection and learnable parameter adjustment methods, the "cold start" problem of zero-sample point cloud abnormal detection in industrial vision detection is solved, and accurate detection in zero-sample situation is achieved, data acquisition costs are reduced, and data collection costs are filled. Vacancies in point cloud visual language model.
Patent Information
- Application Number
- CN202410359413.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-27
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-03-27
AI Technical Summary
In industrial vision detection, it is difficult to accurately detect point cloud abnormalities in zero samples, and the cost of point cloud data acquisition is high, resulting in the "cold start" problem being difficult to solve.
Convert three-dimensional point clouds into depth images through multi-view projection, extract features using image encoder and aggregate them into point clouds, calculate abnormal scores based on the cosine similarity of text encoder, and adjust the image encoder and text prompts using learning parameters to adjust the image encoder and text prompts to Break the domain gap to realize point cloud abnormality detection.
Accurate detection of point cloud anomalies is achieved in the zero-sample situation, filling the gap in the point cloud visual language model, improving the accuracy and generalization performance of detection, and reducing data acquisition costs.
Smart Images

Figure CN118052809B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of point cloud anomaly detection, and more specifically, relates to a zero-sample point cloud anomaly detection method and system considering prompt learning. Background Art
[0002] In today's industrial production, it is crucial to ensure product quality. In this context, the application of industrial visual inspection technology has become particularly important. Industrial visual inspection technology uses an automated visual system to inspect products to identify defects, flaws or products that do not meet specifications, thereby ensuring that product quality meets standards. In recent years, anomaly detection technology has been widely used in surface defect quality inspection of industrial products. In industrial visual inspection, by analyzing the surface image of the product, anomaly detection can identify a variety of defects such as scratches, dents, impurities, etc., so that defective products can be eliminated in the early stages of production. However, although image anomaly detection technology performs well in many aspects, it has limitations when dealing with certain complex scenes. For example, for objects with unclear texture, it is difficult for the image to accurately capture its surface details, and image anomaly detection will fail. In contrast, point cloud data can provide three-dimensional geometric information of objects, making detection more comprehensive and accurate.
[0003] Despite the obvious advantages of point cloud anomaly detection, it still faces a series of challenges in practical applications. The most notable of these is the so-called "cold start" problem and the high cost of point cloud data collection. The cold start problem refers to the inability to collect enough labeled data to effectively train deep learning models in new or unknown environments. The collection of point cloud data usually requires expensive equipment and a lot of time, which further limits the feasibility of large-scale data collection, especially in the ever-changing real-world industrial environments.
[0004] In view of these limitations, there is an urgent need to develop methods for zero-shot point cloud anomaly detection, that is, to develop methods that can complete anomaly detection without collecting data on the object to be tested. At present, zero-shot image anomaly detection has made some progress with the help of visual language models, but due to the lack of visual language models designed specifically for point clouds, it is unable to cope with the "cold start" problem of industrial point cloud anomaly detection and the high cost of point cloud acquisition. Summary of the invention
[0005] In view of the above defects or improvement needs of the prior art, the present invention provides a zero-sample point cloud anomaly detection method and system taking prompt learning into consideration, the purpose of which is to achieve accurate detection of point cloud anomalies in the case of zero samples.
[0006] To achieve the above object, according to a first aspect of the present invention, a zero-sample point cloud anomaly detection method considering prompt learning is proposed, comprising the following steps:
[0007] Obtain a 3D point cloud of the object to be detected, and project the 3D point cloud onto a 2D plane through multi-view projection to obtain multiple depth images;
[0008] The features of the depth image are extracted through the image encoder, and the features of the depth image are aggregated into the points of the point cloud according to the projection relationship to obtain the features of each point in the point cloud;
[0009] Two text prompts are set to describe the normal or abnormal state respectively, and learnable parameters are added to the text prompts; according to the text prompts, text features are extracted through a text encoder;
[0010] According to the cosine similarity between the features of the points in the point cloud and the text features, the anomaly score of the point in the point cloud is obtained, and the anomaly of each point in the point cloud is judged according to the anomaly score.
[0011] As a further preferred embodiment, the image encoder includes N l Transformer layers, from which m key layers {f1,f2,…,f m};
[0012] Adding learnable parameters to the transformer layer, the output of each transformer layer is:
[0013]
[0014] Among them, L j represents the jth transformer layer, B j represents the learnable parameters added to the jth transformer layer, W j Represents the output of the j-th transformer layer.
[0015] As a further preferred method, aggregating the features of the depth image into points of the point cloud according to the projection relationship includes:
[0016] For the kth depth image, after it is input into the image encoder, the output of the lth key layer is Will The channel is expanded into an image to obtain a feature map; then based on the feature map, the point feature of the i-th point in the point cloud on the l-th key layer of the k-th depth image is obtained according to the projection relationship
[0017] Aggregate the point features on all depth images to obtain the features of each point in the point cloud, where the feature of the i-th point for:
[0018]
[0019] Where N is the total number of depth images.
[0020] As a further preference, the key layer selection method is fixed interval selection, that is, a transformer layer is selected as the key layer every certain number of layers.
[0021] As a further preferred embodiment, two text prompts are set to describe the normal or abnormal state respectively, and learnable parameters are added to the text prompts. Specifically, the text prompt includes a state description word State and a category description word CLS, and then the learnable parameters are added thereto.
[0022] As a further preferred embodiment, the learnable parameters in the text prompt include U and S, wherein U is a learnable parameter shared by the two text prompts; S is a dedicated learnable parameter, which includes a learnable parameter S used in the normal state text prompt. n , and the learnable parameter S used in the abnormal state text prompt a .
[0023] As a further preferred embodiment, before calculating the anomaly score of a point in the point cloud, the anomaly score of the entire point cloud is calculated first. If the anomaly score of the entire point cloud exceeds a preset threshold, the anomaly score of the point in the point cloud is further calculated; otherwise, the three-dimensional point cloud is considered to have no anomaly.
[0024] As a further preferred method, the calculation method of the abnormal score ξ of the entire point cloud is:
[0025]
[0026] Where N is the total number of depth images, and softmax represents the activation function; is the class token of the last transformer layer of the image encoder for the kth depth image; G is the text feature extracted by the text encoder.
[0027] According to a second aspect of the present invention, a zero-sample point cloud anomaly detection system considering prompt learning is provided, comprising a processor, wherein the processor is used to execute the above-mentioned zero-sample point cloud anomaly detection method considering prompt learning.
[0028] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the zero-sample point cloud anomaly detection method considering prompt learning is implemented.
[0029] In general, the above technical solution conceived by the present invention has the following technical advantages compared with the prior art:
[0030] 1. Based on multi-view projection and image encoder, the present invention obtains aggregated point features, and then uses the cosine similarity between the aggregated point features and the text features extracted by the text encoder to calculate the level anomaly score, thereby determining the abnormality of the point cloud. It can fill the gap in the point cloud visual language model, expand the image visual language model to the zero-sample point cloud anomaly detection task, and realize accurate detection of point cloud anomalies.
[0031] 2. The features obtained in the pre-trained image encoder are usually strongly correlated with the distribution of the pre-training dataset, which may lead to the domain gap problem. Specifically, for the image encoder, since there are few industrial scenes in its pre-training dataset, the proportion of input images related to defects is very small, which leads to the mismatch between image features and defect information; therefore, the present invention proposes key layer visual cues and adds learnable parameters to fine-tune the image encoder to bridge the domain gap.
[0032] 3. Considering that text prompts can be not only discrete and clear human language expressions, but also continuous abstract vectors; therefore, the present invention uses adaptive text prompts, that is, adding learnable parameters to dynamically generate text prompts, and adopts two types of learnable parameters, common and dedicated, which can better describe the global image content and local defects. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 A flow chart of a zero-sample point cloud anomaly detection method considering prompt learning in an embodiment of the present invention;
[0034] Figure 2 The figure is a schematic diagram of the multi-view projection process and results according to an embodiment of the present invention. DETAILED DESCRIPTION
[0035] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0036] The embodiment of the present invention provides a zero-sample point cloud anomaly detection method considering prompt learning, such as Figure 1 As shown, the following steps are included:
[0037] S1. Obtain a three-dimensional point cloud of the object to be detected, and project the three-dimensional point cloud onto a two-dimensional plane through multi-view projection to obtain multiple depth images.
[0038] Specifically, due to the unstructured nature of point cloud data, the current point cloud feature extraction network performance is insufficient, and there are certain challenges in directly processing point clouds. Considering the equivalence of point clouds and multi-view projected depth images, projecting 3D point cloud data onto a 2D plane and converting it into an image form provides an effective way to process point cloud data. Multi-view projected depth images carry complete information about the object from multiple perspectives, so that enough data can be extracted from these 2D images to understand the original point cloud morphology and features.
[0039] T is a coordinate system from the point cloud {T p} to the camera coordinate system {T c When performing multi-view projection, multiple different homogeneous transformation matrices T are pre-set k , after projection and rendering, N depth images V can be obtained k ,1≤k≤N, such as Figure 2 As shown. Then for the i-th point P in the point cloud P i , which is in the depth image V k The corresponding points in are Then P i and The projection relationship between them is:
[0040]
[0041] in, YesP i In the camera coordinate system {T ck}, the Z coordinate in the camera coordinate system {T ck} corresponds to the homogeneous transformation matrix T k , K is the intrinsic parameter matrix of the camera.
[0042] definition is the mapping relationship between image pixels and points in the point cloud, then we have:
[0043]
[0044] Where R k It is T k The rotation matrix in , t k It is T k The translation vector in . Pos k It will be used in multi-view feature aggregation to align features of different views and obtain point features of the point cloud.
[0045] S2. Extract the features of the depth image through the image encoder, and aggregate the features of the depth image into the points of the point cloud according to the projection relationship to obtain the features of each point in the point cloud.
[0046] The specific steps include:
[0047] S21, obtaining deep image features based on key layer visual cues;
[0048] Let f be the image encoder of the pre-trained model, f has N l Transformer layers, the pre-trained model can be selected as a large visual-language model, such as the CLIP model. The features obtained in the pre-trained image encoder are usually strongly correlated with the distribution of the pre-training dataset, which leads to the domain gap problem, that is, the features extracted by the encoder do not fully express the information of the target domain image. Specifically, there are few industrial scenes in the pre-training dataset, and the proportion of input images related to defects is very small, which leads to a mismatch between image features and defect information. Therefore, it is considered to use visual cues, that is, to add learnable parameters to fine-tune the image encoder to bridge the domain gap.
[0049] Since different layers of the image encoder extract different levels of semantic information, m key layers are selected: l ∈{f1,f2,…,f m},1≤f l ≤N l To fuse multi-scale features of images.
[0050] Then the learnable parameters are added to the transformer layer, and the output of the transformer layer is:
[0051]
[0052] Among them, L j represents the jth transformer layer, B j represents the learnable parameters added to the jth transformer layer, W j Denotes the output of the jth transformer layer. After adding learnable parameters, the features that are more conducive to distinguishing abnormal distributions can be obtained. With the adjustment of visual cues, the image encoder pays more attention to the defect area, improving the generalization performance of the anomaly detection model.
[0053] Furthermore, the key layer selection method is fixed interval selection, that is, a transformer layer is selected as the key layer every certain number of layers. For example, the number of transformer layers in the image encoder is N l =10, the key layer number m=3, f1=3, f2=6, f3=9, that is, the 3rd, 6th and 9th layers are taken as the key layers.
[0054] S22, multi-view feature aggregation, obtaining point features;
[0055] For the kth depth image, after the image encoder processes it, the output of the lth key layer is Will The channel is expanded into an image to obtain the feature map Then, based on the feature map, the point feature of the i-th point in the point cloud on the l-th key layer of the k-th depth image is obtained according to the projection relationship.
[0056]
[0057] in, The feature map The feature of the i-th point in .
[0058] Aggregate the point features on the depth image to get the feature of the i-th point in the point cloud
[0059]
[0060] S3. Set two text prompts to describe the normal or abnormal state respectively, and extract text features through a text encoder based on the text prompts.
[0061] Specifically, anomaly detection can be transformed into a binary classification problem. The visual language model sets two text prompts to describe normal or abnormal images / points, and then generates anomaly scores by evaluating the similarity between the features of the image / point and the text prompts. Define the state description word State to represent different states, and the category description word CLS to represent different categories to be detected, then the simple text prompt can be obtained:
[0062] T prompt =h([State,CLS])
[0063] Among them, h([State,CLS]) can generate a complete text prompt. For example, define the state description words State∈{perfect,normal,damaged,abnormal,etc.}, where perfect and normal are normal descriptions, damaged and abnormal are abnormal descriptions; category description words CLS∈{bagel,carrot,airplane,etc.}; then the generated normal text prompt t n Can be "a photo of a perfect bagel", unusual text prompt t a It can be “a photo of a damaged bagel”.
[0064] However, it is challenging to design text prompts tailored for point cloud anomaly detection. Considering that text prompts can be not only discrete and clear human language expressions, but also continuous abstract vectors, we consider using adaptive text prompts, that is, adding learnable parameters to dynamically generate text prompts:
[0065]
[0066] in U is a common learnable parameter, which is used in both normal and abnormal text prompts; S is a dedicated learnable parameter, and each prompt has a separate dedicated parameter. For example, the normal and abnormal text prompts mentioned above become:
[0067]
[0068]
[0069] Among them, n u is the number of shared learnable parameters; n s is the number of dedicated learnable parameters, that is, the learnable parameters S dedicated to the normal state text prompt n , a learnable parameter S dedicated to abnormal state text prompts a Both have n s indivual.
[0070] If only U is used, the text features of normal and abnormal images are similar, which will have a negative impact on anomaly detection. If only S is used, it may over-enhance the obvious and simple local description of the image and ignore the global information. The advantage of using both parameters is that both global image content and local defects can be better described.
[0071] After obtaining the text prompt, the text feature G is extracted through the text encoder g of the pre-trained model:
[0072] G=g(t n ,t a )
[0073] S4. According to the cosine similarity between the aggregated point features and the text features, the target-level anomaly score and the point-level anomaly score are calculated to determine the point cloud anomaly.
[0074] The specific steps include:
[0075] S41. Calculate the anomaly score of the point cloud as a whole, i.e., the target-level anomaly score ξ; if the anomaly score of the point cloud as a whole exceeds a preset threshold, proceed to step S42 to further calculate the anomaly score of the point in the point cloud and determine the specific anomaly point; otherwise, the three-dimensional point cloud is considered to have no anomaly.
[0076] The target-level anomaly score ξ is calculated as:
[0077]
[0078] in, is the classtoken (category token) of the last transformer layer of the image encoder for the kth depth image; G is the text feature extracted by the text encoder.
[0079] S42. Calculate the point-level anomaly score based on the cosine similarity between the point feature and the text feature; the anomaly score A of the i-th point in the point cloud i The calculation is as follows:
[0080]
[0081] After obtaining the anomaly score of each point in the point cloud, we can determine whether the point is abnormal based on whether the anomaly score of the point exceeds the preset threshold, thereby realizing point cloud anomaly detection.
[0082] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A zero-shot point cloud anomaly detection method considering hint learning, It is characterized in that The steps include: Obtain a 3D point cloud of the object to be detected, and project the 3D point cloud onto a 2D plane through multi-view projection to obtain multiple depth images; The features of the depth image are extracted through the image encoder, and the features of the depth image are aggregated into the points of the point cloud according to the projection relationship to obtain the features of each point in the point cloud; The image encoder includes N l Transformer layers, from which m key layers {f 1 ,f 2 ,…,f m }; Add learnable parameters to the transformer layer, and the output of each transformer layer is: Among them, L j represents the jth transformer layer, B j represents the learnable parameters added to the jth transformer layer, W j represents the output of the jth transformer layer; Aggregate the features of the depth image into the points of the point cloud based on the projection relationship, including: For the kth depth image, after it is input into the image encoder, the output of the lth key layer is Will The channel is expanded into an image to obtain a feature map; then based on the feature map, the point feature of the i-th point in the point cloud on the l-th key layer of the k-th depth image is obtained according to the projection relationship Aggregate the point features on all depth images to obtain the features of each point in the point cloud, where the feature of the i-th point for: Where N is the total number of depth images; Two text prompts are set to describe the normal or abnormal state respectively, and learnable parameters are added to the text prompts; according to the text prompts, text features are extracted through a text encoder; The learnable parameters in the text prompt include U and S, where U is the learnable parameter shared by the two text prompts; S is a dedicated learnable parameter, which includes the learnable parameter S used in the normal state text prompt n , and the learnable parameter S used in the abnormal state text prompt a ; According to the cosine similarity between the features of the points in the point cloud and the text features, the anomaly score of the points in the point cloud is obtained, and the anomaly of each point in the point cloud is judged according to the anomaly score; the steps include: S41, calculating the anomaly score of the entire point cloud. If the anomaly score of the entire point cloud exceeds a preset threshold, step S42 is performed to further calculate the anomaly score of the point in the point cloud to determine the specific anomaly point; otherwise, it is considered that the three-dimensional point cloud has no anomaly; The calculation method of the anomaly score ξ of the point cloud as a whole is: Where N is the total number of depth images, and softmax represents the activation function; is the class token of the last transformer layer of the image encoder for the kth depth image; G is the text feature extracted by the text encoder; S42. Calculate the anomaly score of the point based on the cosine similarity between the feature of the point and the feature of the text; the anomaly score A of the i-th point in the point cloud i The calculation is as follows: After obtaining the anomaly score of each point in the point cloud, the point is judged whether it is abnormal based on whether the anomaly score of the point exceeds the preset threshold, thus realizing point cloud anomaly detection.
2. The zero-shot point cloud anomaly detection method considering hint learning as claimed in claim 1, It is characterized in that The key layer selection method is fixed interval selection, that is, a transformer layer is selected as the key layer every certain number of layers.
3. The zero-shot point cloud anomaly detection method considering hint learning as claimed in claim 1, It is characterized in that Two text prompts are set to describe the normal or abnormal state respectively, and learnable parameters are added to the text prompts. Specifically, the text prompt includes a state description word State and a category description word CLS, and then the learnable parameters are added thereto.
4. A zero-shot point cloud anomaly detection system considering hint learning, It is characterized in that It includes a processor, which is used to execute the zero-sample point cloud anomaly detection method considering prompt learning as described in any one of claims 1-3.
5. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method for zero-sample point cloud anomaly detection taking prompt learning into consideration is implemented as described in any one of claims 1 to 3.
