Point cloud anomaly detection method, device and medium

Abstract

The present invention belongs to the technical field of industrial point cloud anomaly detection, and discloses a point cloud anomaly detection method, device and medium, including calculating an anomaly score map for each point; grouping point clouds through the farthest distance sampling algorithm, and performing preliminary feature extraction; extracting local anomaly features and global anomaly features through a local anomaly perception network and a global anomaly perception network, and connecting text feature vectors, point cloud embeddings and anomaly map embeddings and inputting them into a multimodal large language model backbone network, so that it has the ability to detect and interpret abnormal characteristics. The present invention significantly improves the accuracy and robustness of anomaly detection by integrating the basic multimodal understanding ability and domain generalization ability of the multimodal large language model and the anomaly detection technology of point cloud data. The method realizes the accurate identification and detailed description of abnormal areas in point cloud data through the collaborative work of local and global anomaly perception networks.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial point cloud anomaly detection, and in particular to a point cloud anomaly detection method, device and medium. Background Art

[0002] In modern industrial production, it is crucial to ensure product quality. In this context, the application of industrial visual inspection technology has become particularly important. Industrial visual inspection technology uses an automated visual system to inspect products to identify defects, flaws or products that do not meet specifications, thereby ensuring that product quality meets standards. In recent years, anomaly detection technology has been widely used in surface defect quality inspection of industrial products. In industrial visual inspection, by analyzing the surface image of the product, anomaly detection can identify various defects such as scratches, dents, impurities, etc., so that defective products can be eliminated in the early stages of production.

[0003] Although image anomaly detection technology performs well in many aspects, it has limitations when dealing with certain complex scenarios. For example, for objects with unclear surface textures, it is difficult for images to accurately capture their surface details, resulting in failure of two-dimensional image anomaly detection. In this context, depth cameras are currently introduced in many practical applications, which can collect depth information of the object to be detected, so that the corresponding point cloud data can be synthesized. Compared with the original two-dimensional image, point cloud data can provide three-dimensional geometric information of the object, making the detection more comprehensive and accurate. However, despite the obvious advantages of point cloud anomaly detection, it still faces a series of challenges in practical applications.

[0004] Point cloud anomaly detection prevents failures and maintains product quality by identifying and locating anomalies that deviate from normal operating modes. Despite significant progress in this field, there are still many challenges in achieving accurate detection and in-depth analysis of anomalies due to the complexity of point cloud data. Specifically, current point cloud anomaly detection methods usually rely on generating an anomaly score for each point in the point cloud, and this anomaly score is usually a decimal between 0 and 1, which is relatively abstract and not suitable for practical analysis. Although this method can reflect the existence of anomalies to a certain extent, its information is too abstract and difficult for operators without professional knowledge to understand and apply. This abstract anomaly score cannot directly provide clear anomaly judgment, resulting in certain limitations in actual industrial applications. For example, it may be difficult for operators to make quick and accurate decisions based on these scores, which affects production efficiency and product quality.

[0005] In addition, the complexity of point cloud data itself also increases the difficulty of anomaly detection. Point cloud data usually contains a large amount of point cloud information, which is not only high-dimensional but also large in volume. Processing and analyzing this data requires powerful computing power and complex algorithms. Existing methods often find it difficult to take into account both global and local anomaly features when processing this data, which affects the accuracy and reliability of the detection results.

[0006] In this context, the field of intelligent industrial applications urgently needs a powerful and unified system that can not only perform well in point cloud anomaly detection, but also provide detailed anomaly feature analysis. This framework should be able to transform complex point cloud data into anomaly information that is easy to understand and apply, thereby helping operators make better decisions and improve production efficiency and product quality.

[0007] In summary, although point cloud anomaly detection is of great significance in industrial manufacturing, the challenges it faces cannot be ignored. Existing methods still have a lot of room for improvement in terms of the accuracy of anomaly detection and the comprehensibility of information. In order to solve these problems, it is urgent to develop new technologies and methods that can better handle the complexity of point cloud data and provide clear and detailed anomaly feature analysis to meet the needs of practical industrial applications. Summary of the invention

[0008] The purpose of the present invention is to provide a point cloud anomaly detection method, device and medium to solve the problems raised in the above background technology.

[0009] To achieve the above object, the present invention provides the following technical solutions:

[0010] A point cloud anomaly detection method, comprising:

[0011] S1. Generate an instruction dataset for point cloud anomaly detection, where GPT-4 is used to generate a variety of instruction question-answer pairs to enrich the training data of the multimodal large language model;

[0012] S2. Obtain the three-dimensional point cloud data P of the object to be detected 0 ∈R n×6 , where P 0 represents the original point cloud, R is the real number field, n represents the number of points, and each point has 6 features, including XYZ coordinates and RGB color information;

[0013] S3. Calculate the anomaly score map of the point cloud data through the point cloud anomaly detector to identify the original anomaly score map S 0 ∈R n×1 ;

[0014] S4, first extract point cloud and anomaly score features, then use cross-modal cross attention and gated fusion modules to obtain local anomaly features, and extract global anomaly features through the set abstraction block;

[0015] S5. Input the given conversation text into the pre-trained text feature extractor to obtain a text feature vector. Then, connect the local abnormal features, the global abnormal features and the text feature vector and input them into the multimodal large language model backbone network to train the multimodal large language model so that the multimodal large language model has the ability to detect abnormalities and explain abnormal characteristics:

[0016] ,

[0017] LLM stands for Large Multimodal Language Model, Concat stands for feature concatenation, and T text represents the pre-extracted text feature vector, y is the output sequence, and the loss function L used in training is as follows:

[0018] ,

[0019] Among them, n 0 is the number of markers, y i is the true label of the i-th token in the output sequence, p i is the predicted probability of the i-th token in the output sequence. The loss function is the cross entropy loss function, which is used to measure the difference between the model prediction result and the true label;

[0020] After training, an optimized multimodal large language model for point cloud anomaly detection is obtained;

[0021] S6. Detect point cloud data through the optimized multimodal large language model.

[0022] Furthermore, the S1 specifically includes:

[0023] S11, obtaining a multi-view rendering image, and projecting the point-level true value of the point cloud onto a two-dimensional image to assist in identifying abnormal areas in the two-dimensional view;

[0024] S12, use the CLIP pre-trained visual language model to filter out the images that highlight the anomalies the most, and highlight them based on the two-dimensional truth value;

[0025] S13, input the preprocessed image into GPT-4 to generate context-related descriptions and detailed anomaly information;

[0026] S14. Combine the generated anomaly description, rendered image, and designed instructions to guide GPT-4 to generate multiple rounds of dialogue to form instruction question-answer pairs for point cloud anomaly detection.

[0027] Further, the S4 includes:

[0028] S41, group the point cloud by the farthest distance sampling algorithm, use MiniPointNet for preliminary feature extraction, and sample the anomaly score to obtain the feature sequence;

[0029] S42. The local anomaly perception network fuses the anomaly score features with the point cloud features through a cross-modal cross-attention mechanism and a gated fusion module to obtain local anomaly features. The global anomaly perception network converts the anomaly score features into anomaly prior tokens through a collection of abstract blocks, and extracts global anomaly features containing local geometry and context information.

[0030] Furthermore, the S41 specifically includes:

[0031] The point cloud is grouped by the farthest point sampling algorithm, and MiniPointNet is used to perform preliminary feature extraction on the grouped point cloud. The anomaly score is sampled with the same farthest point sampling algorithm sampling index to obtain the anomaly score feature sequence:

[0032] ,

[0033] ,

[0034] ,

[0035] Among them, S 0 represents the original anomaly score, FPS represents the farthest distance sampling algorithm, m is the number of FPS samples, d is the feature dimension, P FPS is the point cloud sampled at the farthest distance, MiniPointNet is the traditional point cloud feature extractor, Expand is the copy dimension operation, and finally the processed point cloud feature F is obtained. P and the anomaly score feature S P , F P ,S P ∈R m×d .

[0036] Furthermore, the S42 specifically includes:

[0037] S421, the local anomaly perception network fuses the anomaly score features with the point cloud features through the cross-modal cross attention mechanism and the gated fusion module to obtain the anomaly enhanced point cloud features:

[0038] ,

[0039] Among them, CA represents the cross-modal attention mechanism, F LAPNis the output result of the local anomaly perception network, where S P As the query in the cross attention, F P As Key and Value, the purpose is to use the prior knowledge in the anomaly score map to guide the point cloud feature extraction, so that the model pays more attention to the abnormal area, so as to respond to the abnormal information later;

[0040] S422, the global anomaly perception network transforms the anomaly score features into anomaly prior tokens by assembling abstract blocks:

[0041] ,

[0042] Among them, SetAbstraction represents the set abstract block. is the output of the global anomaly perception network, L represents the number of tokens, and D LLM Represents the embedding dimension of the multimodal large language model.

[0043] The present invention also provides a point cloud anomaly detection device, comprising one or more processors, for implementing a point cloud anomaly detection method as described above.

[0044] The present invention also provides a readable storage medium on which a program is stored. When the program is executed by a processor, the point cloud anomaly detection method as described above is implemented.

[0045] Compared with the prior art, the present invention has the following beneficial effects: the present invention significantly improves the accuracy and robustness of anomaly detection by integrating the basic multimodal understanding ability and domain generalization ability of the multimodal large language model and the anomaly detection technology of point cloud data. The method realizes the accurate identification and detailed description of abnormal areas in point cloud data through the collaborative work of local and global anomaly perception networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of the point cloud anomaly detection method of the present invention.

[0047] Figure 2 Schematic diagram of the point cloud preprocessing process.

[0048] Figure 3 It is a schematic diagram of the structure of the multimodal large language model of the present invention.

[0049] Figure 4 It is a structural schematic diagram of a point cloud anomaly detection device of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] See also Figure 1-Figure 3 , a point cloud anomaly detection method, comprising the following steps:

[0052] S1. Generate a command dataset for point cloud anomaly detection. Generate a variety of command question-answer pairs through the currently widely used powerful large language model GPT-4, enrich the training data of the multimodal large language model, and further improve the model's anomaly detection ability and ability to describe anomalies in detail. The specific steps include:

[0053] S11. Obtain a multi-view rendering image and project the point-level true value of the point cloud onto a two-dimensional image to assist in identifying abnormal areas in the two-dimensional view.

[0054] S12. Use the CLIP pre-trained visual language model to filter out the images that best highlight the anomalies. CLIP is a multimodal model that can process images and text simultaneously. It understands the association between images and text through contrastive learning and highlights them based on the two-dimensional true value to more clearly display the abnormal area.

[0055] S13. Input the preprocessed image into GPT-4 to generate context-related descriptions and detailed anomaly information.

[0056] S14. Combine the generated anomaly description, rendered image, and carefully designed instructions to guide GPT-4 to generate multiple rounds of dialogue to form command question-answer pairs for point cloud anomaly detection.

[0057] S2, such as Figure 2 As shown in FIG. 1 , the left side is the point cloud anomaly detection dataset used for training in this embodiment: the RGB image with anomalies of the biscuit category in the MVTEC 3D-AD dataset, and the right side is the depth map corresponding to the left side in the dataset. 0 ∈R n×6 , where P 0 Represents the original point cloud, R is the real number field, n represents the number of points, and each point has 6 features, specifically, including XYZ coordinates and RGB color information.

[0058] S3. Calculate the anomaly score map of the point cloud data through the existing point cloud anomaly detector. The point cloud anomaly detector can select any method previously used for 3D anomaly detection, such as BTF and M3DM. These point cloud anomaly detectors can analyze the geometric and structural features of the point cloud data and generate an anomaly score map representing each point, thereby identifying the original anomaly score map S 0 ∈R n×1 , R is the field of real numbers.

[0059] S4, first extract the point cloud and anomaly score features, then use the cross-modal cross attention and gated fusion modules to obtain local anomaly features, and extract global anomaly features by assembling abstract blocks, including:

[0060] S41. Group the point cloud using the farthest distance sampling algorithm, and use MiniPointNet to perform preliminary feature extraction on the grouped point cloud. Sample the anomaly score with the same FPS sampling index to obtain the anomaly score feature sequence, preparing for the subsequent Transformer feature extraction and multimodal feature fusion:

[0061] ,

[0062] ,

[0063] ,

[0064] Among them, S 0 represents the original anomaly score, FPS represents the farthest distance sampling algorithm, m is the number of FPS samples, d is the feature dimension, P FPS MiniPointNet is a simplified version of PointNet network, which focuses on efficiently extracting local point cloud features. Expand is a copy dimension operation, and finally the processed point cloud feature F is obtained. P and the anomaly score feature S P , F P ,S P ∈R m×d .

[0065] S42, extracting point cloud features and abnormal features through the local abnormality perception network and the global abnormality perception network, specifically including the following steps:

[0066] S421. The local anomaly perception network includes a cross-modal cross-attention mechanism and a gated fusion module, which fuses the anomaly score features with the point cloud features to obtain local anomaly features. The cross-modal cross-attention mechanism is used to establish associations between different modalities and capture the correlation between image and point cloud features. The gated fusion module controls the feature fusion process through a gating mechanism to ensure the effective fusion of anomaly score features and point cloud features:

[0067] ,

[0068] Among them, CA represents the cross-modal attention mechanism, F LAPN is the output result of the local anomaly perception network. Here S P As the query in the cross attention, F P As Key and Value, the purpose is to use the prior knowledge in the anomaly score map to guide the point cloud feature extraction, so that the model pays more attention to the abnormal area, so as to respond to the abnormal information later.

[0069] S422, the global anomaly perception network uses a collection of abstract blocks to transform anomaly score features into anomaly prior tokens:

[0070] ,

[0071] Among them, SetAbstraction represents the set abstract block. is the output of the global anomaly perception network, L represents the number of tokens, and D LLM Represents the embedding dimension of a multimodal large language model. This architecture draws on the feature extraction operation of point clouds in PointNet++ so that the extracted features can contain the geometric information and contextual information of the local area, and better summarize the global anomalies of the point cloud.

[0072] S5. Input the given dialogue text into the pre-trained text feature extractor to obtain the text feature vector. The pre-trained text feature extractor can be the text encoder in CLIP. Then, the local anomaly features, global anomaly features and text feature vectors are connected and input into the multimodal large language model backbone network to train the multimodal large language model. Using these embeddings, the multimodal large language model trained with the point cloud anomaly detection dataset can have the ability to detect anomalies and explain abnormal characteristics:

[0073] ,

[0074] Among them, LLM stands for Large Language Model, Concat stands for feature concatenation, and T text represents the pre-extracted text feature vector, and y is the output sequence. Finally, the loss function used in model training is as follows:

[0075] ,

[0076] Among them, n 0 is the number of markers, y i is the true label of the i-th token in the output sequence, p i is the predicted probability of the i-th token in the output sequence. The loss function is the cross entropy loss function, which is used to measure the difference between the model prediction result and the true label.

[0077] S6. Point cloud data is detected through an optimized multimodal large language model, which can accurately identify and interpret abnormal areas in point clouds. It combines text, point clouds, and abnormal graph embedding to achieve efficient fusion of multimodal features, thereby improving the accuracy and robustness of anomaly detection.

[0078] In the practical application of the present invention, the preliminary anomaly detection model is firstly trained using the point cloud in the training set and the generated multiple rounds of dialogues, and the trained point cloud anomaly detection model is used to perform anomaly detection on the industrial point cloud data to be detected.

[0079] See also Figure 4 A point cloud anomaly detection device provided in an embodiment of the present invention includes one or more processors for implementing a point cloud anomaly detection method in the above embodiment.

[0080] An embodiment of a point cloud anomaly detection device of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From the hardware level, if Figure 4 As shown, it is a hardware structure diagram of any device with data processing capability in which a point cloud anomaly detection device of the present invention is located, except Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in the embodiments may also include other hardware, usually based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0081] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0082] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0083] An embodiment of the present invention further provides a readable storage medium on which a program is stored. When the program is executed by a processor, a point cloud anomaly detection method in the above embodiment is implemented.

[0084] The readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The readable storage medium may also be an external storage device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the readable storage medium may also include both an internal storage unit of any device with data processing capability and an external storage device. The readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0085] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A point cloud anomaly detection method, characterized in that: include: S1. Generate an instruction dataset for point cloud anomaly detection, where GPT-4 is used to generate a variety of instruction question-answer pairs to enrich the training data of the multimodal large language model; S2. Obtain the three-dimensional point cloud data P0∈R of the object to be detected n×6 , where P0 represents the original point cloud, R is the real number field, n represents the number of points, and each point has 6 features, including XYZ coordinates and RGB color information; S3, calculate the anomaly score map of the point cloud data through the point cloud anomaly detector, and then identify the original anomaly score map S0∈R n×1 ; S4, first extract point cloud and anomaly score features, then use cross-modal cross attention and gated fusion modules to obtain local anomaly features, and extract global anomaly features through the set abstraction block; S5. Input the given conversation text into the pre-trained text feature extractor to obtain a text feature vector. Then, connect the local abnormal features, the global abnormal features and the text feature vector and input them into the multimodal large language model backbone network to train the multimodal large language model so that the multimodal large language model has the ability to detect abnormalities and explain abnormal characteristics: ， LLM stands for Large Multimodal Language Model, Concat stands for feature concatenation, and T text represents the pre-extracted text feature vector, y is the output sequence, and the loss function L used in training is as follows: ， Where n0 is the number of markers, y i is the true label of the i-th token in the output sequence, p i is the predicted probability of the i-th token in the output sequence. The loss function is the cross entropy loss function, which is used to measure the difference between the model prediction result and the true label; After training, an optimized multimodal large language model for point cloud anomaly detection is obtained; S6. Detect point cloud data through the optimized multimodal large language model.

2. A point cloud anomaly detection method according to claim 1, characterized in that: The S1 specifically includes: S11, obtaining a multi-view rendering image, and projecting the point-level true value of the point cloud onto a two-dimensional image to assist in identifying abnormal areas in the two-dimensional view; S12, use the CLIP pre-trained visual language model to filter out the images that highlight the anomalies the most, and highlight them based on the two-dimensional truth value; S13, input the preprocessed image into GPT-4 to generate context-related descriptions and detailed anomaly information; S14. Combine the generated anomaly description, rendered image, and designed instructions to guide GPT-4 to generate multiple rounds of dialogue to form instruction question-answer pairs for point cloud anomaly detection.

3. A point cloud anomaly detection method according to claim 1, characterized in that: The S4 includes: S41, group the point cloud by the farthest distance sampling algorithm, use MiniPointNet for preliminary feature extraction, and sample the anomaly score to obtain the feature sequence; S42. The local anomaly perception network fuses the anomaly score features with the point cloud features through a cross-modal cross-attention mechanism and a gated fusion module to obtain local anomaly features. The global anomaly perception network converts the anomaly score features into anomaly prior tokens through a collection of abstract blocks, and extracts global anomaly features containing local geometry and context information.

4. A point cloud anomaly detection method according to claim 3, characterized in that: The S41 specifically includes: The point cloud is grouped by the farthest point sampling algorithm, and MiniPointNet is used to perform preliminary feature extraction on the grouped point cloud. The anomaly score is sampled with the same farthest point sampling algorithm sampling index to obtain the anomaly score feature sequence: ， ， ， Among them, S0 represents the original anomaly score, FPS represents the farthest distance sampling algorithm, m is the number of FPS samples, d is the feature dimension, P FPS is the point cloud sampled at the farthest distance, MiniPointNet is the traditional point cloud feature extractor, Expand is the copy dimension operation, and finally the processed point cloud feature F is obtained. P and the anomaly score feature S P , F P ,S P ∈R m×d .

5. A point cloud anomaly detection method according to claim 4, characterized in that: The S42 specifically includes: S421, the local anomaly perception network fuses the anomaly score features with the point cloud features through the cross-modal cross attention mechanism and the gated fusion module to obtain the anomaly enhanced point cloud features: ， Among them, CA represents the cross-modal attention mechanism, F LAPN is the output result of the local anomaly perception network, where S P As the query in the cross attention, F P As Key and Value, the purpose is to use the prior knowledge in the anomaly score map to guide the point cloud feature extraction, so that the model pays more attention to the abnormal area, so as to respond to the abnormal information later; S422, the global anomaly perception network transforms the anomaly score features into anomaly prior tokens by assembling abstract blocks: ， Among them, SetAbstraction represents the set abstract block. is the output of the global anomaly perception network, L represents the number of tokens, and D LLM Represents the embedding dimension of the multimodal large language model.

6. A point cloud anomaly detection device, characterized in that: The method comprises one or more processors for implementing a point cloud anomaly detection method according to any one of claims 1 to 5.

7. A readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, a point cloud anomaly detection method according to any one of claims 1 to 5 is implemented.

Images