Point cloud anomaly detection method and system based on cross-modal self-supervised learning, medium and device
By employing a cross-modal self-supervised learning method and leveraging the multi-view image generation task and feature diversity constraints, the problem of insufficient fine-grained feature extraction in point cloud self-supervised learning is solved. This enables comprehensive capture of fine-grained geometric features of point clouds and improves the performance of unsupervised point cloud anomaly detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- UNIV OF CHINESE ACAD OF SCI
- Filing Date
- 2026-03-31
- Publication Date
- 2026-07-14
AI Technical Summary
Existing self-supervised learning methods for point clouds are insufficient in capturing fine-grained geometric features, are easily affected by local point distribution, and limit the comprehensive understanding of 3D shapes.
A cross-modal self-supervised learning method is adopted. By constructing a proxy task for multi-view image generation, PointNet++ and Transformer models are used to extract a coarse subset and three-dimensional feature representation of point clouds. Two-dimensional feature representation is generated through a view conditional attention module. Combined with cross-modal reconstruction loss and feature diversity constraints, fine-grained feature extraction of point clouds is achieved.
It effectively improves the performance of unsupervised point cloud anomaly detection, can capture fine-grained geometric features more comprehensively, reduces redundancy between views, and promotes a richer understanding of 3D geometric features.
Smart Images
Figure CN122391698A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of point cloud self-supervised learning technology, and in particular to a point cloud anomaly detection method, system, medium, and device based on cross-modal self-supervised learning. Background Technology
[0002] 3D vision has garnered significant attention due to its ability to understand the real world, finding wide application in fields such as autonomous driving and robotics. Point clouds, as the most common data representation in 3D vision, are widely used to solve tasks such as object classification, detection, and segmentation, which typically require fully supervised training from scratch. However, collecting and labeling point cloud data is often costly and labor-intensive, hindering the development of fully supervised 3D representation learning. Self-supervised learning has emerged as a dominant paradigm for addressing this challenge. This approach leverages the rich statistical and structural cues contained in large-scale unlabeled data to enhance the model's representational capabilities. Inspired by this, self-supervised learning for point clouds has rapidly developed, benefiting from low labeling costs and strong transferability to downstream tasks. Point cloud self-supervised learning can be broadly categorized into two main methods: contrastive methods and generative methods. For contrastive methods in point cloud self-supervised learning, the goal is to learn globally discriminative representations by maximizing mutual information across different viewpoints. For example, PointContrast achieves view transformation invariance by extending the InfoNCE objective. Contrast pre-training typically relies on abundant training data to prevent degenerate overfitting; however, data remains scarce in the field of point cloud object research. Generative methods often leverage known parts to predict missing parts, thereby enhancing the model's reasoning and global understanding capabilities. For example, PointMAE and PointM2AE can also generate point clouds with arbitrary missing parts from the original point cloud, reducing data requirements and making generative self-supervised methods more competitive in point cloud self-supervised learning. Due to the unordered nature of point clouds, reconstruction constraints must employ global set-to-set matching losses (e.g., chamfer distance).
[0003] Although the above methods effectively solve the permutation invariance problem to achieve point cloud self-supervised learning training, they construct a fuzzy target that is susceptible to the influence of local point distribution, which may hinder the capture of fine geometry and limit the model's comprehensive understanding of 3D shapes. Summary of the Invention
[0004] To address the aforementioned problems, the present invention aims to provide a point cloud anomaly detection method, system, medium, and device based on cross-modal self-supervised learning. By constructing a proxy task for multi-view image generation, cross-modal self-supervised learning of point clouds is achieved, enabling the model to understand the fine-grained geometric features of point clouds.
[0005] To achieve the above objectives, in a first aspect, the technical solution adopted by the present invention is as follows: a point cloud anomaly detection method based on cross-modal self-supervised learning, comprising: extracting a rough subset of the point cloud and a three-dimensional feature representation from the input point cloud using a point cloud feature extractor; for a given set of multiple different viewpoints, mapping the three-dimensional coordinates of each point in the rough subset to a two-dimensional grid according to the rendering matrix of each viewpoint, and generating a viewpoint conditional mask based on the depth of all points in the same grid to determine the visibility of each point in the current viewpoint, and obtaining the two-dimensional position code of each point in the current viewpoint; and using a viewpoint conditional attention module, determining the visibility of each point in the current viewpoint based on the two-dimensional position code and the three-dimensional feature representation of the point cloud. The method involves generating attention masks for different viewpoint conditions, projecting the 3D features of the point cloud into the 2D feature space under different viewpoints, and obtaining the 2D feature representations under each viewpoint. The 2D feature representations under each viewpoint are input into the same 2D decoder, which generates the 2D image of the corresponding viewpoint and establishes a cross-modal reconstruction loss with the real projected image. The training of the point cloud self-supervised model is achieved by minimizing this loss. Using a pre-trained point cloud feature extractor, features are extracted from the normal samples trained and a normal prototype is constructed. During the inference stage, the features extracted from the input point cloud are compared with the normal prototype, and point clouds that deviate from the normal prototype are classified as abnormal.
[0006] Furthermore, a hierarchical point cloud feature extractor is constructed using PointNet++ and the Transformer model to obtain a coarse subset of the point cloud and a 3D feature representation from the input point cloud.
[0007] Furthermore, obtain the two-dimensional position code of each point from the current viewpoint, including:
[0008] Based on the extracted rough subset of point cloud, given an arbitrary viewpoint, the three-dimensional spatial coordinates of all points in the rough subset are projected onto a two-dimensional mesh using a rendering matrix to obtain the coordinates of the point on the two-dimensional mesh and the depth information of the point. Traverse all points in the rough subset of the point cloud to obtain the 2D grid coordinates of all points from the current viewpoint. By comparing the depths between points projected onto the same pixel, the visibility mask corresponding to the point with the smallest depth is set to 1, and the remaining points are set to invisible. Applying a visibility mask to each point yields a two-dimensional positional code for each point from the current viewpoint.
[0009] Furthermore, the 3D features of the point cloud are projected into 2D feature spaces from different viewpoints to obtain 2D feature representations from each viewpoint, including: The obtained two-dimensional grid coordinates are encoded as a set of query tokens using a multilayer perceptron; The obtained 3D feature representation is concatenated with 3D spatial coordinates and a fixed 3D mesh to generate the key set and value set in the self-attention mechanism; The two-dimensional image features under the current viewpoint are calculated through a cross-attention mechanism, and the generated feature maps are reorganized into two-dimensional feature representations through a Reshape operation.
[0010] Furthermore, the cross-modal reconstruction loss is a weighted pixel alignment loss, and the specific construction process includes: Two-dimensional feature representations from various viewpoints are input into the same two-dimensional decoder, which generates two-dimensional images of the corresponding viewpoints. The generated two-dimensional images and the corresponding real projection images are then divided into foreground and background regions, respectively. Calculate the mean square error of the generated 2D image and the real projected image in the foreground region, and the mean square error of the generated 2D image and the real projected image in the background region, respectively. Assign a first weight to the mean square error of the foreground region and a second weight to the mean square error of the background region; use the sum of the weighted mean square errors of the foreground region and the background region as the weighted pixel alignment loss.
[0011] Furthermore, the training of the point cloud self-supervised model also includes calculating the feature diversity constraint loss, and the specific implementation process includes: Based on two-dimensional feature representations from multiple different perspectives, the average pairwise dot product between the two-dimensional feature representations from each perspective is calculated to measure the similarity between features from different perspectives. The average pairwise dot product is used as the feature diversity constraint loss. By minimizing this loss, the features corresponding to different viewpoints tend to be orthogonal, thereby reducing feature redundancy between views and promoting feature diversity.
[0012] Furthermore, using a pre-trained point cloud feature extractor, features are extracted from trained normal samples to construct a normal prototype. During the inference phase, the features extracted from the input point cloud are compared with the normal prototype, and point clouds that deviate from the normal prototype are classified as anomalies, including: Using a pre-trained point cloud feature extractor, features are extracted from multiple normal point cloud samples to obtain multiple normal feature vectors. Multiple normal feature vectors are stored in a memory pool to build a normal prototype library; During the inference phase, a pre-trained point cloud feature extractor is used to extract features from the point cloud to be detected, resulting in a feature vector to be detected. Calculate the distance or similarity between the feature vector to be detected and each normal feature vector in the normal prototype library; When the feature vector to be detected deviates from the distribution of the normal prototype library, the point cloud to be detected is determined to be an abnormal sample.
[0013] Secondly, the technical solution adopted by this invention is as follows: a point cloud anomaly detection system based on cross-modal self-supervised learning, comprising: a point cloud feature extraction unit, which extracts a rough subset of the point cloud and a three-dimensional feature representation from the input point cloud through a point cloud feature extractor; a view projection and masking unit, which, for a given set of multiple different viewpoints, maps the three-dimensional coordinates of each point in the rough subset to a two-dimensional grid according to the rendering matrix of each viewpoint, and generates a view conditional mask based on the depth of all points in the same grid to determine the visibility of each point in the current viewpoint and obtain the two-dimensional position code of each point in the current viewpoint; and a view conditional feature projection unit, which, through a view conditional attention module, projects the three-dimensional coordinates of each point in the rough subset to a two-dimensional grid based on the two-dimensional position code of the point cloud and the three-dimensional feature representation of the point cloud. The three-dimensional feature representation generates attention masks for different viewpoint conditions, projecting the three-dimensional features of the point cloud into two-dimensional feature spaces under different viewpoints to obtain two-dimensional feature representations under each viewpoint. The cross-modal reconstruction training unit inputs the two-dimensional feature representations under each viewpoint into the same two-dimensional decoder, which generates two-dimensional images of the corresponding viewpoints and establishes a cross-modal reconstruction loss with the real projected images. The training of the point cloud self-supervised model is achieved by minimizing this loss. The anomaly detection unit uses a pre-trained point cloud feature extractor to extract features from normal samples during training and construct a normal prototype. During the inference phase, the features extracted from the input point cloud are compared with the normal prototype, and point clouds that deviate from the normal prototype are classified as anomalies.
[0014] Thirdly, the technical solution adopted by the present invention is: a computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any of the methods described above.
[0015] Fourthly, the technical solution adopted by the present invention is: a computing device comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods described above.
[0016] The present invention has the following advantages due to the adoption of the above technical solutions: This invention constructs a fine-grained self-supervised learning method for point clouds through a multi-view image generation task. This method presents the target point cloud as a multi-view image. Figure 2Multi-view image generation is used to create scalable cross-modal generation targets and implement stricter cross-modal self-supervised constraints, enabling comprehensive capture of fine-grained geometric features. Multi-view image generation avoids learning one-sided representations, employs a feature diversity constraint loss, reduces redundancy between views, promotes more diverse 2D features, and thus provides a richer and more complete understanding of 3D geometric features, effectively improving the performance of unsupervised point cloud anomaly detection. Attached Figure Description
[0017] Figure 1 This is a flowchart of the point cloud anomaly detection method based on cross-modal self-supervised learning in an embodiment of the present invention; Figure 2 This is a cross-modal point cloud self-supervised learning framework in this embodiment of the invention; Figure 3 This is a schematic diagram of the viewpoint condition Transformer structure in an embodiment of the present invention. Detailed Implementation
[0018] To address the insufficient fine-grained feature extraction in existing technologies, this invention proposes a point cloud anomaly detection method, system, medium, and device based on cross-modal self-supervised learning. By establishing a novel point cloud self-supervised learning paradigm and implementing precise supervision constraints during training, the fine-grained feature extraction capability of the model is improved. This invention uses the PointNet++ model and Point Transformer to construct a point cloud feature extractor. A cross-modal multi-view image generation proxy task is designed based on a multi-view projection strategy to train the point cloud self-supervised learning model. A designed viewpoint-conditional Transformer module transforms 3D features into 2D features under different viewpoints, and a shared 2D decoder generates multi-view projected images. Finally, based on the pre-trained point cloud feature extractor, an unsupervised point cloud anomaly detection task is achieved. This invention can combine cross-modal image information to apply precise supervision signals to point cloud self-supervised learning, enabling the model to learn fine-grained geometric features, and has been successfully applied to unsupervised point cloud anomaly detection tasks.
[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.
[0020] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0021] In one embodiment of the present invention, a point cloud anomaly detection method based on cross-modal self-supervised learning is provided. In this embodiment, as shown... Figure 1 , Figure 2 As shown, the method includes the following steps: 1) Extract a rough subset of the point cloud and a 3D feature representation from the input point cloud using a point cloud feature extractor; 2) For a given number of different viewpoints t, based on the rendering matrix of each viewpoint... The three-dimensional coordinates of each point in the rough subset are mapped to a two-dimensional grid, and a view condition mask is generated based on the depth of all points in the same grid to determine the visibility of each point in the current view and obtain the two-dimensional position code of each point in the current view. 3) Through the viewpoint conditional attention module, attention masks with different viewpoint conditions are generated based on the two-dimensional position encoding and the three-dimensional feature representation of the point cloud under different viewpoints. The three-dimensional features of the point cloud are projected into the two-dimensional feature space under different viewpoints to obtain the two-dimensional feature representation under each viewpoint. 4) Input the two-dimensional feature representations from each viewpoint into the same two-dimensional decoder. The shared two-dimensional decoder generates two-dimensional images from the corresponding viewpoints and establishes a cross-modal reconstruction loss with the real projected images. The training of the point cloud self-supervised model is achieved by minimizing this loss. 5) Using a pre-trained point cloud feature extractor, features are extracted from the trained normal samples and a normal prototype is constructed. During the inference stage, the features extracted from the input point cloud are compared with the normal prototype, and point clouds that deviate from the normal prototype are classified as abnormal.
[0022] In step 1) above, a hierarchical point cloud feature extractor is constructed using PointNet++ and the Transformer model to obtain a rough subset of the point cloud and a 3D feature representation from the input point cloud.
[0023] In this embodiment, specifically, the input point cloud... Similar to PointNet++, it aggregates local features through three layers of local feature extraction, and then uses Point Transformer for further processing to extract point cloud context information. During the point cloud pre-training stage, this point cloud feature extractor can obtain a coarse subset of the point cloud from the input point cloud. With three-dimensional feature representation .
[0024] In step 2) above, obtaining the two-dimensional position code of each point from the current viewpoint includes the following steps: 2.1) Based on the extracted coarse subset of point cloud Given an arbitrary viewpoint t, this rough subset All points are rendered through the matrix. By projecting the three-dimensional spatial coordinates onto a two-dimensional grid, we can obtain the coordinates of the point on the two-dimensional grid and the depth information of the point. 2.2) Traverse all points in the rough subset of the point cloud to obtain the two-dimensional grid coordinates of all points in the current viewpoint; 2.3) By comparing the depths between points projected onto the same pixel, the visibility mask corresponding to the point with the smallest depth is set to 1, and the remaining points are set to invisible; 2.4) Apply the visibility mask to each point to obtain the two-dimensional position code of each point in the current view.
[0025] In this embodiment, specifically, such as Figure 3 As shown, based on the designed viewpoint condition Transformer module, the 3D feature representation of the point cloud is transformed into 2D features under different viewpoints. First, given an arbitrary viewpoint t, the 3D feature representation of the point cloud is transformed into 2D features under different viewpoints. All points are projected onto a 2D mesh, that is, for any... (i=1, 2, ..., N), through the rendering matrix Projecting 3D spatial coordinates onto a 2D mesh:
[0026] in, For point In the coordinates of the 2D grid, This provides the depth information for that point. Given the three-dimensional spatial coordinates of this point, traverse all points in the rough subset of the point cloud and record... For all points in the view 2D grid coordinates below; This indicates the number of points in the downsampled point cloud.
[0027] At the same time, considering the phenomenon of self-occlusion in point clouds, some points are at a certain viewpoint Since the bottom is invisible, for each view, by comparing the depths between points projected onto the same pixel, the visibility mask corresponding to the point with the smallest depth is set to 1, and the remaining points are set to invisible. That is:
[0028] in, For point The corresponding visibility mask, Indicates perspective Projected onto 2D mesh The depth set of all points in , The indicator function is then applied to each point using a visibility mask, i.e.:
[0029] in," " is the Hadamard product; Represented as the 2D grid coordinates of all points at viewpoint t; Represents the coordinates of the 2D grid after masking.
[0030] In step 3) above, the three-dimensional features of the point cloud are projected into two-dimensional feature spaces from different viewpoints to obtain two-dimensional feature representations from each viewpoint. This includes the following steps: 3.1) The obtained two-dimensional grid coordinates are encoded as a query token set using a multilayer perceptron; 3.2) The obtained 3D feature representation is spliced with 3D spatial coordinates and a fixed 3D mesh to generate the key set and value set in the self-attention mechanism; 3.3) Calculate the two-dimensional image features under the current viewpoint through the cross-attention mechanism, and reorganize the generated feature maps into two-dimensional feature representations through the Reshape operation.
[0031] In this embodiment, specifically, the 2D grid coordinates obtained in step 2) are encoded into a query token set using a multilayer perceptron. ,Right now:
[0032] The obtained three-dimensional feature representation will then be... with three-dimensional spatial coordinates And by stitching together 3D meshes, the key set and value set in the self-attention mechanism are generated, namely:
[0033] in," "Indicates a splicing operation, For a fixed 3D mesh, These are the key set and the value set, respectively, followed by the perspective. The two-dimensional image features can be represented as follows:
[0034] in Cross attention in Transformer blocks As a scaling factor, the generated feature maps are reorganized into two-dimensional features through the Reshape operation. .
[0035] In step 4) above, the cross-modal reconstruction loss is a weighted pixel alignment loss, and the specific construction process includes the following steps: 4.1) Input the two-dimensional feature representations from each viewpoint into the same two-dimensional decoder. The shared two-dimensional decoder generates two-dimensional images from the corresponding viewpoints. The generated two-dimensional images and the corresponding real projection images are divided into foreground and background regions, respectively. 4.2) Calculate the mean square error of the generated 2D image and the real projected image in the foreground region, and the mean square error of the generated 2D image and the real projected image in the background region, respectively; 4.3) Assign a first weight to the mean square error of the foreground region and a second weight to the mean square error of the background region; use the sum of the weighted mean square errors of the foreground region and the background region as the weighted pixel alignment loss.
[0036] In this embodiment, specifically, steps 1) to 3) are used to obtain 2D feature representations from different viewpoints. These are then fed into a shared 2D decoder to generate 2D images from different perspectives. Pixel-by-pixel alignment of multi-view images is performed by calculating the mean squared error (MSE) between each image and the Ground Truth (GT). Considering that 3D objects only constitute a portion of the rendered image and background pixels contain almost no information, foreground and background are separated when specifically calculating the MSE, and different weights are assigned to their MSE terms, effectively performing weighted pixel alignment.
[0037] in, , It refers to the foreground and background of the generated image; , These are their corresponding Ground Truth images; These are the balanced weights for the foreground and background, respectively. The mean squared error loss function is represented by T; T represents the number of samples. This represents the mean square error.
[0038] In step 4) above, the training of the point cloud self-supervised model also includes calculating the feature diversity constraint loss. The specific implementation process includes the following steps: 4.4) Based on two-dimensional feature representations from multiple different perspectives, calculate the average pairwise dot product between the two-dimensional feature representations from each perspective to measure the similarity between features from different perspectives; 4.5) The average pairwise dot product is used as the feature diversity constraint loss. By minimizing this loss, the features corresponding to different viewpoints tend to be orthogonal, thereby reducing feature redundancy between views and promoting feature diversity.
[0039] In this embodiment, specifically considering the redundancy between 2D features from different viewpoints, a feature diversity constraint loss is designed to encourage significant differences in fine-grained features between different views, enabling the model to understand unique morphological features from different views. Specifically, this is achieved by calculating the feature diversity constraint loss for each view. Figure 2 The average pairwise dot product between D features is used to measure the similarity between views. It reflects the differences in their underlying representations and highlights the variations in point cloud geometry across different views. Feature diversity constraint loss. Represented as:
[0040] in, Using the identity matrix, minimizing the loss encourages the model to reduce similarity between views. Features corresponding to any pair of views tend to be orthogonal, thereby reducing redundancy and promoting feature diversity, while also ensuring a holistic understanding of the geometry of the 3D point cloud.
[0041] In this embodiment, the optimization objective of cross-modal point cloud self-supervised learning is... It can be represented as:
[0042] in, To balance the hyperparameters.
[0043] In step 5) above, a pre-trained point cloud feature extractor is used to extract features from the trained normal samples and construct a normal prototype. During the inference phase, the features extracted from the input point cloud are compared with the normal prototype, and point clouds that deviate from the normal prototype are classified as abnormal. This includes the following steps: 5.1) Using a pre-trained point cloud feature extractor, feature extraction is performed on multiple normal point cloud samples to obtain multiple normal feature vectors. 5.2) Store multiple normal feature vectors in a memory pool to construct a normal prototype library; 5.3) During the inference phase, the pre-trained point cloud feature extractor is used to extract features from the point cloud to be detected, thereby obtaining the feature vector to be detected; 5.4) Calculate the distance or similarity between the feature vector to be detected and each normal feature vector in the normal prototype library; 5.5) When the feature vector to be detected deviates from the distribution of the normal prototype library, the point cloud to be detected is determined to be an abnormal sample.
[0044] In this embodiment, the training of the cross-modal point cloud self-supervised learning model is achieved through steps 1) to 4). The point cloud feature extractor in step 1) is used to extract features from normal point cloud samples and construct a normal prototype and store it in the memory pool. In the inference stage, the point cloud feature extractor in step 1) is also used to extract features from the point cloud to be detected and compare the extracted features with the normal prototype. If the features deviate from the distribution of the normal prototype, the input point cloud is determined to be an abnormal sample; otherwise, it is a normal sample.
[0045] In summary, this invention obtains the normal sample distribution from normal samples based on a self-supervised learning model, and identifies point clouds that deviate from the normal sample distribution as anomalies, thereby achieving unsupervised point cloud anomaly detection.
[0046] In one embodiment of the present invention, a point cloud anomaly detection system based on cross-modal self-supervised learning is provided, comprising: The point cloud feature extraction unit extracts a rough subset of the point cloud and a three-dimensional feature representation from the input point cloud through a point cloud feature extractor. The view projection and masking unit, for a given number of different viewpoints, maps the 3D coordinates of each point in the rough subset to a 2D mesh according to the rendering matrix of each viewpoint, and generates a view condition mask according to the depth of all points in the same mesh to determine the visibility of each point in the current viewpoint and obtain the 2D position code of each point in the current viewpoint. The viewpoint conditional feature projection unit, through the viewpoint conditional attention module, generates attention masks for different viewpoint conditions based on the two-dimensional position encoding and the three-dimensional feature representation of the point cloud under different viewpoints, and projects the three-dimensional features of the point cloud into the two-dimensional feature space under different viewpoints to obtain the two-dimensional feature representation under each viewpoint. The cross-modal reconstruction training unit inputs the two-dimensional feature representations from each viewpoint into the same two-dimensional decoder. The shared two-dimensional decoder generates two-dimensional images from the corresponding viewpoints and establishes a cross-modal reconstruction loss with the real projected images. The training of the point cloud self-supervised model is achieved by minimizing this loss. The anomaly detection unit uses a pre-trained point cloud feature extractor to extract features from trained normal samples and construct a normal prototype. During the inference phase, the features extracted from the input point cloud are compared with the normal prototype, and point clouds that deviate from the normal prototype are classified as anomalies.
[0047] In the above embodiments, a hierarchical point cloud feature extractor is constructed using PointNet++ and the Transformer model to obtain a rough subset of the point cloud and a 3D feature representation from the input point cloud.
[0048] In the above embodiments, obtaining the two-dimensional position code of each point in the current view includes: Based on the extracted rough subset of point cloud, given an arbitrary viewpoint, the three-dimensional spatial coordinates of all points in the rough subset are projected onto a two-dimensional mesh using a rendering matrix to obtain the coordinates of the point on the two-dimensional mesh and the depth information of the point. Traverse all points in the rough subset of the point cloud to obtain the 2D grid coordinates of all points from the current viewpoint. By comparing the depths between points projected onto the same pixel, the visibility mask corresponding to the point with the smallest depth is set to 1, and the remaining points are set to invisible. Applying a visibility mask to each point yields a two-dimensional positional code for each point from the current viewpoint.
[0049] In the above embodiments, the three-dimensional features of the point cloud are projected into a two-dimensional feature space from different viewpoints to obtain two-dimensional feature representations from each viewpoint, including: The obtained two-dimensional grid coordinates are encoded as a set of query tokens using a multilayer perceptron; The obtained 3D feature representation is concatenated with 3D spatial coordinates and a fixed 3D mesh to generate the key set and value set in the self-attention mechanism; The two-dimensional image features under the current viewpoint are calculated through a cross-attention mechanism, and the generated feature maps are reorganized into two-dimensional feature representations through a Reshape operation.
[0050] In the above embodiments, the cross-modal reconstruction loss is a weighted pixel alignment loss, and the specific construction process includes: Two-dimensional feature representations from various viewpoints are input into the same two-dimensional decoder, which generates two-dimensional images of the corresponding viewpoints. The generated two-dimensional images and the corresponding real projection images are then divided into foreground and background regions, respectively. Calculate the mean square error of the generated 2D image and the real projected image in the foreground region, and the mean square error of the generated 2D image and the real projected image in the background region, respectively. Assign a first weight to the mean square error of the foreground region and a second weight to the mean square error of the background region; use the sum of the weighted mean square errors of the foreground region and the background region as the weighted pixel alignment loss.
[0051] In the above embodiments, the training of the point cloud self-supervised model also includes calculating the feature diversity constraint loss, and the specific implementation process includes: Based on two-dimensional feature representations from multiple different perspectives, the average pairwise dot product between the two-dimensional feature representations from each perspective is calculated to measure the similarity between features from different perspectives. The average pairwise dot product is used as the feature diversity constraint loss. By minimizing this loss, the features corresponding to different viewpoints tend to be orthogonal, thereby reducing feature redundancy between views and promoting feature diversity.
[0052] In the above embodiments, a pre-trained point cloud feature extractor is used to extract features from trained normal samples and construct a normal prototype. During the inference phase, the features extracted from the input point cloud are compared with the normal prototype, and point clouds that deviate from the normal prototype are classified as abnormal, including: Using a pre-trained point cloud feature extractor, features are extracted from multiple normal point cloud samples to obtain multiple normal feature vectors. Multiple normal feature vectors are stored in a memory pool to build a normal prototype library; During the inference phase, a pre-trained point cloud feature extractor is used to extract features from the point cloud to be detected, resulting in a feature vector to be detected. Calculate the distance or similarity between the feature vector to be detected and each normal feature vector in the normal prototype library; When the feature vector to be detected deviates from the distribution of the normal prototype library, the point cloud to be detected is determined to be an abnormal sample.
[0053] The system provided in this embodiment is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.
[0054] In one embodiment of the present invention, a computing device is provided. This computing device can be a terminal and may include a processor, a communication interface, a memory, a display screen, and an input device. The processor, communication interface, and memory communicate with each other via a communication bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. When the computer programs are executed by the processor, they implement the methods described in the above embodiments. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input device can be a touch layer covering the display screen, or buttons, a trackball, or a touchpad mounted on the casing of the computing device, or an external keyboard, touchpad, or mouse. The processor can call logical instructions stored in the memory.
[0055] In one embodiment of the present invention, a computer program product is provided, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to perform the methods provided in the above-described method embodiments.
[0056] In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided, which stores server instructions that cause a computer to perform the methods provided in the above embodiments.
[0057] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A point cloud anomaly detection method based on cross-modal self-supervised learning, characterized in that, include: A coarse subset of the point cloud and a 3D feature representation are extracted from the input point cloud using a point cloud feature extractor. For a given set of multiple different viewpoints, the 3D coordinates of each point in the rough subset are mapped to a 2D mesh based on the rendering matrix of each viewpoint. A viewpoint condition mask is generated based on the depth of all points in the same mesh to determine the visibility of each point in the current viewpoint and obtain the 2D position code of each point in the current viewpoint. The viewpoint conditional attention module generates attention masks for different viewpoint conditions based on the two-dimensional position encoding and the three-dimensional feature representation of the point cloud under different viewpoints. The three-dimensional features of the point cloud are then projected into the two-dimensional feature space under different viewpoints to obtain the two-dimensional feature representation under each viewpoint. Two-dimensional feature representations from different viewpoints are input into the same two-dimensional decoder, which generates two-dimensional images of the corresponding viewpoints. A cross-modal reconstruction loss is established with the real projected image, and the training of the point cloud self-supervised model is achieved by minimizing this loss. Using a pre-trained point cloud feature extractor, features are extracted from normal samples and normal prototypes are constructed. During the inference phase, the features extracted from the input point cloud are compared with the normal prototypes, and point clouds that deviate from the normal prototypes are classified as anomalies.
2. The point cloud anomaly detection method based on cross-modal self-supervised learning as described in claim 1, characterized in that, A hierarchical point cloud feature extractor is constructed using PointNet++ and the Transformer model to obtain a coarse subset of the point cloud and a 3D feature representation from the input point cloud.
3. The point cloud anomaly detection method based on cross-modal self-supervised learning as described in claim 1, characterized in that, Obtain the two-dimensional position code of each point from the current viewpoint, including: Based on the extracted rough subset of point cloud, given an arbitrary viewpoint, the three-dimensional spatial coordinates of all points in the rough subset are projected onto a two-dimensional mesh using a rendering matrix to obtain the coordinates of the point on the two-dimensional mesh and the depth information of the point. Traverse all points in the rough subset of the point cloud to obtain the 2D grid coordinates of all points from the current viewpoint. By comparing the depths between points projected onto the same pixel, the visibility mask corresponding to the point with the smallest depth is set to 1, and the remaining points are set to invisible. Applying a visibility mask to each point yields a two-dimensional positional code for each point from the current viewpoint.
4. The point cloud anomaly detection method based on cross-modal self-supervised learning as described in claim 1, characterized in that, The 3D features of the point cloud are projected onto 2D feature spaces from different viewpoints to obtain 2D feature representations from each viewpoint, including: The obtained two-dimensional grid coordinates are encoded as a set of query tokens using a multilayer perceptron; The obtained 3D feature representation is concatenated with 3D spatial coordinates and a fixed 3D mesh to generate the key set and value set in the self-attention mechanism; The two-dimensional image features under the current viewpoint are calculated through a cross-attention mechanism, and the generated feature maps are reorganized into two-dimensional feature representations through a Reshape operation.
5. The point cloud anomaly detection method based on cross-modal self-supervised learning as described in claim 1, characterized in that, The cross-modal reconstruction loss is a weighted pixel alignment loss, and its specific construction process includes: Two-dimensional feature representations from various viewpoints are input into the same two-dimensional decoder, which generates two-dimensional images of the corresponding viewpoints. The generated two-dimensional images and the corresponding real projection images are then divided into foreground and background regions, respectively. Calculate the mean square error of the generated 2D image and the real projected image in the foreground region, and the mean square error of the generated 2D image and the real projected image in the background region, respectively. Assign a first weight to the mean square error of the foreground region and a second weight to the mean square error of the background region; use the sum of the weighted mean square errors of the foreground region and the background region as the weighted pixel alignment loss.
6. The point cloud anomaly detection method based on cross-modal self-supervised learning as described in claim 1, characterized in that, The training of a point cloud self-supervised model also includes calculating the feature diversity constraint loss. The specific implementation process includes: Based on two-dimensional feature representations from multiple different perspectives, the average pairwise dot product between the two-dimensional feature representations from each perspective is calculated to measure the similarity between features from different perspectives. The average pairwise dot product is used as the feature diversity constraint loss. By minimizing this loss, the features corresponding to different viewpoints tend to be orthogonal, thereby reducing feature redundancy between views and promoting feature diversity.
7. The point cloud anomaly detection method based on cross-modal self-supervised learning as described in claim 1, characterized in that, Using a pre-trained point cloud feature extractor, features are extracted from normal samples during training to construct a normal prototype. During the inference phase, the features extracted from the input point cloud are compared with the normal prototype, and point clouds deviating from the normal prototype are classified as anomalies, including: Using a pre-trained point cloud feature extractor, features are extracted from multiple normal point cloud samples to obtain multiple normal feature vectors. Multiple normal feature vectors are stored in a memory pool to build a normal prototype library; During the inference phase, a pre-trained point cloud feature extractor is used to extract features from the point cloud to be detected, resulting in a feature vector to be detected. Calculate the distance or similarity between the feature vector to be detected and each normal feature vector in the normal prototype library; When the feature vector to be detected deviates from the distribution of the normal prototype library, the point cloud to be detected is determined to be an abnormal sample.
8. A point cloud anomaly detection system based on cross-modal self-supervised learning, characterized in that, include: The point cloud feature extraction unit extracts a rough subset of the point cloud and a three-dimensional feature representation from the input point cloud through a point cloud feature extractor. The view projection and masking unit, for a given number of different viewpoints, maps the 3D coordinates of each point in the rough subset to a 2D mesh according to the rendering matrix of each viewpoint, and generates a view condition mask according to the depth of all points in the same mesh to determine the visibility of each point in the current viewpoint and obtain the 2D position code of each point in the current viewpoint. The viewpoint conditional feature projection unit, through the viewpoint conditional attention module, generates attention masks for different viewpoint conditions based on the two-dimensional position encoding and the three-dimensional feature representation of the point cloud under different viewpoints, and projects the three-dimensional features of the point cloud into the two-dimensional feature space under different viewpoints to obtain the two-dimensional feature representation under each viewpoint. The cross-modal reconstruction training unit inputs the two-dimensional feature representations from each viewpoint into the same two-dimensional decoder. The shared two-dimensional decoder generates two-dimensional images from the corresponding viewpoints and establishes a cross-modal reconstruction loss with the real projected images. The training of the point cloud self-supervised model is achieved by minimizing this loss. The anomaly detection unit uses a pre-trained point cloud feature extractor to extract features from trained normal samples and construct a normal prototype. During the inference phase, the features extracted from the input point cloud are compared with the normal prototype, and point clouds that deviate from the normal prototype are classified as anomalies.
9. A computer-readable storage medium for storing one or more programs, characterized in that, The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any of the methods described in claims 1 to 7.
10. A computing device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods described in claims 1 to 7.