Three-dimensional anomaly detection method and device based on point cloud density adaptive upsampling

Abstract

The invention discloses a three-dimensional anomaly detection method and device based on point cloud density adaptive upsampling, and relates to the technical field of three-dimensional anomaly detection, and the method comprises the steps: collecting a multi-modal sample of a to-be-detected product; the multi-modal sample comprises an RGB image modal and a to-be-processed point cloud modal; the RGB image modals and the point cloud modals to be processed are mapped one by one on pixels; performing point cloud density uniformity improvement operation on the to-be-processed point cloud modality by adopting a preset density adaptive technology to obtain a target point cloud modality; extracting sample feature vectors based on the target point cloud modality and the RGB image modality to obtain a plurality of corresponding point cloud modality feature vectors and a plurality of corresponding image modality feature vectors; and based on feature vector information in a preset feature template library, determining an anomaly detection result corresponding to the to-be-detected product by using each point cloud modal feature vector and each image modal feature vector. In this way, the situation that the density of the point cloud modals in the multi-modal sample is not uniform can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional anomaly detection, and in particular to a three-dimensional anomaly detection method and device based on point cloud density adaptive upsampling. Background Art

[0002] Industrial anomaly detection tasks play an important role in improving the efficiency of industrial product detection and reducing the problem of false detection caused by worker fatigue. Due to the particularity of the industrial production environment, industrial products present various forms of anomalies, and products of specific anomaly types are not easy to obtain. Therefore, in recent years, research on industrial anomaly detection tasks has largely adopted unsupervised methods to detect anomalies of industrial products. Among them, most existing methods use the image modality of the product to be detected for anomaly detection.

[0003] In order to solve the problem of detecting subtle structural anomalies that may appear on the surface of industrial products, some works have introduced the point cloud modal information of the product to be detected into the anomaly detection task, showing better anomaly detection performance than using only the image modality of the product to be detected. However, most of the existing 3D anomaly detection technologies focus on the design of feature extraction model structure and anomaly detection strategy, while ignoring the data analysis of point cloud modalities. The surface structure of industrial products is diverse, with a large number of curved structures and hole structures, resulting in uneven point cloud density in the collected point cloud modalities. Using point cloud modalities with uneven density for feature extraction will produce feature vectors with inconsistent receptive fields, which will adversely affect the performance of anomaly detection.

[0004] It can be seen from the above that how to improve the density unevenness of point cloud modes appearing in multimodal samples in the process of three-dimensional anomaly detection based on point cloud density adaptive upsampling is a problem that needs to be solved urgently. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a three-dimensional anomaly detection method and device based on point cloud density adaptive upsampling, which can improve the density unevenness of point cloud modes appearing in multimodal samples, thereby improving the quality of data collected in actual industrial scenes, and further improving the accuracy of anomaly detection results. The specific scheme is as follows:

[0006] In a first aspect, the present application provides a three-dimensional anomaly detection method based on point cloud density adaptive upsampling, comprising:

[0007] Collecting a multimodal sample of the product to be inspected; the multimodal sample includes an RGB image modality and a point cloud modality to be processed; the RGB image modality and the point cloud modality to be processed are mapped one by one on pixels;

[0008] Using a preset density adaptive technology to perform a point cloud density uniformity improvement operation on the to-be-processed point cloud mode in the multimodal sample to obtain a target point cloud mode;

[0009] Extracting sample feature vectors based on the target point cloud modality and the RGB image modality to obtain corresponding point cloud modality feature vectors and image modality feature vectors;

[0010] Based on the feature vector information in the preset feature template library, and using each of the point cloud modal feature vectors and each of the image modal feature vectors, an abnormality detection result corresponding to the product to be detected is determined.

[0011] Optionally, before the step of using a preset density adaptation technology to improve the uniformity of the point cloud density of the to-be-processed point cloud modality in the multimodal sample, the step further includes:

[0012] Determine coordinate information corresponding to each three-dimensional point in the to-be-processed point cloud modality in the multimodal sample, and determine a neighborhood range based on the coordinate information corresponding to each three-dimensional point, so as to determine a neighborhood point set and coordinate information corresponding to each neighborhood point in the neighborhood point set based on the neighborhood range;

[0013] Using a preset local coordinate system establishment technology, and based on the coordinate information corresponding to each three-dimensional point in the point cloud modality to be processed and the coordinate information corresponding to each neighborhood point in the neighborhood point set, the point cloud modality is converted from a global coordinate system to a local coordinate system;

[0014] The FPFH algorithm is used to extract local feature descriptors from each three-dimensional point in the point cloud modality to be processed and each neighborhood point in the neighborhood point set, and the local coordinate system and the local feature descriptors are fit using a preset neural network model to obtain the local surface height; wherein, there is a mapping relationship between the local coordinate system, the local feature descriptor and the local surface height.

[0015] Optionally, the adopting a preset density adaptation technology to perform a point cloud density uniformity improvement operation on the to-be-processed point cloud modality in the multimodal sample to obtain a target point cloud modality includes:

[0016] Calculating the point cloud local density corresponding to the point cloud modality to be processed based on each three-dimensional point in the point cloud modality to be processed and the neighborhood point set;

[0017] A local surface corresponding to the target point cloud modality is determined based on the local surface height, adaptive sampling is performed on the local surface based on the local density of the point cloud, and the sampled point cloud modality to be converted is converted from the local coordinate system to the global coordinate system to obtain the target point cloud modality.

[0018] Optionally, the extracting of sample feature vectors based on the target point cloud modality and the RGB image modality to obtain corresponding point cloud modality feature vectors and image modality feature vectors includes:

[0019] Performing feature extraction on the RGB image modality using a first preset deep neural network, and setting a result output by a first preset network layer of the first preset deep neural network as an image modality feature vector corresponding to the RGB image modality;

[0020] A second preset deep neural network is used to perform feature extraction on the target point cloud modality, and a result output by a second preset network layer of the second preset deep neural network is set as a point cloud modality feature vector corresponding to the target point cloud modality.

[0021] Optionally, after extracting sample feature vectors based on the target point cloud modality and the RGB image modality to obtain corresponding point cloud modality feature vectors and image modality feature vectors, the method further includes:

[0022] Each of the point cloud modal feature vectors and the image modal feature vector mapped on the pixel are spliced ​​one by one according to the pixel to obtain a plurality of multimodal sample feature vectors, so as to determine the abnormality detection result of the product to be detected based on each of the multimodal sample feature vectors.

[0023] Optionally, before determining the abnormality detection result corresponding to the product to be detected based on the feature vector information in the preset feature template library and using each of the point cloud modal feature vectors and each of the image modal feature vectors, the method further includes:

[0024] Extracting sample feature vectors of several standard products respectively to obtain corresponding several point cloud modal feature vectors and several image modal feature vectors;

[0025] Each of the point cloud modal feature vectors and each of the image modal feature vectors are spliced ​​pixel by pixel to obtain a plurality of multimodal sample feature vectors, and the plurality of multimodal sample feature vectors are stored in the preset feature template library.

[0026] Optionally, the determining of the abnormality detection result corresponding to the product to be detected based on the feature vector information in the preset feature template library and using each of the point cloud modal feature vectors and each of the image modal feature vectors includes:

[0027] Concatenating each of the point cloud modal feature vectors and the corresponding image modal feature vector to obtain a multimodal feature vector group; the multimodal feature vector group includes a plurality of multimodal feature vectors;

[0028] Calculating first Euclidean distances respectively corresponding to each of the multimodal feature vectors in the multimodal feature vector group and each feature vector information in the preset feature template library;

[0029] Determine the feature vector information corresponding to the first Euclidean distance with the smallest value among the first Euclidean distances as the template vector information corresponding to the multimodal feature vector group;

[0030] Calculating the second Euclidean distances respectively corresponding to each of the multimodal feature vectors in the multimodal feature vector group and the template vector information, and setting the second Euclidean distance with the largest value among the second Euclidean distances as the target Euclidean distance;

[0031] The target Euclidean distance is determined as an abnormality score corresponding to the corresponding product to be detected, so as to determine an abnormality detection result corresponding to the product to be detected based on the abnormality score.

[0032] In a second aspect, the present application provides a three-dimensional anomaly detection device based on point cloud density adaptive upsampling, comprising:

[0033] A modal sample acquisition module is used to acquire multimodal samples of the product to be tested; the multimodal samples include an RGB image modality and a point cloud modality to be processed; the RGB image modality and the point cloud modality to be processed are mapped one by one on pixels;

[0034] A density improvement module, configured to perform a point cloud density uniformity improvement operation on the to-be-processed point cloud modality in the multimodal sample by using a preset density adaptation technology, so as to obtain a target point cloud modality;

[0035] A vector extraction module, used to extract sample feature vectors based on the target point cloud modality and the RGB image modality, respectively, to obtain corresponding point cloud modality feature vectors and image modality feature vectors;

[0036] The abnormality detection result determination module is used to determine the abnormality detection result corresponding to the product to be detected based on the feature vector information in the preset feature template library and using each of the point cloud modal feature vectors and each of the image modal feature vectors.

[0037] Optionally, the target point cloud modality acquisition module includes:

[0038] A local density calculation unit, configured to calculate a point cloud local density corresponding to the point cloud modality to be processed based on each three-dimensional point in the point cloud modality to be processed and the neighborhood point set;

[0039] The target point cloud modality acquisition subunit is used to determine the local surface corresponding to the target point cloud modality based on the local surface height, to perform adaptive sampling on the local surface based on the local density of the point cloud, and to convert the sampled point cloud modality to be converted from the local coordinate system to the global coordinate system to obtain the target point cloud modality.

[0040] Optionally, the abnormality detection result determination module includes:

[0041] A vector group determination unit, used for splicing each of the point cloud modal feature vectors and the corresponding image modal feature vectors to obtain a multimodal feature vector group; the multimodal feature vector group includes a plurality of multimodal feature vectors;

[0042] A first Euclidean distance determination unit, used to calculate first Euclidean distances respectively corresponding to each of the multimodal feature vectors in the multimodal feature vector group and each feature vector information in the preset feature template library;

[0043] a template vector information determining unit, configured to determine feature vector information corresponding to the first Euclidean distance having the smallest value among the first Euclidean distances as template vector information corresponding to the multimodal feature vector group;

[0044] a second Euclidean distance determining unit, configured to calculate second Euclidean distances respectively corresponding to each of the multimodal feature vectors in the multimodal feature vector group and the template vector information, and set the second Euclidean distance with the largest value among the second Euclidean distances as a target Euclidean distance;

[0045] The abnormality score determining unit is used to determine the target Euclidean distance as an abnormality score corresponding to the corresponding product to be detected, so as to determine an abnormality detection result corresponding to the product to be detected based on the abnormality score.

[0046] As can be seen from the above, before performing three-dimensional anomaly detection based on point cloud density adaptive upsampling, the present application needs to collect multimodal samples of the product to be detected; the multimodal samples include RGB image modality and point cloud modality to be processed; the RGB image modality and the point cloud modality to be processed are mapped one-to-one on pixels; a preset density adaptive technology is used to perform a point cloud density uniformity improvement operation on the point cloud modality to be processed in the multimodal sample to obtain a target point cloud modality; based on the target point cloud modality and the RGB image modality, sample feature vectors are extracted respectively to obtain corresponding point cloud modality feature vectors and image modality feature vectors; based on the feature vector information in the preset feature template library, and using each of the point cloud modality feature vectors and each of the image modality feature vectors, the anomaly detection result corresponding to the product to be detected is determined. It can be seen that the present application collects the RGB image modality and the point cloud modality to be processed of the product to be inspected, and uses the preset density adaptive technology to improve the point cloud density uniformity of the point cloud modality to be processed in the multimodal sample to obtain the target point cloud modality; then, the sample feature vectors are extracted based on the target point cloud modality and the RGB image modality respectively to obtain the corresponding several point cloud modality feature vectors and several image modality feature vectors; finally, based on the feature vector information in the preset feature template library, and using each point cloud modality feature vector and each image modality feature vector, the abnormal detection result corresponding to the product to be inspected is determined. In this way, the density unevenness of the point cloud modality appearing in the multimodal sample is improved, thereby improving the quality of the data collected in the actual industrial scene, and further improving the accuracy of the abnormal detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0048] Figure 1 A flow chart of a three-dimensional anomaly detection method based on point cloud density adaptive upsampling disclosed in this application;

[0049] Figure 2 A schematic diagram of a specific point cloud density adaptive upsampling technology disclosed in this application;

[0050] Figure 3 A pseudo code schematic diagram of a specific point cloud density adaptive sampling disclosed in this application;

[0051] Figure 4A flowchart of a specific three-dimensional anomaly detection method based on point cloud density adaptive upsampling disclosed in this application;

[0052] Figure 5 This is a schematic diagram of the structure of a three-dimensional anomaly detection device based on point cloud density adaptive upsampling disclosed in this application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] Most of the existing three-dimensional anomaly detection technologies focus on the design of feature extraction model structures and anomaly detection strategy design, while ignoring the data analysis of point cloud modalities. The surface structures of industrial products are diverse, and there are a large number of curved structures and hole structures, which leads to uneven point cloud density in the collected point cloud modalities. Using point cloud modalities with uneven density for feature extraction will produce feature vectors with inconsistent receptive fields, which will in turn have an adverse effect on the performance of anomaly detection. To this end, the present application provides a three-dimensional anomaly detection method based on adaptive upsampling of point cloud density, which can improve the uneven density of point cloud modalities appearing in multimodal samples, thereby improving the quality of data collected in actual industrial scenarios.

[0055] See also Figure 1 As shown, the embodiment of the present invention discloses a three-dimensional anomaly detection method based on point cloud density adaptive upsampling, comprising:

[0056] Step S11, collecting a multimodal sample of the product to be inspected; the multimodal sample includes an RGB image modality and a point cloud modality to be processed; the RGB image modality and the point cloud modality to be processed are mapped one by one on pixels.

[0057] In this embodiment, in order to enable the image modal features and point cloud modal features of the multimodal samples to be spliced ​​pixel by pixel when multimodal feature extraction is performed on the multimodal samples of the product to be inspected, the RGB (Red, Green, Blue) image modality and point cloud modality of the multimodal samples corresponding to the product to be inspected are mapped one-to-one on the pixels.

[0058] Step S12: using a preset density adaptive technology to perform a point cloud density uniformity improvement operation on the to-be-processed point cloud modality in the multimodal sample to obtain a target point cloud modality.

[0059] In this embodiment, in the process of improving the uniformity of the point cloud density of the point cloud mode to be processed in the multimodal sample, it is necessary to perform point cloud coordinate conversion, point cloud local surface fitting, point cloud density adaptive sampling and point cloud coordinate inverse conversion. Among them, the embodiment of the present application first needs to convert the point cloud coordinates, that is, to establish a local coordinate system using each three-dimensional point in the point cloud mode of the multimodal sample and the neighborhood point set corresponding to each three-dimensional point, and based on the established local coordinate system, the point cloud mode of the multimodal sample is converted from the global coordinate system to the local coordinate system. Secondly, the embodiment of the present application needs to fit the local surface height of the point cloud: the FPFH (Fast Point Feature Histograms, i.e., a local feature descriptor for 3D point cloud data) algorithm is used to extract the local FPFH feature descriptor from each three-dimensional point and the neighborhood point set in the point cloud mode of the multimodal sample, and the preset neural network model is used to fit the mapping relationship between the local coordinate system and the local FPFH feature descriptor to the local surface height. Then, the embodiment of the present application needs to perform adaptive sampling based on the point cloud density: the local density of the point cloud is calculated using each three-dimensional point and the neighborhood point set in the point cloud modality of the multimodal sample, and the local surface is determined based on the mapping relationship between the local coordinate system and the local FPFH feature descriptor to the local surface height, so as to perform adaptive sampling in the local surface according to the local density of the point cloud. Finally, the embodiment of the present application needs to perform an inverse transformation on the point cloud coordinates: the adaptively sampled target point cloud modality is transformed from the local coordinate system to the global coordinate system.

[0060] Specifically, before using the preset density adaptive technology to improve the point cloud density uniformity of the point cloud modality to be processed in the multimodal sample, it can also include: determining the coordinate information corresponding to each three-dimensional point in the point cloud modality to be processed in the multimodal sample, and determining the neighborhood range based on the coordinate information corresponding to each three-dimensional point, so as to determine the neighborhood point set and the coordinate information corresponding to each neighborhood point in the neighborhood point set based on the neighborhood range; using the preset local coordinate system establishment technology, and based on the coordinate information corresponding to each three-dimensional point in the point cloud modality to be processed and the coordinate information corresponding to each neighborhood point in the neighborhood point set, the point cloud modality is converted from the global coordinate system to the local coordinate system; using the FPFH algorithm to extract local feature descriptors from each three-dimensional point in the point cloud modality to be processed and each neighborhood point in the neighborhood point set, so as to fit the local coordinate system and the local feature descriptor using a preset neural network model to obtain the local surface height; wherein, there is a mapping relationship between the local coordinate system and the local feature descriptor and the local surface height.

[0061] In a specific embodiment, the neighborhood point set of each three-dimensional point in the point cloud modality of the multimodal sample is a neighborhood point set that is further constructed around the center point of the point cloud block by selecting the center point of the point cloud block using a preset farthest point sampling technique.

[0062] It is understandable that if Figure 2 As shown, the original point cloud modality corresponding to the product to be inspected must undergo point cloud coordinate transformation, that is, from the global coordinate system to the local coordinate system, so as to perform point cloud local surface fitting and point cloud density adaptive sampling operations in the local coordinate system. After the point cloud density is adaptively sampled, because the multimodal features of the point cloud modality can only be extracted in the global coordinate system, the embodiment of the present application needs to perform a point cloud coordinate inverse transformation operation on the point cloud modality, that is, convert the point cloud modality from the local coordinate system to the global coordinate system. It is worth mentioning that the process of point cloud local surface fitting includes local feature encoding, local position encoding, and neural network model fitting. Among them, the local feature encoding is the local point cloud structure feature characterized by the FPFH feature descriptor; the local position encoding is the local coordinate position information encoded by the sine function. By performing local feature encoding and local position encoding on the point cloud modality, the preset neural network model's ability to understand the position information is enhanced. In a specific implementation, the preset neural network model fitting is implemented by a multilayer perceptron model (multilayer perceptron), which is used to fit the mapping relationship from the point cloud modal plane projection coordinates in the local coordinate system and the local point cloud modal FPFH feature descriptor to the local surface height.

[0063] It can be understood that point cloud density adaptive sampling is to calculate the local density of the point cloud based on each three-dimensional point and its neighborhood point set in the point cloud mode in the multimodal sample, and then adaptively sample in the local surface according to the local density of the point cloud. Specifically, the point cloud density uniformity improvement operation is performed on the point cloud mode to be processed in the multimodal sample using a preset density adaptive technology to obtain the target point cloud mode, which may include: calculating the local density of the point cloud corresponding to the point cloud mode to be processed based on each three-dimensional point and the neighborhood point set in the point cloud mode to be processed; determining the local surface corresponding to the target point cloud mode based on the local surface height, adaptively sampling on the local surface based on the local density of the point cloud, and converting the sampled point cloud mode to be converted from the local coordinate system to the global coordinate system to obtain the target point cloud mode.

[0064] In a specific implementation, the pseudo code for adaptive sampling of point cloud density is as follows: Figure 3 As shown:

[0065] The input of the pseudo code is: the original point cloud mode P converted to local coordinates, the number of points N of the original point cloud mode P, the target point cloud block density d t , the average sampling density d and the number of 3D points in the point cloud block K. Then the local surface of the point cloud obtained by fitting based on the original point cloud mode P is sampled, and the sampling ratio in each point cloud block is the same to obtain the over-sampling point set P over, and the up-sampled point set P sel Initialize to an empty set. The pseudo code is as follows:

[0066] for i=1 to N do

[0067] l←0,r←K

[0068] while l≤r do

[0069] n i ←l+⌊(r - l) / 2⌋

[0070]

[0071] if value≤d t then

[0072] l←n i +1

[0073] else

[0074] r←n i -1

[0075] end if

[0076] end while

[0077] n i ←l

[0078] At the oversampling point set P over The upsampling points corresponding to the i-th point cloud block are randomly selected n i Points constitute the point set P isel .

[0079] P sel ←P sel ∪P isel

[0080] end for

[0081] Furthermore, after running the above code, the embodiment of the present application obtains a point cloud that has undergone point cloud density adaptive upsampling: up ←P∪P sel .

[0082] It can be understood that by selecting a number of three-dimensional points from the set of over-sampling points obtained by sampling the local surface of the fitted point cloud, and adding the selected three-dimensional points to the corresponding point cloud blocks, each point cloud block can reach the specified target point cloud block density, thereby improving the uniformity of the point cloud mode of the original multimodal sample.

[0083] Step S13: extracting sample feature vectors based on the target point cloud modality and the RGB image modality respectively to obtain corresponding point cloud modality feature vectors and image modality feature vectors.

[0084] In this embodiment, after obtaining a target point cloud modality with uniform density, the embodiment of the present application needs to extract sample feature vectors for the target point cloud modality and the RGB image modality respectively. Among them, the models used for feature extraction of the target point cloud modality and the RGB image modality are different. Specifically, based on the target point cloud modality and the RGB image modality, respectively, sample feature vectors are extracted to obtain corresponding point cloud modality feature vectors and image modality feature vectors, which may include: using a first preset deep neural network to extract features from the RGB image modality, and setting the result of the output of the first preset network layer of the first preset deep neural network as the image modality feature vector corresponding to the RGB image modality; using a second preset deep neural network to extract features from the target point cloud modality, and setting the result of the output of the second preset network layer of the second preset deep neural network as the point cloud modality feature vector corresponding to the target point cloud modality. Among them, the first preset network layer and the second preset network layer can be set based on the needs of the developer, and will not be specifically elaborated here.

[0085] In one specific implementation, the present application embodiment uses a preset Vision Transformer (i.e., a neural network architecture based on a self-attention mechanism) deep neural network to extract features from the RGB image modality in the multimodal sample, and uses the output of the last layer of the Vision Transformer deep neural network as the feature vector of the image modality. In another specific implementation, the present application embodiment uses a preset Point Transformer (i.e., a network structure that introduces position encoding in point cloud processing) deep neural network to extract features from the point cloud modality in the multimodal sample, and uses the output of the middle layer of the Point Transformer deep neural network as the feature vector of the point cloud modality.

[0086] It can be understood that the Vision Transformer deep neural network consists of an embedding layer and a Transformer encoder. Furthermore, the RGB image modality of the multimodal sample of the product to be inspected needs to be evenly divided into several small image blocks before being input into the Vision Transformer deep neural network. The embedding layer in the Vision Transformer deep neural network will arrange the obtained small image blocks into an image sequence according to the positional relationship, and map the image sequence and its corresponding position information into the input feature vector of the Transformer encoder, wherein the Transformer encoder is formed by cascading several Transformer blocks of the same structure, and each Transformer block is mainly composed of a multi-head attention layer and a feedforward layer. The multi-head attention layer is used to project the feature vector of the image block into different subspaces, and calculate the self-attention mechanism in the subspace, so that the calculated attention score will be used to weight the feature vector. In a specific embodiment, the calculation method of the self-attention mechanism in a projection subspace of the multi-head attention layer is as follows:

[0087] ;

[0088] Where Q is the query vector; K is the key vector, which matches the query vector Q; V is the value vector; d k Represents the dimension of the key vector K.

[0089] It is worth mentioning that the self-attention mechanism can effectively capture the structural and semantic dependencies between each image block in the image sequence, that is, after merging the weighted feature vectors in different projection subspaces, the obtained feature vector will be input into the feedforward layer for further feature processing of the feature vector in the feedforward layer. Among them, the feedforward layer includes two linear layers, and a nonlinear activation function is contained between the two linear layers to use the nonlinear activation function to characterize the nonlinear relationship. Each Transformer block of the Transformer encoder has a residual connection and layer normalization mechanism. After passing through N cascaded Transformer block structures, the Transformer encoder will abstract the image sequence formed by the small image blocks that have been input according to the positional relationship into a high-dimensional feature vector. The above high-dimensional feature vector is the image modality feature vector and will be used in the subsequent anomaly detection process.

[0090] It can be understood that the Point Transformer deep neural network consists of an embedding layer and a Transformer encoder. The embedding layer maps the point cloud block sequence and the center point position information arranged according to the position relationship of the point cloud modality after point cloud density adaptive upsampling into the input feature vector of the Transformer encoder. The Transformer encoder is formed by cascading several Transformer blocks with the same structure. It is worth mentioning that the above explanation of the Transformer encoder of the Vision Transformer deep neural network is also applicable to the Transformer encoder of the Point Transformer deep neural network, and will not be repeated here.

[0091] Furthermore, after obtaining several RGB image modal feature vectors and several point cloud modal feature vectors corresponding to the product to be inspected, the embodiment of the present application needs to perform an interpolation operation on the point cloud modal feature vectors, and align and splice the point cloud modal feature vectors obtained after the interpolation operation with the RGB image modal feature vectors pixel by pixel, so as to obtain a multimodal feature vector.

[0092] It can be understood that after extracting sample feature vectors from the target point cloud modality and the RGB image modality respectively, and obtaining a number of point cloud modality feature vectors and a number of image modality feature vectors, the embodiment of the present application needs to splice the several point cloud modality feature vectors and the several image modality feature vectors one by one according to the pixels, so as to obtain a number of multimodal sample feature vectors, so as to achieve information complementarity between the RGB image modality and the point cloud modality, and then be able to perform more detailed abnormality detection on the subtle structure and color morphology of the product to be inspected, and determine the abnormality score of the product to be inspected based on the multimodal sample feature vector. Specifically, after extracting sample feature vectors based on the target point cloud modality and the RGB image modality respectively, so as to obtain a number of corresponding point cloud modality feature vectors and a number of image modality feature vectors, it can also include: splicing each point cloud modality feature vector with the image modality feature vector mapped on the pixel one by one according to the pixels, so as to obtain a number of multimodal sample feature vectors, so as to determine the abnormality detection result of the product to be inspected based on each multimodal sample feature vector.

[0093] Step S14: Based on the feature vector information in the preset feature template library, and using each of the point cloud modal feature vectors and each of the image modal feature vectors, determine the abnormality detection result corresponding to the product to be detected.

[0094] In this embodiment, before using each point cloud modal feature vector and each image modal feature vector to determine the abnormal detection result corresponding to the product to be detected, the embodiment of the present application needs to construct a preset feature template library based on the feature vector corresponding to the standard product, so as to use the preset feature template library to determine whether the product to be detected is abnormal. Specifically, based on the feature vector information in the preset feature template library, and before using each point cloud modal feature vector and each image modal feature vector to determine the abnormal detection result corresponding to the product to be detected, it can also include: extracting sample feature vectors of several standard products respectively to obtain corresponding several point cloud modal feature vectors and several image modal feature vectors; splicing each point cloud modal feature vector and each image modal feature vector one by one according to pixels to obtain several multimodal sample feature vectors, and storing several multimodal sample feature vectors in the preset feature template library. Among them, the multimodal sample feature vectors in the preset feature template library can be obtained using a preset core set selection technology.

[0095] It is worth mentioning that Figure 4 As shown in the figure, the acquisition of abnormal detection results of the product to be detected includes the offline feature library construction stage and the online product to be detected abnormal detection stage. Among them, the offline feature library construction stage mainly performs the operation of constructing the preset feature template library, and the online product to be detected abnormal detection stage mainly uses the feature template library and several multimodal sample feature vectors corresponding to the product to be detected to calculate the abnormal score of the product to be detected. Specifically, based on the feature vector information in the preset feature template library, and using each point cloud modal feature vector and each image modal feature vector to determine the abnormality detection result corresponding to the product to be detected, it can include: splicing each point cloud modal feature vector and the corresponding image modal feature vector to obtain a multimodal feature vector group; the multimodal feature vector group includes a plurality of multimodal feature vectors; calculating the first Euclidean distances corresponding to each multimodal feature vector in the multimodal feature vector group and each feature vector information in the preset feature template library; determining the feature vector information corresponding to the first Euclidean distance with the smallest value among the first Euclidean distances as the template vector information corresponding to the multimodal feature vector group; calculating the second Euclidean distances corresponding to each multimodal feature vector in the multimodal feature vector group and the template vector information, and setting the second Euclidean distance with the largest value among the second Euclidean distances as the target Euclidean distance; determining the target Euclidean distance as the abnormality score corresponding to the corresponding product to be detected, so as to determine the abnormality detection result corresponding to the product to be detected based on the abnormality score.

[0096] It can be understood that the larger the value of the anomaly score is, the more it can reflect the greater the degree of deviation between the feature vector of the multimodal sample corresponding to the product to be detected and the feature vector of the multimodal sample corresponding to the standard product in the feature template library. Therefore, in the stage of anomaly detection of the product to be detected, the maximum value of the Euclidean distance between the feature vector extracted from the multimodal sample corresponding to the product to be detected and the corresponding template feature vector in the template library is used as the anomaly score of the product to be detected, and the degree of abnormality of the product to be detected is judged based on the anomaly score.

[0097] As can be seen from the above, before the embodiment of the present application performs three-dimensional anomaly detection based on point cloud density adaptive upsampling, it is necessary to collect the RGB image modality and the point cloud modality to be processed of the product to be detected, and use the preset density adaptive technology to improve the uniformity of the point cloud density of the point cloud modality to be processed in the multimodal sample to obtain the target point cloud modality; then extract the sample feature vector based on the target point cloud modality and the RGB image modality respectively to obtain the corresponding several point cloud modality feature vectors and several image modality feature vectors; finally, based on the feature vector information in the preset feature template library, and using each point cloud modality feature vector and each image modality feature vector to determine the anomaly detection result corresponding to the product to be detected. In this way, the embodiment of the present application uses the image modality and point cloud modality of the multimodal sample of the product to be detected to perform anomaly detection on the subtle structure and color morphology of the product to be detected, and improves the density unevenness of the point cloud modality in the multimodal sample, thereby improving the quality of the data collected in the actual industrial scene, and then improving the accuracy of the anomaly detection result.

[0098] Accordingly, see Figure 5 As shown, the present application also provides a three-dimensional anomaly detection device based on point cloud density adaptive upsampling, comprising:

[0099] The modal sample acquisition module 11 is used to acquire multimodal samples of the product to be tested; the multimodal samples include RGB image modality and point cloud modality to be processed; the RGB image modality and the point cloud modality to be processed are mapped one by one on pixels;

[0100] A target point cloud modality acquisition module 12 is used to perform a point cloud density uniformity improvement operation on the to-be-processed point cloud modality in the multimodal sample by using a preset density adaptive technology to obtain a target point cloud modality;

[0101] A vector extraction module 13, used to extract sample feature vectors based on the target point cloud modality and the RGB image modality, respectively, to obtain a corresponding number of point cloud modality feature vectors and a number of image modality feature vectors;

[0102] The abnormality detection result determination module 14 is used to determine the abnormality detection result corresponding to the product to be detected based on the feature vector information in the preset feature template library and using each of the point cloud modal feature vectors and each of the image modal feature vectors.

[0103] As can be seen from the above, before performing three-dimensional anomaly detection based on adaptive upsampling of point cloud density, the embodiment of the present application needs to collect the RGB image modality and the point cloud modality to be processed of the product to be detected, and use the preset density adaptive technology to improve the uniformity of the point cloud density of the point cloud modality to be processed in the multimodal sample to obtain the target point cloud modality; then, based on the target point cloud modality and the RGB image modality, the sample feature vectors are extracted respectively to obtain the corresponding several point cloud modality feature vectors and several image modality feature vectors; finally, based on the feature vector information in the preset feature template library, and using each three-dimensional point cloud modality feature vector and each image modality feature vector, the anomaly detection result corresponding to the product to be detected is determined. In this way, the density unevenness of the point cloud modality appearing in the multimodal sample is improved, thereby improving the quality of the data collected in the actual industrial scene, thereby improving the accuracy of the anomaly detection results and enhancing the user experience.

[0104] In some specific implementations, the three-dimensional anomaly detection device based on point cloud density adaptive upsampling may further include:

[0105] A coordinate information determination unit, used to determine the coordinate information corresponding to each three-dimensional point in the to-be-processed point cloud modality in the multimodal sample, and determine a neighborhood range based on the coordinate information corresponding to each three-dimensional point, so as to determine a neighborhood point set and coordinate information corresponding to each neighborhood point in the neighborhood point set based on the neighborhood range;

[0106] A local coordinate system conversion unit, used to convert the point cloud modality from the global coordinate system to the local coordinate system based on the coordinate information corresponding to each three-dimensional point in the point cloud modality to be processed and the coordinate information corresponding to each neighborhood point in the neighborhood point set by using a preset local coordinate system establishment technology;

[0107] A local surface height acquisition unit is used to extract local feature descriptors from each three-dimensional point in the point cloud modality to be processed and each neighborhood point in the neighborhood point set using an FPFH algorithm, so as to fit the local coordinate system and the local feature descriptors using a preset neural network model to obtain the local surface height; wherein a mapping relationship exists between the local coordinate system, the local feature descriptors and the local surface height.

[0108] In some specific implementations, the target point cloud modality acquisition module 12 may specifically include:

[0109] A local density calculation unit, configured to calculate a point cloud local density corresponding to the point cloud modality to be processed based on each three-dimensional point in the point cloud modality to be processed and the neighborhood point set;

[0110] The target point cloud modality acquisition subunit is used to determine the local surface corresponding to the target point cloud modality based on the local surface height, to perform adaptive sampling on the local surface based on the local density of the point cloud, and to convert the sampled point cloud modality to be converted from the local coordinate system to the global coordinate system to obtain the target point cloud modality.

[0111] In some specific implementations, the vector extraction module 13 may specifically include:

[0112] A first vector extraction unit, configured to perform feature extraction on the RGB image modality using a first preset deep neural network, and set a result output by a first preset network layer of the first preset deep neural network as an image modality feature vector corresponding to the RGB image modality;

[0113] A second vector extraction unit is used to perform feature extraction on the target point cloud modality using a second preset deep neural network, and set a result output by a second preset network layer of the second preset deep neural network as a point cloud modality feature vector corresponding to the target point cloud modality.

[0114] In some specific implementations, the three-dimensional anomaly detection device based on point cloud density adaptive upsampling may further include:

[0115] A vector stitching unit is used to stitch each of the point cloud modal feature vectors with the image modal feature vectors mapped on the pixels one by one according to the pixels to obtain a plurality of multimodal sample feature vectors, so as to determine the abnormality detection result of the product to be detected based on each of the multimodal sample feature vectors.

[0116] In some specific implementations, the three-dimensional anomaly detection device based on point cloud density adaptive upsampling may further include:

[0117] A third vector extraction unit is used to extract sample feature vectors of a plurality of standard products respectively, so as to obtain a plurality of corresponding point cloud modal feature vectors and a plurality of image modal feature vectors;

[0118] A vector storage unit is used to splice each of the point cloud modal feature vectors and each of the image modal feature vectors one by one according to pixels to obtain a plurality of multimodal sample feature vectors, and store the plurality of multimodal sample feature vectors in the preset feature template library.

[0119] In some specific implementations, the abnormal detection result determination module 14 may specifically include:

[0120] A vector group determination unit, used for splicing each of the point cloud modal feature vectors and the corresponding image modal feature vectors to obtain a multimodal feature vector group; the multimodal feature vector group includes a plurality of multimodal feature vectors;

[0121] A first Euclidean distance determination unit, used to calculate first Euclidean distances respectively corresponding to each of the multimodal feature vectors in the multimodal feature vector group and each feature vector information in the preset feature template library;

[0122] a template vector information determining unit, configured to determine feature vector information corresponding to the first Euclidean distance having the smallest value among the first Euclidean distances as template vector information corresponding to the multimodal feature vector group;

[0123] a second Euclidean distance determining unit, configured to calculate second Euclidean distances respectively corresponding to each of the multimodal feature vectors in the multimodal feature vector group and the template vector information, and set the second Euclidean distance with the largest value among the second Euclidean distances as a target Euclidean distance;

[0124] The abnormality score determining unit is used to determine the target Euclidean distance as an abnormality score corresponding to the corresponding product to be detected, so as to determine an abnormality detection result corresponding to the product to be detected based on the abnormality score.

[0125] Furthermore, the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0126] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0127] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0128] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0129] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A three-dimensional anomaly detection method based on point cloud density adaptive upsampling, characterized in that: include: Collecting multimodal samples of the product to be inspected; the multimodal samples include RGB image modality and point cloud modality to be processed; The RGB image modality and the point cloud modality to be processed are mapped one by one on pixels; Using a preset density adaptive technology to perform a point cloud density uniformity improvement operation on the to-be-processed point cloud mode in the multimodal sample to obtain a target point cloud mode; Extracting sample feature vectors based on the target point cloud modality and the RGB image modality to obtain corresponding point cloud modality feature vectors and image modality feature vectors; Based on the feature vector information in the preset feature template library, and using each of the point cloud modal feature vectors and each of the image modal feature vectors, an abnormality detection result corresponding to the product to be detected is determined.

2. The three-dimensional anomaly detection method based on point cloud density adaptive upsampling according to claim 1 is characterized in that: Before the preset density adaptation technology is used to improve the uniformity of the point cloud density of the to-be-processed point cloud mode in the multimodal sample, the method further includes: Determine coordinate information corresponding to each three-dimensional point in the to-be-processed point cloud modality in the multimodal sample, and determine a neighborhood range based on the coordinate information corresponding to each three-dimensional point, so as to determine a neighborhood point set and coordinate information corresponding to each neighborhood point in the neighborhood point set based on the neighborhood range; Using a preset local coordinate system establishment technology, and based on the coordinate information corresponding to each three-dimensional point in the point cloud modality to be processed and the coordinate information corresponding to each neighborhood point in the neighborhood point set, the point cloud modality is converted from a global coordinate system to a local coordinate system; The FPFH algorithm is used to extract local feature descriptors from each three-dimensional point in the point cloud modality to be processed and each neighborhood point in the neighborhood point set, and the local coordinate system and the local feature descriptors are fit using a preset neural network model to obtain the local surface height; wherein, there is a mapping relationship between the local coordinate system, the local feature descriptor and the local surface height.

3. The three-dimensional anomaly detection method based on point cloud density adaptive upsampling according to claim 2 is characterized in that: The step of using a preset density adaptation technology to perform a point cloud density uniformity improvement operation on the to-be-processed point cloud modality in the multimodal sample to obtain a target point cloud modality includes: Calculating the point cloud local density corresponding to the point cloud modality to be processed based on each three-dimensional point in the point cloud modality to be processed and the neighborhood point set; A local surface corresponding to the target point cloud modality is determined based on the local surface height, adaptive sampling is performed on the local surface based on the local density of the point cloud, and the sampled point cloud modality to be converted is converted from the local coordinate system to the global coordinate system to obtain the target point cloud modality.

4. The three-dimensional anomaly detection method based on point cloud density adaptive upsampling according to claim 1, characterized in that: The extracting of sample feature vectors based on the target point cloud modality and the RGB image modality to obtain corresponding point cloud modality feature vectors and image modality feature vectors includes: Performing feature extraction on the RGB image modality using a first preset deep neural network, and setting a result output by a first preset network layer of the first preset deep neural network as an image modality feature vector corresponding to the RGB image modality; A second preset deep neural network is used to perform feature extraction on the target point cloud modality, and a result output by a second preset network layer of the second preset deep neural network is set as a point cloud modality feature vector corresponding to the target point cloud modality.

5. The three-dimensional anomaly detection method based on point cloud density adaptive upsampling according to claim 1, characterized in that: After extracting sample feature vectors based on the target point cloud modality and the RGB image modality to obtain corresponding point cloud modality feature vectors and image modality feature vectors, the method further includes: Each of the point cloud modal feature vectors and the image modal feature vectors mapped on the pixels are spliced ​​one by one according to the pixels to obtain a plurality of multimodal sample feature vectors, so as to determine the abnormality detection result of the product to be detected based on each of the multimodal sample feature vectors.

6. The three-dimensional anomaly detection method based on point cloud density adaptive upsampling according to claim 5, characterized in that: Before determining the abnormality detection result corresponding to the product to be detected based on the feature vector information in the preset feature template library and using each of the point cloud modal feature vectors and each of the image modal feature vectors, the method further includes: Extracting sample feature vectors of several standard products respectively to obtain corresponding several point cloud modal feature vectors and several image modal feature vectors; Each of the point cloud modal feature vectors and each of the image modal feature vectors are spliced ​​pixel by pixel to obtain a plurality of multimodal sample feature vectors, and the plurality of multimodal sample feature vectors are stored in the preset feature template library.

7. The three-dimensional anomaly detection method based on point cloud density adaptive upsampling according to any one of claims 1 to 6, characterized in that: The determining of the abnormality detection result corresponding to the product to be detected based on the feature vector information in the preset feature template library and using each of the point cloud modal feature vectors and each of the image modal feature vectors includes: Concatenating each of the point cloud modal feature vectors and the corresponding image modal feature vector to obtain a multimodal feature vector group; the multimodal feature vector group includes a plurality of multimodal feature vectors; Calculating first Euclidean distances respectively corresponding to each of the multimodal feature vectors in the multimodal feature vector group and each feature vector information in the preset feature template library; Determine the feature vector information corresponding to the first Euclidean distance with the smallest value among the first Euclidean distances as the template vector information corresponding to the multimodal feature vector group; Calculating the second Euclidean distances respectively corresponding to each of the multimodal feature vectors in the multimodal feature vector group and the template vector information, and setting the second Euclidean distance with the largest value among the second Euclidean distances as the target Euclidean distance; The target Euclidean distance is determined as an abnormality score corresponding to the corresponding product to be detected, so as to determine an abnormality detection result corresponding to the product to be detected based on the abnormality score.

8. A three-dimensional anomaly detection device based on point cloud density adaptive upsampling, characterized in that: include: A modal sample acquisition module is used to acquire multimodal samples of the product to be tested; the multimodal samples include an RGB image modality and a point cloud modality to be processed; the RGB image modality and the point cloud modality to be processed are mapped one by one on pixels; A target point cloud modality acquisition module is used to perform a point cloud density uniformity improvement operation on the to-be-processed point cloud modality in the multimodal sample by using a preset density adaptive technology to obtain a target point cloud modality; A vector extraction module, used to extract sample feature vectors based on the target point cloud modality and the RGB image modality, respectively, to obtain corresponding point cloud modality feature vectors and image modality feature vectors; The abnormality detection result determination module is used to determine the abnormality detection result corresponding to the product to be detected based on the feature vector information in the preset feature template library and using each of the point cloud modal feature vectors and each of the image modal feature vectors.

9. The three-dimensional anomaly detection device based on point cloud density adaptive upsampling according to claim 8, characterized in that: The target point cloud modality acquisition module includes: A local density calculation unit, configured to calculate a point cloud local density corresponding to the point cloud modality to be processed based on each three-dimensional point in the point cloud modality to be processed and the neighborhood point set; The target point cloud modality acquisition subunit is used to determine the local surface corresponding to the target point cloud modality based on the local surface height, to perform adaptive sampling on the local surface based on the local density of the point cloud, and to convert the sampled point cloud modality to be converted from the local coordinate system to the global coordinate system to obtain the target point cloud modality.

10. The three-dimensional anomaly detection device based on point cloud density adaptive upsampling according to claim 8, characterized in that: The abnormal detection result determination module comprises: A vector group determination unit, used for splicing each of the point cloud modal feature vectors and the corresponding image modal feature vectors to obtain a multimodal feature vector group; the multimodal feature vector group includes a plurality of multimodal feature vectors; A first Euclidean distance determination unit, used to calculate first Euclidean distances respectively corresponding to each of the multimodal feature vectors in the multimodal feature vector group and each feature vector information in the preset feature template library; a template vector information determining unit, configured to determine feature vector information corresponding to the first Euclidean distance having the smallest value among the first Euclidean distances as template vector information corresponding to the multimodal feature vector group; a second Euclidean distance determining unit, configured to calculate second Euclidean distances respectively corresponding to each of the multimodal feature vectors in the multimodal feature vector group and the template vector information, and set the second Euclidean distance with the largest value among the second Euclidean distances as a target Euclidean distance; The abnormality score determining unit is used to determine the target Euclidean distance as an abnormality score corresponding to the corresponding product to be detected, so as to determine an abnormality detection result corresponding to the product to be detected based on the abnormality score.

Images