A complementary pseudo-multimodal feature system and method

Abstract

The present invention belongs to the technical field related to industrial image processing, and discloses a complementary pseudo-multimodal feature system and method. The method performs multimodal feature extraction on point cloud data through three-dimensional and two-dimensional modal feature extraction modules. The two-dimensional modal feature extraction module includes three steps: multi-view rendering, image feature extraction, and feature back-projection. After extracting different modal features, they are fused to obtain complementary pseudo-multimodal features that contain both local structural information and global semantic information. This feature can be used for point cloud anomaly detection. In the anomaly detection training phase, a feature library containing all training data is constructed; in the testing phase, the nearest neighbor search is used to calculate the Euclidean distance from the test feature to the most similar normal feature in the feature library. This distance can be used to characterize the degree of anomaly. This method can effectively improve the performance of existing point cloud defect detection, reduce labor costs, and improve the automation and intelligence level of production line quality inspection.

Description

Technical Field

[0001] The present invention belongs to the technical field related to industrial image processing, and more specifically, relates to a complementary pseudo-multimodal feature system and method. Background Art

[0002] Currently, various industries, such as machinery, electronics, and food, are undergoing intelligent transformation of production equipment. Defect detection in product quality control is particularly important. In the early days, product quality was primarily determined by skilled workers through manual inspections, but manual experience varies and workers are prone to fatigue, making it difficult to guarantee a high pass rate for product quality after manual screening.

[0003] In recent years, intelligent product quality inspection methods have emerged, among which the inspection method based on machine vision is the main one. With the development of related methods, product quality inspection has gradually expanded from flat defect detection of mobile phone screens in 3C electronics to defect detection of complex curved parts. Due to the complexity of the curved surface, it is difficult to capture the entire surface to be tested with a single perspective, and the acquisition process is prone to problems such as uneven lighting and reflection. The application of traditional image acquisition methods has encountered bottlenecks. Therefore, three-dimensional sensors that can sense the geometric morphology of objects have gradually attracted the attention of companies. Three-dimensional sensors can directly sense the size of the object to be tested and convert relevant information into point cloud data, which is expected to greatly improve the defect detection effect of complex curved parts. In summary, the present invention mainly focuses on point cloud defect detection methods.

[0004] Currently, there is little research on point cloud defect detection methods, and the detection methods achieved in relevant literature have been ineffective. This is mainly due to the fact that the features extracted during the detection process only focus on the local point cloud structure and fail to capture the global point cloud semantics. Therefore, this paper aims to propose more descriptive point cloud features that simultaneously contain local structural information and global semantic information, thereby achieving better point cloud defect detection results.

[0005] In summary, point cloud defect detection is a crucial research direction and technology in industrial quality control. However, existing point cloud defect detection methods lack the ability to capture the semantic information of point cloud data, and their detection performance still needs to be improved. Therefore, this study considers how to develop more descriptive point cloud features to further enhance the robustness and accuracy of existing defect detection models. Summary of the Invention

[0006] In response to the above-mentioned defects or improvement needs of the prior art, the present invention provides a complementary pseudo-multimodal feature system and method to solve the problem of poor effect in point cloud defect detection.

[0007] To achieve the above objectives, according to one aspect of the present invention, a complementary pseudo-multimodal feature system is provided. The complementary pseudo-multimodal feature system includes three modules: a three-dimensional modal feature extraction module, a two-dimensional modal feature extraction module, and a feature fusion module, wherein:

[0008] The three-dimensional modal feature extraction module is used to extract local geometric information of the point cloud data to be processed, so as to obtain the three-dimensional modal features of the point cloud data to be processed;

[0009] The two-dimensional modal feature extraction module will be used to render the point cloud data to be processed to generate a multi-view image, and then use the pre-trained network to extract features from the multi-view image. The image features at each view obtained by the feature extraction are then back-projected into the point cloud space to obtain the point features corresponding to the view. Finally, the point features corresponding to all view angles are fused to obtain the two-dimensional modal features of each point in the point cloud data to be processed.

[0010] The feature fusion module is used to fuse the three-dimensional modal features and the two-dimensional modal features to obtain complementary pseudo multimodal features.

[0011] Further preferably, the three-dimensional modal feature extraction is performed according to the following relationship:

[0012]

[0013] Among them, P 3D is the input point cloud data, N is the number of points contained in the point cloud; is the three-dimensional modal feature extraction function; F 3D To extract the three-dimensional modal features, D 3D is the dimension of the three-dimensional modal feature.

[0014] Further preferably, the two-dimensional modal feature extraction module obtains the multi-view image according to the following steps:

[0015] First, the point cloud data to be processed is rotated to different angles using different rotation angles;

[0016] Then, the rotated point cloud is rendered to obtain a multi-view image.

[0017] Further preferably, the point cloud data to be processed is rotated to different angles according to the following relationship using different rotation angles:

[0018] P 3D,k,i =R k P 3D,i

[0019] Among them, Rk represents the kth rotation matrix, P 3D,k represents the kth point cloud obtained by rotation, and i∈[0,N] represents the i-th point in the point cloud.

[0020] The two-dimensional coordinates of the rendered image and the points in the image are determined according to the following relationship:

[0021]

[0022]

[0023] in, Represents the rendering function, which rotates the kth point cloud P 3D,k Render the image I of the kth perspective k ; K is the intrinsic matrix of the camera used for rendering; T is the corresponding extrinsic matrix; 1 / Z c is the normalized coordinate coefficient; i∈[0,N] represents the i-th point in the point cloud.

[0024] Further preferably, the image feature extraction is performed according to the following relationship:

[0025]

[0026] in, is the feature extraction function of the image, is the extracted feature map of the k-th perspective.

[0027] Further preferably, the reverse projection of the image features into the point cloud space is performed according to the following relationship:

[0028]

[0029] Among them, P 2D,k,i 、F 2D,k,i are the coordinate points and point features corresponding to the i-th point in the k-th perspective, Representing image features Medium P 2D,k,i Features at the location.

[0030] Further preferably, the two-dimensional modal feature is performed according to the following relationship:

[0031]

[0032] Among them, F 2D is the final two-dimensional modal point feature, k represents the k-th perspective, N V Indicates the total number of viewing angles.

[0033] Further preferably, the feature fusion module performs feature fusion according to the following relationship:

[0034]

[0035] in, The simplified point cloud P 3D The CPMF feature extraction function expression is obtained by CPMF It also includes information extracted from three-dimensional and two-dimensional modes.

[0036] According to another aspect of the present invention, a method for detecting geometric defects in a point cloud using the system described above is provided, the method comprising the following steps:

[0037] S1 uses the three-dimensional modal extraction module features to extract three-dimensional modal features from the point cloud data to be processed;

[0038] S2 uses the two-dimensional modal extraction module features to extract two-dimensional modal features from the point cloud data to be processed;

[0039] S3 uses the feature fusion module to fuse the three-dimensional modal features and the two-dimensional modal features to obtain complementary pseudo multimodal features;

[0040] S4 uses the complementary pseudo multimodal features to perform point cloud geometric defect anomaly detection.

[0041] Further preferably, in step S4, the point cloud geometric defect anomaly detection is performed according to the following steps:

[0042] S41 extracts complementary pseudo-multimodal features from the point cloud data of the normal sample, thereby obtaining complementary pseudo-multimodal features of all points of the normal sample, and forming a feature library of complementary pseudo-multimodal features of the normal sample;

[0043] S42 performs complementary pseudo-multimodal feature analysis on the point cloud data of the sample to be processed, thereby obtaining complementary pseudo-multimodal features of all points of the sample to be processed, and calculating the minimum Euclidean distance between the complementary pseudo-multimodal features of the sample to be processed and the normal sample point by point;

[0044] S43: When the Euclidean distance obtained in step S42 is greater than a preset threshold, it is abnormal.

[0045] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0046] 1. This invention fully exploits the implicit information in the 3D geometric structure of objects for point cloud anomaly detection. It describes the input object point cloud data from both 3D and 2D modalities, extracting complementary 3D and 2D modal features. This significantly improves the performance of point cloud defect detection and has good adaptability to point cloud defect detection for different types of industrial products.

[0047] 2. The complementary pseudo-multimodal feature system provided by the present invention has features described in three-dimensional mode that can better describe the local geometric structure, while features described in two-dimensional mode have stronger global semantic information. The point cloud feature properties extracted in different modes are highly different and complementary. Therefore, the CPMF obtained by fusing the two has excellent point cloud anomaly detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a framework of a complementary pseudo-multimodal feature method constructed according to a preferred embodiment of the present invention;

[0049] Figure 2 Schematic diagram of generating multi-view images by rotating point cloud data according to a preferred embodiment of the present invention, wherein (a) is a schematic diagram of point cloud data before rotation; (b) is a schematic diagram of point cloud data after rotation;

[0050] Figure 3 is a generated multi-view image constructed according to a preferred embodiment of the present invention;

[0051] Figure 4 This is an example of the detection results of the point cloud geometric defect detection method constructed according to the preferred embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0053] A system based on complementary pseudo-multimodal features (CPMF) consists of a 3D modal feature extraction module and a 2D modal feature extraction module. These two modules can extract highly discriminative point cloud features.

[0054] CPMF first transforms the input point cloud P 3D Input these two modules respectively to extract the three-dimensional modal features F3D and the two-dimensional modal characteristics F 2D In the 3D modal feature extraction module, the present invention directly uses traditional point cloud manual descriptors to extract local geometric information of point clouds.

[0055] The 2D modal feature extraction module first transforms the input point cloud P 3D Generate multi-view images by rendering, that is, a pseudo two-dimensional modality generated by using three-dimensional information. Then, CPMF uses the pre-trained network to extract features from the multi-view images and further obtain the two-dimensional modal features of each point through feature back projection. CPMF uses the feature fusion module to obtain the three-dimensional modal features F 3D and the two-dimensional modal characteristics F 2D Fusion is performed to obtain the final complementary pseudo multimodal feature F CPMF .

[0056] F CPMF It can be directly used for subsequent anomaly detection. The framework of CPMF is as follows Figure 1 shown.

[0057] The following will introduce the three-dimensional modal feature extraction module, two-dimensional modal feature extraction module, feature fusion module and the corresponding anomaly detection process respectively.

[0058] (1) 3D modal feature extraction module

[0059] The disordered nature of point cloud data makes point cloud feature extraction difficult. Early manual point cloud descriptors constructed point cloud features by describing the local neighborhood structure of the point cloud. This type of feature can accurately and effectively describe the local geometric information of the point cloud. Therefore, this invention uses manual descriptors to describe the point cloud in a three-dimensional modality. The method for obtaining three-dimensional modal features is as follows:

[0060]

[0061] in, is the three-dimensional modal feature extraction function; F 3D To extract the three-dimensional modal features, D 3D is the dimension of the three-dimensional modal feature.

[0062] (2) Two-dimensional modal feature extraction module

[0063] Manual point cloud descriptors have shown preliminary capabilities in point cloud anomaly detection, but the heuristic design of manual point cloud descriptors, which describes local neighborhood structures, makes it difficult to capture global semantic information. Global semantic information is crucial for anomaly detection, so the present invention captures global semantic information by describing it in a pseudo-two-dimensional mode. Specifically, the present invention first uses multi-view rendering to obtain multi-view images corresponding to the input point cloud. Where H×W is the image resolution of the rendered image, N V is the number of perspectives of a multi-view image. V A suitable value needs to be chosen because the larger the N V It can describe the point cloud information more comprehensively, but it will also increase memory consumption and time usage. Then, the pre-trained network is used to extract image features from multi-view images. Among them D 2D is the two-dimensional modal feature dimension. Finally, the present invention uses feature back projection to map the image features to obtain the two-dimensional modal feature F of each point 2D , The following will introduce multi-view rendering, image feature extraction, feature fusion, and anomaly detection respectively.

[0064] Multi-view rendering: rendering functions The input point cloud P can be rendered 3D To picture I, the process can be described as,

[0065]

[0066] For the subsequent feature back projection, it is necessary to obtain the correspondence between the three-dimensional point cloud space coordinate points and the two-dimensional image coordinate system. Let P 3D,i is the three-dimensional space coordinate of the i-th point in the point cloud, P 2D,i is the corresponding two-dimensional image coordinate system coordinate, Their corresponding relationship can be described as:

[0067]

[0068] Where K is the intrinsic matrix of the camera used for rendering; T is the corresponding extrinsic matrix; 1 / Z c are the normalized coordinate coefficients.

[0069] The rendering operation projects the original data from 3D to 2D, which will cause information loss. In order to obtain more complete point cloud information, the present invention converts the original point cloud P 3D Using different rotation matrices Rotate to different angles and render the rotated point cloud to obtain multi-view images. For the k-th perspective, the k-th rotated point cloud P 3D,k The i-th point is:

[0070] P 3D,k,i =R k P 3D,i

[0071] Its corresponding rendering image and two-dimensional coordinates are,

[0072]

[0073]

[0074] In the present invention, the rotation matrix R is given by the angle θ with the x, y, and z axes x ,θ y ,θ z Decide,

[0075]

[0076] Since the surface to be inspected is usually the top surface of the point cloud, the point cloud sensor is usually placed above the object to be inspected. In order to ensure the information integrity of the rendered image during the rendering process, that is, to ensure that the rendered image still contains the complete surface to be inspected, the point cloud will only be rotated by a small angle. In this invention, θ x ,θ y ,θ z Belongs to {-π / 16,0,π / 16}. Figure 2 Shows the details of point cloud rotation and rendering. Using the obtained rotation matrix The present invention can obtain several rendering images And their corresponding image coordinate system coordinates Figure 3 Shows an example of rendering an image.

[0077] Image feature extraction: 2D pre-trained networks have powerful representation capabilities. This invention uses pre-trained networks such as ResNet to extract image features. The features of the kth image can be obtained as follows:

[0078]

[0079] in is the feature extraction function of the image. The present invention uses interpolation operation to interpolate the image features to the original Figure 1 The same resolution is convenient for subsequent feature back projection, that is,

[0080] Feature back projection: To obtain the two-dimensional modal features of each point, the image features need to be back-projected into the point cloud space:

[0081]

[0082] Among them, P 2D,k,i 、F 2D,k,i are the coordinate points and point features corresponding to the i-th point in the k-th perspective, Representing image features Medium P 2D,k,i Features at the location.

[0083] Then, the present invention fuses the corresponding point features under all viewing angles to obtain two-dimensional modal point features with more complete information.

[0084]

[0085] Among them, F 2D is the final two-dimensional modal point feature.

[0086] (3) Feature Fusion

[0087] For a given point cloud P 3D The present invention uses three-dimensional and two-dimensional modal feature extraction modules to extract corresponding features respectively, and then fuses the two to obtain a CPMF feature F with stronger descriptive ability. CPMF , During fusion, the present invention firstly 2D With F 3D Normalize them to unit vectors respectively to ensure that the three-dimensional and two-dimensional modal features have the same amplitude and have equal importance in subsequent anomaly detection.

[0088]

[0089] in and is the normalized feature. Then, and are spliced ​​together to obtain the CPMF features,

[0090]

[0091] in Represents the concatenation function.

[0092] In summary, let the CPMF feature extraction function be F CPMF The extraction process can be simplified as follows:

[0093]

[0094] F CPMF It is used for subsequent anomaly detection.

[0095] (4) Anomaly Detection

[0096] The present invention uses a feature library-based method to perform anomaly detection, which can be divided into two stages: training and testing.

[0097] Specifically, in the training phase, the CPMF method proposed in this invention is used to extract the point cloud features of all training normal point cloud samples and store them in the corresponding feature library.

[0098]

[0099] During the testing phase, the present invention first extracts the CPMF features of the point cloud to be tested and uses a nearest neighbor search, which calculates the minimum Euclidean distance between the feature to be tested and the features in the feature library point by point. This distance represents the similarity between the feature to be tested and the normal feature. The larger the distance, the lower the similarity with the normal feature, and vice versa. Therefore, it can be used to characterize the abnormality level of the feature.

[0100]

[0101]

[0102] in, is the point cloud to be measured, The obtained anomaly score can be used to guide industrial detection. Figure 4 Several detection result examples of the proposed method are shown.

[0103] The above-mentioned abnormality detection method is an example of a method of performing abnormality detection using CPMF, but is not limited to this method.

[0104] The present invention proposes a point cloud defect detection method based on complementary pseudo-multimodal features, which can be used to solve the problem of geometric anomaly detection in point clouds, improve the accuracy and robustness of defect detection in industrial quality inspection links, and effectively improve the level of industrial inspection automation in production lines based on point cloud data.

[0105] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A complementary pseudo-multimodal feature system, characterized in that: The complementary pseudo-multimodal feature system consists of three modules: a three-dimensional modal feature extraction module, a two-dimensional modal feature extraction module, and a feature fusion module, wherein: The three-dimensional modal feature extraction module is used to extract local geometric information of the point cloud data to be processed, so as to obtain the three-dimensional modal features of the point cloud data to be processed; The two-dimensional modal feature extraction module will be used to render the point cloud data to be processed to generate a multi-view image, and then use the pre-trained network to extract features from the multi-view image. The image features at each view obtained by the feature extraction are then back-projected into the point cloud space to obtain the point features corresponding to the view. Finally, the point features corresponding to all view angles are fused to obtain the two-dimensional modal features of each point in the point cloud data to be processed. The feature fusion module is used to fuse the three-dimensional modal features and the two-dimensional modal features to obtain complementary pseudo multimodal features; The two-dimensional modal feature extraction module obtains multi-view images according to the following steps: First, the point cloud data to be processed is rotated to different angles using different rotation angles; Then, the rotated point cloud is rendered to obtain a multi-view image; The point cloud data to be processed is rotated to different angles according to the following relationship: P 3D,k,i =R k P 3D,i Among them, R k represents the kth rotation matrix, P 3D,k represents the kth point cloud obtained by rotation, i∈[0,N] represents the i-th point in the point cloud; The two-dimensional coordinates of the rendered image and the points in the image are determined according to the following relationship: in, Represents the rendering function, which rotates the kth point cloud P 3D,k Render the image I of the kth perspective k ; K is the intrinsic matrix of the camera used for rendering; T is the corresponding extrinsic matrix; 1 / Z c is the normalized coordinate coefficient; i∈[0,N] represents the i-th point in the point cloud; The reverse projection of the image features to the point cloud space is performed according to the following relationship: Among them, P 2D,k,i 、F 2D,k,i are the coordinate points and point features corresponding to the i-th point in the k-th perspective, Representing image features Medium P 2D,k,i Features at the location.

2. A complementary pseudo-multimodal feature system according to claim 1, characterized in that: The three-dimensional modal feature extraction is performed according to the following relationship: Among them, P 3D is the input point cloud data, N is the number of points contained in the point cloud; is the three-dimensional modal feature extraction function; F 3D To extract the three-dimensional modal features, D 3D is the dimension of the three-dimensional modal feature.

3. A complementary pseudo-multimodal feature system according to claim 1 or 2, characterized in that: The image feature extraction is performed according to the following relationship: in, is the feature extraction function of the image, is the extracted feature map of the k-th perspective.

4. A complementary pseudo-multimodal feature system according to claim 2, characterized in that: The two-dimensional modal features are performed according to the following relationship: Among them, F 2D is the final two-dimensional modal point feature, k represents the k-th perspective, N V Indicates the total number of viewing angles.

5. A complementary pseudo-multimodal feature system according to claim 4, characterized in that: The feature fusion module performs feature fusion according to the following relationship: in, The simplified point cloud P 3D The CPMF feature extraction function expression is obtained by CPMF It also includes information extracted from three-dimensional and two-dimensional modes.

6. A method for detecting geometric defects in point clouds using the system according to any one of claims 1 to 5, characterized in that: The method comprises the following steps: S1 uses the three-dimensional modal extraction module features to extract three-dimensional modal features from the point cloud data to be processed; S2 uses the two-dimensional modal extraction module features to extract two-dimensional modal features from the point cloud data to be processed; S3 uses the feature fusion module to fuse the three-dimensional modal features and the two-dimensional modal features to obtain complementary pseudo multimodal features; S4 uses the complementary pseudo multimodal features to perform point cloud geometric defect anomaly detection.

7. The method according to claim 6, wherein In step S4, the point cloud geometric defect anomaly detection is performed according to the following steps: S41 extracts complementary pseudo-multimodal features from the point cloud data of the normal sample, thereby obtaining complementary pseudo-multimodal features of all points of the normal sample, and forming a feature library of complementary pseudo-multimodal features of the normal sample; S42 performs complementary pseudo-multimodal feature analysis on the point cloud data of the sample to be processed, thereby obtaining complementary pseudo-multimodal features of all points of the sample to be processed, and calculating the minimum Euclidean distance between the complementary pseudo-multimodal features of the sample to be processed and the normal sample point by point; S43: When the Euclidean distance obtained in step S42 is greater than a preset threshold, it is abnormal.

Images