Multi-mode auto-encoder part intelligent detection method and device fused with 3D point cloud

Abstract

The invention discloses a multi-mode auto-encoder part intelligent detection method and device fused with 3D point cloud, and relates to the field of image processing, and the method comprises the steps: S1, obtaining the size, weight and 3D point cloud data of parts of various models; normalization and 2D depth image conversion are carried out; s2, carrying out cutting and feature extraction on the 2D depth image, and carrying out feature fusion on the 2D depth image and normalized weight and volume data to obtain multi-modal features; s3, carrying out graying processing to obtain a multi-modal feature grey-scale map; s4, dividing a training set and a test set; training an auto-encoder model by using the training set; s5, inputting the test set into the trained auto-encoder model to obtain feature codes of parts of various models; and S6, inputting the multi-modal feature grey-scale map of the to-be-detected spare and accessory part into the trained auto-encoder model to obtain a feature code of the to-be-detected spare and accessory part, and performing feature comparison to obtain model information. According to the invention, the auto-encoder is used for extracting the multi-modal features fused with the 3D point cloud, and the detection accuracy of spare and accessory parts is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method and device for intelligent detection of multimodal autoencoder parts by integrating 3D point clouds. Background Art

[0002] With the rapid development of computer vision technology and the growing demand for factory intelligence, how to achieve efficient and intelligent detection and classification of spare parts in factory workshops has become a difficult problem that factories urgently need to solve. Due to the refinement and high standardization of the manufacturing industry, there are extremely high requirements for the accuracy of spare parts model detection. Traditional detection methods often rely on complex deep learning models. Although these models can provide high-precision detection results, they often face problems such as high computing resource consumption and poor real-time performance in practical applications. At the same time, traditional image detection methods can often only accept single-modal data, which greatly limits the accuracy of spare parts model detection.

[0003] Therefore, a multimodal autoencoder intelligent detection method for factory spare parts model detection that integrates 3D point clouds is needed. It can not only collect the weight and volume data of spare parts, but also integrate the 3D point cloud depth information of spare parts to provide more useful information for judging spare parts models, thereby improving the accuracy of spare parts model detection. At the same time, the lightweight method has lower computational complexity and faster response speed, and is easy to deploy on edge devices, which makes this method have broad application prospects in scenarios such as industrial production lines, intelligent manufacturing and automated detection. Summary of the invention

[0004] In view of the above problems, the present invention proposes a method and device for intelligent detection of spare parts by a multimodal autoencoder that integrates 3D point clouds. By acquiring the volume and weight information of spare parts and the multimodal data of 3D point clouds for spare parts detection, the multimodal data features of spare parts can be effectively extracted, thereby helping to improve the detection accuracy of spare parts. The autoencoder model used has a simple network structure and a small number of parameters. The trained model has lower computational overhead and memory occupancy during the inference process, and is more efficient in real-time detection and deployment, and is particularly suitable for application scenarios with limited computing resources.

[0005] On the one hand, the multimodal autoencoder parts intelligent detection method integrating 3D point cloud has the following specific steps:

[0006] S1, obtaining volume data, weight data and 3D point cloud data of spare parts of various models; normalizing the weight data and volume data of spare parts, and converting the 3D point cloud data into a 2D depth image;

[0007] S2, after cropping the 2D depth image, extracting features to obtain a feature map, and fusing the feature map with the normalized weight data and volume data to obtain multimodal features of spare parts;

[0008] S3, graying the multimodal features of the spare parts to obtain a multimodal feature grayscale image;

[0009] S4, dividing the multimodal feature grayscale image into a training set and a test set; using the training set to train the constructed autoencoder model to obtain a trained autoencoder model;

[0010] S5, input the test set into the trained autoencoder model to obtain the feature encoding of each model of spare parts;

[0011] S6, the spare parts to be detected are processed by S1-S3 to obtain a multimodal feature grayscale image of the spare parts to be detected, and the image is input into the trained autoencoder model to obtain the feature coding of the spare parts to be detected, and the feature coding of the spare parts to be detected is compared with the feature coding of spare parts of each model to obtain the model information of the spare parts to be detected.

[0012] Preferably, the 3D point cloud data is converted into a 2D depth image as follows:

[0013] The points (x, y, z) of the 3D point cloud data in the laser radar coordinate system are represented as a set of points in the polar coordinate system, that is, , the conversion formula is expressed as:

[0014] ;

[0015] Among them, based on the laser radar coordinate system, r represents the distance from the point to the origin, and θ represents the angle of the point in the horizontal direction. Represents the angle of a point in the vertical direction; (x, y, z) represent the coordinates of each axis of the point in the Cartesian coordinate system;

[0016] The polar coordinate system coordinate values ​​are mapped to a grayscale image; the pixel grayscale values ​​of the grayscale image represent the depth information of the point cloud; the depth information is the distance from each point in the 3D point cloud data in the three-dimensional space to the origin of the coordinate system.

[0017] Preferably, the feature extraction is performed after cropping the 2D depth image, specifically as follows:

[0018] Crop the 2D depth image to a uniform size;

[0019] The cropped image is input into the convolution kernel for feature extraction; the convolution kernel size is 3×3;

[0020] The output of the convolution kernel is input to the Relu activation function.

[0021] Preferably, the feature map is subjected to feature fusion with the normalized weight data and volume data, as follows:

[0022] The normalized spare parts weight value is subtracted from the normalized spare parts volume value, and the obtained difference is added to the extracted feature map pixel by pixel to obtain a multimodal feature containing the spare parts volume information, weight information and 3D point cloud information.

[0023] Preferably, the multimodal features of the spare parts are grayed to obtain a multimodal feature grayscale map, as follows:

[0024] The multimodal features of spare parts are input into a 1×1 convolution module for dimensionality reduction to obtain a single-channel grayscale image, i.e., a multimodal feature grayscale image.

[0025] Preferably, the loss function of the autoencoder model is expressed as:

[0026] ;

[0027] in, represents the loss function of the autoencoder model; represents the Euclidean norm; f() represents a mapping function that maps the multimodal features of spare parts to coding features; max(,) represents taking a larger value; α represents a preset edge, which is used to limit the minimum distance between the coding features obtained from different models of spare parts; f(a) and f(b) represent the coding features obtained by mapping the multimodal features of spare parts of different models respectively; f(a k ) and f(b j ) represent the coded features obtained by multimodal feature mapping of each component of the same model.

[0028] Preferably, feature similarity is used to compare the feature codes of the parts to be detected with the feature codes of the parts of each model; the feature similarity is expressed as:

[0029] ;

[0030] ;

[0031] in, Indicates feature similarity; represents the cosine value; x represents the coding features of the parts to be tested, and y represents the coding features of the parts of each model; represents the Euclidean norm; n represents the total number of features in the feature vector; represents the i-th encoded feature in x; represents the i-th encoded feature in y.

[0032] Preferably, the feature code of the to-be-detected spare parts is compared with the feature codes of the spare parts of each model to detect and obtain the model information of the to-be-detected spare parts, as follows:

[0033] The feature similarities between the feature codes of the spare parts to be detected and the feature codes of spare parts of each model are calculated respectively. If the feature similarity corresponding to a certain model of spare parts is greater than the preset similarity threshold, the model detection of the spare parts to be detected is successful, and the model of the spare parts to be detected is the model of spare parts; otherwise, the model detection fails.

[0034] Preferably, if the model detection fails, the characteristic code of the spare parts to be detected is added to the characteristic code of each model of spare parts for the next model detection.

[0035] On the other hand, the multimodal autoencoder parts intelligent detection device integrating 3D point cloud includes the following:

[0036] The data acquisition and processing module is used to acquire the volume data, weight data and 3D point cloud data of each model of spare parts; normalize the weight data and volume data of spare parts, and convert the 3D point cloud data into a 2D depth image;

[0037] The multimodal feature acquisition module is used to extract features from the 2D depth image after cropping to obtain a feature map, and to fuse the feature map with the normalized weight data and volume data to obtain the multimodal features of spare parts;

[0038] A multimodal feature grayscale image acquisition module is used to grayscale the multimodal features of spare parts to obtain a multimodal feature grayscale image;

[0039] The model training module is used to divide the multimodal feature grayscale image into a training set and a test set; the constructed autoencoder model is trained using the training set to obtain a trained autoencoder model;

[0040] The feature code acquisition module of each model of spare parts is used to input the test set into the trained autoencoder model to obtain the feature code of each model of spare parts;

[0041] The detection module for spare parts to be detected is used to process the spare parts to be detected through the data acquisition and processing module, the multimodal feature acquisition module and the multimodal feature grayscale image acquisition module to obtain the multimodal feature grayscale image of the spare parts to be detected, and input the trained autoencoder model to obtain the feature coding of the spare parts to be detected, and perform feature comparison between the feature coding of the spare parts to be detected and the feature coding of spare parts of various models to obtain the model information of the spare parts to be detected.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] (1) The present invention performs spare parts detection by acquiring the volume, weight information and 3D point cloud multimodal data of spare parts, which can effectively extract the multimodal data features of spare parts, thereby helping to improve the detection accuracy of spare parts;

[0044] (3) The autoencoder model of the present invention has a simple network structure and a small number of parameters. The trained model has low computational overhead and memory usage during the inference process, and is more efficient in real-time detection and deployment, and is particularly suitable for application scenarios with limited computing resources. It avoids the drawbacks of traditional image detection methods that increase the depth of the network and the complexity of the model in exchange for detection accuracy.

[0045] (3) The autoencoder model of the present invention can effectively reduce overfitting and improve the generalization ability of the model through optimized feature learning and data compression capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The present invention is further described in detail below in conjunction with the accompanying drawings;

[0047] Figure 1 It is a flow chart of a multi-modal autoencoder parts intelligent detection method integrating 3D point cloud according to an embodiment of the present invention;

[0048] Figure 2 A detection model architecture diagram of a multimodal autoencoder component intelligent detection method integrating 3D point cloud according to an embodiment of the present invention;

[0049] Figure 3 It is an autoencoder code diagram of the multimodal autoencoder spare parts intelligent detection method integrating 3D point cloud according to an embodiment of the present invention;

[0050] Figure 4 This is a structural block diagram of a multi-modal autoencoder parts intelligent detection device integrating 3D point cloud according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The present invention is further described below through specific implementation modes.

[0052] like Figure 1 and Figure 2 As shown in the figure, the multimodal autoencoder parts intelligent detection method integrating 3D point cloud has the following specific steps:

[0053] S1, obtaining volume data, weight data and 3D point cloud data of spare parts of various models; normalizing the weight data and volume data of spare parts, and converting the 3D point cloud data into a 2D depth image.

[0054] Specifically, the weight of the spare parts is measured by an electronic balance in milligrams; the volume of the spare parts is calculated by a 3D scanner in cubic centimeters; the 3D point cloud of the spare parts is obtained by a laser 3D scanner. Then, the weight and volume data of the spare parts are normalized through normalization operations, and the 3D point cloud of the spare parts is converted into a 2D depth image. The weight of the spare parts is normalized, and the weight normalization formula is defined as follows:

[0055] ;

[0056] Among them, max represents the maximum mass of all current parts, and min represents the minimum mass of all current parts. It represents the average quality of all the spare parts currently, and m represents the quality of the spare parts currently to be tested. Represents the normalized spare parts mass.

[0057] Normalize the volume of spare parts. The volume normalization formula is defined as follows:

[0058] ;

[0059] Among them, max represents the maximum volume of all current parts, and min represents the minimum volume of all current parts. represents the average volume of all the current spare parts, v represents the volume of the spare parts that need to be tested. Represents the normalized volume of a component.

[0060] The role of data normalization is mainly to help improve the training effect of the model and accelerate the convergence process in machine learning and deep learning model training.

[0061] The conversion operation of the 3D point cloud of the spare parts into a 2D depth image is as follows:

[0062] This method first expresses the point (x, y, z) in the laser radar coordinate system as a set of points in the polar coordinate system, that is, , the conversion formula is:

[0063] ;

[0064] Among them, based on the radar coordinate system, r represents the distance from the point to the origin, and θ represents the angle of the point in the horizontal direction. Indicates the angle of a point in the vertical direction. (x, y, z) indicates the coordinates of a point in the Cartesian coordinate system.

[0065] In practical applications, since the projection process is continuous, the pixels of the 2D image may not completely correspond to the projection position of the 3D point. Therefore, interpolation processing is usually required to map the discrete points in the 3D point cloud to the 2D image grid. Common interpolation methods include nearest neighbor interpolation and bilinear interpolation.

[0066] S2, after cropping the 2D depth image, extracting features to obtain a feature map, and merging the feature map with the normalized weight data and volume data to obtain the multimodal features of the spare parts.

[0067] Specifically, the 2D depth image is uniformly cropped to a size of S×S pixels, where S is defined as:

[0068] ;

[0069] in, , Represent the maximum width and height of the spare parts size, and max (,) means taking the larger value. The cropped spare parts image retains the original information of the spare parts and removes the interference of background information to a great extent.

[0070] The cropped image is then input into a 3×3 convolution kernel for feature extraction, and the output features are then input into the Relu activation function to introduce nonlinear relationships, allowing the network to represent more complex features, thereby improving the model's expressiveness and generalization capabilities. The convolution kernel stride is 1 and the padding is 1 to ensure that the output feature map is consistent with the input image size.

[0071] The feature fusion method of the feature map with the normalized weight data and the normalized volume data is as follows: subtract the normalized spare parts weight value from the normalized spare parts volume value, and then add the obtained difference to the feature map pixel by pixel, so as to obtain a multimodal feature map containing the volume information, weight information and 3D point cloud information of the spare parts. Compared with traditional deep learning methods, through this multimodal feature fusion module, the spare parts detection model can use more comprehensive spare parts information, thereby improving the accuracy of spare parts detection. This innovative design significantly enhances the model's understanding of spare parts features and helps to improve the model's detection performance.

[0072] S3, graying the multimodal features of the spare parts to obtain a multimodal feature grayscale image.

[0073] Use a 1×1 convolution kernel to grayscale the multimodal features of spare parts and perform channel dimensionality reduction to obtain a single-channel grayscale image. This can reduce the number of model parameters and prevent overfitting. It also helps reduce training parameters and make the model lighter.

[0074] S4, dividing the multimodal feature grayscale image into a training set and a test set; using the training set to train the constructed autoencoder model to obtain a trained autoencoder model.

[0075] The autoencoder model construction code for this implementation case can be found in Figure 3 As shown, the code based on the auto-encoder model is designed to train the auto-encoder model with the following loss function. The formula of the loss function is expressed as:

[0076] ;

[0077] in, represents the Euclidean distance, f() represents a mapping function that maps the multimodal features of spare parts to coding features, max represents a larger value, and α is a preset margin, which is used to limit the minimum distance between the coding features obtained from different models of spare parts. f(a) and f(b) represent the coding features obtained by mapping the multimodal features of spare parts of different models, respectively. f(a) k ) and f(b j ) represent the coded features obtained by multimodal feature mapping of each component of the same model.

[0078] The advantage of using the designed loss function to train the Auto-encoder network in this embodiment is that the mapping distance between the coding features obtained based on the multimodal features of the same model parts can be as close as possible, while ensuring that the mapping distance between the coding features obtained based on the multimodal features of different models of parts is as far as possible within an appropriate range. The benefit of such a design is that the feature encoding of the same model of parts can be as similar as possible, and the feature encoding of different models of parts can be as different as possible.

[0079] S5, input the test set into the trained autoencoder model to obtain the feature encoding of each model of spare parts.

[0080] Specifically, the grayscale image of the multimodal features of each model of spare parts is input into the trained autoencoder model to obtain the feature codes of each model of spare parts, and the feature codes of each model are used to construct a feature vector database of each model of spare parts. The feature vector database established in this embodiment is a database of feature codes of each model of spare parts in the training sample, and each model of spare parts stores a feature code in the database for subsequent detection of spare parts to be detected.

[0081] S6, the spare parts to be detected are processed by S1-S3 to obtain a multimodal feature grayscale image of the spare parts to be detected, and the image is input into the trained autoencoder model to obtain the feature coding of the spare parts to be detected, and the feature coding of the spare parts to be detected is compared with the feature coding of spare parts of each model to obtain the model information of the spare parts to be detected.

[0082] After the database is established, the weight, volume data and 3D point cloud of the parts to be detected are obtained, and the data of the parts to be detected are input into the network. The data processing process of the parts to be detected is the same as that of the parts used for training. The grayscale image of the multimodal features of the parts to be detected is obtained based on these data, and then the grayscale image of the multimodal features of the parts to be detected is input into the trained Auto-encoder model to obtain the coding features of the parts to be detected. Finally, the obtained coding features of the parts to be detected are compared with the coding features of each model of parts stored in the database. If the feature similarity between the coding features of the parts to be detected and the coding features of a certain model of parts in the database is not less than the set threshold, the model of the parts is output and the detection is successful. Otherwise, if the coding features of the parts to be detected have not been stored in the database, a model name is given to the parts to be detected, and the coding features of the parts to be detected are stored in the database.

[0083] Specifically, the calculation method of the feature similarity between the coding features of different spare parts in this embodiment refers to the cosine similarity calculation formula, which is as follows:

[0084] ;

[0085] In the above formula, It is expressed as:

[0086] ;

[0087] Among them, x represents the encoding features obtained by the Auto-encoder model based on the multimodal features of the parts to be detected, and y represents the encoding features of each model of parts stored in the database. i represents the i-th feature in the feature vector, i is an integer from 1 to n, indicating the specific dimension of the feature vector, and n represents the total number of features in the feature vector, that is, how many features or dimensions there are in the vector. is the i-th feature of the feature vector x, is the i-th feature of the feature vector y. is the Euclidean norm. Since the value range of cos(x,y) is [-1,1], the value range of g(x,y) is [0,2]. The closer the calculation result of g(x,y) is to 2, the higher the feature similarity between x and y. If the feature similarity threshold is set to θ (θ is greater than 1), when g(x,y) is greater than or equal to the set similarity threshold, there is a coding feature of a spare part of the same model as the spare part to be detected in the database, the detection is successful, and the model name of this spare part is output. Otherwise, the detection fails, and the coding feature of the spare part to be detected has not been stored in the database, then a model name is given to the spare part to be detected, and the coding feature of the spare part to be detected is stored in the database.

[0088] like Figure 4 As shown, the present invention also discloses a multi-modal autoencoder parts intelligent detection device integrating 3D point cloud, comprising:

[0089] The data acquisition and processing module 401 is used to acquire the volume data, weight data and 3D point cloud data of each model of spare parts; normalize the weight data and volume data of the spare parts, and convert the 3D point cloud data into a 2D depth image;

[0090] The multimodal feature acquisition module 402 is used to extract features from the 2D depth image after cropping to obtain a feature map, and to fuse the feature map with the normalized weight data and volume data to obtain multimodal features of spare parts;

[0091] The multimodal feature grayscale image acquisition module 403 is used to grayscale the multimodal features of the spare parts to obtain a multimodal feature grayscale image;

[0092] The model training module 404 is used to divide the multimodal feature grayscale image into a training set and a test set; use the training set to train the constructed autoencoder model to obtain a trained autoencoder model;

[0093] The feature code acquisition module 405 for each model of spare parts is used to input the test set into the trained autoencoder model to obtain the feature code of each model of spare parts;

[0094] The spare parts detection module 406 to be detected is used to obtain a multimodal feature grayscale image of the spare parts to be detected by processing the spare parts to be detected through the data acquisition and processing module, the multimodal feature acquisition module and the multimodal feature grayscale image acquisition module, and input the trained autoencoder model to obtain the feature coding of the spare parts to be detected, and perform feature comparison between the feature coding of the spare parts to be detected and the feature coding of spare parts of each model to detect and obtain the model information of the spare parts to be detected.

[0095] The specific implementation of the multimodal autoencoder parts intelligent detection device integrating 3D point cloud is the same as the multimodal autoencoder parts intelligent detection method integrating 3D point cloud, and will not be repeated in this embodiment.

[0096] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.

Claims

1. A multi-modal autoencoder parts intelligent detection method integrating 3D point cloud, characterized in that: The steps include: S1, obtaining volume data, weight data and 3D point cloud data of spare parts of various models; normalizing the weight data and volume data of spare parts, and converting the 3D point cloud data into a 2D depth image; S2, after cropping the 2D depth image, extracting features to obtain a feature map, and fusing the feature map with the normalized weight data and volume data to obtain multimodal features of spare parts; S3, graying the multimodal features of the spare parts to obtain a multimodal feature grayscale image; S4, dividing the multimodal feature grayscale image into a training set and a test set; using the training set to train the constructed autoencoder model to obtain a trained autoencoder model; S5, input the test set into the trained autoencoder model to obtain the feature encoding of each model of spare parts; S6, the spare parts to be detected are processed by S1-S3 to obtain a multimodal feature grayscale image of the spare parts to be detected, and the image is input into the trained autoencoder model to obtain the feature coding of the spare parts to be detected, and the feature coding of the spare parts to be detected is compared with the feature coding of spare parts of each model to obtain the model information of the spare parts to be detected.

2. The multi-modal autoencoder parts intelligent detection method for 3D point cloud fusion according to claim 1 is characterized in that: The 3D point cloud data is converted into a 2D depth image as follows: The points (x, y, z) of the 3D point cloud data in the laser radar coordinate system are represented as a set of points in the polar coordinate system, that is, , the conversion formula is expressed as: ； Among them, based on the laser radar coordinate system, r represents the distance from the point to the origin, and θ represents the angle of the point in the horizontal direction. Represents the angle of a point in the vertical direction; (x, y, z) represent the coordinates of each axis of the point in the Cartesian coordinate system; The polar coordinate system coordinate values ​​are mapped to a grayscale image; the pixel grayscale values ​​of the grayscale image represent the depth information of the point cloud; the depth information is the distance from each point in the 3D point cloud data in the three-dimensional space to the origin of the coordinate system.

3. The multi-modal autoencoder parts intelligent detection method for 3D point cloud fusion according to claim 1 is characterized in that: The feature extraction is performed after the 2D depth image is cropped, as follows: Crop the 2D depth image to a uniform size; The cropped image is input into the convolution kernel for feature extraction; the convolution kernel size is 3×3; The output of the convolution kernel is input to the Relu activation function.

4. The multi-modal autoencoder parts intelligent detection method for 3D point cloud fusion according to claim 1 is characterized in that: The feature fusion of the feature map with the normalized weight data and volume data is as follows: The normalized spare parts weight value is subtracted from the normalized spare parts volume value, and the obtained difference is added to the extracted feature map pixel by pixel to obtain a multimodal feature containing the spare parts volume information, weight information and 3D point cloud information.

5. The multi-modal autoencoder parts intelligent detection method for 3D point cloud fusion according to claim 1 is characterized in that: The multimodal features of the spare parts are grayed to obtain a multimodal feature grayscale map, which is as follows: The multimodal features of spare parts are input into a 1×1 convolution module for dimensionality reduction to obtain a single-channel grayscale image, i.e., a multimodal feature grayscale image.

6. The multi-modal autoencoder parts intelligent detection method for 3D point cloud fusion according to claim 1 is characterized in that: The loss function of the autoencoder model is expressed as: ； in, represents the loss function of the autoencoder model; represents the Euclidean norm; f() represents a mapping function that maps the multimodal features of spare parts to coding features; max(,) represents taking a larger value; α represents a preset edge, which is used to limit the minimum distance between the coding features obtained from different models of spare parts; f(a) and f(b) represent the coding features obtained by mapping the multimodal features of spare parts of different models respectively; f(a k ) and f(b j ) represent the coded features obtained by multimodal feature mapping of each component of the same model.

7. The multi-modal autoencoder parts intelligent detection method for 3D point cloud fusion according to claim 1 is characterized in that: The feature similarity is used to compare the feature codes of the parts to be tested with the feature codes of the parts of each model; the feature similarity is expressed as: ； ； in, Indicates feature similarity; represents the cosine value; x represents the coding features of the parts to be tested, and y represents the coding features of the parts of each model; represents the Euclidean norm; n represents the total number of features in the feature vector; represents the i-th encoded feature in x; represents the i-th encoded feature in y.

8. The multi-modal autoencoder parts intelligent detection method for 3D point cloud fusion according to claim 7 is characterized in that: The feature codes of the parts to be detected are compared with the feature codes of the parts of each model to obtain the model information of the parts to be detected, as follows: The feature similarities between the feature codes of the spare parts to be detected and the feature codes of spare parts of each model are calculated respectively. If the feature similarity corresponding to a certain model of spare parts is greater than the preset similarity threshold, the model detection of the spare parts to be detected is successful, and the model of the spare parts to be detected is the model of spare parts; otherwise, the model detection fails.

9. The multi-modal autoencoder parts intelligent detection method for 3D point cloud fusion according to claim 8 is characterized in that: If the model detection fails, the characteristic code of the spare parts to be detected is added to the characteristic codes of the spare parts of each model for the next model detection.

10. A multi-modal autoencoder parts intelligent detection device integrating 3D point cloud, comprising the following: The data acquisition and processing module is used to acquire the volume data, weight data and 3D point cloud data of each model of spare parts; normalize the weight data and volume data of spare parts, and convert the 3D point cloud data into a 2D depth image; The multimodal feature acquisition module is used to extract features from the 2D depth image after cropping to obtain a feature map, and to fuse the feature map with the normalized weight data and volume data to obtain the multimodal features of spare parts; A multimodal feature grayscale image acquisition module is used to grayscale the multimodal features of spare parts to obtain a multimodal feature grayscale image; The model training module is used to divide the multimodal feature grayscale image into a training set and a test set; the constructed autoencoder model is trained using the training set to obtain a trained autoencoder model; The feature code acquisition module of each model of spare parts is used to input the test set into the trained autoencoder model to obtain the feature code of each model of spare parts; The detection module for spare parts to be detected is used to process the spare parts to be detected through the data acquisition and processing module, the multimodal feature acquisition module and the multimodal feature grayscale image acquisition module to obtain the multimodal feature grayscale image of the spare parts to be detected, and input the trained autoencoder model to obtain the feature coding of the spare parts to be detected, and perform feature comparison between the feature coding of the spare parts to be detected and the feature coding of spare parts of various models to obtain the model information of the spare parts to be detected.

Images