Multi-mode printed circuit board defect detection method and device based on deep learning and storage medium

Abstract

The invention relates to a multi-mode printed circuit board defect detection method and device based on deep learning and a storage medium, and the method comprises the steps: collecting three-dimensional point cloud data and two-dimensional image data, and carrying out the registration, alignment and preprocessing of the three-dimensional point cloud data and the two-dimensional image data; inputting the three-dimensional point cloud data into a double-view fusion module, mapping to generate an aerial view BV and a front view FV, respectively performing primary feature extraction and interpolation on the aerial view BV and the front view FV, and finally fusing to obtain multi-view fusion feature information; repeatedly inputting the multi-view fusion feature information and the two-dimensional image data into an attention module for multiple times to generate fusion features; and inputting the fusion features into a detection module, and outputting prediction results of defect types and bounding boxes. Through the target detection method based on bidirectional cross-modal feature transmission, the problem that small target defect information on a printed circuit board is easy to lose in the extraction process is solved.

Description

Technical Field

[0001] The present invention relates to the field of defect detection, and in particular to a multimodal printed circuit board defect detection method, device and storage medium based on deep learning. Background Art

[0002] The manufacturing process of printed circuit boards includes multiple complex steps, such as exposure, etching, electroplating, welding, etc. In each production stage, differences in process details may lead to defects, including scratches, short circuits, open circuits, etc. These defects may directly affect the electrical performance and product quality of the printed circuit board, resulting in the product not being able to work properly and eventually being judged as scrapped. In order to improve production efficiency, defect detection is a key link. Timely and accurate detection and identification of these defects can detect potential defects in the early stages of production, thereby reducing the flow of defective products into subsequent links, thereby reducing rework and material waste. Through defect detection, not only can product quality be guaranteed, but also rework and scrap rates can be reduced, which is of great significance for ensuring product quality and improving production efficiency. How to effectively use the collected data to achieve accurate defect detection is the main problem that needs to be solved at present.

[0003] There are several main methods for defect detection: (1) The traditional rule-based image processing method relies on some classic image processing algorithms and mainly identifies defects in images by designing specific rules. For example, based on edge detection, edge detection algorithms such as Sobel and Canny are used to identify edge discontinuities in images and detect defects such as cracks or breaks.

[0004] (2) Based on traditional machine learning methods, defect detection is performed by using manually designed features combined with traditional machine learning models. For example, based on manually designed features, this method has a deep understanding of the characteristics of printed circuit boards, and manually extracts specific features of images (such as HOG, SIFT, etc.) and inputs them into traditional classifiers (such as SVM, KNN, etc.) for defect identification.

[0005] (3) Deep learning-based methods allow the model to automatically learn features from data to handle more complex defect detection tasks. For example, based on the convolutional neural network (CNN), local and global features in two-dimensional images are extracted through the convolutional neural network, and then these extracted features are used to classify and locate different types of defects.

[0006] In traditional printed circuit board defect detection technology, since such methods rely on manually designed features and heuristic target area generation methods, their adaptability and robustness are poor. When dealing with complex and varied scenes, these methods often fail to achieve ideal detection results and cannot be effectively applied to different types of printed circuit boards. At present, deep learning-based methods are the most commonly used defect detection technology, but there are some significant problems when dealing with small targets. Since only two-dimensional image data is often used for defect detection, and the area of ​​small targets in the image is relatively small, their detailed features may be blurred or disappear during the feature extraction process. In addition, during the feature fusion process, the features of small targets are easily covered by high-level features after multiple convolutional layers, resulting in important information being ignored or lost. These factors often make it difficult to extract sufficient feature information of small targets, especially in high-noise environments or low-contrast conditions, which seriously affects the accuracy and reliability of detection. Summary of the invention

[0007] The present invention provides a multimodal printed circuit board defect detection method, device and storage medium based on deep learning, aiming to solve at least one of the technical problems existing in the prior art.

[0008] The technical solution of the present invention is a multimodal printed circuit board defect detection method, device and storage medium based on deep learning. The multimodal printed circuit board defect detection method based on deep learning is applied to a multimodal printed circuit board defect detection device based on deep learning. The multimodal printed circuit board defect detection device based on deep learning includes: an acquisition module, which is used to acquire, register, align and preprocess three-dimensional point cloud data and two-dimensional image data; a dual-view fusion module, which is used to map and perform feature extraction and interpolation on the three-dimensional point cloud data, and the dual-view fusion module includes a bird's-eye view convolution submodule and a front view convolution submodule; an attention module, which includes a point cloud feature extraction submodule connected to the dual-view fusion module, an image feature extraction submodule connected to the acquisition module, and an attention fusion submodule connected to the point cloud feature extraction submodule and the image feature extraction submodule respectively; a detection module, which is used to output the prediction results of the defect type and the bounding box. The multimodal printed circuit board defect detection method based on deep learning includes the following steps: S100, collecting three-dimensional point cloud data and two-dimensional image data, and respectively registering, aligning and preprocessing the three-dimensional point cloud data and the two-dimensional image data; S200, inputting the 3D point cloud data into a dual-view fusion module, mapping to generate a bird's-eye view BV and a front view FV, performing feature extraction and interpolation on the bird's-eye view BV and the front view FV respectively, and finally fusing them to obtain multi-view fusion feature information; S300, repeatedly inputting multi-view fusion feature information and two-dimensional image data into the attention module to generate fusion features; S400, input the fused features into the detection module, and output the prediction results of the defect type and the bounding box.

[0009] Further, the step S100 includes: S110, collecting three-dimensional point cloud data and two-dimensional image data respectively, wherein the three-dimensional point cloud data includes spatial coordinate information , the two-dimensional image data includes RGB information; S120, registering and aligning the three-dimensional point cloud data and the two-dimensional image data to ensure that the three-dimensional point cloud data and the two-dimensional image data correspond to each other; S130, preprocessing the three-dimensional point cloud data and the two-dimensional image data respectively, wherein the preprocessing at least includes denoising, filtering and downsampling operations on the three-dimensional point cloud data, and denoising and image enhancement operations on the two-dimensional image data.

[0010] Further, the step S120 includes: S121, calculating the conversion matrix between the laser radar coordinate system and the camera coordinate system through the feature points; S122. Projecting the three-dimensional point cloud data into the camera coordinate system through a transformation matrix to achieve spatial consistency between the three-dimensional point cloud data and the two-dimensional image data.

[0011] Further, the step S200 includes: S210, mapping the three-dimensional point cloud data to a first plane to generate a bird's-eye view BV and mapping the three-dimensional point cloud data to a second plane to generate a front view FV, wherein the first plane is an XY plane and the second plane is an XZ plane; S220, extracting features in the bird's-eye view BV and the front view FV through the bird's-eye view convolution submodule and the front view convolution submodule respectively, to obtain a bird's-eye view feature map and a front view feature map; S230, performing bilinear interpolation operations on the bird's-eye view feature map and the front view feature map respectively to obtain a bird's-eye view feature map and a front view feature map with consistent spatial sizes; S240, concatenating and fusing the bird's-eye view feature map and the front view feature map through a concat operation to obtain multi-view fusion feature information.

[0012] Further, in step S220, the structure of the bird's-eye view convolution module and the front view convolution module is consistent, and the bird's-eye view convolution module or the front view convolution module includes a first convolution layer Conv1, batch normalization BN, Relu and a first adder connected in sequence, and the input of the first convolution layer Conv1 is connected to the second input terminal of the first adder.

[0013] Further, the step S300 includes: S310, inputting the two-dimensional image data into the point cloud feature extraction submodule, inputting the multi-view fusion feature information into the image feature extraction submodule, performing secondary feature extraction respectively, and obtaining corresponding three-dimensional point cloud features and two-dimensional image features, wherein the attention module includes a point cloud feature extraction submodule and an image feature extraction submodule; S320, combining the three-dimensional point cloud features and the two-dimensional image features with the original features respectively; S330, re-inputting the three-dimensional point cloud features and the two-dimensional image features combined with the original features into the point cloud feature extraction submodule and the image feature extraction submodule, and repeating steps S310 and S320 for a preset number of times to obtain final features including the final features of the three-dimensional point cloud and the final features of the two-dimensional image; S340, inputting the final features including the final features of the three-dimensional point cloud and the final features of the two-dimensional image into the attention fusion submodule to obtain fused features, wherein the attention module includes the attention fusion submodule.

[0014] Further, in step S310, The point cloud feature extraction module and the image feature extraction module have the same structure. The point cloud feature extraction module or the image feature extraction module includes a second convolutional layer Conv2, a third convolutional layer Conv3, a fourth convolutional layer Conv4, a second adder, a multi-layer perceptron MLP and a third adder connected in sequence. The input of the second convolutional layer Conv2 is connected to the second input end of the second adder, and the output end of the second adder is connected to the second input end of the third adder. The convolution kernel size of the second convolutional layer Conv2 is 1*1, the convolution kernel size of the third convolutional layer Conv3 is 3*3, and the convolution kernel size of the fourth convolutional layer Conv4 is 1*1.

[0015] Furthermore, in step S340, the attention fusion submodule includes: For 3D point cloud features and 2D image features , ; ; in, and Represent point cloud features The number and two-dimensional image features The number of and Represent point cloud features The feature dimension and two-dimensional image features The characteristic dimension of For point cloud features Generate a query ,key Sum ,in ; For image features Generate a query ,key Sum ,in ; in, , , , , , These are the weight matrices learned during the training of the attention fusion submodule. , , , , , Linear transformation of point cloud and image data to generate corresponding query Q, key K, value V; Then calculate the cross attention and , taking image information as Q, point cloud information as K and V, we can calculate , and we can calculate , ; ; in, is the dimension of key K, Final Output To fuse features: .

[0016] Furthermore, the present invention also proposes a multimodal printed circuit board defect detection device based on deep learning, which is used to execute the multimodal printed circuit board defect detection method based on deep learning. The multimodal printed circuit board defect detection device based on deep learning includes: Acquisition module, used for acquisition, registration, alignment and preprocessing of 3D point cloud data and 2D image data; A dual-view fusion module, used for mapping and performing feature extraction and interpolation on the three-dimensional point cloud data, the dual-view fusion module comprising a bird's-eye view convolution submodule and a front view convolution submodule, and the dual-view fusion module is connected to the acquisition module; An attention module, comprising at least one attention module, wherein the attention module is used to perform secondary feature extraction and multiple feature fusion on two-dimensional image data and multi-view fusion feature information, wherein the attention module comprises a point cloud feature extraction submodule connected to the dual-view fusion module, an image feature extraction submodule connected to the acquisition module, and an attention fusion submodule connected to the point cloud feature extraction submodule and the image feature extraction submodule respectively; A detection module is used to output the prediction results of the defect type and the bounding box, and the detection module is connected to the attention module.

[0017] Furthermore, the present invention also proposes a computer-readable storage medium having program instructions stored thereon, and when the program instructions are executed by a processor, the multimodal printed circuit board defect detection method based on deep learning is implemented.

[0018] The beneficial effects of the present invention are: The multimodal printed circuit board defect detection method based on deep learning solves the problem that small target defect information on the printed circuit board is easily lost during the extraction process through a target detection method based on bidirectional cross-modal feature transfer. The present invention significantly improves the richness and detail retention ability of the final feature information by integrating the detail information extracted from the two-dimensional image data with the spatial structure information extracted from the three-dimensional point cloud data, ensuring that the feature information contains more key details and differential features, thereby improving the accuracy of small target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The overall flow chart of the multimodal printed circuit board defect detection method based on deep learning.

[0020] Figure 2 Schematic diagram of the overall process of multimodal printed circuit board defect detection method based on deep learning.

[0021] Figure 3 Schematic diagram of the structure of the dual-view fusion module in the multimodal printed circuit board defect detection method based on deep learning.

[0022] Figure 4 It is a structural schematic diagram of a bird's-eye view convolution module or a front view convolution module in a multimodal printed circuit board defect detection method based on deep learning.

[0023] Figure 5 It is a structural schematic diagram of a point cloud feature extraction module or an image feature extraction module in a multimodal printed circuit board defect detection method based on deep learning.

[0024] Figure 6Schematic diagram of the structure of the attention fusion submodule in the multimodal printed circuit board defect detection method based on deep learning. DETAILED DESCRIPTION

[0025] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0026] It should be noted that, unless otherwise specified, when a feature is referred to as being "fixed" or "connected" to another feature, it may be directly fixed or connected to the other feature, or it may be indirectly fixed or connected to the other feature. In addition, the descriptions of up, down, left, right, top, bottom, etc. used in the present invention are only relative to the relative positional relationship of the components of the present invention in the drawings.

[0027] In addition, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art. The terms used in this specification are only for describing specific embodiments, not for limiting the present invention. The term "and / or" used herein includes any combination of one or more related listed items.

[0028] It should be understood that, although the terms first, second, third, etc. may be used to describe various elements in the present disclosure, these elements should not be limited to these terms. These terms are only used to distinguish elements of the same type from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element.

[0029] Reference Figures 1 to 6 In some embodiments, the technical solution of the present invention is a multimodal printed circuit board defect detection method based on deep learning, and the multimodal printed circuit board defect detection method based on deep learning is applied to a multimodal printed circuit board defect detection device based on deep learning, and the multimodal printed circuit board defect detection device based on deep learning includes: an acquisition module, which is used to acquire, register, align and preprocess three-dimensional point cloud data and two-dimensional image data; a dual-view fusion module, which is used to map and perform feature extraction and interpolation on the three-dimensional point cloud data, and the dual-view fusion module includes a bird's-eye view convolution submodule and a front view convolution submodule; an attention module, including a point cloud feature extraction submodule connected to the dual-view fusion module, an image feature extraction submodule connected to the acquisition module, and an attention fusion submodule connected to the point cloud feature extraction submodule and the image feature extraction submodule respectively; a detection module, which is used to output the prediction results of the defect type and the bounding box, refer to Figure 1 and Figure 2 , the multimodal printed circuit board defect detection method based on deep learning comprises the following steps: S100, collecting three-dimensional point cloud data and two-dimensional image data, and respectively registering, aligning and preprocessing the three-dimensional point cloud data and the two-dimensional image data; S200, inputting the 3D point cloud data into a dual-view fusion module, mapping to generate a bird's eye view (BV) and a front view (FV), performing feature extraction and interpolation on the bird's eye view BV and the front view FV respectively, and finally fusing them to obtain multi-view fusion feature information; S300, repeatedly inputting multi-view fusion feature information and two-dimensional image data into the attention module to generate fusion features; S400, input the fused features into the detection module, and output the prediction results of the defect type and the bounding box.

[0030] The beneficial effects of the present invention are: The multimodal printed circuit board defect detection method, device and storage medium based on deep learning solve the problem that small target defect information on the printed circuit board is easily lost during the extraction process through a target detection method based on bidirectional cross-modal feature transfer. The present invention significantly improves the richness and detail retention ability of the final feature information by integrating the detail information extracted from the two-dimensional image data with the spatial structure information extracted from the three-dimensional point cloud data, ensuring that the feature information contains more key details and differential features, thereby improving the accuracy of small target detection.

[0031] Specifically, the technical solution of the present invention significantly improves the detection accuracy of small target defects on printed circuit boards by introducing a multimodal detection method with bidirectional fusion. Compared with traditional technical solutions that rely on a single modality or manual feature design, the present invention combines the texture and color information of a two-dimensional image with the depth and geometric features of a three-dimensional point cloud, thereby effectively solving the problem that small target features are easily lost during the extraction process. This technical solution directly brings about the following positive effects: more accurate feature extraction, more comprehensive defect recognition capabilities, and higher detection robustness, especially in the case of high noise or complex backgrounds. Compared with methods that only use two-dimensional image data, the present invention functionally increases the richness of information and the retention of features through multimodal data fusion, and no longer relies on manually designed features or heuristic target generation methods, and thus has significant advantages in adaptability and generalization.

[0032] Further, refer to Figure 1 and Figure 2 , the step S100 comprises: S110, collecting three-dimensional point cloud data and two-dimensional image data respectively, wherein the three-dimensional point cloud data includes spatial coordinate information , the two-dimensional image data includes RGB information; S120, registering and aligning the three-dimensional point cloud data and the two-dimensional image data to ensure that the three-dimensional point cloud data and the two-dimensional image data correspond to each other; S130, preprocessing the three-dimensional point cloud data and the two-dimensional image data respectively, wherein the preprocessing at least includes denoising, filtering and downsampling operations on the three-dimensional point cloud data, and denoising and image enhancement operations on the two-dimensional image data.

[0033] Further, the step S120 includes: S121, calculating the conversion matrix between the laser radar coordinate system and the camera coordinate system through the feature points; S122. Projecting the three-dimensional point cloud data into the camera coordinate system through a transformation matrix to achieve spatial consistency between the three-dimensional point cloud data and the two-dimensional image data.

[0034] Specifically, in step S100, the collected 3D point cloud data and 2D image data are acquired, wherein the 3D point cloud data includes spatial coordinate information (x, y, z), and the original image data includes RGB information. The acquired point cloud data and image data are registered and aligned to ensure that the two are accurately aligned in the same coordinate system. Then, the point cloud data and image data are preprocessed.

[0035] Further, refer to Figures 1 to 3 , the step S200 comprises: S210, mapping the three-dimensional point cloud data to a first plane to generate a bird's-eye view BV and mapping the three-dimensional point cloud data to a second plane to generate a front view FV, wherein the first plane is an XY plane and the second plane is an XZ plane; S220, extracting features in the bird's-eye view BV and the front view FV through the bird's-eye view convolution submodule and the front view convolution submodule respectively, to obtain a bird's-eye view feature map and a front view feature map; S230, performing bilinear interpolation operations on the bird's-eye view feature map and the front view feature map respectively to obtain a bird's-eye view feature map and a front view feature map with consistent spatial sizes; S240, concatenating and fusing the bird's-eye view feature map and the front view feature map through a concat operation to obtain multi-view fusion feature information.

[0036] Further, refer to Figure 4In step S220, the structure of the bird's-eye view convolution module and the front view convolution module is consistent. The bird's-eye view convolution module or the front view convolution module includes a first convolution layer Conv1, batch normalization BN, Relu and a first adder connected in sequence, and the input of the first convolution layer Conv1 is connected to the second input terminal of the first adder.

[0037] Specifically, in step S200, the point cloud data is mapped to the XY plane and the XZ plane to generate a bird's-eye view and a front view, and then a two-dimensional convolutional network is used to extract features from the two views to generate corresponding feature maps. The two feature maps are bilinearly interpolated to keep their spatial sizes consistent, and then they are spliced ​​in the channel dimension to form multi-view fusion features and input into the subsequent feature extraction network.

[0038] Further, refer to Figure 1 and Figure 2 , the step S300 comprises: S310, inputting the two-dimensional image data into the point cloud feature extraction submodule, inputting the multi-view fusion feature information into the image feature extraction submodule, performing secondary feature extraction respectively, and obtaining corresponding three-dimensional point cloud features and two-dimensional image features, wherein the attention module includes a point cloud feature extraction submodule and an image feature extraction submodule; S320, combining the three-dimensional point cloud features and the two-dimensional image features with the original features respectively; S330, re-inputting the three-dimensional point cloud features and the two-dimensional image features combined with the original features into the point cloud feature extraction submodule and the image feature extraction submodule, and repeating steps S310 and S320 for a preset number of times to obtain final features including the final features of the three-dimensional point cloud and the final features of the two-dimensional image; S340, inputting the final features including the final features of the three-dimensional point cloud and the final features of the two-dimensional image into the attention fusion submodule to obtain fused features, wherein the attention module includes the attention fusion submodule.

[0039] Specifically, in step S300, the preprocessed image data in step S100 and the multi-view fusion features in step S200 are input into the feature extraction network, and feature extraction is performed respectively to obtain corresponding three-dimensional point cloud features and image features. The corresponding point cloud features and image features obtained are input into the attention fusion module to generate fusion features containing rich information, which are then combined with the original features to enhance the feature expression. Subsequently, feature extraction is performed again, and the above process is repeated N times to gradually enrich the feature information.

[0040] Further, refer to Figure 5 In step S310, The point cloud feature extraction module and the image feature extraction module have the same structure. The point cloud feature extraction module or the image feature extraction module includes a second convolutional layer Conv2, a third convolutional layer Conv3, a fourth convolutional layer Conv4, a second adder, a multi-layer perceptron MLP and a third adder connected in sequence. The input of the second convolutional layer Conv2 is connected to the second input end of the second adder, and the output end of the second adder is connected to the second input end of the third adder. The convolution kernel size of the second convolutional layer Conv2 is 1*1, the convolution kernel size of the third convolutional layer Conv3 is 3*3, and the convolution kernel size of the fourth convolutional layer Conv4 is 1*1.

[0041] Further, refer to Figure 6 In step S340, the attention fusion submodule includes: For 3D point cloud features and 2D image features , ; ; in, and Represent point cloud features The number and two-dimensional image features The number of and Represent point cloud features The feature dimension and two-dimensional image features The characteristic dimension of For point cloud features Generate a query ,key Sum ,in ; For image features Generate a query ,key Sum ,in ; in, , , , , , These are the weight matrices learned during the training of the attention fusion submodule. , , , , , Linear transformation of point cloud and image data to generate corresponding query Q, key K, value V; Then calculate the cross attention and , taking the image information as Q and the point cloud information as K and V, we can calculate , and we can calculate , ; ; in, is the dimension of key K, Final Output To fuse features: .

[0042] Further, refer to Figure 2 The present invention also proposes a multimodal printed circuit board defect detection device based on deep learning, which is used to execute the multimodal printed circuit board defect detection method based on deep learning. The multimodal printed circuit board defect detection device based on deep learning includes: Acquisition module, used for acquisition, registration, alignment and preprocessing of 3D point cloud data and 2D image data; A dual-view fusion module, used for mapping and performing feature extraction and interpolation on the three-dimensional point cloud data, the dual-view fusion module comprising a bird's-eye view convolution submodule and a front view convolution submodule, and the dual-view fusion module is connected to the acquisition module; An attention module, comprising at least one attention module, wherein the attention module is used to perform secondary feature extraction and multiple feature fusion on two-dimensional image data and multi-view fusion feature information, wherein the attention module comprises a point cloud feature extraction submodule connected to the dual-view fusion module, an image feature extraction submodule connected to the acquisition module, and an attention fusion submodule connected to the point cloud feature extraction submodule and the image feature extraction submodule respectively; A detection module is used to output the prediction results of the defect type and the bounding box, and the detection module is connected to the attention module.

[0043] Furthermore, the present invention also proposes a computer-readable storage medium having program instructions stored thereon, and when the program instructions are executed by a processor, the multimodal printed circuit board defect detection method based on deep learning is implemented.

[0044] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of protection of the present disclosure. All should belong to the protection scope of the present invention. Within the protection scope of the present invention, its technical scheme and / or implementation method can have various modifications and changes.

Claims

1. A multimodal printed circuit board defect detection method based on deep learning, wherein the multimodal printed circuit board defect detection method based on deep learning is applied to a multimodal printed circuit board defect detection device based on deep learning, wherein the multimodal printed circuit board defect detection device based on deep learning comprises: An acquisition module is used for acquiring, registering, aligning and preprocessing 3D point cloud data and 2D image data; a dual-view fusion module is used for mapping and performing feature extraction and interpolation on the 3D point cloud data, and the dual-view fusion module includes a bird's-eye view convolution submodule and a front view convolution submodule; An attention module, comprising a point cloud feature extraction submodule connected to the dual-view fusion module, an image feature extraction submodule connected to the acquisition module, and an attention fusion submodule connected to the point cloud feature extraction submodule and the image feature extraction submodule respectively; The detection module is used to output the prediction results of the defect type and the bounding box, wherein the multimodal printed circuit board defect detection method based on deep learning comprises the following steps: S100, collecting three-dimensional point cloud data and two-dimensional image data, and respectively registering, aligning and preprocessing the three-dimensional point cloud data and the two-dimensional image data; S200, inputting the 3D point cloud data into a dual-view fusion module, mapping to generate a bird's-eye view BV and a front view FV, performing feature extraction and interpolation on the bird's-eye view BV and the front view FV respectively, and finally fusing them to obtain multi-view fusion feature information; S300, repeatedly inputting multi-view fusion feature information and two-dimensional image data into the attention module to generate fusion features; S400, input the fused features into the detection module, and output the prediction results of the defect type and the bounding box.

2. The multimodal printed circuit board defect detection method based on deep learning according to claim 1 is characterized in that: The step S100 includes: S110, collecting three-dimensional point cloud data and two-dimensional image data respectively, wherein the three-dimensional point cloud data includes spatial coordinate information , the two-dimensional image data includes RGB information; S120, registering and aligning the three-dimensional point cloud data and the two-dimensional image data to ensure that the three-dimensional point cloud data and the two-dimensional image data correspond to each other; S130, preprocessing the three-dimensional point cloud data and the two-dimensional image data respectively, wherein the preprocessing at least includes denoising, filtering and downsampling operations on the three-dimensional point cloud data, and denoising and image enhancement operations on the two-dimensional image data.

3. The multimodal printed circuit board defect detection method based on deep learning according to claim 2 is characterized in that: The step S120 includes: S121, calculating the conversion matrix between the laser radar coordinate system and the camera coordinate system through the feature points; S122. Projecting the three-dimensional point cloud data into the camera coordinate system through a transformation matrix to achieve spatial consistency between the three-dimensional point cloud data and the two-dimensional image data.

4. The multimodal printed circuit board defect detection method based on deep learning according to claim 1, characterized in that: The step S200 includes: S210, mapping the three-dimensional point cloud data to a first plane to generate a bird's-eye view BV and mapping the three-dimensional point cloud data to a second plane to generate a front view FV, wherein the first plane is an XY plane and the second plane is an XZ plane; S220, extracting features in the bird's-eye view BV and the front view FV through the bird's-eye view convolution submodule and the front view convolution submodule respectively, to obtain a bird's-eye view feature map and a front view feature map; S230, performing bilinear interpolation operations on the bird's-eye view feature map and the front view feature map respectively to obtain a bird's-eye view feature map and a front view feature map with consistent spatial sizes; S240, concatenating and fusing the bird's-eye view feature map and the front view feature map through a concat operation to obtain multi-view fusion feature information.

5. The multimodal printed circuit board defect detection method based on deep learning according to claim 4 is characterized in that: In step S220, the structure of the bird's-eye view convolution module and the front view convolution module is consistent, and the bird's-eye view convolution module or the front view convolution module includes a first convolution layer Conv1, batch normalization BN, activation function Relu and a first adder connected in sequence, and the input of the first convolution layer Conv1 is connected to the second input terminal of the first adder.

6. The multimodal printed circuit board defect detection method based on deep learning according to claim 1, characterized in that: The step S300 includes: S310, inputting the two-dimensional image data into the point cloud feature extraction submodule, inputting the multi-view fusion feature information into the image feature extraction submodule, performing secondary feature extraction respectively, and obtaining corresponding three-dimensional point cloud features and two-dimensional image features, wherein the attention module includes a point cloud feature extraction submodule and an image feature extraction submodule; S320, combining the three-dimensional point cloud features and the two-dimensional image features with the original features respectively; S330, re-inputting the three-dimensional point cloud features and the two-dimensional image features combined with the original features into the point cloud feature extraction submodule and the image feature extraction submodule, and repeating steps S310 and S320 for a preset number of times to obtain final features including the final features of the three-dimensional point cloud and the final features of the two-dimensional image; S340, inputting the final features including the final features of the three-dimensional point cloud and the final features of the two-dimensional image into the attention fusion submodule to obtain fused features, wherein the attention module includes the attention fusion submodule.

7. The multimodal printed circuit board defect detection method based on deep learning according to claim 6 is characterized in that: In the step S310, The point cloud feature extraction module and the image feature extraction module have the same structure. The point cloud feature extraction module or the image feature extraction module includes a second convolutional layer Conv2, a third convolutional layer Conv3, a fourth convolutional layer Conv4, a second adder, a multi-layer perceptron MLP and a third adder connected in sequence. The input of the second convolutional layer Conv2 is connected to the second input end of the second adder, and the output end of the second adder is connected to the second input end of the third adder. The convolution kernel size of the second convolutional layer Conv2 is 1*1, the convolution kernel size of the third convolutional layer Conv3 is 3*3, and the convolution kernel size of the fourth convolutional layer Conv4 is 1*1.

8. The multimodal printed circuit board defect detection method based on deep learning according to claim 6, characterized in that: In step S340, the attention fusion submodule includes: For 3D point cloud features and 2D image features , ； ； in, and Represent point cloud features The number and two-dimensional image features The number of and Represent point cloud features The feature dimension and two-dimensional image features The characteristic dimension of For point cloud features Generate a query ,key Sum ,in ； For image features Generate a query ,key Sum ,in ； in, , , , , , These are the weight matrices learned during the training of the attention fusion submodule. , , , , , Linear transformation of point cloud and image data to generate corresponding query Q, key K, value V; Then calculate the cross attention and , taking image information as Q, point cloud information as K and V, we can calculate , and we can calculate , ； ； in, is the dimension of key K, Final Output To fuse features: 。 9. A multimodal printed circuit board defect detection device based on deep learning, used to execute the multimodal printed circuit board defect detection method based on deep learning as claimed in any one of claims 1 to 8, characterized in that: The multimodal printed circuit board defect detection device based on deep learning includes: Acquisition module, used for acquisition, registration, alignment and preprocessing of 3D point cloud data and 2D image data; A dual-view fusion module, used for mapping and performing feature extraction and interpolation on the three-dimensional point cloud data, the dual-view fusion module comprising a bird's-eye view convolution submodule and a front view convolution submodule, and the dual-view fusion module is connected to the acquisition module; An attention module, comprising at least one attention module, wherein the attention module is used to perform secondary feature extraction and multiple feature fusion on two-dimensional image data and multi-view fusion feature information, wherein the attention module comprises a point cloud feature extraction submodule connected to the dual-view fusion module, an image feature extraction submodule connected to the acquisition module, and an attention fusion submodule connected to the point cloud feature extraction submodule and the image feature extraction submodule respectively; A detection module is used to output the prediction results of the defect type and the bounding box, and the detection module is connected to the attention module. 10 . A computer-readable storage medium having program instructions stored thereon, wherein the program instructions are executed by a processor to implement the method according to claim 1 .

Images