A 3D reconstruction method for electronic components based on semi-supervised learning
Through a three-dimensional reconstruction method based on semi-supervised learning, the improved Monodepth network and camera internal references are used to solve the problems of human work dependence and low accuracy of unsupervised algorithms in electronic component assembly, and efficient three-dimensional reconstruction and assembly accuracy are achieved.
Patent Information
- Application Number
- CN202110964549.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-19
- Publication Date
- 2024-12-10
- Estimated Expiration
- 2041-08-19
AI Technical Summary
The existing electronic component assembly methods rely on human work, are inefficient and costly, have poor accuracy of traditional three-dimensional reconstruction algorithms, and the reconstruction accuracy of unsupervised algorithms based on deep learning is not high, making it difficult to apply in assembly scenarios.
Using a three-dimensional reconstruction method of electronic components based on semi-supervised learning, the improved Monodepth network combines dense residual attention and channel attention modules to obtain the three-dimensional information of components through semantic segmentation and unsupervised training, and realize the three-dimensional reconstruction with camera internal reference. .
It improves assembly accuracy and efficiency, reduces assembly costs, and realizes accurate three-dimensional reconstruction tasks in assembly scenarios.
Smart Images
Figure CN113902807B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a three-dimensional reconstruction method, and in particular to a three-dimensional reconstruction method of electronic components based on semi-supervised learning, and belongs to the field of machine vision. Background Art
[0002] The application of manipulators in electronic component assembly scenarios plays an extremely important role in improving production efficiency and reducing manual labor; therefore, providing the manipulator with reliable spatial position information of components so that the manipulator can accurately grasp the components is the key to achieving correct assembly. However, in the current electronic component assembly methods, plug-in components are mainly based on manual operations, or the assembly environment is highly restricted, which has the problems of low efficiency and high cost. The present invention proposes a three-dimensional reconstruction method for electronic components based on semi-supervised learning, which can effectively calculate the position and posture information of the electronic components to be assembled through a convolutional neural network, thereby improving the assembly accuracy, improving the assembly efficiency, and reducing the assembly cost.
[0003] Three-dimensional reconstruction is one of the research difficulties and hotspots in the field of computer vision. It is a technology that obtains disparity values from two-dimensional plane image pairs and then restores depth information. The accuracy of three-dimensional reconstruction directly affects the accuracy of depth calculation and the effect of three-dimensional reconstruction of the scene. Its uses involve three-dimensional environment perception and modeling, robot navigation, driverless cars, object tracking and detection, etc.
[0004] At present, 3D reconstruction algorithms mainly include traditional algorithms and algorithms based on deep learning. Traditional algorithms have problems such as poor reconstruction accuracy and long calculation time, and lack widespread use; 3D reconstruction algorithms based on deep learning are mainly divided into supervised learning algorithms and unsupervised learning algorithms. Supervised learning algorithms can achieve good reconstruction accuracy, but when training the network, the Groundtruth value of the image must be provided as a reference, and backpropagation is performed based on its matching cost. However, the method of obtaining the Groundtruth image is very difficult, so it is difficult to apply it in assembly scenarios; the three-dimensional reconstruction algorithm based on unsupervised learning does not need to provide the Groundtruth image, but its reconstruction accuracy is often not as good as that of the supervised learning algorithm. Summary of the invention
[0005] In view of the problems of high dependence on manual work in existing electronic component assembly scenarios and low reconstruction accuracy of existing unsupervised deep learning 3D reconstruction algorithms, the present invention proposes a 3D reconstruction method for electronic components based on semi-supervised learning.
[0006] The present invention achieves the above technical objectives through the following technical means.
[0007] A method for three-dimensional reconstruction of electronic components based on semi-supervised learning comprises the following steps:
[0008] Use the calibration plate to calibrate the industrial binocular camera, obtain the camera's intrinsic parameter information and store it in the camera's intrinsic parameter storage module; use the industrial binocular camera to capture the two-dimensional RGB image of the electronic components, obtain the left view and right view, and use the camera's intrinsic parameter information to perform image correction on the captured left view and right view, so that the same position of the left view and the right view is on the same epipolar line; divide the acquired images into a semantic segmentation dataset and a 3D reconstruction dataset; use the Labelme tool to provide labels for the semantic segmentation dataset and divide the semantic segmentation dataset into a training set, a validation set, a test set and a label set in proportion; divide the 3D reconstruction dataset into a training set, a validation set and a test set in proportion; divide the divided semantic segmentation dataset into a training set, a validation set and a test set The semantic segmentation dataset is input into the improved Monodepth network for semantic segmentation training to obtain the semantic information of electronic components; the trained semantic segmentation network weight information is saved in the semantic information storage module for subsequent use; the divided 3D reconstruction dataset and the semantic segmentation weight information stored in the semantic information storage module are input into the improved Monodepth network for unsupervised 3D reconstruction training; the disparity weight learned from the 3D reconstruction training is stored in the 3D information storage module, and the disparity weight is used to predict the disparity map; the predicted disparity map and the camera intrinsic parameter information of the camera calibration are combined to obtain the real depth information of the electronic components, and realize the 3D reconstruction of the electronic components.
[0009] The improved Monodepth network includes a dense residual attention module; the dense residual attention module combines a dense residual structure with a channel attention module; the dense residual module includes three 3x3 convolutions, three LeakeyRelu activation functions and one cascade module, each layer of the dense residual attention module obtains additional input from all previous layers, and passes its own feature map to all subsequent layers, and finally fuses the information of all layers through cascading to output a dense residual feature map; the channel attention module includes one global average pooling layer, two fully connected layers, a Leakey Relu activation function and a 1x1 convolution layer, and the output feature map of the dense residual attention module is used as the input information of the channel attention module, and is input into the global average pooling layer and the fully connected layer to obtain the channel attention feature. Figure 1 , the obtained channel attention features Figure 1 Perform element-wise multiplication with the dense residual feature map to output channel attention features Figure 2 , the channel attention feature Figure 2 Input to the 1x1 convolution layer to obtain channel attention features Figure 3 , and finally the channel attention feature Figure 3Add the original input x to the original input element by element and produce the output y.
[0010] Furthermore, after completing the camera calibration task, the system will automatically store the learned camera intrinsic parameter information into the camera intrinsic parameter information storage module; the camera intrinsic parameter information is used for image calibration and depth map acquisition.
[0011] Furthermore, after completing the semantic segmentation training task, the system will automatically store the learned semantic segmentation network weight information into the semantic segmentation storage module; the semantic segmentation feature information is used to provide semantic segmentation prior knowledge for subsequent 3D reconstruction tasks.
[0012] Furthermore, after completing the 3D reconstruction training task, the system will automatically store the learned disparity weight information in the 3D reconstruction information storage module; the disparity weight information is used for subsequent electronic component disparity information prediction tasks.
[0013] Furthermore, it includes an image acquisition unit; the image acquisition unit uses two color cameras with a resolution of 1920×10.8 million pixels to obtain visual information of electronic components in the material box, and uses these images as input for the three-dimensional reconstruction model.
[0014] Beneficial effects:
[0015] The present invention proposes a three-dimensional reconstruction method for electronic components based on semi-supervised learning. During depth estimation, the system combines the predicted disparity map with the camera intrinsic parameter information in the camera intrinsic parameter storage module to obtain the depth information of the electronic components and realize the three-dimensional reconstruction task of the electronic components.
[0016] The present invention applies an improved Monodepth network model, a camera intrinsic parameter information storage module, a semantic information storage module and a three-dimensional information storage module to achieve the task of accurate three-dimensional reconstruction of electronic components in assembly scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 It is a flow chart of a method for three-dimensional reconstruction of electronic components based on semi-supervised learning of the present invention;
[0018] Figure 2 It is a specific flowchart of the semantic segmentation stage in the present invention;
[0019] Figure 3 is a schematic diagram of the dense residual attention module;
[0020] Figure 4 It is a flowchart of applying the prior knowledge learned from semantic segmentation to the semi-supervised 3D reconstruction task. DETAILED DESCRIPTION
[0021] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0022] It should be noted that the industrial binocular color camera, camera intrinsic parameter, Opencv, Matlab, Lableme, semantic segmentation, Monodepth, and residual network have the usual meanings understood by persons skilled in the technical field of the present invention, and are not described in detail here.
[0023] The dense residual attention module obtains more diverse texture information while maintaining attention to channel information, enabling the network to learn feature information with richer details.
[0024] A method for three-dimensional reconstruction of electronic components based on semi-supervised learning comprises the following steps:
[0025] Use the calibration plate to calibrate the industrial binocular camera, obtain the camera's intrinsic parameter information and store it in the camera's intrinsic parameter storage module; use the industrial binocular camera to capture the two-dimensional RGB image of the electronic components, obtain the left view and right view, and use the camera's intrinsic parameter information to perform image correction on the captured left view and right view, so that the same position of the left view and the right view is on the same epipolar line; divide the acquired image into a semantic segmentation dataset and a 3D reconstruction dataset; use the Labelme tool to provide labels for the semantic segmentation dataset and divide the semantic segmentation dataset into a training set, a validation set, a test set and a label set in proportion; divide the 3D reconstruction dataset into a training set, a validation set and a test set in proportion; divide the divided The semantic segmentation dataset is input into the improved Monodepth network for semantic segmentation training to obtain the semantic information of electronic components; the trained semantic segmentation network weight information is saved in the semantic information storage module for subsequent use; the divided 3D reconstruction dataset and the semantic segmentation weight information stored in the semantic information storage module are input into the improved Monodepth network for unsupervised 3D reconstruction training; the weights learned from the 3D reconstruction training are stored in the 3D information storage module, and the weights are used to predict the disparity map; the predicted disparity map and the camera intrinsic parameter information of the camera calibration are combined to obtain the real depth information of the electronic components, and the 3D reconstruction of the electronic components is realized.
[0026] The improved Monodepth network includes a dense residual attention module; the dense residual attention module combines a dense residual structure with a channel attention module; the dense residual module includes three 3x3 convolutions, three LeakeyRelu activation functions and one cascade module, each layer of the dense residual attention module obtains additional input from all previous layers, and passes its own feature map to all subsequent layers, and finally fuses the information of all layers through cascading to output a dense residual feature map; the channel attention module includes one global average pooling layer, two fully connected layers, a Leakey Relu activation function and a 1x1 convolution layer, and the output feature map of the dense residual attention module is used as the input information of the channel attention module, and is input into the global average pooling layer and the fully connected layer to obtain the channel attention feature. Figure 1 , the obtained channel attention features Figure 1 Perform element-wise multiplication with the dense residual feature map to output channel attention features Figure 2 , the channel attention feature Figure 2 Input to the 1x1 convolution layer to obtain channel attention features Figure 3 , and finally the channel attention feature Figure 3 Add the original input x to the original input element by element and produce the output y.
[0027] Example
[0028] Combined with Figure 1 As shown, this embodiment is a three-dimensional reconstruction system of electronic components based on semi-supervised learning, including an industrial binocular camera 1, a camera calibration 2, a camera intrinsic parameter information storage module 3, an image correction 4, a semantic segmentation data set 5, a three-dimensional reconstruction data set 6, an improved Monodepth network 7, a semantic segmentation network weight 8, a semantic information storage module 9, a three-dimensional reconstruction information storage module 10, a disparity map 11 and a depth map 12.
[0029] Combined with Figure 1 As shown, in specific applications, the binocular camera 1 obtains a binocular RGB image of the electronic components in the current field of view in the material box, including a left view and a right view; the binocular camera 1 is used to shoot the chessboard calibration plate, and the captured image is input into the Matlab binocular calibration toolbox for camera calibration 2, and the camera intrinsic parameters are obtained and the parameter information is stored in the camera intrinsic parameter storage module 3.
[0030] Use the binocular camera 1 to photograph electronic components in different postures, and divide the captured images into two categories, one for semantic segmentation and the other for 3D reconstruction.
[0031] The captured images are randomly selected. The total number of images in the semantic segmentation dataset is 1,000; the total number of images in the 3D reconstruction dataset is 30,000.
[0032] Combined with Figure 1 , import the camera intrinsic parameter information stored in the camera intrinsic parameter information storage module 3 into Opencv; at the same time, import the three-dimensional reconstruction data set, and perform image correction processing 4 on the left view and right view of the three-dimensional reconstruction data set.
[0033] The stereoRectify function in Opencv is used to perform image correction on the left view and the right view. Its purpose is to make the corresponding points in the left view and the right view be at the same epipolar position.
[0034] The follow-up is mainly divided into two steps, refer to Figure 1 , including a specific flowchart of the semantic segmentation calculation stage 13 and a flowchart of using the prior knowledge learned by the semantic segmentation for semi-supervised 3D reconstruction 14.
[0035] Step 1: Semantic segmentation task stage.
[0036] Combined with Figure 2 As shown in the figure, the image correction is performed for semantic segmentation, and the semantic segmentation dataset is divided into training set, validation set and test set in a ratio of 8:1:1. At the same time, the Labelme tool is used to label the training set samples to generate a Groudtruth dataset, the number of which is equal to that of the training set. The above steps form a new semantic segmentation dataset.
[0037] The new semantic segmentation dataset constructed in the previous step is input into the improved Monodepth network 7 for training the semantic segmentation dataset.
[0038] It should be noted that when performing semantic segmentation tasks, the improved Monodepth network will output disparity maps corresponding to the left view and the right view; and the semantic segmentation label contains the true values corresponding to the left view and the right view.
[0039] Figure 3 What is shown is the improved part of the Monodepth network structure, namely the dense residual attention module. The original structure in the Monodepth network is the residual module.
[0040] refer to Figure 3 , the dense residual attention module works as follows: input a two-dimensional tensor Where C represents the number of channels of the tensor, H represents the height of the image tensor, and W represents the width of the image tensor; this tensor is input into the first convolution 15, and the output dimension is The convolution kernel size is 3×3, the step size is 1, and the padding is 1; input x2 into the LeakyRelu activation function, and the output is Similarly, x and x3 are cascaded in the channel dimension to form The output dimension of the input to the convolution and Leaky Relu activation function is Similarly, x, x3, and x5 are cascaded in the channel dimension to form The output dimension of the input to the convolution and LeakyRelu activation function is Finally, x, x3, x5, and x7 are all cascaded together through the cascade module to output the dimension
[0041] refer to Figure 3 , the output of the previous step First, an adaptive global pooling operation is performed. The operation of each channel is:
[0042]
[0043] Among them, Z c Represents the value after global average pooling operation on the Cth channel; x c (i,j) represents the value of the element at position (i,j) on the Cth channel.
[0044] refer to Figure 3 , the previous step is Output after global average pooling operation
[0045] refer to Figure 3 ,Will Input into a fully connected layer and output Where r is the drop rate, usually 16.
[0046] refer to Figure 3 ,Will The result of the input to a LeakyRelu activation function is then input to a fully connected layer, and the output
[0047] refer to Figure 3 ,Will and Multiply element by element, output where x 12 The channel attention information is included.
[0048] refer to Figure 3 , and replace x in the previous step 12 Input to convolution, output The convolution kernel size is 1×1, the step size is 1, and the padding is 0.
[0049] refer to Figure 3 , the previous step With input Perform element-wise addition operation to output the final dense residual attention feature
[0050] The above shows the specific process of the dense residual attention module.
[0051] It should be noted that the dense residual attention module can extract information of more scales in the network, and combined with the channel attention module, it can obtain more detailed texture information, which helps to improve the accuracy of 3D reconstruction.
[0052] After inputting the latest semantic segmentation dataset into the improved Monodepth network training, the semantic segmentation network weight 8 is obtained and automatically saved in the semantic information storage module 9. So far, the semantic segmentation training task has been completed.
[0053] Step 2: 3D reconstruction stage.
[0054] refer to Figure 4 ,The 3D reconstruction dataset after image correction in the above steps is divided into training set, validation set and test set in a ratio of 8:1:1 to form the latest 3D reconstruction dataset 6.
[0055] It should be noted that the dataset for 3D reconstruction consists of 30,000 images, or 15,000 pairs of images, and each left view corresponds to a right view.
[0056] refer to Figure 4 , the latest 3D reconstruction dataset 6 in the previous step and the semantic segmentation information stored in the semantic information storage module 9 are input into the improved Monodepth network for semi-supervised learning, the 3D reconstruction network weight information of electronic components is obtained, and this information is stored in the 3D reconstruction information storage module 10.
[0057] refer to Figure 4 , the test set in the latest 3D reconstruction dataset is combined with the 3D reconstruction network weight information stored in the 3D information storage module in the previous step to obtain the predicted electronic component disparity map 11.
[0058] refer to Figure 4 , the camera intrinsic parameter information stored in the camera intrinsic parameter information storage module 3 is combined with the disparity map 11 to calculate the depth information of the electronic components. The specific calculation method is:
[0059]
[0060] Wherein, depth represents the depth information of the electronic component; f represents the normalized focal length of the binocular camera; baseline represents the distance between the optical centers of the two cameras; and disp represents the disparity value in the disparity map 11.
[0061] Through the above steps, the present invention avoids the problem of difficulty in obtaining three-dimensional real images through supervised learning; our solution utilizes the prior knowledge of semantic segmentation learning, so that the trained network can well learn most of the semantic information of electronic components during unsupervised learning, which helps to improve the accuracy of unsupervised three-dimensional reconstruction of electronic components and avoids problems such as low reconstruction accuracy and unclear contours.
[0062] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0063] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and intent of the present invention.
Claims
1. A method for three-dimensional reconstruction of electronic components based on semi-supervised learning, characterized in that: The steps include: Use the calibration board to calibrate the industrial binocular camera, obtain the camera's internal parameter information and store it in the camera's internal parameter storage module; An industrial binocular camera is used to capture two-dimensional RGB images of electronic components, obtain the left view and the right view, and use the camera's intrinsic parameter information to perform image correction on the captured left view and the right view, so that the same position of the left view and the right view is on the same epipolar line; The acquired images are divided into a semantic segmentation dataset and a 3D reconstruction dataset; the Labelme tool is used to provide labels for the semantic segmentation dataset and divide the semantic segmentation dataset into a training set, a validation set, a test set, and a label set in proportion; the 3D reconstruction dataset is divided into a training set, a validation set, and a test set in proportion; Input the divided semantic segmentation dataset into the improved Monodepth network for semantic segmentation training to obtain the semantic information of electronic components; save the trained semantic segmentation network weight information into the semantic information storage module for subsequent use; The divided 3D reconstruction data set and the semantic segmentation weight information stored in the semantic information storage module are input into the improved Monodepth network for unsupervised 3D reconstruction training; the disparity weight learned from the 3D reconstruction training is stored in the 3D information storage module, and the disparity weight is used to predict the disparity map; the predicted disparity map and the camera intrinsic parameter information of the camera calibration are combined to obtain the real depth information of the electronic components and realize the 3D reconstruction of the electronic components; The improved Monodepth network includes a dense residual attention module; the dense residual attention module combines a dense residual structure with a channel attention module; the dense residual module includes three 3x3 convolutions, three LeakeyRelu activation functions and one cascade module, each layer of the dense residual attention module obtains additional input from all previous layers, and passes its own feature map to all subsequent layers, and finally fuses the information of all layers through cascading to output a dense residual feature map; the channel attention module includes one global average pooling layer, two fully connected layers, a Leakey Relu activation function and a 1x1 convolution layer, take the output feature map of the dense residual attention module as the input information of the channel attention module, input it to the global average pooling layer and the fully connected layer to obtain the channel attention feature map 1, multiply the obtained channel attention feature map 1 with the dense residual feature map element-wise to output the channel attention feature map 2, input the channel attention feature map 2 into the 1x1 convolution layer to obtain the channel attention feature map 3, and finally perform element-wise addition operation on the channel attention feature map 3 and the original input x to output y.
2. The method for three-dimensional reconstruction of electronic components based on semi-supervised learning according to claim 1, characterized in that: After completing the camera calibration task, the system will automatically store the learned camera intrinsic parameter information into the camera intrinsic parameter information storage module; the camera intrinsic parameter information is used for image calibration and depth map acquisition.
3. The method for three-dimensional reconstruction of electronic components based on semi-supervised learning according to claim 1, characterized in that: After completing the semantic segmentation training task, the system will automatically store the learned semantic segmentation network weight information into the semantic segmentation; the semantic segmentation feature information is used to provide semantic segmentation prior knowledge for subsequent 3D reconstruction tasks.
4. The method for 3D reconstruction of electronic components based on semi-supervised learning according to claim 1, characterized in that: After completing the 3D reconstruction training task, the system will automatically store the learned disparity weight information in the 3D reconstruction information storage module; the disparity weight information is used for subsequent electronic component disparity information prediction tasks.
5. The method for 3D reconstruction of electronic components based on semi-supervised learning according to claim 1, characterized in that: It includes an image acquisition unit; the image acquisition unit uses two color cameras with a resolution of 1920×10.8 million pixels to obtain visual information of electronic components in the material box, and uses these images as input for the three-dimensional reconstruction model.