Micro-nano manipulation visual guidance method and system for nanowire assembly

Abstract

The present invention relates to a micro-nano operation visual guidance method applied to nanowire assembly, including using a convolutional neural network model to perform noise reduction processing on an image to obtain a pure image; performing Canny detection on the image to obtain its binary image, using Hough transform to identify the edge contour and spatial position information of the target; obtaining the mask image and the estimated light source projection shape of the image, inputting the pure image, the mask image and the estimated light source projection shape into the SFS algorithm for three-dimensional reconstruction, and obtaining the relative depth value of the target and the substrate; according to the relative depth value of the target and the substrate, controlling the micro-nano operation robot to slowly descend and gradually approach the electrode surface after the target is picked up and transported to the top of the electrode until the target contacts the substrate to complete the assembly operation. The present invention can provide quantitative guidance for the placement operation of the nanowire, which can not only greatly reduce the difficulty of the operation and save a lot of time, but also greatly improve the assembly accuracy and efficiency.

Description

Technical Field

[0001] The present invention relates to the field of micro-nanoscale automated operation visual technology, and in particular to a micro-nano operation visual guidance method and system applied to nanowire assembly. Background Art

[0002] The task of assembling nanowires is to transfer a single, appropriately sized, unbent and entangled nanowire from a complex substrate to an electrode chip. The main operations include pulling the nanowire from the substrate and lowering the nanowire onto the chip electrode. The imaging principle of a scanning electron microscope is to scan the image point by point through an electron beam. The image only contains the brightness information of the sample plane. In order to achieve efficient and accurate operation with the micro-nano operation platform integrated with the SEM, other methods must be used to obtain the three-dimensional information of the target and guide the micro-nano operation. Restoring the three-dimensional morphology of the target through the image has always been one of the research hotspots in computer vision. More and more three-dimensional reconstruction methods based on two-dimensional plane images have been proposed one after another. These methods have been deeply studied and widely used in the three-dimensional reconstruction of targets in macroscopic scenes. With the continuous in-depth research of micro-nano technology, since the application background of various measurement and reconstruction methods was originally in the macroscopic optical imaging environment, it is of great significance to combine the physical meaning of these macroscopic methods with the SEM imaging environment.

[0003] At present, the domestic nanowire assembly scheme adopts the following key points: (1) The positional relationship of nanowires on the substrate is relatively complex, and the nanowires are interlaced and stacked. For this problem, the general nano-manipulation strategy is to locally magnify the interlaced and stacked parts, manually identify the nanowires located below, and then adjust to a suitable magnification. Move the tip of the AFM probe to the bottom of the manually identified nanowire and control the AFM probe to move slowly upward. By judging whether the nanowire moves with the probe, it is determined whether the target nanowire to be picked up is in contact with the probe surface under the action of van der Waals force, and then further adjust the position and posture of the probe to pull the target nanowire out of the nanowire agglomerate. However, this method is time-consuming and requires continuous adjustment of the electron microscope parameters and manual judgment of the positional relationship between nanowires to select the target. It is inefficient; (2) Three-dimensional reconstruction based on shadow recovery structure: ① Using only one image to restore the structure not only requires more constraints, but the uniqueness and accuracy of the solution may not be guaranteed. ② Using multiple images with different illumination directions to perform three-dimensional reconstruction requires adding multiple detectors to the SEM. ③Argyriou and Petrou proposed an iterative PS algorithm in the case of multiple highlights and shadows. This algorithm is applicable to working environments with at least three light sources and can identify unreliable light sources based on unreliable pixel intensities. However, research on the influence of SEM variables and measurement parameters on 3D reconstruction accuracy cannot meet actual needs; (3) The three-dimensional reconstruction algorithm based on motion recovery structure (Struct From Motion, SFM) is a method of restoring the coordinates of three-dimensional points by matching two-dimensional feature points from images of two or more perspectives. Acquiring multi-perspective images in SEM involves the positioning and tracking of the platform in the SEM. In order to obtain images from multiple perspectives, a deflection coil can be added to the SEM device to obtain images from two perspectives. At the same time, a camera can be used to obtain images from two perspectives by tilting or moving the sample to achieve the effect of two cameras.

[0004] Combining the above technologies, it is not difficult to find that the existing technologies all use scanning electron microscopes to take images of micro-nano-scale targets as input images, extract certain features such as texture, shadow, focus and other information for three-dimensional reconstruction. However, the imaging mechanism in SEM is quite different from optical imaging. If the above method is used, the camera parameters of the SEM detector need to be calibrated. In addition, in order to implement certain specific reconstruction algorithms, additional control devices or detectors need to be added. Due to the complex environmental interference in SEM, the image quality collected is not as high as that collected by optical imaging equipment, which will have a certain impact on the reconstruction accuracy. When SFM obtains images from different perspectives, the rotation of the operating table easily causes the sample to leave the field of view, and manual adjustment to move back to the field of view is time-consuming and complicated. In addition, the matching algorithm in the existing three-dimensional reconstruction algorithm has a high mismatch rate for weak-textured micro-target objects, which leads to poor stability of reconstruction initialization and incomplete model after reconstruction. Summary of the invention

[0005] To this end, the technical problem to be solved by the present invention is to overcome the problems existing in the prior art and propose a micro-nano operation visual guidance method and system applied to nanowire assembly, which can provide quantitative guidance for the placement operation of nanowires, which can not only greatly reduce the difficulty of operation and save a lot of time, but also greatly improve the assembly accuracy and efficiency.

[0006] In order to solve the above technical problems, the present invention provides a micro-nano operation visual guidance method applied to nanowire assembly, comprising:

[0007] The convolutional neural network model is used to perform noise reduction on the acquired SEM image. The input of the convolutional neural network model is the noisy SEM image. The residual training strategy of the convolutional neural network model is used to estimate the noise in the SEM image, separate the noise from the SEM image, and obtain a pure SEM image.

[0008] Perform Canny detection on the pure SEM image to obtain the binary image after detection, and use Hough transform to identify the edge contour and spatial position information of the target in the binary image;

[0009] Obtain the mask image and estimated light source projection shape of the clean SEM image after noise reduction, input the clean SEM image, mask image and estimated light source projection shape into the SFS algorithm for 3D reconstruction, and obtain the relative depth value between the target and the substrate;

[0010] According to the relative depth value between the target and the substrate, the end effector of the micro-nano manipulation robot is controlled to slowly descend after picking up the target and moving it above the electrode. This process continuously changes the SEM imaging focal length to always keep the target structure clear. When the target is close enough to the electrode surface, the SEM focal plane is adjusted to the electrode surface, and the Z-direction step size of the end effector of the micro-nano manipulation robot is reduced to gradually approach the electrode surface until the target contacts the substrate to complete the assembly operation.

[0011] In one embodiment of the present invention, each layer of the convolutional neural network model is provided with convolution, each layer from the first layer to the (d-1) layer is provided with a ReLU activation function, and each layer from the second layer to the (d-2) layer is provided with batch normalization, where d represents the depth of the convolutional neural network.

[0012] In one embodiment of the present invention, the first layer of the convolutional neural network model has 64 filters of size 3×3×c and ReLU activation function units; the second to (d-2) layers each have 64 filters of size 3×3×64, and batch normalization is provided between the convolution and ReLU activation function units; the last layer has 3×3×64 filters, where c represents the number of channels of the input image.

[0013] In one embodiment of the present invention, the Hough transform is expressed as follows:

[0014]

[0015]

[0016] Where I(xi,yi) is the grayscale value of the N×M image pixel, θ is the angle from the x-axis to the vertical line, K means that θ takes k discrete values ​​uniformly in the interval [0,π), and ρ q is the number of samples of the vertical distance from the parameter space line to the origin.

[0017] In one embodiment of the present invention, a method for performing three-dimensional reconstruction using the SFS algorithm includes:

[0018] A normal vector model of the height of any position of the target and any point on the target surface is established, and the normal vector model is substituted into the Lambert surface reflection model of the target. The Lambert surface reflection model is minimized, and its equation is converted into the form of an error function. An integrable constraint is added to convert the problem into a functional extreme value problem and the functional is varied. The Gauss-Seidel iterative method is used to solve the height value of the target surface, which is the relative depth value of the target and the substrate.

[0019] In one embodiment of the present invention, a method for establishing a normal vector model of a height at any position of a target and any point on a surface of the target includes:

[0020] The height of any point on the target is represented by z=z(x,y), and the normal vector at any point on the target surface is represented by vector n=(n1,n2,n3). The relationship between the two is expressed as follows:

[0021]

[0022]

[0023] n1=l sin V cosf, n2=l sin V sinf, n3=l cos V

[0024] Where p and q represent the gradient values ​​of the target surface in the x and y directions, l represents the modulus of the normal vector at the scanning point on the sample surface, V represents the inclination angle of the target surface, and F represents the deflection angle of the target surface.

[0025] In one embodiment of the present invention, the Lambertian surface reflection model is expressed as:

[0026]

[0027] Where (-p0, -q0) represents the direction of the light source, E(x, y) represents the normalized brightness of the scan point image, and R(p, q) represents the reflection function.

[0028] In one embodiment of the present invention, the method of minimizing the Lambert surface reflection model and converting its equation into the form of an error function includes:

[0029] The Lambert surface reflection model expression is converted into the error function form as J1 = ∫∫ K (E(x,y)R(p(x,y),q(x,y))) 2 dxdy, where K is the value range of (x, y); the smooth surface model is expressed as Where p x ,p y ,q x ,q y are the partial derivatives of functions p and q with respect to x and y respectively.

[0030] In one embodiment of the present invention, the method of adding an integrable constraint to convert the problem into a functional extremum problem and calculating the variation of the functional includes:

[0031] Define the integrable constraint as J3 = ∫∫ K [(z x -p) 2 +(z y -q) 2 ]dxdy, where z x 、z yis the first-order partial derivative of the height function z(x,y) with respect to x,y. The problem is transformed into a functional extreme value problem as J=J1+λJ2+μJ3, where λ and μ are Lagrange multipliers, J, J1, J2 and J3 are functionals. The functional J is varied to obtain the necessary conditions for the existence of its extreme value, and the first-order partial derivatives of the height function and the gradient function with respect to x,y and the Laplace operator are discretized through the staggered grid method.

[0032] The present invention also provides a micro-nano operation visual guidance system for nanowire assembly, comprising:

[0033] An image processing module, wherein the image processing module is used to perform noise reduction processing on the acquired SEM image using a convolutional neural network model, wherein the input of the convolutional neural network model is a noisy SEM image, and the residual training strategy of the convolutional neural network model is used to estimate the noise in the SEM image, separate the noise from the SEM image, and obtain a pure SEM image;

[0034] A target recognition module is used to perform Canny detection on the pure SEM image to obtain a binary image after detection, and use Hough transform to identify the edge contour and spatial position information of the target in the binary image;

[0035] A three-dimensional reconstruction module, wherein the three-dimensional reconstruction module is used to obtain a mask image and an estimated light source projection shape of a pure SEM image after noise reduction, and input the pure SEM image, the mask image and the estimated light source projection shape into an SFS algorithm for three-dimensional reconstruction to obtain a relative depth value between the target and the substrate;

[0036] The target assembly guidance module is used to control the end effector of the micro-nano manipulation robot to slowly descend after the target is picked up and moved to the top of the electrode according to the relative depth value of the target and the substrate. This process continuously changes the SEM imaging focal length to always keep the target structure clear. When the target is close enough to the electrode surface, the SEM focal plane is adjusted to the electrode surface, and the Z-direction step of the end effector of the micro-nano manipulation robot is reduced to gradually approach the electrode surface until the target contacts the substrate to complete the assembly operation.

[0037] The above technical solution of the present invention has the following advantages compared with the prior art:

[0038] The present invention proposes a micro-nano operation visual guidance method for nanowire assembly. Aiming at the image noise caused by the imaging mechanism and working environment of the SEM, a neural network is used to pre-process the image, and various noises in the image are removed while retaining the detailed features of the image. The edge contour and spatial position information of the nanowire are obtained, which is beneficial to the extraction operation of the nanowire from the substrate. By reconstructing the three-dimensional structure of the micro-nano scale and calculating the relative depth value of the nanowire and the substrate, the placement operation of the nanowire can be quantitatively guided, which can not only greatly reduce the difficulty of the operation and save a lot of time, but also greatly improve the assembly accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings.

[0040] Figure 1 It is a schematic flow chart of the micro-nano operation visual guidance method applied to nanowire assembly of the present invention.

[0041] Figure 2 It is a structural schematic diagram of the convolutional neural network model of the present invention.

[0042] Figure 3 It is a flow chart of the three-dimensional reconstruction algorithm based on secondary electron grayscale information of the present invention.

[0043] Figure 4 It is a schematic diagram of some prior conditions of the present invention.

[0044] Figure 5 This is the result of three-dimensional reconstruction of the nanowire-electrode of the present invention.

[0045] Figure 6 This is the three-dimensional reconstruction result of the image after the nanowires of the present invention are assembled. DETAILED DESCRIPTION

[0046] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0047] See also Figure 1 As shown, this embodiment provides a micro-nano operation visual guidance method applied to nanowire assembly, comprising the following steps:

[0048] S10: using a convolutional neural network model to perform denoising on the acquired SEM image, the input of the convolutional neural network model is a noisy SEM image, the residual training strategy of the convolutional neural network model is used to estimate the noise in the SEM image, and the noise is separated from the SEM image to obtain a pure SEM image;

[0049] S20: Perform Canny detection on the pure SEM image to obtain a binary image after detection, and use Hough transform to identify the edge contour and spatial position information of the target in the binary image;

[0050] S30: obtaining a mask image and an estimated light source projection shape of the clean SEM image after noise reduction, inputting the clean SEM image, the mask image and the estimated light source projection shape into the SFS algorithm for three-dimensional reconstruction, and obtaining a relative depth value between the target and the substrate;

[0051] S40: According to the relative depth value between the target and the substrate, the end effector of the micro-nano manipulation robot is controlled to slowly descend after the target is picked up and moved to the top of the electrode. This process continuously changes the focal length of the SEM imaging to always keep the target structure clear. When the target is close enough to the electrode surface, the SEM focal plane is adjusted to the electrode surface, and the Z-direction step of the end effector of the micro-nano manipulation robot is reduced to gradually approach the electrode surface until the target contacts the substrate to complete the assembly operation.

[0052] The present invention proposes a micro-nano operation visual guidance method for nanowire assembly. Aiming at the image noise caused by the imaging mechanism and working environment of the SEM, a neural network is used to pre-process the image, and various noises in the image are removed while retaining the detailed features of the image. The edge contour and spatial position information of the nanowire are obtained, which is beneficial to the extraction operation of the nanowire from the substrate. By reconstructing the three-dimensional structure of the micro-nano scale and calculating the relative depth value of the nanowire and the substrate, the placement operation of the nanowire can be quantitatively guided, which can not only greatly reduce the difficulty of the operation and save a lot of time, but also greatly improve the assembly accuracy and efficiency.

[0053] In a micro-nano operation visual guidance method for nanowire assembly disclosed in the present invention, the above steps S20 and S30 can be performed simultaneously.

[0054] In a micro-nano operation visual guidance method for nanowire assembly disclosed in the present invention, for S10 of the above-mentioned embodiment, extracting a single nanowire of suitable size, no entanglement and curling, and high surface cleanliness through an SEM image is the first step of micro-nano assembly. In order to accurately achieve this goal, this embodiment performs noise reduction processing on the acquired SEM image, and performs noise reduction on the noise in the imaging environment unique to the SEM through a convolutional neural network. The input of the convolutional neural network model is a noisy image y=x+v, y is a noisy image, x is a pure image, and v is the noise in the image. The residual training strategy of the convolutional neural network is used to estimate the noise in the image. Construct a data set to train the convolutional neural network so that the residual mapping function R(y)≈v, and then perform the operation x=yR(y) to separate the noise from the image and obtain a pure image. The loss function is the mean square error function between the residual mapping function value and the true noise, and is defined as shown in the formula.

[0055]

[0056] Use the ReLU function as the activation function, as shown below:

[0057] y=max(0,x) (2)

[0058] The depth of the convolutional neural network is D = 20 layers, each layer has a convolution, a total of 20 convolutions. Figure 2 As shown, each layer from the first layer to the (d-1) layer has a ReLU activation function, and from the second layer to the (d-2) layer, each layer has a batch normalization. The first layer has 64 filters of size 3×3×c (c is the number of channels of the input image) and a ReLU activation function unit. The second layer to the (d-2) layer each has 64 filters of size 3×3×64, and there is a batch normalization between the convolution and ReLU units. The last layer has 3×3×64 filters to construct the output.

[0059] The above denoising process can make the image clearer while significantly reducing noise, especially when the details of the edge and internal features of the nanowires are significantly increased, the image quality is significantly improved. The image denoised by this method has a PSNR of 26.628 and an MSE of 141.323, which is significantly better than Gaussian filtering (PSNR 22.475MSE 367.751), bilateral filtering (PSNR25.013MSE 204.983) and median filtering (PSNR 21.586MSE 451.329).

[0060] In a micro-nano operation visual guidance method for nanowire assembly disclosed in the present invention, for S20 of the above-mentioned embodiment, the prepared metal nanowires and carbon nanowires are distributed in clusters and multiple aggregations at the edge of the substrate, and the spatial positions are staggered and disorderly, stacked and nested. Therefore, the present invention judges the positional relationship of the nanostructure in the depth direction, selects a suitable picking target and realizes the recognition of the target position, and visually assists the picking operation. Specifically, the problem of detecting geometric features is converted into a problem of statistical peaks by transforming the coordinate space of the geometric features in the image to the parameter space to form a peak. The Hough transform and Canny detection operator are used to identify the nanostructure in the SEM image. After the edges of the carbon nanowires are identified using the Hough transform (mainly the junctions of the carbon nanowires), as shown in Figure 4 As shown, observe the blocked part where the carbon nanowires intersect. If the geometric edge of the nanowire is continuous, it means that the nanowire is above the discontinuous carbon nanowire. The Hough transform can be expressed as:

[0061]

[0062]

[0063] Where I(xi,yi) is the grayscale value of the N×M image pixel, θ is the angle from the x-axis to the vertical line, K means that θ takes k discrete values ​​uniformly in the interval [0,π), and ρ q is the number of samples of the vertical distance from the parameter space line to the origin.

[0064] The present invention proposes a target recognition method based on denoised images to identify mutually occluded and overlapping micro-nano structure targets, and at the same time judges their positional relationship with each other, assists in the selection of nano targets, optimizes micro-nano operation strategies, and improves operation efficiency.

[0065] In a micro-nano operation visual guidance method for nanowire assembly disclosed in the present invention, for S30 of the above-mentioned embodiment, the current assembly process of the three-dimensional field effect chip is analyzed. After the nanowire is picked up, it is moved to the top of the chip electrode and then slowly descends. This process continuously changes the focal length of the electron microscope imaging to always keep the nanowire structure clear. When the nanowire is close enough to the electrode surface, the surface morphology of both are relatively clear. At this time, the focal plane of the electron microscope is adjusted to the electrode surface, and the Z-direction step size of the micro-nano operation robot end effector is reduced, gradually approaching the electrode surface until the shape of the nanowire changes, and then the nanowire and the electrode are fixed by electron beam deposition or focused ion beam deposition, or the nanowire posture is continuously adjusted to assemble the nanowire and the electrode through van der Waals force. However, the above process requires constant adjustment of the focusing state of the SEM, which is time-consuming, labor-intensive and inefficient. Therefore, the present invention inputs some prior conditions and uses the three-dimensional reconstruction results of the secondary electron image grayscale information to guide the assembly of the nanowire-electrode. It uses the grayscale information of the sample surface morphology in a single secondary electron image to calculate the relative depth of each point morphology, wherein the prior conditions include a pure SEM image, a mask image and an estimated light source projection shape.

[0066] In the real SEM imaging environment, the grayscale value of the target image is affected by many factors. The 3D reconstruction algorithm of the present invention makes the following simplifications and assumptions: the imaging "light source" is a point "light source" at infinite distance, and the reflection model of the incident "light source" and the sample surface is a Lambert surface reflection model. Figure 3 As shown, under the condition that the above situation is established, when the SEM shooting parameters are determined (SEM parameters are shown in the embodiment), the brightness of the final image at each electron beam scanning point on the target surface is determined only by the cosine of the incident angle of the electron beam scanning at that point. Relative to a certain height reference plane of the target sample surface manually selected during imaging, the height of any point at the target position is expressed as z=z(x,y), and the vector n=(n1,n2,n3) is used to represent the normal vector at any point on the target surface. The relationship between them is expressed as follows:

[0067]

[0068]

[0069] n1=l sin V cosf, n2=l sin V sinf, n3=l cos V (7)

[0070] Where p, q are the gradient values ​​of the target surface in the x and y directions; L is the modulus of the normal vector at the scanning point on the target surface; V is the inclination angle of the target surface; F is the deflection angle of the target surface;

[0071] At the same time, the Lambert surface reflection model can be expressed as:

[0072]

[0073] Where (-p0, -q0) represents the direction of the light source; E(x, y) represents the normalized brightness of the scanned point image; R(p, q) represents the reflection function;

[0074] The equation (8) obtained from this is an ill-conditioned equation with no unique solution. In order to solve this problem, other constraints are introduced. It is assumed that the target surface is continuous and smooth, that is, the second-order partial derivative of the target surface height function z with respect to points x and y is continuous. This assumption of the smoothness of the target surface has constrained its shape. Then, other known boundary conditions such as the morphological boundary conditions of the sample itself are combined with the reflection model of the object surface to establish a regularized model. There are many regularized models. The present invention adopts the minimization method proposed by Horn.

[0075] The minimization method of the present invention combines the surface brightness equation (8) with the surface continuous smooth model, and classifies it as a functional extreme value problem, and further solves the variation of the functional as the minimization solution. First, the error between the grayscale information of the secondary electron image and the target surface brightness function is analyzed, and the brightness equation (8) is converted into the form of an error function:

[0076] J1=∫∫ K (E(x,y)-R(p(x,y),q(x,y))) 2 dx dy (9)

[0077] Where K is the value range of (x, y), and then the smooth surface model is expressed as follows:

[0078]

[0079] Where p x ,p y ,q x ,q y ——Partial derivatives of functions p,q with respect to x,y;

[0080] This constraint means that the normal vectors of the tangent planes of adjacent points on the smooth surface are relatively consistent, that is, the second-order partial derivatives of the height z(x,y) of the target surface with respect to x and y are continuous. In order to solve more accurately, an integrability constraint can generally be added:

[0081] J3=∫∫ K [(zx-p) 2 +(zy-q) 2 ]dx dy (11)

[0082] where z x ,z yis the first-order partial derivative of the height function z(x,y) with respect to x and y, which transforms the problem into a functional extreme value problem:

[0083] J=J1+λJ2+μJ3 (12)

[0084] Among them, λ, μ are Lagrange multipliers, J, J1, J2, J3 are functionals. The functional J is varied to obtain the necessary conditions for the existence of its extreme value, and the height function and the first-order partial derivative of the gradient function with respect to x, y and the Laplace operator are discretized by the staggered grid method. Finally, the Gauss-Seidel iteration method is used to solve the height z of the target sample surface, and finally reconstruction is performed based on the obtained depth value.

[0085] In a micro-nano operation visual guidance method for nanowire assembly disclosed in the present invention, for S40 of the above embodiment, finally, according to the relative depth between the nanowire and the substrate obtained by reconstruction, the Z axis of the SmarAct three-axis micro-motion platform is controlled to slowly descend after the nanowire is picked up and moved to the top of the chip electrode. This process continuously changes the imaging focal length of the electron microscope to always keep the nanowire structure clear. When the nanowire is close enough to the electrode surface, the surface morphology of both is relatively clear. At this time, the focal plane of the electron microscope is adjusted to the electrode surface, and the Z-direction step of the micro-nano operation robot end effector is adjusted to be small, and gradually approach the electrode surface until the shape of the nanowire changes, indicating that the nanowire is in contact with the substrate. The Picmotor actuator is rotated so that the stress on the nanowire exceeds the strength limit and breaks, and the nanowire remains on the substrate surface, and is in contact with the substrate under the action of van der Waals force, thus completing the assembly operation.

[0086] The method used in the present invention does not require the addition of additional detector equipment. Compared with the 5% to 20% error of other non-contact three-dimensional reconstruction algorithms combined with computer vision in SEM, the reconstruction method proposed in this paper has a more obvious improvement in accuracy. The reconstruction method of the present invention has a faster calculation speed and does not require the placement of additional physical equipment in the SEM chamber or the modification of its imaging system, so it has strong applicability.

[0087] The following is a demonstration of a micro-nano assembly example using the method of the present invention in conjunction with the accompanying drawings, in order to provide a more detailed description of the present invention:

[0088] First, the implementation basis of the present invention is introduced: a micro-nano robot operating system is integrated in the scanning electron microscope cavity of the present invention, which is composed of four operating units, each of which is composed of a three-axis micro-motion platform providing XYZ three-axis linear motion and a piezoelectric brake providing 360° rotational motion.

[0089] First, extract a single nanowire. Add deionized water to disperse the prepared 70nm diameter Pt nanowire powder in a 5ml test tube and centrifuge it. Use a pipette to take 0.5ml of liquid and place it at 1cm 2 Place the copper sheet on a square copper sheet and place it on a heating table at 90°C for 5 minutes to evaporate its moisture. Place the copper sheet on the micro-nano operation robot manipulator in the SEM chamber and ground it well. The robot manipulator at the other end is equipped with an AFM probe. The acceleration voltage is 5kV, the working distance range is between 4-11mm, and the imaging detector uses an SE2 detector. Move the copper sheet and the AFM probe slowly to the same height and place them in the center of the field of view at an appropriate magnification. The total length of the nanowire is 12.58μm. During the extraction process, images of extraction at different times are collected to assist the extraction work. The SEM images of the extraction process are analyzed to identify the edge information of the nanowire and determine the spatial position information of the nanowire. The end effector picks up the nanowire at the bottom of the two nanowires.

[0090] Next, visual guidance is provided for the placement of the nanowires. The bottom layer of the three-dimensional field effect chip substrate is Si, and a layer of SiO2 is coated on it. The electrode material on top of SiO2 is 2nm thick Cr and 45nm thick Au. The electrode channel width is 500nm, the width of a single electrode is 2μm, and the diameter of the nanowire is 70nm. In order to be able to build and fix it on the electrode, the length of a single nanowire covering the electrode must be greater than 500nm. The length of the nanowire can exceed 4.5μm while ensuring that both ends of the nanowire do not contact the substrate. After pulling out the nanowire of appropriate length, open the warehouse and install the AFM Cantilever on the side of the clamp, remove the copper sheet with the Pt nanowire, replace it with a three-dimensional electrode chip and ground it well. Bring the AFM Cantilever that has picked up the nanowire close to the electrode to be assembled.

[0091] When the nanowire moves above the electrode, the electron microscope parameters are adjusted to capture a clear secondary electron image, the focal plane is adjusted to the horizontal plane where the nanowire is located, and the nanowire is reconstructed in three dimensions using a three-dimensional reconstruction method based on the grayscale information of the secondary electron image to obtain the depth information of the nanowire relative to the electrode surface.

[0092] First, the image to be reconstructed is denoised, then its mask shape is obtained, the electron beam "light source" is defined and the optimization parameters are calculated, and finally the SFS algorithm is used for 3D reconstruction, such as Figure 4 and 5As shown, according to the nanowire-electrode surface height value obtained from the reconstruction result (i.e., the depth information of the nanowire relative to the electrode surface), the height values ​​of ten points are randomly taken from each surface, and the average value is used as the average height of the electrode relative to the substrate. The calculated average height of the electrode surface is 53.8nm. The average height of the highest point of the nanowire is calculated by the same method to be 253.5nm. It is approximately assumed that the radial diameter of the nanowire is the same as 70nm, and the relative distance between the lowest part of the Pt nanowire and the upper surface of the electrode is 129.7nm.

[0093] Use the acquired depth value to visually guide the placement operation. Control the Z axis of the SmarAct three-axis micro-motion platform to move downward from this position according to the basic operation strategy, and record the movement displacement of the Z axis. First control the Z axis to move downward by 80% of the calculated relative distance, and move downward by 104nm. Observe the process of the Z axis moving downward by 104nm. No contact deformation between the nanowire and the electrode occurs during the process. Continue to control the Z axis to move downward with a minimum step of 1nm. When it continues to move downward by 7nm, that is, when the total Z movement stroke is 111nm, the target Pt nanowire touches the electrode surface, and the nanowire morphology changes.

[0094] The Picmotor piezoelectric actuator is controlled to rotate at a fixed rate of 5 steps each time, with a single-step rotation angle of less than 1 μrad, until the stress in the nanowire reaches its strength limit and the nanowire breaks. The broken nanowire crosses the electrode surface under the action of the van der Waals force on the Au electrode surface and fits in contact with it. The length of the broken nanowire is about 7 μm. The grayscale information of the secondary electron image is used to reconstruct the broken nanowire and electrode in three dimensions and measure its depth information. The results are as follows Figure 6 As shown. Figure 6 The reconstruction results were analyzed by randomly taking the height values ​​of ten points on each surface of the electrode, and the average value was used as the average height of the electrode relative to the substrate. The average height of the electrode surface was 49.6nm, and the height of the highest point of the nanowire was 125.3nm. According to the electrode processing parameters, the electrode thickness is 47nm, the electrode height reconstruction accuracy is 94.47%, and the nanowire diameter is approximately 70nm, and the nanowire surface reconstruction accuracy is 92.91%. It can be basically determined that the assembly operation of the nanowire has been completed.

[0095] A micro-nano operation visual guidance system for nanowire assembly disclosed in an embodiment of the present invention is introduced below. The micro-nano operation visual guidance system for nanowire assembly described below and the micro-nano operation visual guidance method for nanowire assembly described above can refer to each other.

[0096] The embodiment of the present invention further provides a micro-nano operation visual guidance system applied to nanowire assembly, comprising:

[0097] An image processing module, wherein the image processing module is used to perform noise reduction processing on the acquired SEM image using a convolutional neural network model, wherein the input of the convolutional neural network model is a noisy SEM image, and the residual training strategy of the convolutional neural network model is used to estimate the noise in the SEM image, separate the noise from the SEM image, and obtain a pure SEM image;

[0098] A target recognition module is used to perform Canny detection on the pure SEM image to obtain a binary image after detection, and use Hough transform to identify the edge contour and spatial position information of the target;

[0099] A three-dimensional reconstruction module, wherein the three-dimensional reconstruction module is used to obtain a mask image and an estimated light source projection shape of a pure SEM image after noise reduction, and input the pure SEM image, the mask image and the estimated light source projection shape into an SFS algorithm for three-dimensional reconstruction to obtain a relative depth value between the target and the substrate;

[0100] The target assembly guidance module is used to control the end effector of the micro-nano manipulation robot to slowly descend after the target is picked up and moved to the top of the electrode according to the relative depth value of the target and the substrate. This process continuously changes the SEM imaging focal length to always keep the target structure clear. When the target is close enough to the electrode surface, the SEM focal plane is adjusted to the electrode surface, and the Z-direction step of the end effector of the micro-nano manipulation robot is reduced to gradually approach the electrode surface until the target contacts the substrate to complete the assembly operation.

[0101] The micro-nano operation visual guidance system applied to nanowire assembly of this embodiment is used to implement part of the embodiment of the aforementioned micro-nano operation visual guidance method applied to nanowire assembly. Therefore, its specific implementation method can refer to the description of the corresponding various partial embodiments and will not be elaborated here.

[0102] In addition, since the micro-nano operation visual guidance system for nanowire assembly of this embodiment is used to implement the aforementioned micro-nano operation visual guidance method for nanowire assembly, its function corresponds to that of the aforementioned method and will not be repeated here.

[0103] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0104] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0105] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0106] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0107] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.

Claims

1. A micro-nano operation visual guidance method for nanowire assembly, characterized in that: include: The convolutional neural network model is used to perform noise reduction on the acquired SEM image. The input of the convolutional neural network model is the noisy SEM image. The residual training strategy of the convolutional neural network model is used to estimate the noise in the SEM image, separate the noise from the SEM image, and obtain a pure SEM image. Perform Canny detection on the pure SEM image to obtain the binary image after detection, and use Hough transform to identify the edge contour and spatial position information of the target in the binary image; Obtain the mask image and estimated light source projection shape of the clean SEM image after noise reduction, input the clean SEM image, mask image and estimated light source projection shape into the SFS algorithm for 3D reconstruction, and obtain the relative depth value between the target and the substrate; Methods for 3D reconstruction using the SFS algorithm include: Establish the height of any position of the target and the normal vector model of any point on the target surface, substitute the normal vector model into the Lambert surface reflection model of the target, minimize the Lambert surface reflection model, convert its equation into the form of error function, add integrable constraints to transform the problem into a functional extreme value problem and calculate the variation of the functional, use the Gauss-Seidel iteration method to solve and obtain the height value of the target surface, which is the relative depth value of the target and the substrate; According to the relative depth value between the target and the substrate, the end effector of the micro-nano manipulation robot is controlled to slowly descend after picking up the target and moving it above the electrode. This process continuously changes the focal length of the SEM imaging to always keep the target structure clear. When the target is close enough to the electrode surface, the SEM focal plane is adjusted to the electrode surface, and the Z-direction step of the end effector of the micro-nano manipulation robot is reduced to gradually approach the electrode surface until the shape of the nanowire changes. The Picmotor actuator is rotated so that the stress on the nanowire exceeds the strength limit and breaks, remaining on the substrate surface, and in contact with the substrate under the action of van der Waals force, thus completing the assembly operation.

2. The micro-nano operation visual guidance method for nanowire assembly according to claim 1, characterized in that: Each layer of the convolutional neural network model is provided with convolution, each layer from the first layer to the (d-1) layer is provided with a ReLU activation function, and each layer from the second layer to the (d-2) layer is provided with batch normalization, where d represents the depth of the convolutional neural network.

3. The micro-nano operation visual guidance method for nanowire assembly according to claim 2, characterized in that: The first layer of the convolutional neural network model has 64 filters of size 3×3×c and ReLU activation function units; the second to (d-2) layers each have 64 filters of size 3×3×64, with batch normalization between convolution and ReLU activation function units; the last layer has 3×3×64 filters, where c represents the number of channels of the input image.

4. The micro-nano operation visual guidance method for nanowire assembly according to claim 1, characterized in that: The Hough transform is expressed as follows: Where I(xi,yi) is the grayscale value of the N×M image pixel, θ is the angle from the x-axis to the vertical line, k means that θ takes k discrete values ​​uniformly in the interval [0,π), and ρ q is the number of samples of the vertical distance from the parameter space line to the origin.

5. The micro-nano operation visual guidance method for nanowire assembly according to claim 1, characterized in that: Methods for establishing a model of the height of any position of a target and a normal vector of any point on the target surface include: The height of any point on the target is represented by z=z(x,y), and the normal vector at any point on the target surface is represented by vector n=(n1,n2,n3). The relationship between the two is expressed as follows: n1=lsinVcosf, n2=lsinVsinf, n3=lcosV Where p and q represent the gradient values ​​of the target surface in the x and y directions, l represents the modulus of the normal vector at the scanning point on the sample surface, V represents the inclination angle of the target surface, and F represents the deflection angle of the target surface.

6. The micro-nano operation visual guidance method for nanowire assembly according to claim 5, characterized in that: The Lambert surface reflection model is expressed as: Where (-p0, -q0) represents the direction of the light source, E(x, y) represents the normalized brightness of the scan point image, and R(p, q) represents the reflection function.

7. The micro-nano operation visual guidance method for nanowire assembly according to claim 6, characterized in that: Methods for minimizing the Lambert surface reflection model and converting its equations into the form of error function include: The Lambert surface reflection model expression is converted into the error function form as J1 = ∫∫ K (E(x,y)-R(p(x,y),q(x,y))) 2 dx dy, where K is the value range of (x, y); the smooth surface model is expressed as Where p x ,p y ,q x ,q y are the partial derivatives of functions p and q with respect to x and y respectively.

8. The micro-nano operation visual guidance method for nanowire assembly according to claim 7, characterized in that: Methods for adding integrability constraints to transform the problem into a functional extremum problem and then calculating the variation of the functional include: Define the integrable constraint as J3 = ∫∫ K [(z x -p)2+(z y -q) 2 ]dx dy, where z x 、z y is the first-order partial derivative of the height function z(x,y) with respect to x,y. The problem is transformed into a functional extreme value problem as J=J1+λJ2+μJ3, where λ and μ are Lagrange multipliers, J, J1, J2 and J3 are functionals. The functional J is varied to obtain the necessary conditions for the existence of its extreme value, and the first-order partial derivatives of the height function and the gradient function with respect to x,y and the Laplace operator are discretized through the staggered grid method.

9. A micro-nano operation visual guidance system for nanowire assembly, characterized in that: include: An image processing module, wherein the image processing module is used to perform noise reduction processing on the acquired SEM image using a convolutional neural network model, wherein the input of the convolutional neural network model is a noisy SEM image, and the residual training strategy of the convolutional neural network model is used to estimate the noise in the SEM image, separate the noise from the SEM image, and obtain a pure SEM image; A target recognition module is used to perform Canny detection on the pure SEM image to obtain a binary image after detection, and use Hough transform to identify the edge contour and spatial position information of the target in the binary image; A three-dimensional reconstruction module, wherein the three-dimensional reconstruction module is used to obtain a mask image and an estimated light source projection shape of a pure SEM image after noise reduction, and input the pure SEM image, the mask image and the estimated light source projection shape into an SFS algorithm for three-dimensional reconstruction to obtain a relative depth value between the target and the substrate; Methods for 3D reconstruction using the SFS algorithm include: Establish the height of any position of the target and the normal vector model of any point on the target surface, substitute the normal vector model into the Lambert surface reflection model of the target, minimize the Lambert surface reflection model, convert its equation into the form of error function, add integrable constraints to transform the problem into a functional extreme value problem and calculate the variation of the functional, use the Gauss-Seidel iteration method to solve and obtain the height value of the target surface, which is the relative depth value of the target and the substrate; The target assembly guidance module is used to control the end effector of the micro-nano manipulation robot to slowly descend after the target is picked up and moved to the top of the electrode according to the relative depth value of the target and the substrate. This process continuously changes the SEM imaging focal length to always keep the target structure clear. When the target is close enough to the electrode surface, the SEM focal plane is adjusted to the electrode surface, and the Z-direction step of the end effector of the micro-nano manipulation robot is gradually reduced to gradually approach the electrode surface until the shape of the nanowire changes, and the Picmotor actuator is rotated so that the stress on the nanowire exceeds the strength limit and breaks, and the nanowire remains on the substrate surface, and is in contact with the substrate under the action of van der Waals force, thereby completing the assembly operation.

Images