Wafer grabbing fool-proof method and device based on 3D visual technology

Abstract

The invention relates to a wafer grabbing fool-proof method and device based on a 3D vision technology. The method comprises the following steps: acquiring a 2D image and 3D point cloud data of a wafer to be identified; judging whether the 2D image is complete or not by adopting a jam angle model matching mode, and if the 2D image is complete, acquiring a jam angle coordinate and performing inverse radiation transformation on the 2D image to obtain a corrected jam image; according to the corrected jam image, adopting a pre-trained deep learning model to identify the two-dimensional code position on each layer of wafer, and according to the two-dimensional code position of each layer of wafer, obtaining the position information of each layer of wafer; obtaining filtered point cloud data of each layer and a clustering result of the point cloud data according to the 3D point cloud data and the position information of each layer of wafer; and according to the two-dimensional code position of each layer of wafer and the clustering result, performing multi-modal fusion judgment to obtain a fusion judgment result. According to the method, fool-proof recognition is more stable, and the stability of the semiconductor automatic production line is ensured.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a wafer grabbing fool-proof method and device based on 3D vision technology. Background Art

[0002] In the semiconductor manufacturing process, due to the use of high-precision technologies such as single-crystal silicon materials and material costs, the safety requirements of automated production lines are extremely high, far exceeding other production scenarios. In order to meet the needs of efficient and stable production, the requirements for semiconductor material grabbing safety protection (or foolproofing) are becoming increasingly urgent.

[0003] Wafers (or wafers) are high-purity silicon wafers used to manufacture semiconductor devices, generally in the form of thin discs. In the production and manufacturing process of wafers, in order to prevent abnormalities, it is necessary to use foolproofing technology to ensure the existence of logistics and whether it meets the current task. The existing foolproofing technologies mainly include:

[0004] 1. Barcode and QR code recognition: Each wafer or cassette (wafer container) usually has a unique barcode or QR code for automatic identification. However, the QR code needs to be attached to the side of the wafer. When the wafer is in the cassette, it cannot be photographed due to the angle of the barcode reader. Therefore, this method cannot be widely used in semiconductor production.

[0005] 2. RFID tag verification mainly uses wireless identification. Similar to the above-mentioned coding method, when the wafers are stacked in containers, it is impossible to use automated identification and verification, and the deployment cost is high.

[0006] 3. Point laser optical anti-mistake, using laser ranging to determine the presence of wafers inside the container by distance. However, due to the limitation of the radius of a single laser point, the installation accuracy of the laser is required to be high, and multiple groups of lasers need to be installed to meet the distance measurement and anti-mistake of multiple wafers, which is costly.

[0007] 4. Mechanical anti-foolproofing, using mechanical fixtures to ensure the presence of wafers. This solution can solve the problem of wafer anti-foolproofing to a certain extent, but it requires a large-scale transformation of the existing plug, which also requires high transformation and maintenance costs.

[0008] 5. Visual anti-foolproofing uses industrial cameras and machine learning to infer whether there are wafers placed on each layer inside the jam. The defects of this solution are also obvious. First of all, this method requires the shooting angle to be basically perpendicular to the jam. In actual application scenarios, the shooting angle cannot be completely perpendicular to the jam, and even the wafer and the jam step cannot be photographed at the same time. In this case, the above method cannot be applied. Secondly, the shooting angle of the jam on each shelf is inconsistent, and there is also a deviation in the shooting angle of the jam each time the anti-foolproofing recognition is performed. In order to ensure stable anti-foolproofing, a large amount of data training is required. This will lead to the inability to change models quickly and increase deployment and use costs.

[0009] Based on the above problems and defects, there is an urgent need for a solution that can make fool-proof recognition more stable. Summary of the invention

[0010] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a wafer grabbing fool-proofing method and device based on 3D vision technology, which makes the fool-proofing recognition more stable and ensures the stability of the semiconductor automated production line.

[0011] In order to achieve the above object, the main technical solutions adopted by the present invention include:

[0012] In a first aspect, an embodiment of the present invention provides a wafer grabbing fool-proof method based on 3D vision technology, comprising:

[0013] S100, obtaining a 2D image and 3D point cloud data of a wafer to be identified;

[0014] S200, judging whether the 2D image is complete by using a jamming angle model matching method according to the 2D image, and if the 2D image is complete, obtaining jamming angle coordinates and performing a reverse radiometric transformation on the 2D image to obtain a corrected jamming image;

[0015] S300, according to the corrected jammed image, using a pre-trained deep learning model to identify the position of the QR code on each layer of wafer, and obtaining the position information of each layer of wafer according to the position of the QR code of each layer of wafer;

[0016] S400, acquiring filtered point cloud data of each layer according to the 3D point cloud data and position information of each layer of wafers, and performing clustering processing on the filtered point cloud data of each layer to obtain a clustering result of each layer;

[0017] S500 , performing multimodal fusion judgment according to the identified two-dimensional code position of each layer of wafers and the clustering result of each layer, and obtaining the result of the fusion judgment.

[0018] Optionally, before S200, the method further includes:

[0019] Construct the Kasai Angle model;

[0020] Or, specifically, based on a predetermined upper left corner image of the jam, binarization is performed, and the binarized image is scaled proportionally to obtain a jam upper left corner image template with a four-layer pyramid structure;

[0021] The upper left corner image template of each layer in the pyramid structure is rotated 360 times, 1 degree each time, and the contour of each upper left corner image template of each layer after rotation is extracted to obtain the template contour map of four layers with equal scale;

[0022] The four-layer proportionally scaled template contour images are mirrored and flipped to obtain the template contour images of the upper right corner, the lower right corner, and the lower left corner, and the template contour images of the upper left corner, the upper right corner, the lower right corner, and the lower left corner are combined to obtain the jamming angle model;

[0023] Or, specifically, based on a predetermined upper left corner image of the jam, binarization is performed, and the binarized image is scaled proportionally to obtain an upper left corner image template of the jam with an M-layer pyramid structure;

[0024] The upper left corner image template of each layer in the pyramid structure is rotated P times, each time by 360 / P degrees, and the contour of each upper left corner image template of each layer after rotation is extracted to obtain the template contour map of M layers with equal scale; P is a natural number greater than or equal to 10; M is a natural number greater than or equal to 1;

[0025] The proportionally scaled template contour map of the M layer is mirror-flipped to obtain the template contour maps of the upper right corner, the lower right corner, and the lower left corner, and the template contour maps of the upper left corner, the upper right corner, the lower right corner, and the lower left corner are combined to obtain the jamming angle model.

[0026] Optionally, the S200 includes:

[0027] S201, matching the 2D image with the upper left corner template contour in the jamming angle model, and obtaining a matching score between each template contour and the 2D image in each layer of the pyramid structure;

[0028] S202, performing non-maximum suppression on the matching scores of template contours of all layers to obtain matching results of the four layers of template contours in the upper left corner;

[0029] Traverse the template contour matching of the upper right corner, the lower right corner and the lower left corner to obtain the matching results of the template contour matching of each layer of the pyramid structure corresponding to the four stuck corners;

[0030] S203, refining the matching results of the four jamming angles, and obtaining the positions of the four jamming angles using normal distribution;

[0031] S204. If the positioning of at least one stuck corner fails, send foolproof reporting information; otherwise, confirm that the 2D image is a complete image, and perform an inverse affine transformation on the 2D image based on the positions of the four stuck corners to obtain a corrected stuck image.

[0032] Optionally, the S201 includes:

[0033] The matching score between each template contour and the 2D image in each layer of the pyramid structure is obtained based on the following formula;

[0034]

[0035] Among them, Sm(u,v) represents the matching score of a certain pixel point (u,v), (u,v) represent the pixel points of the 2D image respectively, G represents the gradient information obtained by calculating the 2D image, T represents the template contour in each layer of the pyramid structure, X represents the length of the template contour, Y represents the width of the template contour, S represents the calculation area in the 2D image, m represents the number of pixels, i ranges from 1 to m; x and y are the point values ​​of the horizontal and vertical coordinates of the template;

[0036] Alternatively, the S203 includes:

[0037] S2031, passing from the top layer of the pyramid structure in the stuck angle model to the bottom layer layer by layer, applying f(k+1)=2*f(k) in order; and passing to the next layer based on the calculation result of the previous layer, until the result of the bottom layer is obtained;

[0038] Among them, f represents the coordinates of the current layer, k represents the number of layers,

[0039] S2032. For the bottom layer result obtained, use normal distribution to obtain the positions of four jamming angles. Specifically,

[0040]

[0041] f(x′) is the current score, σ is 1, x' is the current angle, and μ is the new angle value of the jamming angle;

[0042] If μ is greater than the specified threshold, the angle value corresponding to μ is used as the position of the jamming angle; otherwise, the position positioning of the jamming angle fails;

[0043] Alternatively, the step of performing an inverse affine transformation on the 2D image based on the positions of the four jamming angles to obtain a corrected jamming image in S204 includes:

[0044] If the position corner points of the Carssell angle are P lt ,P rt ,P rb ,P lb, based on the following formula, we can get the four corner points P after transformation l ' t ,P r ' t ,P r ' b ,P l ' b ;

[0045]

[0046] P l ' t =(0,0),P r ' t =(w,0),P r ' b =(w,h),P l ' b =(0,h)

[0047] Where w and h represent the width and height of the jammer in the rectified 2D image, respectively. rt .x and P rt .y represent corner points P rt The x and y coordinates of P lt .x、P lt .y represent corner points P lt The x and y coordinates of P rb .x and P rb .y represent corner points P rb In this embodiment, the four points after correction are a rectangle. After the three points of the standard rectangle are confirmed, the fourth point is naturally determined, so the above does not reflect P lb This corner point.

[0048] Optionally, before S300, the method further includes:

[0049] Acquire a training image set, and train the deep learning model according to the training image set;

[0050] or,

[0051] Acquire a training image set, and train the deep learning model according to the training image set;

[0052] Get a training image set, including:

[0053] Pre-collecting a plurality of first 2D images including a texture image of a two-dimensional code on a wafer, and marking the position and area of ​​the two-dimensional code on the first 2D image;

[0054] Performing an affine transformation on the annotated first 2D image to obtain an enhanced training image set;

[0055] The deep learning model is trained using the enhanced training image set to obtain a trained deep learning model.

[0056] Optionally, the step of obtaining the position information of each layer of wafers according to the position of the two-dimensional code of each layer of wafers in S300 includes:

[0057] The starting position P of each layer of wafer is calculated according to the following formula: i :

[0058] P start =H·(1-s)

[0059]

[0060] P i =P start +d offset ·i

[0061] Similarly: P i+1 =P start +d offset (i+1)

[0062] Where H is the height of the jam, P start Indicates the position of the first wafer on the top, P i represents the starting position of the i-th wafer, W represents the width of the corrected jam image, and when the Y coordinate of the QR code position label is at P i and P i+1 When it is between , it means that the wafer to which the label belongs is located in the Pth i Layer, d offset is the offset of each layer of wafer.

[0063] Optionally, the S400 includes:

[0064] According to the position information of each layer of wafer, the point cloud Pc of the position of each layer of wafer is extracted from the 3D point cloud data. i , uniform filtering is performed to remove noise, and each layer of point cloud is evenly distributed to obtain the processed point cloud Ds i ;

[0065] S34 Point Cloud Ds i Perform radius filtering;

[0066] Specifically, a KD-TreeTree (D) is established. For each independent point p, the data of its neighboring points within a certain radius range r is calculated. If N is less than the threshold, the point is marked as a noise point, otherwise it is retained; all points in the point cloud are traversed to obtain the filtered point cloud Fs i ;

[0067] S35 for the filtered point cloud Fs i Run the Euclidean clustering operation to obtain the largest cluster point; count the data of the largest cluster point, if it is greater than the set threshold, the 3D point cloud in the current area is judged to have a wafer, otherwise it is judged to have no wafer, and the clustering results of the 3D point cloud of each layer of wafer are obtained.

[0068] Optionally, the S500 includes:

[0069] If the clustering result of each layer of 3D point cloud indicates that there is a wafer, and the layer contains information of a QR code, then it is confirmed that there is a wafer;

[0070] Otherwise, if there is a layer of 3D point cloud whose clustering result is no wafer, but the layer has the information of QR code, it is confirmed that there is a wafer;

[0071] If the clustering result of a layer of 3D point cloud indicates that there is a wafer, but there is no QR code information in this layer, it is confirmed that there is a wafer.

[0072] In a second aspect, an embodiment of the present invention provides a wafer grabbing fool-proof device based on 3D vision technology, comprising:

[0073] An acquisition unit, used for acquiring 2D images and 3D point cloud data of the wafer to be identified;

[0074] A two-dimensional image correction unit, configured to determine whether the 2D image is complete by using a jamming angle model matching method according to the 2D image, and if the 2D image is complete, obtain the jamming angle coordinates and perform an anti-radiation transformation on the 2D image to obtain a corrected jamming image;

[0075] A two-dimensional image judgment unit is used to identify the position of the two-dimensional code on each layer of wafers according to the corrected jam image and a pre-trained deep learning model, and obtain the position information of each layer of wafers according to the position of the two-dimensional code of each layer of wafers;

[0076] A three-dimensional data processing unit, used to obtain filtered point cloud data of each layer according to the 3D point cloud data and position information of each layer of wafers, and perform clustering processing to obtain clustering results;

[0077] The modal fusion unit is used to perform multi-modal fusion judgment based on the identified QR code position of each layer of wafer and the clustering result of 3D point cloud data to obtain the result of fusion judgment.

[0078] In a third aspect, an embodiment of the present invention further provides a wafer grabbing and fool-proofing system based on 3D vision technology, comprising a 3D sensor and an electronic device, wherein the electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program in the memory, and executes any of the wafer grabbing and fool-proofing methods based on 3D vision technology described in the first aspect above;

[0079] The 3D sensor is arranged at a designated position to capture a jammed image of a wafer to be identified, and the 3D sensor collects 2D images and 3D point cloud data.

[0080] The beneficial effects of the present invention are:

[0081] The wafer grabbing fool-proofing method based on 3D vision technology in the embodiment of the present invention ensures high efficiency, stability and safety on the semiconductor automated production line.

[0082] Compared with the above-mentioned traditional scheme, the anti-fool-proof recognition in the method of the present invention is more stable, that is, it adopts the dual-modality method of 2D image and 3D point cloud data simultaneously output by the 3D camera, and combines the corresponding recognition methods of 2D and 3D, so that the anti-fool-proof effect is more accurate and stable.

[0083] The method of the embodiment of the present invention is easy to use. When the model used in the method requires training samples, a small number of training samples can be used, and then the entire training data set can be obtained through sample enhancement. There is no need for a large amount of labeled training, and it is easy to deploy.

[0084] The system in the embodiment of the present invention is more adaptable to cameras, has stronger compatibility with camera shooting positions, postures, etc., is better adaptable to sizes of wafers and plugs, and is convenient for replacing models on the production line. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 A flowchart of a wafer grabbing fool-proof method based on 3D vision technology provided by an embodiment of the present invention;

[0086] Figure 2 A flowchart of another wafer grabbing fool-proofing method based on 3D vision technology provided by another embodiment of the present invention;

[0087] Figure 3 A flowchart of another wafer grabbing fool-proofing method based on 3D vision technology provided in the third embodiment of the present invention;

[0088] Figure 4 It is a detailed flow chart of some steps in the third embodiment of the present invention;

[0089] Figure 5 is a schematic diagram of a partial contour diagram of a jamming angle model in an embodiment of the present invention;

[0090] Figure 6 and Figure 7 A schematic diagram of a successful matching result of a corner in an embodiment of the present invention;

[0091] Figure 8 is a schematic diagram of a 2D image after correction in an embodiment of the present invention;

[0092] Fig. 9 Schematic diagram of the position of a two-dimensional code obtained in an embodiment of the present invention;

[0093] Fig.10 A schematic diagram of obtaining the position information of each layer of wafer based on a QR code in an embodiment of the present invention;

[0094] Fig.11 The diagram is a schematic diagram showing the division of layers based on the position of each wafer layer in an embodiment of the present invention. DETAILED DESCRIPTION

[0095] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation modes in conjunction with the accompanying drawings.

[0096] In the solution of the embodiment of the present invention, a 3D camera is used as a sensor. The 3D camera can collect 2D images and 3D point cloud data at the same time. The specific processing according to the information uploaded by the sensor is as follows: First, the template matching method is used on the 2D image to locate the jam; secondly, the DNN (deep neural network) method is used to locate the QR code label of the wafer in the jam, and then the position of each layer of the jam is inferred according to the label position; finally, according to the position information of each layer, combined with the 2D image and 3D point cloud information, a dual-modal method is used to determine whether the wafer exists. The method of the embodiment of the present invention has good adaptability to the camera's shooting position and posture, and the result recognition is more stable. In the pre-DNN training and learning process, the amount of label recognition and annotation data required for the jam is also small.

[0097] In practical applications, the 3D camera of this embodiment can be installed horizontally in front of the wafer, for example about 30-50 cm in front of the wafer. The 3D camera of this embodiment can be a 3D sensor, and simultaneously collects 2D images with texture information and 3D point cloud data.

[0098] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0099] Example 1

[0100] See also Figure 1 , Figure 1 This is a flow chart of a wafer grabbing fool-proof method based on 3D vision technology provided in Example 1 of the present invention. The method of this embodiment includes the following steps:

[0101] S100, obtaining a 2D image and 3D point cloud data of a wafer to be identified;

[0102] S200, judging whether the 2D image is complete by using a jamming angle model matching method according to the 2D image, and if the 2D image is complete, obtaining jamming angle coordinates and performing a reverse radiometric transformation on the 2D image to obtain a corrected jamming image;

[0103] S300, based on the corrected jammed image, using a pre-trained deep learning model to identify the position of the QR code on each layer of wafer, and obtaining the position information of each layer of wafer based on the position of the QR code of each layer of wafer.

[0104] For example, this step may include: calculating the starting position P of each layer of wafer according to the following formula: i :

[0105] P start =H·(1-s)

[0106]

[0107] P i =P start +d offset ·i

[0108] Similarly: P i+1 =P start +d offset (i+1)

[0109] Where H is the height of the jam, P start Indicates the position of the first wafer on the top, P i represents the starting position of the i-th wafer, W represents the width of the corrected jam image, and when the Y coordinate of the QR code position label is at P i and P i+1 When it is between P and i

[0110] Layer, d offset is the offset of each layer of wafer.

[0111] In this step, if the position of the two-dimensional code on at least one layer of the wafer is not recognized in step S300, the foolproof reporting is to send the foolproof reporting information without performing the subsequent steps.

[0112] S400 , acquiring each layer of filtered point cloud data according to the 3D point cloud data and the position information of each layer of wafers, and performing clustering processing on each layer of filtered point cloud data to obtain a clustering result of each layer.

[0113] For example, according to the position information of each layer of wafer, the point cloud Pc of the position of each layer of wafer is extracted from the 3D point cloud data. i , uniform filtering is performed to remove noise, and each layer of point cloud is evenly distributed to obtain the processed point cloud Ds i ;

[0114] Point cloud Ds i Perform radius filtering;

[0115] Specifically, a KD-Tree Tree (D) is established. For each independent point p, the data of its neighboring points within a certain radius r are calculated. If N is less than the threshold, the point is marked as a noise point, otherwise it is retained. This scheme is run for all points to obtain the filtered point cloud Fs i ;

[0116] For the filtered point cloud Fs i Run the Euclidean clustering operation to obtain the largest cluster point. Count the data of the largest cluster point. If it is greater than the set threshold, the 3D point cloud in the current area is judged to have a wafer, otherwise it is judged to be absent, and the clustering result based on the 3D point cloud is obtained.

[0117] S500 , performing multimodal fusion judgment according to the identified QR code position of each layer of wafer and the clustering result of the 3D point cloud data, and obtaining the result of the fusion judgment.

[0118] For example, if the clustering result of each layer of 3D point cloud is that there is a wafer, and the layer has the information of the QR code, then it is confirmed that there is a wafer; otherwise, if the clustering result of a layer of 3D point cloud is that there is no wafer, but the layer has the information of the QR code, then it is confirmed that there is a wafer;

[0119] If the clustering result of a layer of 3D point cloud indicates that there is a wafer, but there is no QR code information in this layer, it is confirmed that there is a wafer.

[0120] The wafer grabbing fool-proofing method based on 3D vision technology of this embodiment ensures high efficiency, stability and safety in semiconductor automated production lines. The method of this embodiment is used for fool-proofing recognition, using a dual-modality method of 2D images and 3D point cloud data output simultaneously by a 3D camera, and matching 2D and 3D corresponding recognition methods, so that the fool-proofing effect is more accurate and stable.

[0121] In addition, according to the location information of each layer of wafers, the present embodiment combines the comprehensive information of 2D images and 3D point clouds to output the specific information of the presence or absence of wafers in each layer. The present embodiment makes full use of 2D and 3D dual-modal information, adjusts the judgment strategy through experience, and improves the stability of the entire wafer grabbing foolproofing system.

[0122] Reference Figure 2 As shown, the method of this embodiment is relative to Figure 1 , including step A1:

[0123] A1: Construct the card angle model;

[0124] For example, based on a predetermined upper left corner image of the jammer, binarization is performed, and the binarized image is scaled proportionally to obtain an upper left corner image template of the jammer with a four-layer pyramid structure;

[0125] The upper left corner image template of each layer in the pyramid structure is rotated 360 times, 1 degree each time, and the contour of each upper left corner image template of each layer after rotation is extracted to obtain the template contour map of four layers with equal scale;

[0126] The four-layer proportionally scaled template contour images are mirrored and flipped to obtain the template contour images at the upper right corner, lower right corner, and lower left corner. The template contour images at the upper left corner, upper right corner, lower right corner, and lower left corner are combined to obtain the jamming angle model.

[0127] In this embodiment, a four-layer pyramid structure is used as an example. In practical applications, the pyramid structure is not limited to a four-layer structure, and may also be a three-layer, five-layer, six-layer structure, etc., determined according to actual needs.

[0128] Accordingly, based on the jam angle model in step A1, step S200 may include the following sub-steps:

[0129] S201, using the upper left corner template contour image in the stuck angle model to match the 2D image, and obtaining the matching score between each template contour and the 2D image in each layer of the pyramid structure.

[0130] For example, the matching score between each template contour and the 2D image in each layer of the pyramid structure is obtained based on the following formula;

[0131]

[0132] Among them, Sm(u,v) represents the matching score of a certain pixel point (u,v), (u,v) represent the pixel points of the 2D image respectively, G represents the gradient information obtained by calculating and processing the 2D image, T represents the template contour in each layer of the pyramid structure, X represents the length of the template contour, Y represents the width of each template contour, S represents the scene, m represents the number of pixels, and i ranges from 1 to m;

[0133] S202, performing non-maximum suppression on the matching scores of template contours of all layers to obtain matching results of the four layers of template contours in the upper left corner;

[0134] Traverse the template contour matching of the upper right corner, the lower right corner and the lower left corner to obtain the matching results of the template contour matching of each layer of the pyramid structure corresponding to the four stuck corners;

[0135] S203, refining the matching results of the four jamming corners, and using normal distribution to obtain the positions of the four jamming corners.

[0136] For example, step S203 may include:

[0137] S2031, passing from the top layer of the pyramid structure in the stuck angle model to the bottom layer layer by layer, applying f(k+1)=2*f(k) in sequence; and based on the calculation result of the previous layer, extracting the calculation result and passing it to the next layer, until the result of the bottom layer is obtained;

[0138] Among them, f represents the coordinates of the current layer, k represents the number of layers,

[0139] S2032. For the bottom layer result obtained, use normal distribution to obtain the positions of four jamming angles. Specifically,

[0140]

[0141] f(x′) is the current score, σ is 1, x' is the current angle, and μ is the new angle value of the jamming angle;

[0142] If μ is greater than the specified threshold, the angle value corresponding to μ is used as the position of the jamming angle; otherwise, the position positioning of the jamming angle fails.

[0143] S204. If the positioning of at least one stuck corner fails, send foolproof reporting information; otherwise, confirm that the 2D image is a complete image, and perform an inverse affine transformation on the 2D image based on the positions of the four stuck corners to obtain a corrected stuck image.

[0144] For example, performing an inverse affine transformation on the 2D image based on the positions of the four jamming angles in step S204 to obtain a corrected jamming image includes:

[0145] If the position corner points of the Carssell angle are P lt ,P rt ,P rb ,P lb , based on the following formula, we can get the four corner points P after transformation l ' t ,P r ' t ,P r ' b ,P l ' b ;

[0146]

[0147] P l ' t=(0,0),P r ' t =(w,0),P r ' b =(w,h),P l ' b =(0,h)

[0148] Where w and h represent the width and height of the jammer in the rectified 2D image, respectively. rt .x and P rt .y represent corner points P rt The x and y coordinates of P lt .x、P lt .y represent corner points P lt The x and y coordinates of P rb .x and P rb .y represent corner points P rb The x and y coordinates of .

[0149] The electronic device of this embodiment can obtain 3D camera data and locate the four corners of the jam using template matching. The four corners of the jam are identified and located based on the jam angle model to obtain accurate corner coordinates. In the specific processing, since the four corners are in a mirror relationship, when selecting a template, you only need to select one and flip the mirror image (up and down, left and right). Based on the obtained results and completeness of the four corners of the jam, first determine whether the jam in the camera field of view is complete. Only on the basis of the complete detection of the four jam corners can the next step be performed. Otherwise, the anti-foolproofing fails and an anti-foolproofing report is performed.

[0150] After the four jamming angles are detected successfully, an inverse affine transformation is performed according to the jamming angle coordinates to obtain a corrected jamming image. In fact, the image is processed according to the coordinates, for example, an irregular image is transformed into a regular image and the image is adjusted to be flat.

[0151] In a possible implementation, before the aforementioned step S300, the method of this embodiment further includes a step A2 not shown in the figure. There is no order relationship between A2 and the aforementioned step A1, and it is configured according to actual needs.

[0152] A2. Obtain a training image set, and train the deep learning model DNN according to the training image set;

[0153] For example, image enhancement is used to obtain a training image set.

[0154] For example: pre-collect multiple first 2D images including the texture image of the QR code on the wafer, and mark the QR code position and area on the first 2D image;

[0155] Performing an affine transformation on the annotated first 2D image to obtain an enhanced training image set;

[0156] The deep learning model is trained using the enhanced training image set to obtain a trained deep learning model. Therefore, during the training and learning process, the amount of label recognition and annotation data required for the card insert is relatively small, and deployment and use are better achieved.

[0157] That is, the QR code attached to the wafer is trained with data collection to obtain a DNN deep learning model for identification during the foolproofing process. It should be noted that the QR code is attached to the wafer, and the camera usually cannot obtain the specific information of the QR code, so it is impossible to use this information to independently complete the foolproofing, and the bottom wafer cannot be photographed. Therefore, the existing technical solution has inaccurate identification, which leads to the inability to meet the foolproofing needs of the actual production line.

[0158] In this embodiment, the enhanced sample is obtained by radially transforming the QR code. Only about 100 to 200 pieces of data need to be collected, and the data are enhanced to thousands of pieces to achieve the expected effect and meet the actual production line anti-mistake needs.

[0159] Example 2

[0160] Refer to 3 to Fig.11 As shown, the method of this embodiment may include the following steps:

[0161] 301. Training the stuck angle model using image pyramid and angle rotation. Figure 5 A partial schematic diagram of the jammer angle model is shown.

[0162] In this embodiment, due to the mirror relationship of the four corner templates, only the upper left corner of the jam is used as an example, and the other three corners can be achieved by mirror flipping. Figure 5 As shown. On this basis, the image pyramid is used to obtain a 4-layer proportionally scaled template ( Figure 5 The template above is reduced in size, such as 60*40, 30*20, 15*10, 7*5, etc., so as to effectively reduce the number of iterations in subsequent processing), and each layer of the template is rotated 360 times, 1 degree each time, such as Figure 5 As shown. After rotation, the canny edge operator is applied to the rotated image to extract the contour, and the obtained contour image is saved. Thus, a 4-layer proportionally scaled template set is obtained, with 360 template contour images in each layer, and a total of 4×360=1440 template contour images, which are used as the jamming angle model.

[0163] 302. The 2D image (referred to as the original image) transmitted by the 3D camera is subjected to template matching based on contours using the jammer angle model, such as Figure 7 shown.

[0164] In this embodiment, the original image is applied to the image pyramid, and the matching score is calculated for the scene of each layer of the pyramid and the template in the corresponding stuck angle model, and then the matching calculation is performed.

[0165] The formula for calculating the matching score is as follows:

[0166]

[0167] Sm(u,v) represents the matching score of a certain pixel point (u,v), (u,v) represents the horizontal and vertical coordinates of the pixel respectively, G represents the gradient information of the image, T represents the calculation template, X, Y represents the length and width of the template respectively, S represents the calculation area of ​​the 2D image (i.e. the calculated scene), and m represents the number of pixels. The score calculated by this formula ranges from 0 to 1. The higher the score, the more similar the template and the scene are.

[0168] 303. Score non-maximum suppression is performed on the matching results (matching scores of the four-layer pyramid) to filter out repeated recognition results at the same position.

[0169] By performing mirror template matching in the above steps, the positions of the four corners of each pyramid layer can be obtained. Each layer has four positions. In this embodiment, a four-layer pyramid is taken as an example, and a total of 16 positions can be obtained.

[0170] 304. Then use the pyramid method to refine it.

[0171] Apply f(k+1)=2*f(k) in sequence from the top pyramid to the bottom.

[0172] Where f represents the coordinates of the current layer, and k represents the number of layers.

[0173] The calculation result of the previous layer, according to the principle of pyramid, passes the positioning position to the next layer, such as Figure 8 shown.

[0174] 305. When calculating the last layer based on step 304, the normal distribution is used to improve the angle detection accuracy.

[0175]

[0176] As shown in the above formula, f(x′) is the current score, σ is 1, x' is the current angle, and μ is the new angle value. Applying this solution, the error of angle positioning will be further reduced.

[0177] 305. Apply the above template matching method and obtain the four corner positions of the jam through the threshold range (the threshold is set to be greater than 0.7) of the template and 2D image matching.

[0178] If the four corners fail to locate, it means that there is no complete jam in the scene, an exception is returned, and the process ends. The successful matching result in the upper right corner is as follows Figure 7 As shown, the four corners are successfully positioned at the same time. Figure 8 shown.

[0179] In this step, the pyramid multi-angle template matching method can adapt to different camera shooting postures, accurately locate the position, and the accuracy can reach within 2 pixels, and the angle accuracy is within 0.5 degrees. This solution makes the entire anti-fool algorithm more adaptable in actual use, such as the impact caused by uneven ground and irregular placement of plugs.

[0180] 306. Perform an inverse affine transformation based on the coordinates of the four corner points of the jam obtained by template matching. The purpose of applying the transformation is to correct the jam identification area so that the captured jam corner points are at a level viewing angle for the next segmentation process. Figure 8 shown.

[0181] The input points of the inverse affine transformation are the four corner points of the card, which are P lt ,P rt ,P rb ,P lb The result point P of the inverse affine transformation is l ' t ,P r ' t ,P r ' b ,P l ' b , calculated according to the following formula:

[0182]

[0183] P l ' t =(0,0),P r ' t =(w,0),P r ' b =(w,h),P l ' b =(0,h)

[0184] Where w and h represent the width and height of the jam in the image, respectively. rt .x and P rt .y represent corner points P rt The x and y coordinates of P lt .x、P lt .y represent corner points P lt The x and y coordinates of P rb .x and P rb .y represent corner points Prb In this embodiment, the four points after correction are a rectangle. After the three points of the standard rectangle are confirmed, the fourth point is naturally determined, so the above does not reflect P lb This corner point.

[0185] The transformation matrix of the transmission transformation is calculated based on the jammed corner points and the result points, and the jammed area after correction is obtained as follows: Fig. 9 .

[0186] 307. Run the DNN deep learning trained based on data enhancement to recognize the QR code on the wafer, such as Fig. 9 As shown, the QR code is used to infer the layer of the wafer.

[0187] By inputting the DNN deep learning model and identifying the QR code on the wafer, the position of the wafer in the image can be obtained. Fig.10 shown.

[0188] The training process can be described as follows: collect the QR code texture image on the wafer, and manually annotate the 2D position and area of ​​the QR code. Enhance the annotated data set through affine transformation. The purpose of affine transformation is to simulate the process of taking texture images at different viewing angles. In this process, the positions of the four corners of the annotated image are recorded as C i (i=1,2,3,4), and assign different weights w to each corner i (i=1,2,3,4). By iteratively adjusting different weight values ​​w i The process of realizing affine transformation. Generally, to simulate real scenes, w i The value range is (0.5,1.5).

[0189] The annotated enhanced data in the above steps is used as input to obtain the model through DNN network training. Due to the use of data enhancement scheme, only 100 to 200 pictures need to be collected to achieve the effect of thousands of pictures, which can cover the scene simulation of the entire operation process.

[0190] 308. Obtain image position information of each layer according to the center position information (x, y) of the recognition result of the two-dimensional code on the wafer. Fig.11 shown.

[0191] Set a scale factor s, and calculate the offset d of each wafer layer from the top for the corrected jam image. offset The wafer starting position P is calculated according to the following formula i :

[0192] P start =H·(1-s)

[0193]

[0194] P i =P start +d offset ·i

[0195] Similarly, we can get

[0196] P i+1 =P start +d offset (i+1)

[0197] In the above formula, H represents the height of the jam, P start Indicates the position of the first wafer on the top, P i represents the starting position of the i-th wafer, and W represents the width of the corrected jam image. i and P i+1 When it is between P and i

[0198] Layer, the judgment result is as follows Fig.11 shown.

[0199] Specifically, Fig.10 Indicates the actual wafer distribution, Fig.11 Indicates the inferred distribution of wafers in each layer, where the dark color represents the wafer starting position P i , the bright color represents the inferred position of the QR code, and the number on the right indicates the corresponding layer number of the current wafer.

[0200] 309. Preprocessing of original 3D point cloud data.

[0201] First, according to the position information obtained in the segmented area of ​​each layer, namely 308, the point cloud Pc of the area is extracted respectively. i ,First, uniform filtering is performed to remove noise and make the upper and lower layers of the point cloud evenly distributed to obtain the processed point cloud Ds i .

[0202] 310. Point Cloud Ds i Perform radius filtering.

[0203] The specific scheme is to first establish a KD-Tree Tree (D). For each independent point p, calculate the data of its neighboring points within a certain radius r. If N is less than the threshold, the point is marked as a noise point, otherwise it is retained. Apply this scheme to all points to obtain the filtered point cloud Fs i .

[0204] 311. For the point cloud Fs after the above processing i Run the Euclidean clustering operation to obtain the maximum number of cluster points (i.e. find the valid number of points).

[0205] The data of the largest cluster point is counted. If it is greater than the set threshold, the 3D point cloud in the current area is judged to have logistics, otherwise it is judged to be none.

[0206] 312. 3D / 2D multimodal fusion judgment.

[0207] If the judgment results of the two schemes are consistent, the current contemporary result of the task has a higher confidence level; 2D means whether there is a QR code, if there is a wafer, if not, there is no wafer;

[0208] If there is any inconsistency, we need to discuss the following cases:

[0209] 1) The result of 2D image judgment is that there is material, but the result of 3D power clustering judgment is that there is no material. This is usually because the material is thin and reflective, resulting in no 3D data. The result of 2D image judgment shall prevail;

[0210] 2) The result of 2D image judgment is that there is no material, and the result of 3D image judgment is that there is material. In this case, the data in the result of 2D image judgment is generally blurred, and the position is determined by 3D judgment result.

[0211] The fusion solution combines the judgment advantages of 2D images and 3D point clouds, which greatly increases the judgment success rate. In actual use, the overall success rate is > 99.99% when 100 to 200 pieces of collected data are used for training.

[0212] Based on the above method, this embodiment has the function of completing an important step in the semiconductor production process through a relatively low-cost visual solution: the fool-proofing task of automated wafer handling, thereby improving the integrity and safety of the entire automated process, being easy to use, and having strong adaptability.

[0213] Through 3D / 2D fusion recognition, the decision measurement of the dual-modal recognition result is determined according to the empirical value, which has a higher success rate and stability compared with the single 3D or 2D recognition scheme.

[0214] Through the affine transformation enhancement scheme, the sample data set is greatly improved, the annotation cost is reduced, and a more complete expected effect is achieved.

[0215] Example 3

[0216] In addition, an embodiment of the present invention further provides a wafer grabbing fool-proof device based on 3D vision technology, the device comprising:

[0217] An acquisition unit, used for acquiring 2D images and 3D point cloud data of the wafer to be identified;

[0218] A two-dimensional image correction unit, configured to determine whether the 2D image is complete by using a jamming angle model matching method according to the 2D image, and if the 2D image is complete, obtain the jamming angle coordinates and perform an anti-radiation transformation on the 2D image to obtain a corrected jamming image;

[0219] A two-dimensional image judgment unit is used to identify the position of the two-dimensional code on each layer of wafers according to the corrected jam image and a pre-trained deep learning model, and obtain the position information of each layer of wafers according to the position of the two-dimensional code of each layer of wafers;

[0220] A three-dimensional data processing unit, used to obtain filtered point cloud data of each layer according to the 3D point cloud data and position information of each layer of wafers, and perform clustering processing to obtain clustering results;

[0221] The modal fusion unit is used to perform multi-modal fusion judgment based on the identified QR code position of each layer of wafer and the clustering result of 3D point cloud data to obtain the result of fusion judgment.

[0222] Compared with the above-mentioned traditional scheme, the anti-fool-proof recognition in the method of this embodiment is more stable, that is, it adopts the dual-modality method of 2D image and 3D point cloud data simultaneously output by the 3D camera, and combines the corresponding recognition methods of 2D and 3D, so that the anti-fool-proof effect is more accurate and stable.

[0223] The method of this embodiment is easy to use. When the model used in the method requires training samples, a small number of training samples can be used, and then the entire training data set can be obtained through sample enhancement. It does not require a large amount of labeled training and is easy to deploy.

[0224] In addition, an embodiment of the present invention further provides a wafer grabbing fool-proofing system based on 3D vision technology, comprising a 3D sensor and an electronic device, wherein the electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program in the memory and executes the wafer grabbing fool-proofing method based on 3D vision technology described in any of the above embodiments;

[0225] The 3D sensor is arranged at a designated position to capture a jammed image of a wafer to be identified, and the 3D sensor collects 2D images and 3D point cloud data.

[0226] The system in the embodiment of the present invention is more adaptable to cameras, has stronger compatibility with camera shooting positions, postures, etc., is better adaptable to sizes of wafers and plugs, and is convenient for replacing models on the production line.

[0227] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0228] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0229] In the present invention, unless otherwise clearly specified and limited, when a first feature is “on” or “below” a second feature, it may be that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediate medium. Moreover, when a first feature is “above”, “above” or “above” a second feature, it may be that the first feature is directly above or obliquely above the second feature, or it may simply mean that the first feature is higher in level than the second feature. When a first feature is “below”, “below” or “below” a second feature, it may be that the first feature is directly below or obliquely below the second feature, or it may simply mean that the first feature is lower in level than the second feature.

[0230] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0231] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may alter, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A wafer grabbing and foolproofing method based on 3D vision technology, characterized in that: include: S100, obtaining a 2D image and 3D point cloud data of a wafer to be identified; S200, judging whether the 2D image is complete by using a jamming angle model matching method according to the 2D image, and if the 2D image is complete, obtaining jamming angle coordinates and performing a reverse radiometric transformation on the 2D image to obtain a corrected jamming image; S300, according to the corrected jammed image, using a pre-trained deep learning model to identify the position of the QR code on each layer of wafer, and obtaining the position information of each layer of wafer according to the position of the QR code of each layer of wafer; S400, acquiring filtered point cloud data of each layer according to the 3D point cloud data and position information of each layer of wafers, and performing clustering processing on the filtered point cloud data of each layer to obtain a clustering result of each layer; S500 , performing multimodal fusion judgment according to the identified two-dimensional code position of each layer of wafers and the clustering result of each layer, and obtaining the result of the fusion judgment.

2. The method according to claim 1, characterized in that: Before S200, the method further includes: Construct the Kasai Angle model; Or, specifically, based on a predetermined upper left corner image of the jam, binarization is performed, and the binarized image is scaled proportionally to obtain a jam upper left corner image template with a four-layer pyramid structure; The upper left corner image template of each layer in the pyramid structure is rotated 360 times, 1 degree each time, and the contour of each upper left corner image template of each layer after rotation is extracted to obtain the template contour map of four layers with equal scale; The four-layer proportionally scaled template contour images are mirrored and flipped to obtain the template contour images of the upper right corner, the lower right corner, and the lower left corner, and the template contour images of the upper left corner, the upper right corner, the lower right corner, and the lower left corner are combined to obtain the jamming angle model; Or, specifically, based on a predetermined upper left corner image of the jam, binarization is performed, and the binarized image is scaled proportionally to obtain an upper left corner image template of the jam with an M-layer pyramid structure; The upper left corner image template of each layer in the pyramid structure is rotated P times, each time by 360 / P degrees, and the contour of each upper left corner image template of each layer after rotation is extracted to obtain the template contour map of M layers with equal scale; P is a natural number greater than or equal to 10; M is a natural number greater than or equal to 1; The proportionally scaled template contour map of the M layer is mirror-flipped to obtain the template contour maps of the upper right corner, the lower right corner, and the lower left corner, and the template contour maps of the upper left corner, the upper right corner, the lower right corner, and the lower left corner are combined to obtain the jamming angle model.

3. The method according to claim 2, characterized in that The S200 includes: S201, matching the 2D image with the upper left corner template contour in the jamming angle model, and obtaining a matching score between each template contour and the 2D image in each layer of the pyramid structure; S202, performing non-maximum suppression on the matching scores of template contours of all layers to obtain matching results of the four layers of template contours in the upper left corner; Traverse the template contour matching of the upper right corner, the lower right corner and the lower left corner to obtain the matching results of the template contour matching of each layer of the pyramid structure corresponding to the four stuck corners; S203, refining the matching results of the four jamming angles, and obtaining the positions of the four jamming angles using normal distribution; S204. If the positioning of at least one stuck corner fails, send foolproof reporting information; otherwise, confirm that the 2D image is a complete image, and perform an inverse affine transformation on the 2D image based on the positions of the four stuck corners to obtain a corrected stuck image.

4. The method according to claim 3, characterized in that The S201 includes: The matching score between each template contour and the 2D image in each layer of the pyramid structure is obtained based on the following formula; Among them, Sm(u,v) represents the matching score of a certain pixel point (u,v), (u,v) represent the pixel points of the 2D image respectively, G represents the gradient information obtained by calculating the 2D image, T represents the template contour in each layer of the pyramid structure, X represents the length of the template contour, Y represents the width of the template contour, S represents the calculation area in the 2D image, m represents the number of pixels, i ranges from 1 to m; x is the horizontal coordinate of the template contour, and y is the vertical coordinate of the template contour; Alternatively, the S203 includes: S2031, passing from the top layer of the pyramid structure in the stuck angle model to the bottom layer layer by layer, applying f(k+1)=2*f(k) in order; and passing to the next layer based on the calculation result of the previous layer, until the result of the bottom layer is obtained; Among them, f represents the coordinates of the current layer, k represents the number of layers, S2032. For the bottom layer result obtained, use normal distribution to obtain the positions of four jamming angles. Specifically, f(x′) is the current score, σ is 1, x' is the current angle, and μ is the new angle value of the jamming angle; If μ is greater than the specified threshold, the angle value corresponding to μ is used as the position of the jamming angle; otherwise, the position positioning of the jamming angle fails; Alternatively, the step of performing an inverse affine transformation on the 2D image based on the positions of the four jamming angles to obtain a corrected jamming image in S204 includes: If the position corner points of the Carssell angle are P lt ,P rt ,P rb ,P lb , based on the following formula, we can get the four corner points P after transformation l ' t ,P r ' t ,P r ' b ,P l ' b ; P l ′ t =(0,0),P r ′ t =(w,0),P r ′ b =(w,h),P l ′ b =(0,h) Where w and h represent the width and height of the jam in the rectified 2D image, respectively. rt .x and P ry .y represent corner points P rt The x and y coordinates of P lt .x、P lt .y represent corner points P lt The x and y coordinates of P rb .x and P rb .y represent corner points P rb The x and y coordinates of .

5. The method according to claim 1, characterized in that: Before S300, the method further includes: Acquire a training image set, and train the deep learning model according to the training image set; or, Acquire a training image set, and train the deep learning model according to the training image set; Get a training image set, including: Pre-collecting a plurality of first 2D images including a texture image of a two-dimensional code on a wafer, and marking the position and area of ​​the two-dimensional code on the first 2D image; Performing an affine transformation on the annotated first 2D image to obtain an enhanced training image set; The deep learning model is trained using the enhanced training image set to obtain a trained deep learning model.

6. The method according to claim 5, characterized in that The step of obtaining the position information of each layer of wafers according to the position of the two-dimensional code of each layer of wafers in S300 includes: The starting position P of each layer of wafer is calculated according to the following formula: i : P start =H·(1-s) P i =P start +d offset ·i Similarly: P i+1 =P start +d offset (i+1) Where H is the height of the jam, P start Indicates the position of the first wafer on the top, P i represents the starting position of the i-th wafer, W represents the width of the corrected jam image, and when the Y coordinate of the QR code position label is at P i and P i+1 When it is between , it means that the wafer to which the label belongs is located in the Pth i Layer, d offset is the offset of each layer of wafer.

7. The method according to claim 5, characterized in that The S400 includes: According to the position information of each layer of wafer, the point cloud Pc of the position of each layer of wafer is extracted from the 3D point cloud data. i , uniform filtering is performed to remove noise, and each layer of point cloud is evenly distributed to obtain the processed point cloud Ds i ; S34, point cloud Ds i Perform radius filtering; Specifically, a KD-Tree Tree (D) is established. For each independent point p, the data of its neighboring points within a certain radius r are calculated. If N is less than the threshold, the point is marked as a noise point, otherwise it is retained. All points in the point cloud are traversed to obtain the filtered point cloud Fs i ; S35, the filtered point cloud Fs i Run the Euclidean clustering operation to obtain the largest cluster point; count the data of the largest cluster point, if it is greater than the set threshold, the 3D point cloud in the current area is judged to have a wafer, otherwise it is judged to have no wafer, and the clustering results of the 3D point cloud of each layer of wafer are obtained.

8. The method according to claim 1, characterized in that The S500 includes: If the clustering result of each layer of 3D point cloud indicates that there is a wafer, and the layer contains information of a QR code, then it is confirmed that there is a wafer; Otherwise, if there is a layer of 3D point cloud whose clustering result is no wafer, but the layer has the information of QR code, it is confirmed that there is a wafer; If the clustering result of a layer of 3D point cloud indicates that there is a wafer, but there is no QR code information in this layer, it is confirmed that there is a wafer.

9. A wafer grabbing fool-proof device based on 3D vision technology, characterized in that: include: An acquisition unit, used for acquiring 2D images and 3D point cloud data of the wafer to be identified; A two-dimensional image correction unit, configured to determine whether the 2D image is complete by using a jamming angle model matching method according to the 2D image, and if the 2D image is complete, obtain the jamming angle coordinates and perform an anti-radiation transformation on the 2D image to obtain a corrected jamming image; A two-dimensional image judgment unit is used to identify the position of the two-dimensional code on each layer of wafers according to the corrected jam image and a pre-trained deep learning model, and obtain the position information of each layer of wafers according to the position of the two-dimensional code of each layer of wafers; A three-dimensional data processing unit, used to obtain filtered point cloud data of each layer according to the 3D point cloud data and position information of each layer of wafers, and perform clustering processing to obtain a clustering result; The modal fusion unit is used to perform multi-modal fusion judgment based on the identified QR code position of each layer of wafer and the clustering result of 3D point cloud data to obtain the result of fusion judgment.

10. A wafer grabbing foolproof system based on 3D vision technology, characterized in that: The device comprises a 3D sensor and an electronic device, wherein the electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program in the memory and executes the wafer grabbing fool-proofing method based on 3D vision technology as described in any one of claims 1 to 8; The 3D sensor is arranged at a designated position to capture a jammed image of a wafer to be identified, and the 3D sensor collects 2D images and 3D point cloud data.

Images