A deep learning-based multi-sensor cooperative wafer pose real-time correction method

Abstract

The application provides a multi-sensor cooperative wafer pose real-time correction method based on deep learning, and belongs to the technical field of semiconductor intelligent manufacturing. The method adopts a heterogeneous perception architecture combining global vision in the eye-off-hand mode and local laser scanning in the eye-on-hand mode, extracts complementary features from texture images and depth point clouds through a specially designed double-flow multi-modal attention network, and realizes sub-millimeter-level prediction of the six-degree-of-freedom pose of a warped wafer. Based on the prediction result, a variable impedance control model is used to generate a compliant compensation trajectory of a mechanical arm, and the stiffness of the mechanical arm is dynamically adjusted at the contact moment, thereby completely solving the stress damage problem in the wafer grabbing process.

Description

Technical Field

[0001] This invention belongs to the field of semiconductor intelligent manufacturing technology, and in particular relates to a multi-sensor collaborative real-time wafer pose correction method based on deep learning. Background Technology

[0002] In advanced semiconductor wafer manufacturing processes, especially at 7nm, 5nm and smaller nodes, wafer manufacturing is extremely complex, involving hundreds of steps such as coating, developing, etching, and thin film deposition. As wafer size has standardized to 12 inches and its thickness continues to decrease to accommodate 3D stacking technology, wafers are highly susceptible to nonlinear warping deformation after undergoing processes such as high-temperature annealing and thin film deposition. The warping shape is usually saddle-shaped, bowl-shaped, or irregularly twisted, and the warping amount can reach hundreds of micrometers or even several millimeters.

[0003] However, current automated wafer handling equipment in semiconductor production lines relies primarily on preset rigid teaching trajectories for wafer gripping and placement. This open-loop control mode cannot perceive the actual three-dimensional shape and pose deviation of the wafer on the transfer disk in real time. If the wafer warps or shifts position, and the end effector of the robotic arm still follows the standard planar trajectory, hard contact collisions are highly likely to occur, leading to wafer edge chipping, microcracks, or microparticle contamination, severely impacting chip yield. Therefore, there is an urgent need for a method that can integrate multi-dimensional sensing data, accurately predict the six degrees of freedom pose of the wafer in real time during dynamic transfer, and guide the actuator to perform active compliant correction.

[0004] The existing wafer pose recognition and correction methods mainly include the following three types:

[0005] 1. Mechanical Notch or Flat Edge Physical Alignment: This method relies on a dedicated mechanical pre-alignment machine. The wafer is vacuum-adsorbed and rotated 360 degrees. A transmission-type photoelectric sensor scans the notch or flat edge of the wafer to determine the center coordinates and rotation angle. The drawback of this method is that it requires physical contact for mechanical rotation, resulting in low processing efficiency. Alignment time per wafer typically exceeds five seconds, severely limiting production line throughput. Furthermore, this method can only correct lateral and longitudinal coordinates and planar rotation angle deviations within a plane, completely failing to detect height fluctuations, roll angles, and pitch angles of the wafer in the vertical axis direction, and cannot address the risk of handling warped wafers.

[0006] 2. Planar positioning method based on traditional 2D machine vision: This method uses an industrial camera mounted on top to acquire grayscale images of the wafer and calculates the wafer position using Canny edge detection, Hough circle transform, or geometric template matching algorithms. The fundamental drawback of this method is the lack of depth information. Monocular vision systems cannot accurately measure the vertical height and spatial normal vector of the wafer surface. When the wafer is warped or the transport disk is tilted, the projection on the 2D image will be distorted, leading to a decrease in positioning accuracy. More seriously, it cannot provide vertical axis height feedback, and the robotic arm cannot dynamically adjust the gripping height according to the wafer warping, which can easily cause wafer crushing accidents.

[0007] 3. Single-point discrete detection method based on laser ranging: This method installs three to four point laser displacement sensors at the end of a robotic arm and fits a plane equation by measuring the distance from a limited number of points to the wafer surface. The drawbacks of this method are sparse data and poor anti-interference capabilities. The limited number of measurement points makes it difficult to reproduce complex nonlinear wafer surface warping patterns, such as wavy warping. Furthermore, the wafer surface material is complex, such as highly reflective aluminum layers or transparent photoresist layers. The significant differences in reflectivity between different materials can easily cause jumps in point laser sensor readings or generate noise, leading to errors in the calculation of the normal vector of the fitted pose plane and preventing high-precision full-attitude correction.

[0008] In summary, existing methods cannot simultaneously meet the requirements of non-contact, high efficiency, six-degree-of-freedom full-attitude sensing, and adaptability to large warp wafers. Summary of the Invention

[0009] To address the aforementioned problems, this invention proposes a multi-sensor collaborative real-time wafer pose correction method based on deep learning. This method goes beyond simply superimposing sensor data; instead, it constructs a complete closed loop encompassing precise physical perception, heterogeneous feature deep inference, simulation and real-world transfer training, and variable impedance compliant control. This fundamentally solves the depth perception blind spots inherent in traditional vision methods when dealing with large-warped wafers, as well as the rigid impact problems arising from traditional position control when contacting non-planar wafers.

[0010] This invention provides a multi-sensor collaborative real-time wafer pose correction method based on deep learning, comprising the following steps:

[0011] S1, based on an industrial camera, acquires a global texture image covering the entire surface of the wafer. High-density local depth contour point cloud data is collected by a robotic arm-driven line laser sensor.

[0012] S2, calculate the hand-eye calibration matrix for spatiotemporal registration; reconstruct the local depth contour point cloud data into continuous original 3D point cloud data based on spatial interpolation and coordinate transformation. ;Will The depth map is generated by back-projecting the depth onto the camera's imaging plane and using a depth buffer to retain only the depth value closest to the camera's optical center. ,Will The three color channels and The single depth channel is stitched together along the channel dimension to obtain a four-channel multimodal input data tensor. ;

[0013] S3, will and Input a trained two-stream deep learning model for wafer six-DOF pose prediction, and output the predicted translation vector of the wafer center. Quaternions for spatial attitude prediction The model includes a texture feature extraction branch, a geometric feature extraction branch, a collaborative attention fusion module, and a multi-task pose regression head.

[0014] S4 firstly uses the inverse operation of the predicted pose and the teaching reference to solve the deviation matrix, and constructs a six-dimensional attitude error vector and a second-order impedance dynamic control law; then, based on the vertical height deviation, the Z-axis stiffness and damping parameters are adjusted in real time to achieve flexible motion correction.

[0015] Preferably, the calculation of the hand-eye calibration matrix specifically involves:

[0016] First, calculate the camera optical center coordinate system. Relative to the robot arm base coordinate system static rigid body transformation matrix , including descriptions of rotational relationships Rotation matrix With description of positional relationship Translation vector ;

[0017] Secondly, using the tool center point, i.e., the TCP calibration method, the coordinate system of the origin of the line laser emission is calculated. Relative to the flange center coordinate system Fixed transformation matrix ;

[0018] The forward kinematics matrix of the flange relative to the base, fed back from the robotic arm controller, is read in real time via industrial Ethernet. Using the chain rule Solve for the coordinate system of the linear laser sensor Relative to the robot arm base coordinate system Real-time transformation matrix .

[0019] Preferably, the texture feature extraction branch specifically includes:

[0020] First, the input has 4 channels and a convolution kernel size of [size missing]. An initial convolutional layer with a stride of 2 is used, followed by a max pooling layer. Downsampling is then applied to reduce the feature map resolution and expand the receptive field, resulting in a shallow feature map containing basic edge information. ;

[0021] The data passes through four cascaded residual layers, each consisting of multiple stacked bottleneck structures, each containing... , , The three convolutional layers, combined with batch normalization and the ReLU activation function, produce a high-dimensional feature map containing rich semantic information at the output of the fourth residual layer, Layer 4. ;

[0022] Will The input is fed into a dimensionality-reducing convolutional layer, which compresses the number of channels to a preset dimension. Then, a global average pooling operation is performed to compress the spatial dimension to 1, and the vector is flattened. Finally, the texture feature vector is output. .

[0023] Preferably, the geometric feature extraction branch extracts the original 3D point cloud data reconstructed in the previous step. Perform farthest point sampling; after sampling, obtain the result containing... Input point set of points Then, through three cascaded set abstraction modules, geometric features from local microstructure to global morphology are extracted step by step:

[0024] First-level SA layer: As input, the farthest point sampling algorithm is used to select... There are several center points; with each center point as the center of the sphere and a radius... Perform a sphere query to construct a local neighborhood; use a coordinate difference calculation method to convert the spatial location information of neighborhood points into local relative coordinates relative to the center point; then input to the channel number. The shared multilayer perceptron is used, and finally, max pooling is used to extract local features with permutation invariance, outputting the first-level feature tensor. ;

[0025] Second-level SA layer: The feature tensor output by the first-level SA layer The coordinates of the center point and its corresponding center point are used as input, and then the farthest point sampling algorithm is used to select the center point. A central point; with an expanded radius Perform a ball query to construct a neighborhood; using the same coordinate difference calculation method as the first-level SA layer, convert the absolute coordinates of each point in the neighborhood to relative coordinates relative to the current center point; then concatenate the relative coordinates with the features and input them into a channel with a number of channels. The shared multilayer perceptron outputs a second-level deep feature tensor. ;

[0026] Third-level SA layer: For input, all Each point is treated as a single global group, and the number of inputs to the channels is [number missing]. The MLP performs a global max pooling operation to aggregate the global geometric feature vectors that characterize the overall warpage of the wafer, denoted as . .

[0027] Preferably, the multi-task pose regression head is divided into two parallel fully connected sub-networks, which take the fused context feature vector that integrates multimodal information output by the collaborative attention fusion module as input, and process the position and pose regression tasks respectively:

[0028] Translation regression branch includes The fully connected layers have a decreasing number of neurons in each layer. ReLU activation functions are added between layers to increase non-linearity. The final output layer contains 3 neurons, corresponding to the X, Y, and Z axis coordinates.

[0029] Rotational regression branch includes The layer is a fully connected layer, and the final output layer contains 4 neurons, corresponding to the four components of a quaternion;

[0030] Finally, the translation regression branch outputs the predicted translation vector of the wafer center, denoted as... ,in These represent the predicted X, Y, and Z axis displacements of the wafer center in the base coordinate system, respectively. The output of the rotation branch, after L2 norm normalization, yields a predicted unit quaternion representing the spatial attitude, denoted as... .

[0031] Preferably, the S4 process specifically includes:

[0032] S41, construct the current pose matrix using the predicted quaternion and translation vector, and calculate the correction transformation matrix containing the translation vector and rotation matrix by inverse operation with the standard teaching reference pose matrix.

[0033] S42 encapsulates translational deviations with different physical dimensions and rotational deviations analyzed by matrix logarithmic mapping algorithm into a six-dimensional attitude error vector, and establishes a second-order impedance dynamics equation describing the relationship between force and motion as the core control law.

[0034] S43, based on the predicted vertical axis height deviation, dynamically adjusts the target stiffness and damping coefficient of the robotic arm in the Z-axis direction in real time, so that the robotic arm exhibits low stiffness and flexible characteristics at the moment of contact in order to achieve posture correction and compliant grasping tasks.

[0035] S44 utilizes the horizontal deviation component to drive the robotic arm to perform concentric correction in the XY plane, while simultaneously adjusting the suction cup normal according to the rotational deviation vector to parallel fit the wafer warp surface, eliminating eccentricity error and the risk of unilateral force.

[0036] Preferably, the specific process of S41 is as follows:

[0037] First, obtain the reference pose matrix for standard teaching. Then, using the output prediction results, including the prediction unit quaternion, denoted as... With the predicted translation vector Construct the current pose matrix The construction process is as follows:

[0038] The predictor unit quaternion is denoted as ,in For the real part, It is the imaginary part;

[0039] Based on the transformation relationship from quaternions to rotation matrices, calculate Orthogonal rotation matrix Perform the splicing operation along the column direction, setting the left region as a rotation matrix. And the region on the right is the predicted translation vector. The dimension is obtained as The middle matrix; then fill the bottom of the matrix with a row of values. The row vectors are ultimately used to generate the standard... Homogeneous transformation matrix ;

[0040] Calculate the correction transformation matrix : The homogeneous transformation matrix Right-multiply the reference pose matrix The inverse matrix, then from the correction transformation matrix Extract the translation deviation vector With rotational deviation matrix The two parameters are directly used to generate the inverse kinematics position compensation command for the robotic arm.

[0041] Preferably, the specific process of S42 includes:

[0042] Six-dimensional attitude error vector The analytical construction unifies translational and rotational deviations with different physical dimensions into a six-dimensional vector. :

[0043] 1) Translation component extraction: Directly extract the translation deviation vector. As The first three components;

[0044] 2) Rotation component transformation: The rotation deviation matrix is ​​processed using a matrix logarithmic mapping algorithm. First, calculate the trace relation. rotation angle Then, based on the antisymmetric property of the rotation matrix, the unit vector of the rotation axis is uniquely determined using the formula. :

[0045]

[0046] in Representative matrix The Middle Line number The column's numerical elements; rotate by angle With unit vector The product is defined as the rotation error vector. Then the translation deviation vector will be shifted. With rotation error vector Concatenate the data sequentially to generate a complete six-dimensional attitude error vector. ;

[0047] Establishment of the second-order impedance dynamics control law: Based on the above error vector, a second-order differential equation describing the relationship between force and motion is constructed as the core control law of the system:

[0048]

[0049] in, , , These represent the target inertia matrix, target damping matrix, and target stiffness matrix, respectively. Set as a preset constant diagonal matrix; and Defined as a matrix of state variables to be determined; The external contact force is fed back in real time by a six-dimensional force sensor at the end of the robotic arm.

[0050] Preferably, the specific process of S43 includes:

[0051] Based on vertical deviation Variable impedance compliant contact control: Extraction of vertical component Then, substitute the values ​​into the exponential decay formula to calculate the target stiffness. When the robotic arm moves away from the wafer, it maintains high stiffness to ensure trajectory accuracy during high-speed movement; when the robotic arm approaches the wafer surface, it reduces stiffness to make the end effector exhibit spring-like flexibility; calculated values Real-time updates to the target stiffness matrix in step S42 The Z-axis component position;

[0052] Using formula Real-time calculation of target damping ,in For the target inertia matrix Diagonal element components along the Z-axis; calculated values The target damping matrix in step S42 will be updated in real time. The Z-axis component position.

[0053] Preferably, the horizontal deviation component output in step S41 is... and As a position compensation command input control loop, it drives the robotic arm to perform precise translational motion in the XY plane, so that the center point of the suction cup is strictly close to the predicted geometric center of the wafer in the horizontal dimension.

[0054] Based on rotational deviation Normal alignment attitude control: The rotation deviation matrix is ​​analyzed using a matrix logarithmic mapping algorithm. First, the rotation angle is uniquely calculated based on the properties of the matrix trace. Then, the unit vector of the rotation axis is constructed using the following analytical formula.

[0055]

[0056] in Representative matrix The Middle Line number The column element values; the final calculated rotation angle With unit vector The product is defined as the rotation vector. And as an attitude correction command, it drives the end flange of the robotic arm to rotate around the spatial axis. Rotation angle The normal vector of the forced suction cup plane actively seeks out and parallels the predicted local normal vector of the wafer surface.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] (1) Spatiotemporal precise alignment of heterogeneous sensors and construction of dense red-green-blue depth multimodal data: A heterogeneous data fusion method based on robot kinematics chain is proposed. The two-dimensional texture information with fixed viewpoint and the three-dimensional depth information of moving scan are registered at the pixel level in a unified world coordinate system. A dense multimodal dataset containing micro-texture and macro-geometric warping of wafer surface is constructed, which breaks through the limitation of insufficient information dimension of single sensor.

[0059] (2) Two-stream geometric texture co-coding and pose regression network: A parallel deep neural network architecture was designed, which includes a ResNet50 convolutional branch for processing texture and a PointNet++ point cloud branch for processing geometry. By introducing a co-attention mechanism, the network can adaptively allocate weights between the two modalities according to feature saliency. For example, when locating the Notch, texture features are emphasized, and when judging the warp height, geometric features are emphasized, thus achieving robust prediction of six-DOF pose under complex conditions.

[0060] (3) Variable impedance compliant correction control based on predicted normal vector: A dynamic mapping model from pose prediction deviation to robot arm impedance parameters is established. Unlike traditional pure position compensation, this invention corrects the posture of the robot arm end in real time based on the predicted wafer surface normal vector deviation, and dynamically adjusts the damping and stiffness coefficients of the robot arm in the contact direction based on the vertical axis prediction error, thus realizing soft landing compliant correction. Attached Figure Description

[0061] To more clearly illustrate the technical solutions of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the following description is only one embodiment of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a flowchart illustrating the overall steps of the visual rotational speed measurement method based on temporal feature analysis and dual-domain refinement estimation of the present invention.

[0063] Figure 2 This is a network architecture diagram of the two-stream deep learning model for wafer six-degree-of-freedom pose prediction in this invention.

[0064] Figure 3 This is a structural diagram of the collaborative attention fusion module of the present invention.

[0065] Figure 4 This is a diagram showing the error distribution of six-degree-of-freedom pose prediction on a wafer in an embodiment of the present invention.

[0066] Figure 5 This is a comparison chart of pose prediction accuracy under different degrees of warping in embodiments of the present invention.

[0067] Figure 6 This is a comparison chart of the contact force variation curves during the wafer gripping process in an embodiment of the present invention. Detailed Implementation

[0068] The present invention will be further described below with reference to embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0069] Please see Figure 1 This invention illustrates a deep learning-based multi-sensor collaborative real-time wafer pose correction method according to an embodiment of the present invention. This method employs a heterogeneous perception architecture combining global vision in an eye-on-hand mode with local laser scanning in an eye-on-hand mode. Through a specially designed dual-stream multimodal attention network, complementary features are extracted from texture images and depth point clouds to achieve sub-millimeter-level prediction of the six-DOF pose of a warped wafer. Based on the prediction results, a compliant compensation trajectory for the robotic arm is generated using a variable impedance control model, dynamically adjusting the robotic arm stiffness at the moment of contact, thus completely solving the stress damage problem during the grasping of warped wafers.

[0070] S1. Multi-sensor heterogeneous data collaborative acquisition and spatiotemporal registration in wafer transfer scenarios

[0071] The core task of this step is to construct a high-precision physical perception benchmark. The system deploys a collaborative perception system at the wafer transfer station, comprising a high-resolution industrial camera and a high-precision line laser profilometer. Given the differences in rigid body transformation between visual texture data and laser depth data in the spatial dimension and the differences in sampling frequency in the temporal dimension, this step utilizes hand-eye calibration matrices and robot kinematics interpolation algorithms to rigorously map these two heterogeneous data sets to a unified base coordinate system, thereby constructing pixel-level aligned multimodal input data.

[0072] (1) Hardware layout and parameter definition of heterogeneous sensors: In order to balance the globality of macroscopic positioning and the precision of microscopic morphology detection, the system adopts a complementary sensor layout scheme.

[0073] Deploy a machine with a resolution of Global shutter industrial cameras, in which height Set to 2048 pixels, width The camera is set to 2048 pixels. Mounted vertically 1000 mm above the wafer transfer stage with a telecentric lens, it is specifically designed to capture global texture images of the wafer surface. Global shutter technology effectively eliminates the rolling shutter effect that may occur during robotic arm movement, ensuring image clarity for edge notches and surface laser coding.

[0074] A transverse sampling point is deployed on the side of the flange at the end of the robotic arm. The blue line laser profile sensor, in which The system is configured with 1280 physical sampling points. A 405 nm blue laser was chosen to minimize noise interference caused by the high reflectivity of the metal coating on the wafer surface. This sensor is responsible for acquiring high-density local depth contour point cloud data at high frequency during the movement of the robotic arm to detect micron-level surface warping changes.

[0075] (2) Hand-eye calibration and coordinate system one: In order to unify the spatial reference, it is necessary to accurately calculate the relative positional relationship between the sensor and the robotic arm and establish the kinematic chain of the whole system.

[0076] First, calculate the camera optical center coordinate system. Relative to the robot arm base coordinate system static rigid body transformation matrix This matrix contains information describing the rotation relationships. Rotation matrix With description of positional relationship Translation vector ;

[0077] Secondly, using the tool center point, i.e., the TCP calibration method, the coordinate system of the origin of the line laser emission is calculated. Relative to the flange center coordinate system Fixed transformation matrix ;

[0078] The system reads the forward kinematics matrix of the flange relative to the base from the robotic arm controller in real time via industrial Ethernet. Using the chain rule Solve for the coordinate system of the linear laser sensor Relative to the robot arm base coordinate system Real-time transformation matrix Based on this global transformation matrix, a precise spatial description of the laser measurement point at any time in the global base coordinate system can be established.

[0079] (3) Data scanning and dense point cloud reconstruction: During the data acquisition phase, the robotic arm is controlled to drive the line laser sensor at a constant speed. A linear scanning motion is performed along the wafer diameter. The line laser sensor operates at a frequency... The system triggers the acquisition of a 2D contour line. It then combines the discrete laser contour data over time with the corresponding robotic arm pose. Perform spatial interpolation and coordinate transformation. For the first... The first frame in the scan data laser points Its world coordinate calculation formula is: .

[0080] By accumulating and stitching together all frame data, continuous original 3D point cloud data is reconstructed, denoted as... This process effectively preserves the nonlinear warping morphology of the wafer surface caused by thermal stress, compensating for the deficiency that a single visual system cannot perceive height information.

[0081] (4) Pixel-level spatiotemporal registration and multimodal data generation: In order to achieve pixel-level correspondence between texture features and geometric features, a transformation matrix is ​​used. The inverse matrix of the 3D point cloud The image is projected backwards onto the camera's imaging plane. For 3D point clouds... In the coordinate system of the robot arm base The lower coordinate is Any point in space, combined with the camera intrinsic parameter matrix Calculate its position in pixel coordinates The following two-dimensional coordinates:

[0082]

[0083] This process employs a depth buffer, or Z-Buffer algorithm, to handle occlusion relationships, i.e., at the same pixel position. If multiple projection points exist, only the depth value closest to the camera's optical center is retained, thus generating a depth map with a resolution strictly consistent with the texture image, denoted as . Simultaneously, a global shutter industrial camera is deployed to acquire two-dimensional images covering the entire surface of the wafer from the same viewpoint, which are denoted as global texture images. .

[0084] Ultimately, The three color channels and A single depth channel performs a stitching operation along the channel dimension, outputting a four-channel multimodal input data tensor, denoted as . Its dimensions are This tensor This will serve as the standard input for the texture branch in subsequent steps of the model, while the downsampled point cloud This will be used as the standard input for the geometry branch.

[0085] S2, Construction of a dual-stream deep learning model for wafer six-DOF pose prediction

[0086] This step constructs a wafer-level six-DOF pose prediction deep learning model as the core inference engine of the entire system. The model structure is as follows: Figure 2 As shown. The input to this model is the multimodal input data tensor containing texture and depth information generated in step S1. and raw 3D point cloud data The model outputs translation vectors and rotation quaternions that characterize the precise spatial state of the wafer in the coordinate system of the robotic arm base.

[0087] The model's function is to establish an end-to-end nonlinear mapping relationship between the raw observation data from heterogeneous sensors and the six-degree-of-freedom physical pose. Its role is to effectively overcome the geometric distortion interference caused by nonlinear warping of the wafer surface through deep feature learning, thereby providing a unique and high-precision decision basis for the compliant correction control of the robotic arm in subsequent step S4. The specific inputs, outputs, and processing logic of each module are as follows:

[0088] (1) Texture feature extraction branch: directly use the four-channel multimodal input data tensor generated in step S1 As input, this input includes not only RGB texture information but also pixel-aligned depth map information, enabling the network to simultaneously perceive the geometric depth of planar textures and two-dimensional projections.

[0089] First, the input has 4 channels and a convolution kernel size of [size missing]. An initial convolutional layer with a stride of 2 is followed by a max-pooling layer. This step aims to reduce the feature map resolution and expand the receptive field through downsampling, outputting a shallow feature map containing basic edge information. ;

[0090] The data passes through four cascaded residual layers, denoted as Layer 1 to Layer 4. Each residual layer consists of multiple stacked bottleneck structures. Each bottleneck structure contains... , , The three convolutional layers are combined with batch normalization (BatchNorm) and the ReLU activation function. This structure can effectively control the number of parameters while increasing the network depth. At the output of Layer 4, a high-dimensional feature map containing rich semantic information is obtained, denoted as . ;

[0091] To align with the feature dimensions of the geometric branches, Input to the convolution kernel is The dimension reduction convolutional layer compresses the number of channels from 2048 to a preset dimension. Then, a global average pooling operation is performed to reduce the spatial dimension. Compress to 1, flatten the vector, and finally output the texture feature vector, denoted as . .

[0092] (2) Geometric feature extraction branch: the original 3D point cloud data reconstructed in step S1 Perform Farthest Point Sampling (FPS). FPS is a general point cloud downsampling algorithm that iteratively selects the point farthest from the currently selected point set as the new sampling point. This embodiment uses the FPS algorithm to ensure that the sampled point set uniformly covers the entire wafer surface, preserving the nonlinear warping morphology of the wafer edges to the greatest extent possible and avoiding the loss of local information due to random sampling. After sampling, a point set containing... Let the set of input points be _ points, denoted as _ _ points _ ... , in which settings .

[0093] The hierarchical feature extraction process comprises three cascaded Set Abstraction (SA) modules, designed to extract geometric features from local microstructure to global morphology step by step:

[0094] First-level SA layer: As input, the farthest point sampling algorithm is used to select... There are several center points; with each center point as the center of the sphere and a radius... Perform a sphere query to construct a local neighborhood; use the coordinate difference calculation method, that is, subtract the absolute coordinates of the corresponding center point from the absolute coordinates of each point in the neighborhood, to convert the spatial position information of the neighborhood points into local relative coordinates relative to the center point; then input to the channel number. The shared multilayer perceptron (MLP) is used, and finally, max pooling is used to extract local features with permutation invariance, outputting the first-level feature tensor. .

[0095] Second-level SA layer: The feature tensor output by the first-level SA layer The coordinates of the center point and its corresponding center point are used as input, and then the farthest point sampling algorithm is used to select the center point. A central point; with an expanded radius Perform a ball query to construct a neighborhood; using the same coordinate difference calculation method as the first-level SA layer, convert the absolute coordinates of each point in the neighborhood to relative coordinates relative to the current center point; then concatenate the relative coordinates with the features and input them into a channel with a number of channels. The shared multilayer perceptron (MLP) outputs a second-level deep feature tensor. .

[0096] Third-level SA layer: For input, all Each point is treated as a single global group, and the number of inputs to the channels is [number missing]. The MLP performs a global max pooling operation to aggregate the global geometric feature vectors that characterize the overall warpage of the wafer, denoted as . And set This ensures that the feature dimensions of the two branches are strictly aligned.

[0097] (3) Collaborative Attention Fusion Module: This module aims to address the semantic gap problem of heterogeneous features. By calculating the correlation between texture features and geometric features, the model automatically focuses on feature regions that are significant in both modalities. The model structure is as follows: Figure 3 As shown, the process is as follows:

[0098] Define query vector key vector value vector Calculate the matrix product of the query vector and the transpose of the key vector. This product physically represents the correlation score matrix between texture semantics and geometric structure;

[0099] For the score matrix Divide by scaling factor To prevent gradient vanishing due to excessively large dot product values, the Softmax function is then applied to obtain attention weight scalars with values ​​between 0 and 1. ;

[0100] scalar of attention weight AND value vector Perform scalar multiplication to obtain the weighted geometric eigenvectors, denoted as... This step enables dynamic adjustment of geometric feature representation weights based on texture correlation strength. Using a residual connection mechanism, the weighted geometric feature vectors are... Compared with the original texture feature vector Perform element-wise addition, that is The output is a fused context feature vector that incorporates multimodal information.

[0101] (4) Multi-task pose regression head: The input data comes from the fusion context feature vector of the fusion module. The network is divided into two parallel fully connected sub-networks (MLP Heads) to handle the position and pose regression tasks respectively:

[0102] Translation regression branch includes The fully connected layers have a decreasing number of neurons in each layer. ReLU activation functions are added between layers to increase non-linearity. The final output layer contains 3 neurons, corresponding to the X, Y, and Z axis coordinates.

[0103] Rotational regression branch includes The layer is a fully connected layer, and the final output layer contains 4 neurons, corresponding to the four components of a quaternion;

[0104] Finally, the translation branch directly outputs the predicted translation vector of the wafer center, denoted as... ,in These represent the predicted X, Y, and Z axis displacements of the wafer center in the base coordinate system, respectively, to distinguish them from the general coordinates of the input point cloud. The output of the rotation branch, after L2 norm normalization, yields a prediction unit quaternion representing the spatial attitude, denoted as . .

[0105] and As the final output of the entire network, it is passed to step S3 for loss calculation and step S4 for deviation control.

[0106] S3. Model Training Based on Domain Randomization and Geometric Consistency Constraints

[0107] This step describes the model's training iterations and parameter optimization process. Addressing the extreme scarcity of large-warp wafer data with precise 6D pose labels in industrial scenarios, this solution employs a two-stage strategy of "simulation pre-training + real-world fine-tuning." The training process first constructs a simulation dataset and a real dataset containing ground truth labels, with the labels being ground truth translation vectors. With the true value unit quaternion The specific implementation steps are as follows:

[0108] (1) Dataset construction and ground truth physical calibration: This step aims to construct high-quality “data-label” pairs to provide a mathematical benchmark for loss function calculation.

[0109] The thermal warpage morphology of the wafer is simulated using a cubic Bézier surface generation algorithm, through random sampling. The height parameter of the control point grid is set to a range of [range]. This generates diverse nonlinear saddle-shaped and bowl-shaped surface meshes. The coordinates of the virtual wafer's geometric center in the virtual base coordinate system are directly extracted as the truth translation vector. Extract its rotation matrix and convert it into a quaternion as the truth unit quaternion. ; Constructing using computer graphics technology that includes A simulation dataset of group samples.

[0110] To obtain high-precision physical true values, a high-precision coordinate measuring machine (CMM) was used to perform contact probe measurements on the surface of a real wafer in a static state. The physical geometric center of the wafer was then fitted and recorded as the true value translation vector. Simultaneously, the rotation matrix is ​​calculated based on the fitted normal vector and the Notch notch direction, and then converted into a true unit quaternion. This calibration process ensures that the physical accuracy of the tags reaches the sub-millimeter level. Data was collected in a physical experimental setup containing... The real dataset of the group samples.

[0111] (2) Construction and calculation of composite loss function: In order to comprehensively constrain the predictive ability of the model, the total loss function is defined. For position loss With rotational loss The weighted sum, i.e.:

[0112]

[0113] in and These are weighting coefficients used to balance the gradient magnitudes for different tasks. The physical meaning and calculation of each loss term are as follows:

[0114] Location loss This item measures the accuracy of the translation prediction. It calculates the predicted translation vector output from step S2. Translation vector with truth value The numerical differences between them are considered. To balance convergence speed and robustness, a smooth L1 loss function is adopted. When the absolute value of the prediction error is less than 1, a squared penalty is used to ensure that the function is differentiable at zero and to accelerate convergence; when the absolute value of the prediction error is greater than or equal to 1, a linear penalty is used to reduce sensitivity to outliers.

[0115] Rotational loss This item measures the accuracy of attitude prediction. It calculates the predicted unit quaternion output in step S2. With the true value unit quaternion The geodesic distance between them. Mathematically, this distance is equivalent to the shortest arc length of the two rotational movements on a four-dimensional hypersphere. The specific calculation process first solves for the dot product of the two, denoted as... Then, the loss formula was constructed using the inverse cosine function. .

[0116] This loss function utilizes the geometric properties of quaternions, effectively avoiding the gimbal lock problem in Euler angle representation and the double-coverage ambiguity problem inherent in quaternions, thus ensuring the uniqueness and continuity of attitude regression.

[0117] (3) Gradient-based parameter iterative optimization: Set the total number of training rounds to 1. Batch size is In each training iteration, a batch of data is input into the model, and forward propagation calculations are performed. Calculate using the backpropagation algorithm The gradient vectors for all weight parameters of the network are calculated. The AdamW optimizer is used to compute the first moment estimate (momentum) and second moment estimate (adaptive learning rate) of the gradients, and a weight decay term is introduced to prevent overfitting. Finally, the gradients are calculated based on the set learning rate. Update network weights.

[0118] (4) Two-stage transfer training strategy:

[0119] Phase 1 Simulation Pre-training: Using... Simulation dataset of group samples Round training, setting a high learning rate The goal of this stage is to enable the model to quickly learn basic wafer edge detection capabilities and geometric warping feature extraction capabilities.

[0120] Phase Two Real Data Fine-tuning: Using Included Real dataset of group samples Round training, setting a low learning rate and At this stage, the front of the frozen texture branch of ResNet50 is... The convolutional parameters (i.e., their weights) are not updated; only the high-level semantic parameters and fully connected layers are updated. This strategy effectively prevents overfitting caused by small sample data, while enabling the model to adapt to the unique lighting variations and texture distributions of real-world conditions.

[0121] S4. Real-time pose deviation calculation and variable impedance compliance control

[0122] This step deploys the converged model trained in S3 to an edge computing controller, enabling real-time inference of the six-DOF pose of the current wafer on the production line. The controller calculates the deviation matrix between the current pose and the standard taught pose, converts the deviation into a Cartesian space trajectory compensation command for the robotic arm's end effector, and dynamically adjusts the stiffness and damping parameters of the impedance controller based on the predicted vertical axis height deviation to drive the robotic arm to perform compliant correction movements and eliminate rigid impacts. The specific control logic is as follows:

[0123] (1) Deviation matrix calculation: First, obtain the reference pose matrix of the standard teaching. Then, the prediction results output by the S2 model, including the prediction unit quaternion, are denoted as... With the predicted translation vector Construct the current pose matrix The specific construction process is as follows:

[0124] The predictor unit quaternion is denoted as ,in For the real part, It is the imaginary part;

[0125] Based on the transformation relationship from quaternions to rotation matrices, calculate Orthogonal rotation matrix The specific formulas for calculating each element in this matrix are as follows:

[0126]

[0127] First, perform the concatenation operation along the column direction, setting the left region as a rotation matrix. And the region on the right is the predicted translation vector. The dimension is obtained as The middle matrix; then fill the bottom of the matrix with a row of values. The row vectors are ultimately used to generate the standard... Homogeneous transformation matrix .

[0128] Calculate the correction transformation matrix : The homogeneous transformation matrix Right-multiply the reference pose matrix The inverse matrix, i.e. Then from this correction transformation matrix Extract the translation deviation vector With rotational deviation matrix These two parameters are directly used to generate the inverse kinematics position compensation command for the robotic arm.

[0129] (2) Construction of target impedance model and analytical solution of error vector: This step aims to construct the dynamic mathematical model framework for the compliant control of the robotic arm end effector. The core task of this step is to define the structure of the second-order differential equation describing the relationship between force and motion, and to complete the vectorization encapsulation of pose deviation data, thereby providing a standardized algebraic interface for the specific stiffness and damping parameters calculated in the subsequent step S4-(3).

[0130] Six-dimensional attitude error vector Analytical Construction: In order for the impedance equation to simultaneously handle position and attitude deviations, translational and rotational deviations with different physical dimensions must be uniformly encapsulated into a six-dimensional vector. The specific construction process follows a uniquely determined mathematical path:

[0131] Translation component extraction: Directly extract the translation deviation vector. As The first three components;

[0132] Rotation component transformation: The rotation deviation matrix output by S4-(1) is processed using the matrix logarithmic mapping algorithm. First, calculate the expression that satisfies the trace relation. rotation angle Subsequently, based on the antisymmetric property of the rotation matrix, the unit vector of the rotation axis is uniquely determined using the following analytical formula. :

[0133]

[0134] in Representative matrix The Middle Line number The column's numerical elements. Rotation angle. With unit vector The product is defined as the rotation error vector. Then the translation deviation vector will be shifted. With rotation error vector Concatenate the data sequentially to generate a complete six-dimensional attitude error vector. This vector physically represents the generalized difference between the current actual pose and the target reference pose in the tangent space, and is set as the virtual equilibrium point of the impedance control system.

[0135] Establishment of the second-order impedance dynamics control law: Based on the above error vector, a second-order differential equation describing the relationship between force and motion is constructed as the core control law of the system:

[0136]

[0137] in, , , These represent the target inertia matrix, target damping matrix, and target stiffness matrix, respectively. It is set to a preset constant diagonal matrix to maintain the basic inertia; and and The state variable matrix defined in this step is to be determined. The specific values ​​of the elements on its diagonal will be explained in detail in the following step S4-(3). The external contact force is fed back in real time by a six-dimensional force sensor at the end of the robotic arm.

[0138] The controller solves the above equations in real time and outputs a virtual corrective force vector. This vector is then converted into a low-level torque drive command and sent directly to the robotic arm's joint controller. The core function of this command is to transform the abstract mechanical constraints calculated by the upper-level algorithm into the actual output torque of the hardware motor, forcing the robotic arm to automatically adjust its compliance according to changes in external contact forces, much like a spring system, while performing pose correction tasks. This mechanism constructs a physical buffer layer in the control loop, fundamentally eliminating the risk of hard collisions that may arise from rigid trajectory planning, thus achieving robust and compliant control.

[0139] Drive the robotic arm to correct pose deviations The process exhibits the characteristics of parameters The determined mass-spring-damped second-order dynamic characteristics. This mechanism ensures that the robotic arm can both eliminate positional errors and adjust... and It gently absorbs the energy it comes into contact with.

[0140] (3) Full-dimensional parameter-driven variable impedance compliant control strategy: This step fully utilizes all the deviation parameters calculated in step S4-(1) and instantiates the impedance model constructed in step S4-(2) into specific robotic arm action commands to achieve high-precision pose correction and compliant grasping tasks:

[0141] Based on vertical deviation Variable impedance compliant contact control: The purpose of this process is to achieve a "soft landing" gripping of fragile wafers while ensuring the stability of the robotic arm's movement. The specific process is as follows:

[0142] Extracting vertical components Then, substitute the values ​​into the exponential decay formula to calculate the target stiffness. Its physical task is to maintain high stiffness when the robotic arm is away from the wafer to ensure trajectory accuracy during high-speed movement; and to reduce stiffness when the robotic arm approaches the wafer surface, making the end effector exhibit spring-like flexibility to absorb collision energy at the moment of physical contact and prevent crushing the wafer; this calculated value The target stiffness matrix will be updated in real time to step S4-(2). The Z-axis component position.

[0143] Using formula Real-time calculation of target damping .in The target inertia matrix defined in step S4-(2) The diagonal component along the Z-axis. This parameter can suppress end-effector oscillations that may be caused by rapid stiffness changes. By matching the critical damping value in real time, it ensures smooth and stable movement of the robotic arm during the transition from "stiff" to "soft," avoiding the risk of secondary collisions due to system overshoot. This calculated value... The target damping matrix in step S4-(2) will be updated in real time. The Z-axis component position.

[0144] (4) The following wafer pose adjustment control is finally achieved:

[0145] Based on horizontal deviation Planar concentricity correction control: The purpose of this step is to eliminate the eccentricity error during wafer gripping. The horizontal deviation component output from step S4-(1) is directly applied. and As inputs to the position compensation command control loop, these two parameters drive the robotic arm to perform precise translational movements in the XY plane, ensuring that the center point of the suction cup is precisely aligned with the predicted geometric center of the wafer in the horizontal dimension, thereby guaranteeing the concentricity of the gripping position.

[0146] Based on rotational deviation Normal alignment attitude control: This process aims to achieve stress-free parallel bonding between the chuck surface and the wafer warp surface. Specifically, a matrix logarithmic mapping algorithm is used to analyze the rotation deviation matrix. First, the rotation angle is uniquely calculated based on the properties of the matrix trace. Then, the unit vector of the rotation axis is constructed using the following analytical formula.

[0147]

[0148] in Representative matrix The Middle Line number The column element values. Finally, the rotation angle calculated above... With unit vector The product is defined as the rotation vector. And as an attitude correction command, it drives the end flange of the robotic arm to rotate around the spatial axis. Rotation angle This forces the normal vector of the chuck plane to actively seek and parallel the predicted local normal vector of the wafer surface, completely eliminating the risk of unilateral force or vacuum leakage caused by angular deviation.

[0149] S5. Experimental Verification and Analysis

[0150] To comprehensively verify the effectiveness and reliability of the deep learning-based multi-sensor collaborative real-time wafer pose correction method proposed in this invention, a high-precision semiconductor wafer transport simulation experimental platform was constructed in this embodiment. The platform is located in a Class 100 cleanroom environment. The hardware system includes a six-axis industrial robotic arm with a repeatability of 0.02 mm, a 5-megapixel global shutter industrial camera mounted on a top support, and a blue line laser profile sensor mounted at the end of the robotic arm. The experimental subjects were standard 12-inch silicon wafers, including flat wafers resulting from different processing methods and non-flat wafers with saddle-shaped warpage. 5000 sets of test data were collected, focusing on evaluating three core indicators: the model's six-DOF pose prediction accuracy, its adaptability to wafers with large warpage, and the contact force characteristics during the compliant correction process.

[0151] 1. Experimental Analysis of Wafer Six-DOF Pose Prediction Accuracy

[0152] This experiment aims to quantitatively evaluate the accuracy of a two-stream attention fusion network in inferring the spatial position and orientation of a wafer. The test set includes wafer samples under different illumination intensities and initial placement positions. The pose parameters predicted by the model are compared one by one with the true values ​​calibrated by a high-precision coordinate measuring machine, and the Euclidean distance error is calculated.

[0153] like Figure 4 The wafer six-DOF pose prediction error distribution diagram is shown. The horizontal axis represents the translation error in the horizontal plane, and the vertical axis represents the comprehensive pose error including vertical height and tilt angle. The color depth maps the confidence score of the model output.

[0154] Experimental data demonstrate that the multi-sensor collaborative method proposed in this invention exhibits extremely high prediction accuracy. Specifically, in terms of translational positioning along the horizontal X and Y axes, the prediction error data points are highly concentrated, with over 95% of the test samples showing an error of less than 0.05 mm. In terms of predicting vertical Z-axis height and spatial tilt angle, the errors are mainly distributed within the range of less than 0.02 mm. Compared to traditional two-dimensional vision methods, this invention effectively supplements the missing information in the depth dimension by introducing a geometric feature extraction branch. The collaborative attention mechanism successfully adaptively weights the edge information in texture features with the curvature information in geometric features, making the model more robust in distinguishing wafer edges from the background, thereby achieving accurate six-degree-of-freedom regression at the sub-millimeter level.

[0155] 2. Comparative Experimental Analysis of Adaptability of Large Warpage Wafers

[0156] This experiment focuses on examining the robustness of the algorithm to the common wafer thermal warpage problem in advanced manufacturing processes. A set of wafer samples that underwent high-temperature annealing were selected, with edge warpage increasing linearly from 0 mm to 3.0 mm. Under the same test conditions, the proposed multi-sensor collaborative method, the traditional 2D machine vision localization method, and the point cloud-based geometric fitting method were run respectively, and the pose prediction success rate under different warpage degrees was recorded. The success rate was determined by a translation error of less than 0.1 mm and an angular error of less than 0.1° between the predicted pose and the true pose.

[0157] like Figure 5 The comparison chart of pose prediction accuracy under different degrees of warping is shown. In the chart, the red solid line represents the method of the present invention, and the blue dashed line represents the traditional two-dimensional vision method.

[0158] Experimental results show that the performance of different methods varies significantly with increasing wafer warpage. The prediction accuracy of traditional 2D vision methods drops drastically after warpage exceeds 1.0 mm, falling below 30% when warpage reaches 3.0 mm. This is because 2D images cannot perceive height changes, and the edge projection distortion caused by warpage severely misleads the center-fitting algorithm. In contrast, the method of this invention exhibits extremely strong stability, maintaining a pose prediction accuracy above 97.2% even under extreme conditions with warpage reaching 3.0 mm.

[0159] The advantage of this method lies in the fact that the PointNet++ point cloud network directly encodes the three-dimensional spatial structure, accurately identifying nonlinear curved surfaces, and eliminating sensor noise interference by combining texture information. This proves that the present invention can effectively overcome the positioning failure problem caused by wafer deformation in automated transmission.

[0160] 3. Experimental Analysis of Compliance Correction Contact Force Characteristics

[0161] This experiment aims to verify the compliance protection effect of the variable impedance control strategy based on predicted pose deviation in actual grasping process. A six-dimensional torque sensor installed at the end flange of the robotic arm recorded the vertical force exerted by the robotic arm's suction cup on the wafer surface in real time at a sampling frequency of 1000 Hz. The experiment compared three control strategies: uncorrected pure position control, correction control based on the traditional proportional-integral-derivative algorithm, and the variable impedance compliant correction control proposed in this invention.

[0162] like Figure 6 The comparison graph of contact force variation during wafer gripping is shown. The horizontal axis represents time in milliseconds, and the vertical axis represents the magnitude of contact force in Newtons (N).

[0163] The experimental waveforms clearly demonstrate the mechanical response characteristics under different strategies. In the uncorrected mode (corresponding to the gray dashed line), a rigid impact peak of up to 18N is generated at the moment of contact, followed by violent oscillations. This impact force can easily cause microcracks in the ultrathin wafer. While the traditional corrected mode (corresponding to the orange solid line) reduces the impact force to around 8N, it still exhibits significant overshoot due to the lack of dynamic adjustment of contact stiffness.

[0164] The green solid line, corresponding to the variable impedance compliant correction mode of this invention, demonstrates the best force control effect. Within the first 50ms of wafer contact, the contact force exhibits a smooth S-shaped upward trend with no overshoot peaks, eventually stabilizing at the preset adsorption force value of 5N. The dynamic force error throughout the process is controlled within 0.2N. This result confirms the effectiveness of the variable impedance control model constructed in step S4. By exponentially reducing the stiffness coefficient of the robotic arm in the Z-axis direction in real time based on the predicted vertical height deviation, the robotic arm exhibits spring-like flexibility at the moment of physical contact. This compensates for pose deviations while physically absorbing contact energy, achieving truly stress-free compliant transmission.

[0165] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0166] While the specific embodiments of the present invention have been described above, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A multi-sensor collaborative real-time wafer pose correction method based on deep learning, characterized in that, Includes the following processes: S1, based on an industrial camera, acquires a global texture image covering the entire surface of the wafer. High-density local depth contour point cloud data is collected by a robotic arm-driven line laser sensor. S2, calculate the hand-eye calibration matrix for spatiotemporal registration; reconstruct the local depth contour point cloud data into continuous original 3D point cloud data based on spatial interpolation and coordinate transformation. ;Will The depth map is generated by back-projecting the depth onto the camera's imaging plane and using a depth buffer to retain only the depth value closest to the camera's optical center. ,Will The three color channels and The single depth channel is stitched together along the channel dimension to obtain a four-channel multimodal input data tensor. ; S3, will and Input a trained six-DOF wafer pose prediction dual-stream deep learning model, and output the predicted translation vector of the wafer center. Quaternions for spatial attitude prediction The model includes a texture feature extraction branch, a geometric feature extraction branch, a collaborative attention fusion module, and a multi-task pose regression head. S4. First, the deviation matrix is ​​solved by inverse operation of the predicted pose and the reference pose matrix of the standard teaching, and a six-dimensional attitude error vector and a second-order impedance dynamic control law are constructed. Then, the Z-axis stiffness and damping parameters are adjusted in real time based on the vertical height deviation to achieve flexible motion correction.

2. The method for real-time wafer pose correction based on deep learning using multi-sensor collaborative technology as described in claim 1, characterized in that: The calculation of the hand-eye calibration matrix is ​​specifically as follows: First, calculate the camera optical center coordinate system. Relative to the robot arm base coordinate system static rigid body transformation matrix , including descriptions of rotational relationships Rotation matrix With description of positional relationship Translation vector ; Secondly, using the tool center point, i.e., the TCP calibration method, the coordinate system of the line laser emission origin is calculated. Relative to the flange center coordinate system Fixed transformation matrix ; The forward kinematics matrix of the flange relative to the base, fed back from the robotic arm controller, is read in real time via industrial Ethernet. Using the chain rule Solve for the coordinate system of the linear laser sensor Relative to the robot arm base coordinate system Real-time transformation matrix .

3. The method for real-time wafer pose correction based on deep learning using multi-sensor collaborative technology as described in claim 1, characterized in that: The texture feature extraction branch is specifically as follows: First, the input has 4 channels and a convolution kernel size of [size missing]. An initial convolutional layer with a stride of 2 is used, followed by a max pooling layer. Downsampling is then applied to reduce the feature map resolution and expand the receptive field, resulting in a shallow feature map containing basic edge information. ; The data passes through four cascaded residual layers, each consisting of multiple stacked bottleneck structures, each containing... , , The three convolutional layers, combined with batch normalization and the ReLU activation function, produce a high-dimensional feature map containing rich semantic information at the output of the fourth residual layer, Layer 4. ; Will The input is fed into a dimensionality-reducing convolutional layer, which compresses the number of channels to a preset dimension. Then, a global average pooling operation is performed to compress the spatial dimension to 1, and the vector is flattened. Finally, the texture feature vector is output. .

4. The method for real-time wafer pose correction based on deep learning using multi-sensor collaborative technology as described in claim 1, characterized in that: The geometric feature extraction branch extracts the original 3D point cloud data reconstructed in the previous steps. Perform farthest point sampling; after sampling, obtain the result containing... Input point set of points Then, through three cascaded set abstraction modules, geometric features from local microstructure to global morphology are extracted step by step: First-level SA layer: As input, the farthest point sampling algorithm is used to select... One central point; With each center point as the center of the sphere, the radius is... Perform a sphere query to construct a local neighborhood; use a coordinate difference calculation method to convert the spatial location information of neighborhood points into local relative coordinates relative to the center point; then input to the channel number. The shared multilayer perceptron is used, and finally, max pooling is used to extract local features with permutation invariance, outputting the first-level feature tensor. ; Second-level SA layer: The feature tensor output by the first-level SA layer The coordinates of the center point and its corresponding center point are used as input, and then the farthest point sampling algorithm is used to select the center point. A central point; with an expanded radius Perform a ball query to construct a neighborhood; using the same coordinate difference calculation method as the first-level SA layer, convert the absolute coordinates of each point in the neighborhood to relative coordinates relative to the current center point; then concatenate the relative coordinates with the features and input them into a channel with a number of channels. The shared multilayer perceptron outputs a second-level deep feature tensor. ; Third-level SA layer: with For input, all Each point is treated as a single global group, and the number of inputs to the channels is [number missing]. The MLP performs a global max pooling operation to aggregate the global geometric feature vectors that characterize the overall warpage of the wafer, denoted as . .

5. The method for real-time wafer pose correction based on deep learning using multi-sensor collaborative technology as described in claim 1, characterized in that: The multi-task pose regression head consists of two parallel fully connected sub-networks, which take the fused context feature vector, which incorporates multimodal information, output by the collaborative attention fusion module as input, and process the position and pose regression tasks respectively. Translation regression branch includes The fully connected layers have a decreasing number of neurons in each layer. ReLU activation functions are added between layers to increase non-linearity. The final output layer contains 3 neurons, corresponding to the X, Y, and Z axis coordinates. Rotational regression branch includes The layer is a fully connected layer, and the final output layer contains 4 neurons, corresponding to the four components of a quaternion; Finally, the translation regression branch outputs the predicted translation vector of the wafer center, denoted as... ,in These represent the predicted X, Y, and Z axis displacements of the wafer center in the base coordinate system, respectively. The output of the rotation branch, after L2 norm normalization, yields a predicted unit quaternion representing the spatial attitude, denoted as... .

6. The method for real-time wafer pose correction based on deep learning using multi-sensor collaborative technology as described in claim 1, characterized in that: The specific process of S4 includes: S41, construct the current pose matrix using the predicted quaternion and translation vector, and calculate the correction transformation matrix containing the translation vector and rotation matrix by inverse operation with the standard teaching reference pose matrix; S42 encapsulates translational deviations with different physical dimensions and rotational deviations analyzed by matrix logarithmic mapping algorithm into a six-dimensional attitude error vector, and establishes a second-order impedance dynamics equation describing the relationship between force and motion as the core control law. S43, based on the predicted vertical axis height deviation, dynamically adjusts the target stiffness and damping coefficient of the robotic arm in the Z-axis direction in real time, so that the robotic arm exhibits low stiffness and flexible characteristics at the moment of contact in order to achieve posture correction and compliant grasping tasks. S44 utilizes the horizontal deviation component to drive the robotic arm to perform concentric correction in the XY plane, while simultaneously adjusting the suction cup normal according to the rotational deviation vector to parallel fit the wafer warp surface, eliminating eccentricity error and the risk of unilateral force.

7. The method for real-time wafer pose correction based on deep learning using multi-sensor collaborative methods as described in claim 6, characterized in that: The specific process of S41 is as follows: First, obtain the reference pose matrix for standard teaching. Then, using the output prediction results, including the prediction unit quaternion, denoted as... With the predicted translation vector Construct the current pose matrix The construction process is as follows: The predictor unit quaternion is denoted as ,in For the real part, It is the imaginary part; Based on the transformation relationship from quaternions to rotation matrices, calculate Orthogonal rotation matrix Perform the splicing operation along the column direction, setting the left region as a rotation matrix. And the region on the right is the predicted translation vector. The dimension is obtained as The middle matrix; then fill the bottom of the matrix with a row of values. The row vectors are ultimately used to generate the standard... Homogeneous transformation matrix ; Calculate the correction transformation matrix : Transform the homogeneous matrix Right-multiply the reference pose matrix The inverse matrix, then from the correction transformation matrix Extract the translation deviation vector With rotational deviation matrix The two parameters are directly used to generate the inverse kinematics position compensation command for the robotic arm.

8. The method for real-time wafer pose correction based on deep learning using multi-sensor collaborative technology as described in claim 7, characterized in that: The specific process of S42 includes: Six-dimensional attitude error vector The analytical construction unifies translational and rotational deviations with different physical dimensions into a six-dimensional vector. : 1) Translation component extraction: Directly extract the translation deviation vector. As The first three components; 2) Rotation component transformation: The rotation deviation matrix is ​​processed using a matrix logarithmic mapping algorithm. First, calculate the trace relation. rotation angle Then, based on the antisymmetric property of the rotation matrix, the unit vector of the rotation axis is uniquely determined using the formula. : in Representative matrix The Middle Line number The column's numerical elements; rotate by angle With unit vector The product is defined as the rotation error vector. Then the translation deviation vector will be shifted. With rotation error vector Concatenate the data sequentially to generate a complete six-dimensional attitude error vector. ; Establishment of the second-order impedance dynamics control law: Based on the above error vector, a second-order differential equation describing the relationship between force and motion is constructed as the core control law of the system: in, , , These represent the target inertia matrix, target damping matrix, and target stiffness matrix, respectively. Set as a preset constant diagonal matrix; and Defined as a matrix of state variables to be determined; The external contact force is fed back in real time by a six-dimensional force sensor at the end of the robotic arm; This is a virtual corrective force vector.

9. The method for real-time wafer pose correction based on deep learning using multi-sensor collaborative methods as described in claim 8, characterized in that: The specific process of S43 includes: Based on vertical deviation Variable impedance compliant contact control: Extraction of vertical component Then, substitute the values ​​into the exponential decay formula to calculate the target stiffness. When the robotic arm moves away from the wafer, it maintains high stiffness to ensure trajectory accuracy during high-speed movement; when the robotic arm approaches the wafer surface, it reduces stiffness to make the end effector exhibit spring-like flexibility; calculated values Real-time update to the target stiffness matrix in step S42 The Z-axis component position; Using formula Real-time calculation of target damping ,in For the target inertia matrix Diagonal element components along the Z-axis; calculated values The target damping matrix in step S42 will be updated in real time. The Z-axis component position.

10. The method for real-time wafer pose correction based on deep learning using multi-sensor collaborative methods as described in claim 9, characterized in that: The horizontal deviation component output in step S41 and As a position compensation command input control loop, it drives the robotic arm to perform precise translational motion in the XY plane, so that the center point of the suction cup is strictly close to the predicted geometric center of the wafer in the horizontal dimension. Based on rotational deviation Normal alignment attitude control: The rotation deviation matrix is ​​analyzed using a matrix logarithmic mapping algorithm. First, the rotation angle is uniquely calculated based on the properties of the matrix trace. Then, the unit vector of the rotation axis is constructed using the following analytical formula. in Representative matrix The Middle Line number The column element values; the final calculated rotation angle With unit vector The product is defined as the rotation vector. And as an attitude correction command, it drives the end flange of the robotic arm to rotate around the spatial axis. Rotation angle The normal vector of the forced suction cup plane actively seeks out and parallels the predicted local normal vector of the wafer surface.

Images