Double-camera precision positioning method and device in bonding system, equipment and storage medium

Abstract

The invention relates to a double-camera precision positioning method and device in a bonding system, equipment and a storage medium, and the method comprises the steps: collecting and continuously monitoring a bonding tool and a bonding interface through a first camera and a second camera, and obtaining a bonding image; carrying out image processing on the bonding image and extracting features, and obtaining the position and posture of a bonding tool and geometric features of a bonding area; obtaining a current moving path of the bonding tool through a pre-constructed bonding path planning model, and outputting an instruction for controlling the bonding tool to carry out chip bonding according to the current moving path; and the first camera and the second camera are utilized to monitor the bonding process of the bonding tool executing the bonding operation according to the current moving path in real time, so that the purpose of performing high-precision and high-efficiency positioning and completing chip bonding through the double cameras in a bonding system in a complex temperature environment is achieved.

Description

Technical Field

[0001] The present invention relates to the field of chip packaging technology, and in particular to a dual-camera precision positioning method, device, equipment and storage medium in a bonding system. Background Art

[0002] In the field of semiconductor manufacturing, chip bonding technology plays a vital role. With the rapid development of technology, especially the rise of advanced packaging technologies such as 3D packaging and system-in-package (SiP), more stringent requirements are put forward for precise positioning and efficient control in the chip bonding process. Although the traditional chip bonding process has incorporated machine vision technology to a certain extent to improve positioning accuracy, most of them are still limited to a single camera perspective, and are unable to cope with complex changes such as thermal expansion of materials and image blur caused by temperature. Temperature changes not only cause small but crucial changes in the physical dimensions of the bonding tool and the bonding interface, but also affect the image acquisition quality, which in turn affects the subsequent image processing and feature extraction steps, and may eventually lead to a shift in the bonding position or a decrease in bonding quality.

[0003] Although some existing technologies attempt to address this problem through complex calibration procedures or fixed temperature compensation strategies, these methods often lack flexibility and cannot adapt to the rapid temperature changes in the actual working environment in real time. In addition, the compensation effects on bonding components of different materials and structures are also uneven. How to maintain high-precision positioning capabilities in chip bonding technology during temperature changes has become a technical problem that needs to be urgently solved in the current semiconductor manufacturing field. Summary of the invention

[0004] The main purpose of the present invention is to provide a dual-camera precision positioning method, device, equipment and storage medium in a bonding system, so as to achieve high-precision and high-efficiency positioning through dual cameras in a bonding system with a complex temperature environment to complete the purpose of chip bonding.

[0005] To achieve the above object, the present invention provides a dual-camera precision positioning method in a bonding system, comprising the following steps:

[0006] The bonding tool and the bonding interface are respectively captured and continuously monitored by the first camera and the second camera to obtain a bonding image, wherein the bonding image includes a bonding tool image and a bonding interface image; wherein the first camera focuses on the bonding tool and the second camera focuses on the bonding interface, and the two cameras synchronously capture image information corresponding to the same time based on a timestamp;

[0007] Performing image processing on the bonding image and extracting features to obtain the bonding tool position, posture and geometric features of the bonding area;

[0008] Based on the position and posture of the bonding tool and the geometric features of the bonding area, the current moving path of the bonding tool is obtained through a pre-constructed bonding path planning model, and an instruction for controlling the bonding tool to perform chip bonding according to the current moving path is output. The bonding path planning model is based on data of various bonding tool positions, postures and geometric features of the bonding area in various historical bonding operations, and is trained using a machine learning algorithm. The input is data related to the position and posture of the bonding tool and the geometric features of the bonding area, and the output is the moving path, moving speed and moving force of the bonding tool when performing chip bonding.

[0009] The chip bonding instruction is executed by the bonding tool, and the first camera and the second camera are used to monitor in real time the bonding process of the bonding tool performing the bonding operation according to the current moving path.

[0010] Furthermore, the step of acquiring and continuously monitoring the bonding tool and the bonding interface by the first camera and the second camera respectively to obtain the bonding image includes:

[0011] Calibrate the first camera and the second camera, determine the internal parameters of the first camera and the second camera and the relative position relationship between the first camera and the second camera, and perform image registration and calibration;

[0012] The first camera collects the image of the bonding tool in real time and continuously monitors it to obtain the bonding tool image, wherein the bonding tool image at least includes the shape and size of the key parts when the bonding action is performed;

[0013] adjusting the focal length and depth of field of the second camera according to the size and depth of the bonding area;

[0014] While the first camera is capturing images of the bonding tool, the second camera is used to capture and continuously monitor the bonding interface between the chip to be synthesized and the substrate to obtain an image of the bonding area.

[0015] Furthermore, the step of performing image processing on the bonding image and extracting features by using an image denoising and enhancement algorithm to obtain the bonding tool position, posture and geometric features of the bonding area includes:

[0016] Performing image preprocessing operations on the acquired bonded image;

[0017] Performing feature extraction on the preprocessed bonding image through dynamic range adjustment and a pre-trained temperature compensation model to respectively determine feature points of the bonding tool and the bonding area;

[0018] Based on the result of feature extraction, the position and posture of the bonding tool and the geometric features of the bonding area are determined.

[0019] Furthermore, the step of extracting features from the preprocessed bonding image by dynamic range adjustment and a pre-trained temperature compensation model to respectively determine feature points of the bonding tool and the bonding area includes:

[0020] Based on the temperature in the current bonding environment acquired in real time by the temperature sensor, the dynamic range of the bonding image is adjusted by using a set image processing algorithm to enhance the image details, and the thermal expansion compensation of the bonding image is performed by using a pre-trained temperature compensation model;

[0021] Based on the dynamic range adjustment and thermal expansion compensation results of the bonding image, feature extraction is performed on the bonding image to obtain feature points of the bonding tool and the bonding area respectively, wherein the feature extraction includes edge detection, shape recognition and corner point detection;

[0022] The first camera and the second camera continuously track the characteristic points of the bonding tool and the characteristic points of the bonding area to monitor the movement trajectory and posture changes of the bonding tool and the morphological changes of the bonding interface.

[0023] Further, the step of acquiring the current moving path of the bonding tool through a pre-built bonding path planning model based on the bonding tool position, posture and geometric features of the bonding area, and outputting instructions for controlling the bonding tool to perform chip bonding according to the current moving path includes:

[0024] Integrate and format the data of the bonding tool position and posture and the data of the geometric features of the bonding area;

[0025] The temperature sensor is used to obtain the current temperature in the bonding environment in real time and perform error compensation;

[0026] Inputting the integrated and formatted data and the current temperature into a pre-trained bonding path planning model to obtain a required moving position and moving posture of the bonding tool relative to the bonding area;

[0027] Determining a current moving path required for the bonding tool to perform chip bonding based on an output result of the bonding path planning model;

[0028] Based on the calculated current moving path, converting executable chip bonding instructions;

[0029] The chip bonding instruction is sent to the execution component of the bonding tool.

[0030] Furthermore, the training step of the bonding path planning model includes:

[0031] Collect bonding image data of bonding tools and bonding interfaces in various positions, postures and temperature environments, and collect parameter data of speed and force actually recorded during each bonding operation, and pre-process the collected data. The sources of the collected data include operation records of various types of chip bonding equipment in actual production and data collected from specially designed bonding experiments;

[0032] Extracting key features for characterizing relevant morphological information of a bonding tool and a bonding interface from the preprocessed bonding image data, wherein the key features include edges, shapes, and corner points;

[0033] The extracted key features and the corresponding parameter data of speed and force are annotated, wherein the annotated content includes the position, posture, geometric features of the bonding area, and the corresponding speed value and force value of the bonding tool, and a bonding image training data set is constructed, wherein the bonding image training data set includes an image of multi-dimensional features and the corresponding position, posture, geometric features, and speed and force annotations;

[0034] The bonding image training data set is input into a deep neural network algorithm model, and a bonding path planning model is obtained through training and optimization, which calculates the movement path, speed and force of the bonding tool when performing chip bonding based on the feature points of the bonding tool and the bonding area.

[0035] Furthermore, the step of executing the chip bonding instruction by the bonding tool and using the first camera and the second camera to monitor in real time the bonding process of the bonding tool performing the bonding operation according to the current moving path includes:

[0036] Based on the chip bonding instruction, driving the bonding tool to move along the current moving path to perform a chip bonding operation;

[0037] During the process of the bonding tool moving and performing the bonding operation, continuously monitoring the position, posture and state of the bonding area of ​​the bonding tool in real time by the first camera and the second camera;

[0038] Based on the current moving path and the real-time monitoring data, calculating the position and posture deviation through the bonding path planning model;

[0039] Establishing a correspondence between the expected feature point coordinates of the bonding tool on the current moving path output by the bonding path planning model and the actual feature point coordinates monitored in real time by the first camera, and determining the position deviation between the bonding tool and the expected feature point coordinates by using a coordinate transformation method;

[0040] Determine the posture deviation of the bonding tool from the expected posture based on the expected posture of the bonding tool at each node obtained in the moving path output by the bonding path planning model and the current actual posture of the prime bonding tool calculated by the posture estimation algorithm by monitoring the bonding tool in real time using the first camera;

[0041] Based on the position deviation and the posture deviation results, the moving path or bonding operation of the bonding tool is adjusted through a bonding path planning model.

[0042] The present invention also provides a dual-camera precision positioning device in a bonding system, comprising:

[0043] A data acquisition module, used to acquire and continuously monitor the bonding tool and the bonding interface through the first camera and the second camera respectively, to obtain a bonding image;

[0044] A data processing module, used to perform image processing on the bonding image and extract features to obtain the bonding tool position, posture and geometric features of the bonding area;

[0045] An information output module, used to obtain the current moving path of the bonding tool through a pre-built bonding path planning model based on the position and posture of the bonding tool and the geometric features of the bonding area, and output an instruction to control the bonding tool to perform chip bonding according to the current moving path;

[0046] The information monitoring module is used to execute the chip bonding instruction through the bonding tool, and use the first camera and the second camera to monitor in real time the bonding process of the bonding tool performing the bonding operation according to the current moving path.

[0047] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the dual-camera precision positioning method in the above-mentioned bonding system are implemented.

[0048] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the dual-camera precision positioning method in the above-mentioned bonding system are implemented.

[0049] The dual-camera precise positioning method, device, equipment and storage medium in the bonding system provided by the present invention have the following beneficial effects:

[0050] The dual-camera (first camera and second camera) system synchronously collects image information corresponding to the same moment based on the timestamp, thereby ensuring the temporal consistency of the images of the bonding tool and the bonding interface, and further improving the positioning accuracy; using the pre-built bonding path planning model, the optimal movement path can be automatically calculated and planned according to the bonding tool position, posture and geometric features of the bonding area. This intelligent control method not only improves the bonding efficiency, but also reduces the risk of errors caused by human operation; by accurately controlling the movement path, movement speed and movement force of the bonding tool, high-quality chip bonding can be achieved, which helps to reduce defects and failures in the bonding process and improve product reliability and service life; real-time monitoring of the status of the bonding tool and the bonding interface, and adjusting the movement path or bonding operation according to the real-time monitoring data can further ensure the stability and consistency of the bonding quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a schematic flow chart of a dual-camera precision positioning method in a bonding system in one embodiment of the present invention;

[0052] Figure 2 is a structural block diagram of a dual-camera precision positioning device in a bonding system in one embodiment of the present invention;

[0053] Figure 3 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0054] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0056] Reference Figure 1 , which is a flow chart of a dual-camera precision positioning method in a bonding system proposed by the present invention, comprising the following steps:

[0057] S1, respectively collecting and continuously monitoring a bonding tool and a bonding interface through a first camera and a second camera to obtain a bonding image, wherein the bonding image includes a bonding tool image and a bonding interface image; wherein the first camera focuses on the bonding tool, and the second camera focuses on the bonding interface, and the two cameras synchronously collect image information corresponding to the same time based on a timestamp;

[0058] S2, performing image processing on the bonding image and extracting features to obtain the bonding tool position, posture and geometric features of the bonding area;

[0059] S3, based on the position, posture and geometric features of the bonding tool and the bonding area, the current moving path of the bonding tool is obtained through a pre-built bonding path planning model, and an instruction for controlling the bonding tool to perform chip bonding according to the current moving path is output, wherein the bonding path planning model is based on data of various bonding tool positions, postures and bonding area geometric features in various historical bonding operations, and is trained using a machine learning algorithm, with inputs being data related to the bonding tool position, posture and bonding area geometric features, and outputs being the moving path, moving speed and moving force of the bonding tool when performing chip bonding;

[0060] S4, executing the chip bonding instruction through the bonding tool, and using the first camera and the second camera to monitor in real time the bonding process of the bonding tool performing the bonding operation according to the current moving path.

[0061] As described in step S1 above, the bonding tool and the bonding interface are respectively captured and continuously monitored by the first camera and the second camera to obtain a bonding image. Among them, the first camera is used to monitor the bonding tool, and the bonding tool refers to a device or apparatus used to connect two or more components (such as chips and substrates) together in the fields of semiconductor manufacturing and microelectronic packaging, and the second camera is used to monitor the bonding interface, which refers to the contact area between the two components under the action of the bonding tool. The image information of the bonding tool and the bonding interface is captured and recorded in real time and uninterruptedly by two cameras, and the image information of the two cameras is ensured to be captured at the same time based on the timestamp.

[0062] As described in the above-mentioned step S2, the bonding image is processed and features are extracted by image denoising and enhancement algorithms to obtain the geometric features of bonding tool position, posture and bonding area. The image that is affected by various noises (such as environmental noise, camera noise, etc.) during data acquisition, resulting in quality degradation, is subjected to denoising, and these noises are reduced or eliminated to improve the clarity of the image, and image enhancement is performed to improve the visual effect of the image, so that the image is more suitable for subsequent image analysis and feature extraction, including adjusting the properties such as contrast, brightness, sharpness of the image, so that the features of the bonding tool and the bonding interface are more obvious. The extracted image features can be the edge, shape, texture, etc. in the image that can represent the key information in the image. By an image processing algorithm, the specific position of the bonding tool in the image is determined by detecting the edge or specific mark of the bonding tool, and the posture information of the bonding tool is extracted at the same time, such as rotation angle, tilt degree, etc., for understanding the motion state of the bonding tool in the bonding process, extracting the geometric features of the bonding interface or bonding area, such as area, perimeter, shape, etc., these features can reflect the state and quality of the bonding interface.

[0063] As described in step S3 above, based on the bonding tool position, posture and geometric features of the bonding area, the current moving path of the bonding tool is obtained through a pre-built bonding path planning model, and an instruction for controlling the bonding tool to perform chip bonding according to the current moving path is output. When new bonding tool position, posture and bonding area geometric feature data are input, the model will output an optimal moving path, which is learned by the model based on historical data, and is intended to ensure that the bonding tool can accurately and efficiently move to the correct position and perform chip bonding at an appropriate speed and force. When new bonding tool position, posture and bonding area geometric feature data are input, the model will output an optimal current moving path. This path is learned by the model based on historical data, and is intended to ensure that the bonding tool can accurately and efficiently move to the correct position and perform chip bonding at an appropriate speed and force. In addition, the model output also includes moving speed and moving force, thereby achieving efficient and accurate chip bonding.

[0064] As described in the above step S4, the chip bonding instruction is executed by the bonding tool, and the first camera and the second camera are used to monitor the bonding process of the bonding tool according to the current moving path in real time. According to the chip bonding instruction calculated and converted in step S3, the bonding tool is driven to start moving along the predetermined current moving path, thereby implementing the chip bonding operation. During the movement of the bonding tool and the actual execution of the bonding operation, the first camera and the second camera are continuously used for real-time monitoring. The first camera focuses on the bonding tool and captures the position, posture, etc. of the bonding tool in real time, such as whether the key parts of the bonding tool are in the expected spatial position, whether its angle posture meets the requirements, etc.; the second camera focuses on the bonding interface, and grasps the state of the bonding area in real time, such as whether there are abnormal changes in the bonding interface between the chip to be synthesized and the substrate, etc., and the dual cameras are used to jointly ensure the full monitoring of the key elements of the entire bonding process from different perspectives. Based on the pre-planned moving path and the data obtained from real-time monitoring, the bonding path planning model is used to calculate the position and posture deviations. According to the calculated position deviation and posture deviation results, the bonding path planning model is used to make corresponding adjustments to the moving path of the bonding tool or the bonding operation.

[0065] In one embodiment, the step S1 of acquiring and continuously monitoring the bonding tool and the bonding interface by the first camera and the second camera respectively to obtain the bonding image includes:

[0066] S11, calibrating the first camera and the second camera, determining internal parameters of the first camera and the second camera and a relative position relationship between the first camera and the second camera, and performing image registration and calibration;

[0067] S12, acquiring an image of the bonding tool in real time through the first camera and continuously monitoring the image to obtain an image of the bonding tool, wherein the image of the bonding tool at least includes a shape and a size of a key part when performing a bonding action;

[0068] S13, adjusting the focal length and depth of field of the second camera according to the size and depth of the bonding area;

[0069] S14, while the first camera is capturing images of the bonding tool, the second camera is used to capture and continuously monitor the bonding interface between the chip to be synthesized and the substrate to obtain a bonding area image.

[0070] As described in step S11 above, by taking images of standard objects of known size and shape, the internal parameters of the camera (such as focal length, principal point position, distortion coefficient, etc.) are calculated using these images, and the relative position relationship between the two cameras is determined using a stereo calibration method, including a rotation matrix and a translation vector. Based on the above parameters, the images taken by the two cameras are registered and calibrated to ensure that the two cameras are aligned in the same coordinate system and can accurately reflect the three-dimensional information of the real world. As described in steps S12-S14 above, the first camera is set to focus on the bonding tool and collect its image in real time. Through the image processing algorithm, the shape and size information of the key parts of the bonding tool during bonding (such as the bonding tool head) are extracted from the collected image. The second camera is set to focus on the bonding interface and collect its image in real time. According to the size of the bonding area, the focal length of the second camera is adjusted to obtain the best image clarity, and according to the depth of the bonding interface, the depth of field of the camera is adjusted to ensure that the entire bonding area is within the range of clear imaging. Through the image processing algorithm, the state of the bonding interface (such as alignment, gap size, etc.) is continuously monitored, and relevant geometric feature information is extracted. The above steps provide accurate and reliable image data for subsequent image processing, feature extraction and bonding path planning through camera calibration, focus and depth of field adjustment, and real-time image acquisition and monitoring, and together form the basis of the dual-camera precision positioning method in the bonding system.

[0071] In one embodiment, the step S2 of performing image processing on the bonding image and extracting features by using an image denoising and enhancement algorithm to obtain the bonding tool position, posture and geometric features of the bonding area includes:

[0072] S21, performing image preprocessing operation on the acquired bonded image;

[0073] S22, performing feature extraction on the preprocessed bonding image through dynamic range adjustment and a pre-trained temperature compensation model, and determining feature points of the bonding tool and the bonding area respectively;

[0074] S23, determining the position and posture of the bonding tool and the geometric features of the bonding area based on the result of feature extraction.

[0075] As described in step S21 above, the preprocessing operation includes but is not limited to removing common noise types such as salt and pepper noise and Gaussian noise in the image, and adjusting the contrast and brightness of the image by methods such as histogram equalization to make the key information in the image clearer and more recognizable. For example, the collected bonding tool image has some randomly distributed bright spots (noise), and these bright spots are removed by a suitable denoising algorithm to make the true shape and contour of the bonding tool more clearly displayed. As described in step S22 above, the bonding image will cause large differences in brightness and contrast in different areas of the image due to factors such as the setting of the acquisition device and the ambient temperature, and the brightness of some areas is uneven, and some details are difficult to see clearly. The brightness and contrast of the image are adjusted to a suitable range by adjusting the dynamic range, so that all details in the image can be presented as clearly as possible, which is convenient for subsequent accurate extraction of feature points. For example, some subtle structures of the bonding tool are not clearly visible in the original image because they are in a darker area. By adjusting the dynamic range, the brightness of the area is increased while maintaining the reasonable display of other areas, so that these subtle structures can be shown, so as to determine their feature points later. Temperature changes can cause slight changes in the physical dimensions of the bonding tool and the bonding interface, and can also affect the image acquisition quality. The pre-trained temperature compensation model is used to compensate the bonding image according to the real-time temperature information, so that the image can accurately reflect the actual state of the bonding tool and the bonding area, without being disturbed by factors such as thermal expansion caused by temperature changes. For example, in a high temperature environment, the bonding tool may expand slightly, resulting in the shape and size of the bonding tool in the collected image being different from that at room temperature. The temperature compensation model can correct this change. Among them, the pre-trained temperature compensation model is trained based on various image data related to the bonding process with temperature markers and corresponding correct feature points. During the chip bonding process, temperature changes will affect the physical dimensions of the bonding tool and the bonding interface, thereby affecting the image acquisition quality. The pre-trained temperature compensation model is to compensate for such image changes caused by temperature changes, so that the extracted feature points can accurately reflect the actual bonding tool and bonding area, rather than false feature points that are biased by temperature.

[0076] As described in step S23 above, the feature points of the bonding tool determined in step S22 are combined with the image coordinate system and some known reference points (such as the optical center of the camera, etc.), and the position coordinates of the bonding tool in the image are determined by the method of geometric calculation, and then its position in the actual space is inferred (combined with the calibration parameters of the camera, etc. for further spatial transformation). According to the relative position relationship between the multiple feature points of the bonding tool and their changes in the continuous images, the posture of the bonding tool (such as horizontal placement, tilting at a certain angle, etc.) is analyzed. For example, if the ordinate of a corner point of the bonding tool gradually decreases in several consecutive frames of images, while the abscissa remains basically unchanged, it means that the bonding tool is gradually tilting downward. Based on the feature points of the bonding area, the geometric features of the bonding area are determined by analyzing the geometric parameters such as the shape, area, and perimeter surrounded by these feature points. For example, the bonding area is a rectangle, and its length, width, area and other geometric properties can be accurately calculated through the feature points. These geometric features are used for subsequent planning of the movement path of the bonding tool and other operations.

[0077] In one embodiment, the step S22 of extracting features from the preprocessed bonding image by dynamic range adjustment and a pretrained temperature compensation model to respectively determine feature points of the bonding tool and the bonding area includes:

[0078] S221, based on the temperature in the current bonding environment acquired in real time by the temperature sensor, using a set image processing algorithm to adjust the dynamic range of the bonding image to enhance image details, and using a pre-trained temperature compensation model to perform thermal expansion compensation on the bonding image;

[0079] S222, based on the dynamic range adjustment and thermal expansion compensation results of the bonding image, extracting features of the bonding image to obtain feature points of the bonding tool and the bonding area respectively, wherein the feature extraction includes edge detection, shape recognition, and corner point detection;

[0080] S223, continuously tracking the bonding tool feature points and the bonding area feature points by the first camera and the second camera, and monitoring the movement trajectory and posture changes of the bonding tool and the morphology changes of the bonding interface.

[0081] As described in step S221 above, based on the temperature in the current bonding environment acquired in real time by the temperature sensor, the dynamic range of the bonding image is adjusted using a set image processing algorithm (such as histogram stretching, logarithmic transformation, etc.) to expand the brightness range of the image, make the dark and bright details of the image clearer, and provide high-quality image input for subsequent feature extraction. At the same time, the pre-trained temperature compensation model is used to compensate for thermal expansion of the bonding image. Since temperature changes will cause thermal expansion and contraction of materials, thereby affecting the size and shape of objects in the image, the pre-trained temperature compensation model predicts and compensates for this thermal expansion effect according to the current temperature to ensure that the size and shape of the object in the image are consistent with the actual. As described in step S222 above, an edge detection algorithm (such as Canny edge detection) is used to identify the edge information of the bonding tool and the bonding area in the image to determine the contour and shape; based on the results of edge detection, the shape features of the object in the image are further identified through template matching and shape context; and a corner detection algorithm (such as Harris corner detection) is used to determine the corner features in the image. Corner points are important local features in the image, i.e., feature points. Through the above-mentioned feature extraction process, the feature points of the bonding tool and the bonding area are obtained respectively for subsequent position, posture and morphology monitoring. As described in the above step S223, the image data captured synchronously and in real time by two cameras are used in combination with a feature point matching algorithm (such as the least squares method) to continuously track the feature points of the bonding tool and the bonding area. By tracking the motion trajectory of the feature points, the movement trajectory and posture changes of the bonding tool are monitored in real time to ensure that the bonding operation is performed according to the predetermined path and posture. At the same time, by monitoring the morphological changes (such as area, perimeter, shape, etc.) of the feature points in the bonding area, potential bonding problems (such as dislocation, fracture, etc.) are discovered and handled in a timely manner.

[0082] In one embodiment, based on the position and posture of the bonding tool and the geometric features of the bonding area, the step S3 of acquiring the current moving path of the bonding tool through a pre-built bonding path planning model, and outputting an instruction for controlling the bonding tool to perform chip bonding according to the current moving path comprises:

[0083] S31, integrating and formatting the data of the bonding tool position and posture and the data of the geometric features of the bonding area;

[0084] S32, obtaining the current temperature in the bonding environment in real time through a temperature sensor and performing error compensation;

[0085] S33, inputting the integrated and formatted data and the current temperature into a pre-trained bonding path planning model to obtain a required moving position and moving posture of the bonding tool relative to the bonding area;

[0086] S34, determining a current moving path required for the bonding tool to perform chip bonding based on an output result of the bonding path planning model;

[0087] S35, converting executable chip bonding instructions based on the calculated current moving path;

[0088] S36, sending the chip bonding instruction to the execution component of the bonding tool.

[0089] As described in step S31 above, the position (such as X, Y, Z coordinates) and posture (such as rotation angle, inclination, etc.) data of the bonding tool, as well as the geometric features of the bonding area, such as shape, size, position, etc., are obtained by the camera sensor (first and second cameras), and these data are integrated together and converted into the input format required by the model, such as a specific data structure. As described in step S32 above, the temperature in the bonding environment is monitored in real time using a temperature sensor, and the position or posture error caused by temperature change is calculated based on the temperature data using a pre-established temperature compensation model, and the error is compensated. As described in steps S33-S36 above, the integrated data (including position, posture and geometric features) and the current temperature are used as inputs to the pre-trained bonding path planning model to calculate the required moving path and posture of the bonding tool from the current position to the target position. The moving path information is extracted from the output of the model, including path points, speed, acceleration, direction, etc. Based on this information, a complete moving path of the bonding tool from the starting position to the target position is constructed. Convert the moving path information into control instructions, such as the number of steps of the stepper motor, the position setting of the servo system, etc., to ensure that the format of the instructions is compatible with the interface of the bonding tool execution component. Send the instructions to the execution component of the bonding tool through the communication interface. The execution component controls the bonding tool to perform precise movement and bonding operations based on the received instructions. The above steps achieve precise planning and control of the bonding tool movement path by integrating data, performing temperature error compensation, using the bonding path planning model for calculation, determining the moving path, converting instructions, and sending instructions, thereby ensuring the accuracy and reliability of chip bonding.

[0090] In one embodiment, the step of training the bonding path planning model includes:

[0091] Collect bonding image data of bonding tools and bonding interfaces in various positions, postures and temperature environments, and collect parameter data of speed and force actually recorded during each bonding operation, and pre-process the collected data. The sources of the collected data include operation records of various types of chip bonding equipment in actual production and data collected from specially designed bonding experiments;

[0092] According to the physical model and kinematic principle of the bonding system in this embodiment, a large amount of combined data of the bonding tool position, posture and bonding area geometric features are generated by computer simulation software. These simulation data cover various possible bonding scenarios, including different chip sizes and shapes, the starting position and posture of the bonding tool, etc. For example, in different three-dimensional space coordinates, the bonding tool approaches the bonding area with different geometric shapes (such as rectangle, circle, etc.) at different inclination angles and rotation directions. For each simulated combined data, the known ideal bonding path (predetermined based on physical rules and process requirements) is used as the corresponding label data, that is, the path information of how the bonding tool should move to accurately complete the bonding, and the above data is organized into a bonding system data set.

[0093] Extracting key features for characterizing relevant morphological information of a bonding tool and a bonding interface from the preprocessed bonding image data, wherein the key features include edges, shapes, and corner points;

[0094] The extracted key features and the corresponding parameter data of speed and force are annotated, wherein the annotated content includes the position, posture, geometric features of the bonding area, and the corresponding speed value and force value of the bonding tool, and a bonding image training data set is constructed, wherein the bonding image training data set includes an image of multi-dimensional features and the corresponding position, posture, geometric features, and speed and force annotations;

[0095] The deep neural network algorithm model is trained using the bonding image training data set and the bonding system data set, and a bonding path planning model is obtained through training and optimization, which calculates the movement path, speed and force of the bonding tool when performing chip bonding based on the feature points of the bonding tool and the bonding area.

[0096] In one embodiment, the step S4 of executing the chip bonding instruction by the bonding tool and using the first camera and the second camera to monitor in real time the bonding process of the bonding tool performing the bonding operation according to the current moving path includes:

[0097] S41, based on the chip bonding instruction, driving the bonding tool to move along the current moving path to perform a chip bonding operation;

[0098] S42, during the process of the bonding tool moving and performing the bonding operation, continuously monitoring the position, posture and state of the bonding area of ​​the bonding tool in real time by using the first camera and the second camera;

[0099] S43, calculating the position and posture deviation through a bonding path planning model based on the current moving path and the real-time monitoring data;

[0100] S44, establishing a correspondence between the expected feature point coordinates of the bonding tool on the current moving path output by the bonding path planning model and the actual feature point coordinates monitored in real time by the first camera, and determining the position deviation between the bonding tool and the expected feature point coordinates by using a coordinate transformation method;

[0101] S45, determining a posture deviation of the bonding tool from the expected posture based on the expected posture of the bonding tool at each node obtained in the moving path output by the bonding path planning model and the current actual posture of the prime bonding tool calculated by the posture estimation algorithm by monitoring the bonding tool in real time using the first camera;

[0102] S46, based on the position deviation and the posture deviation results, adjusting the moving path or bonding operation of the bonding tool through a bonding path planning model.

[0103] As described in the above step S41, the control system receives the chip bonding instruction, and the control system controls the bonding tool to move along the current moving path through the driving mechanism (such as a stepper motor, a servo motor, etc.) according to the instruction. As described in the above steps S42-S43, the bonding tool and the bonding area are respectively imaged by the first camera and the second camera, and the collected image data is transmitted to the image processing system in real time to extract the position and posture information of the bonding tool and the state information of the bonding area. The real-time monitored bonding tool position and posture data are compared with the expected data output by the bonding path planning model, and the deviation value of the position and posture is calculated by the algorithm. As described in the above steps S44-S46, according to the current moving path output by the bonding path planning model, a series of key feature points and their coordinates are preset, the actual feature points on the bonding tool are monitored in real time by the first camera, and their coordinates are extracted. The coordinates of the actual feature points are compared with the expected feature point coordinates by the coordinate transformation method, and the position deviation is calculated. Based on the expected posture of each node in the current moving path output by the bonding path planning model, the actual posture of the bonding tool is monitored in real time with the first camera, and the current actual posture is calculated using the posture estimation algorithm. The expected posture and the actual posture are compared to calculate the posture deviation of the bonding tool. The calculated position deviation and posture deviation are input into the bonding path planning model. The model calculates the adjusted moving path or bonding operation parameters according to the deviation value. The control system adjusts the moving path or bonding operation of the bonding tool according to the adjusted parameters to reduce the deviation and improve the bonding quality. The above steps realize the precise control and optimization of the bonding process by real-time monitoring of the bonding process, calculating the position and posture deviation, determining the deviation value, and adjusting the moving path or bonding operation using the bonding path planning model, thereby improving the accuracy and reliability of chip bonding.

[0104] Reference Figure 2, is a structural block diagram of a dual-camera precision positioning device in a bonding system in one embodiment of the present invention, comprising:

[0105] A data acquisition module, used to acquire and continuously monitor the bonding tool and the bonding interface through the first camera and the second camera respectively, to obtain a bonding image;

[0106] A data processing module, used to perform image processing on the bonding image and extract features to obtain the bonding tool position, posture and geometric features of the bonding area;

[0107] An information output module, used to obtain the current moving path of the bonding tool through a pre-built bonding path planning model based on the position and posture of the bonding tool and the geometric features of the bonding area, and output an instruction to control the bonding tool to perform chip bonding according to the current moving path;

[0108] The information monitoring module is used to execute the chip bonding instruction through the bonding tool, and use the first camera and the second camera to monitor in real time the bonding process of the bonding tool performing the bonding operation according to the current moving path.

[0109] For the specific implementation of each device in the above device example, please refer to the above method embodiment, which will not be repeated here.

[0110] Reference Figure 3 In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0111] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0112] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0113] In summary, the steps of the dual-camera precision positioning method in the bonding system include: respectively collecting and continuously monitoring the bonding tool and the bonding interface through the first camera and the second camera to obtain a bonding image; performing image processing on the bonding image and extracting features to obtain the bonding tool position, posture and geometric features of the bonding area; based on the bonding tool position, posture and geometric features of the bonding area, obtaining the current moving path of the bonding tool through a pre-built bonding path planning model, and outputting instructions for controlling the bonding tool to perform chip bonding according to the current moving path; executing chip bonding instructions through the bonding tool, and using the first camera and the second camera to monitor in real time the bonding process of the bonding tool performing the bonding operation according to the current moving path, so as to achieve high-precision and high-efficiency positioning through dual cameras in a bonding system with a complex temperature environment to achieve the purpose of chip bonding.

[0114] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0115] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0116] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A dual-camera precision positioning method in a bonding system, characterized in that: The following steps are involved: The bonding tool and the bonding interface are respectively captured and continuously monitored by the first camera and the second camera to obtain a bonding image, wherein the bonding image includes a bonding tool image and a bonding interface image; wherein the first camera focuses on the bonding tool and the second camera focuses on the bonding interface, and the two cameras synchronously capture image information corresponding to the same time based on a timestamp; Performing image processing on the bonding image and extracting features to obtain the bonding tool position, posture and geometric features of the bonding area; Based on the position and posture of the bonding tool and the geometric features of the bonding area, the current moving path of the bonding tool is obtained through a pre-constructed bonding path planning model, and an instruction for controlling the bonding tool to perform chip bonding according to the current moving path is output. The bonding path planning model is based on data of various bonding tool positions, postures and geometric features of the bonding area in various historical bonding operations, and is trained using a machine learning algorithm. The input is data related to the position and posture of the bonding tool and the geometric features of the bonding area, and the output is the moving path, moving speed and moving force of the bonding tool when performing chip bonding. The chip bonding instruction is executed by the bonding tool, and the first camera and the second camera are used to monitor in real time the bonding process of the bonding tool performing the bonding operation according to the current moving path.

2. The dual-camera precision positioning method in a bonding system according to claim 1, characterized in that: The step of acquiring and continuously monitoring the bonding tool and the bonding interface by the first camera and the second camera respectively to obtain a bonding image includes: Calibrate the first camera and the second camera, determine the internal parameters of the first camera and the second camera and the relative position relationship between the first camera and the second camera, and perform image registration and calibration; The first camera collects the image of the bonding tool in real time and continuously monitors it to obtain the bonding tool image, wherein the bonding tool image at least includes the shape and size of the key parts when the bonding action is performed; adjusting the focal length and depth of field of the second camera according to the size and depth of the bonding area; While the first camera is capturing images of the bonding tool, the second camera is used to capture and continuously monitor the bonding interface between the chip to be synthesized and the substrate to obtain an image of the bonding area.

3. The dual-camera precision positioning method in a bonding system according to claim 1, characterized in that: The step of performing image processing on the bonding image and extracting features to obtain the bonding tool position, posture and geometric features of the bonding area includes: Performing image preprocessing operations on the acquired bonded image; Performing feature extraction on the preprocessed bonding image through dynamic range adjustment and a pre-trained temperature compensation model to respectively determine feature points of the bonding tool and the bonding area; Based on the result of feature extraction, the position and posture of the bonding tool and the geometric features of the bonding area are determined.

4. The dual-camera precision positioning method in a bonding system according to claim 3, characterized in that: The step of extracting features from the preprocessed bonding image by dynamic range adjustment and a pretrained temperature compensation model to respectively determine feature points of the bonding tool and the bonding area comprises: Based on the temperature in the current bonding environment acquired in real time by the temperature sensor, the dynamic range of the bonding image is adjusted by using a set image processing algorithm to enhance the image details, and the thermal expansion compensation of the bonding image is performed by using a pre-trained temperature compensation model; Based on the dynamic range adjustment and thermal expansion compensation results of the bonding image, feature extraction is performed on the bonding image to obtain feature points of the bonding tool and the bonding area respectively, wherein the feature extraction includes edge detection, shape recognition and corner point detection; The first camera and the second camera continuously track the characteristic points of the bonding tool and the characteristic points of the bonding area to monitor the movement trajectory and posture changes of the bonding tool and the morphological changes of the bonding interface.

5. The dual-camera precision positioning method in a bonding system according to claim 1, characterized in that: The step of acquiring the current moving path of the bonding tool through a pre-built bonding path planning model based on the bonding tool position, posture and geometric features of the bonding area, and outputting instructions for controlling the bonding tool to perform chip bonding according to the current moving path includes: Integrate and format the data of the bonding tool position and posture and the data of the geometric features of the bonding area; The temperature sensor is used to obtain the current temperature in the bonding environment in real time and perform error compensation; Inputting the integrated and formatted data and the current temperature into a pre-trained bonding path planning model to obtain a required moving position and moving posture of the bonding tool relative to the bonding area; Determining a current moving path required for the bonding tool to perform chip bonding based on an output result of the bonding path planning model; Based on the calculated current moving path, converting executable chip bonding instructions; The chip bonding instruction is sent to the execution component of the bonding tool.

6. The dual-camera precision positioning method in a bonding system according to claim 5, characterized in that: The training steps of the bonding path planning model include: Collect bonding image data of bonding tools and bonding interfaces in various positions, postures and temperature environments, and collect parameter data of speed and force actually recorded during each bonding operation, and pre-process the collected data. The sources of the collected data include operation records of various types of chip bonding equipment in actual production and data collected from specially designed bonding experiments; Extracting key features for characterizing relevant morphological information of a bonding tool and a bonding interface from the preprocessed bonding image data, wherein the key features include edges, shapes, and corner points; The extracted key features and the corresponding parameter data of speed and force are annotated, wherein the annotated content includes the position, posture, geometric features of the bonding area, and the corresponding speed value and force value of the bonding tool, and a bonding image training data set is constructed, wherein the bonding image training data set includes an image of multi-dimensional features and the corresponding position, posture, geometric features, and speed and force annotations; The bonding image training data set is input into a deep neural network algorithm model, and a bonding path planning model is obtained through training and optimization, which calculates the movement path, speed and force of the bonding tool when performing chip bonding based on the feature points of the bonding tool and the bonding area.

7. The dual-camera precision positioning method in a bonding system according to claim 1, characterized in that: The step of executing the chip bonding instruction by the bonding tool and using the first camera and the second camera to monitor in real time the bonding process of the bonding tool performing the bonding operation according to the current moving path includes: Based on the chip bonding instruction, driving the bonding tool to move along the current moving path to perform a chip bonding operation; During the process of the bonding tool moving and performing the bonding operation, continuously monitoring the position, posture and state of the bonding area of ​​the bonding tool in real time by the first camera and the second camera; Based on the current moving path and the real-time monitoring data, calculating the position and posture deviation through the bonding path planning model; Establishing a correspondence between the expected feature point coordinates of the bonding tool on the current moving path output by the bonding path planning model and the actual feature point coordinates monitored in real time by the first camera, and determining the position deviation between the bonding tool and the expected feature point coordinates by using a coordinate transformation method; Determine the posture deviation of the bonding tool from the expected posture based on the expected posture of the bonding tool at each node obtained in the moving path output by the bonding path planning model and the current actual posture of the prime bonding tool calculated by the posture estimation algorithm by monitoring the bonding tool in real time using the first camera; Based on the position deviation and the posture deviation results, the moving path or bonding operation of the bonding tool is adjusted through a bonding path planning model.

8. A dual-camera precision positioning device in a bonding system, characterized in that: include: A data acquisition module, used to acquire and continuously monitor the bonding tool and the bonding interface through the first camera and the second camera respectively, to obtain a bonding image; A data processing module, used to perform image processing on the bonding image and extract features to obtain the bonding tool position, posture and geometric features of the bonding area; An information output module, used to obtain the current moving path of the bonding tool through a pre-built bonding path planning model based on the position and posture of the bonding tool and the geometric features of the bonding area, and output an instruction to control the bonding tool to perform chip bonding according to the current moving path; The information monitoring module is used to execute the chip bonding instruction through the bonding tool, and use the first camera and the second camera to monitor in real time the bonding process of the bonding tool performing the bonding operation according to the current moving path.

9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the dual-camera precision positioning method in the bonding system described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the dual-camera precision positioning method in a bonding system according to any one of claims 1 to 7 are implemented.

Images