Dual-camera precise positioning method, device, equipment and storage medium in bonding system

Abstract

The present invention relates to a dual-camera precision positioning method, device, equipment and storage medium in a bonding system, wherein the method comprises the following steps: respectively capturing and continuously monitoring a bonding tool and a bonding interface by a first camera and a second camera to obtain a bonding image; performing image processing on the bonding image and extracting features to obtain the position, posture and geometric features of the bonding tool in the bonding area; thereby obtaining the current movement path of the bonding tool through a pre-built bonding path planning model, and outputting instructions for controlling the bonding tool to perform chip bonding according to the current movement path; utilizing the first camera and the second camera to perform real-time monitoring of the bonding process of the bonding tool performing a bonding operation according to the current movement path, so as to achieve high-precision and high-efficiency positioning by the dual cameras in a bonding system with a complex temperature environment, thereby completing the purpose of chip bonding.

Description

Technical Field

[0001] The present invention relates to the field of chip packaging technology, and in particular to a dual-camera precision positioning method, device, equipment and storage medium in a bonding system. Background Art

[0002] In the field of semiconductor manufacturing, chip bonding technology plays a vital role. With the rapid development of technology, especially the rise of advanced packaging technologies such as 3D packaging and system-in-package (SiP), more stringent requirements are placed on precise positioning and efficient control during the chip bonding process. Although traditional chip bonding processes have incorporated machine vision technology to a certain extent to improve positioning accuracy, most are still limited to a single camera perspective and are unable to cope with complex changes such as thermal expansion of materials and image blur caused by temperature. Temperature changes not only cause small but significant changes in the physical dimensions of the bonding tool and the bonding interface, but also affect the quality of image acquisition, which in turn affects the subsequent image processing and feature extraction steps, and may ultimately lead to a shift in the bonding position or a decrease in bonding quality.

[0003] Although some existing technologies attempt to address this problem through complex calibration procedures or fixed temperature compensation strategies, these methods often lack flexibility and cannot adapt to the rapid temperature changes in the actual working environment in real time. In addition, the compensation effects on bonding components of different materials and structures are also uneven. How to maintain high-precision positioning capabilities in chip bonding technology during temperature changes has become a technical problem that needs to be urgently solved in the current semiconductor manufacturing field. Summary of the Invention

[0004] The main purpose of the present invention is to provide a dual-camera precision positioning method, device, equipment and storage medium in a bonding system, so as to achieve high-precision and high-efficiency positioning through dual cameras in a bonding system with a complex temperature environment to complete the purpose of chip bonding.

[0005] To achieve the above objectives, the present invention provides a dual-camera precision positioning method in a bonding system, comprising the following steps:

[0006] Capturing and continuously monitoring a bonding tool and a bonding interface using a first camera and a second camera, respectively, to obtain a bonding image, the bonding image including an image of the bonding tool and an image of the bonding interface; wherein the first camera focuses on the bonding tool and the second camera focuses on the bonding interface, and the first camera and the second camera synchronously capture image information corresponding to the same moment based on a timestamp;

[0007] Performing image processing on the bonding image and extracting features to obtain the bonding tool position, posture, and geometric features of the bonding area;

[0008] Based on the position and posture of the bonding tool and the geometric features of the bonding area, a pre-built bonding path planning model is used to obtain the current movement path of the bonding tool, and an instruction for controlling the bonding tool to perform chip bonding according to the current movement path is output. The bonding path planning model is based on data on the positions and postures of various bonding tools and the geometric features of the bonding area in various historical bonding operations, and is trained using a machine learning algorithm. The input is the data related to the position and posture of the bonding tool and the geometric features of the bonding area, and the output is the movement path, movement speed, and movement force of the bonding tool when performing chip bonding;

[0009] The chip bonding instruction is executed by the bonding tool, and the first camera and the second camera are used to monitor in real time the bonding process of the bonding tool performing the bonding operation according to the current moving path.

[0010] Furthermore, the step of acquiring and continuously monitoring the bonding tool and the bonding interface by the first camera and the second camera respectively to obtain a bonding image includes:

[0011] Calibrate the first camera and the second camera to determine internal parameters of the first camera and the second camera and the relative position relationship between the first camera and the second camera, and perform image registration and calibration;

[0012] The first camera collects an image of the bonding tool in real time and continuously monitors the image to obtain an image of the bonding tool, wherein the image of the bonding tool includes at least the shape and size of a key part when performing the bonding action;

[0013] adjusting the focal length and depth of field of the second camera according to the size and depth of the bonding area;

[0014] While the first camera is capturing images of the bonding tool, the second camera is used to capture and continuously monitor the bonding interface between the chip to be synthesized and the substrate to obtain an image of the bonding area.

[0015] Furthermore, the step of performing image processing on the bonding image and extracting features by image denoising and enhancement algorithms to obtain the position and posture of the bonding tool and the geometric features of the bonding area includes:

[0016] Performing image preprocessing on the acquired bonded image;

[0017] Performing feature extraction on the pre-processed bonding image through dynamic range adjustment and a pre-trained temperature compensation model to determine feature points of the bonding tool and the bonding area respectively;

[0018] Based on the result of feature extraction, the position and posture of the bonding tool and the geometric features of the bonding area are determined.

[0019] Furthermore, the step of extracting features from the pre-processed bonding image by dynamic range adjustment and a pre-trained temperature compensation model to determine the feature points of the bonding tool and the bonding area respectively includes:

[0020] Based on the temperature in the current bonding environment acquired in real time by the temperature sensor, a set image processing algorithm is used to adjust the dynamic range of the bonding image to enhance image details, and a pre-trained temperature compensation model is used to compensate for thermal expansion of the bonding image;

[0021] Based on the dynamic range adjustment and thermal expansion compensation results of the bonding image, feature extraction is performed on the bonding image to obtain feature points of the bonding tool and the bonding area respectively, wherein the feature extraction includes edge detection, shape recognition, and corner detection;

[0022] The first camera and the second camera continuously track the characteristic points of the bonding tool and the characteristic points of the bonding area to monitor the movement trajectory and posture changes of the bonding tool and the morphological changes of the bonding interface.

[0023] Furthermore, the step of obtaining the current movement path of the bonding tool through a pre-built bonding path planning model based on the position and posture of the bonding tool and the geometric features of the bonding area, and outputting an instruction for controlling the bonding tool to perform chip bonding according to the current movement path, includes:

[0024] Integrating and formatting the data of the bonding tool position and posture and the data of the geometric features of the bonding area;

[0025] The current temperature in the bonding environment is acquired in real time through the temperature sensor, and error compensation is performed;

[0026] Inputting the integrated and formatted data and the current temperature into a pre-trained bonding path planning model to obtain a desired movement position and movement posture of the bonding tool relative to the bonding area;

[0027] Determining a current moving path of the bonding tool required for chip bonding based on an output result of the bonding path planning model;

[0028] Based on the calculated current moving path, converting executable chip bonding instructions;

[0029] The chip bonding instruction is sent to the execution component of the bonding tool.

[0030] Furthermore, the training step of the bonding path planning model includes:

[0031] Collect bonding image data covering bonding tools and bonding interfaces in various positions, postures, and temperature environments. Also collect parameter data on speed and force recorded during each bonding operation, and pre-process the collected data. The data is sourced from operational records of various chip bonding equipment models in actual production and data collected from specially designed bonding experiments.

[0032] Extracting key features for characterizing relevant morphological information of the bonding tool and the bonding interface from the pre-processed bonding image data, wherein the key features include edges, shapes, and corners;

[0033] Annotating the extracted key features and corresponding parameter data of speed and force, wherein the annotation content includes the position, posture, geometric features of the bonding area, and corresponding speed and force values of the bonding tool, and constructing a bonding image training dataset, wherein the bonding image training dataset includes images of multi-dimensional features and their corresponding position, posture, geometric features, speed, and force annotations;

[0034] The bonding image training data set is input into a deep neural network algorithm model, and through training and optimization, a bonding path planning model is obtained that calculates the movement path, speed and force of the bonding tool during chip bonding based on the feature points of the bonding tool and the bonding area.

[0035] Furthermore, the step of executing the chip bonding instruction by the bonding tool and using the first camera and the second camera to monitor in real time the bonding process of the bonding tool performing the bonding operation according to the current movement path includes:

[0036] Based on the chip bonding instruction, driving the bonding tool to move along the current moving path to perform a chip bonding operation;

[0037] During the process of the bonding tool moving and performing the bonding operation, continuously monitoring the position and posture of the bonding tool and the state of the bonding area in real time by the first camera and the second camera;

[0038] Calculating position and posture deviations using a bonding path planning model based on the current movement path and real-time monitoring data;

[0039] Establishing a correspondence between the expected feature point coordinates of the bonding tool on the current moving path output by the bonding path planning model and the actual feature point coordinates monitored in real time by the first camera, and determining the position deviation between the bonding tool and the expected feature point coordinates using a coordinate transformation method;

[0040] Determining a posture deviation of the bonding tool from the expected posture based on an expected posture of the bonding tool at each node obtained in the movement path output by the bonding path planning model and a current actual posture of the prime bonding tool calculated by a posture estimation algorithm using real-time monitoring of the bonding tool using the first camera;

[0041] Based on the position deviation and the posture deviation results, the moving path or bonding operation of the bonding tool is adjusted through a bonding path planning model.

[0042] The present invention also provides a dual-camera precision positioning device in a bonding system, comprising:

[0043] a data acquisition module, configured to acquire and continuously monitor the bonding tool and the bonding interface through the first camera and the second camera respectively, to obtain a bonding image;

[0044] a data processing module, configured to perform image processing on the bonding image and extract features to obtain the bonding tool position, posture, and geometric features of the bonding area;

[0045] An information output module is configured to obtain a current movement path of the bonding tool based on the position and posture of the bonding tool and the geometric features of the bonding area through a pre-built bonding path planning model, and output an instruction for controlling the bonding tool to perform chip bonding according to the current movement path;

[0046] The information monitoring module is used to execute the chip bonding instruction through the bonding tool, and use the first camera and the second camera to monitor in real time the bonding process of the bonding tool performing the bonding operation according to the current moving path.

[0047] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the dual-camera precision positioning method in the above-mentioned bonding system are implemented.

[0048] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the dual-camera precision positioning method in the above-mentioned bonding system.

[0049] The dual-camera precision positioning method, device, equipment, and storage medium provided by the present invention in a bonding system have the following beneficial effects:

[0050] The dual-camera (first camera and second camera) system synchronously captures image information corresponding to the same moment based on the timestamp, thereby ensuring the temporal consistency of the images of the bonding tool and the bonding interface, further improving positioning accuracy; using the pre-built bonding path planning model, the optimal movement path can be automatically calculated and planned based on the bonding tool position, posture, and geometric characteristics of the bonding area. This intelligent control method not only improves bonding efficiency, but also reduces the risk of error caused by human operation; by precisely controlling the movement path, movement speed, and movement force of the bonding tool, high-quality chip bonding is achieved, which helps to reduce defects and failures in the bonding process and improve product reliability and service life; real-time monitoring of the status of the bonding tool and the bonding interface, and adjustment of the movement path or bonding operation based on the real-time monitoring data, can further ensure the stability and consistency of the bonding quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 1 is a flow chart of a dual-camera precision positioning method in a bonding system according to one embodiment of the present invention;

[0052] Figure 2 This is a structural block diagram of a dual-camera precision positioning device in a bonding system according to one embodiment of the present invention;

[0053] Figure 3 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0054] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0056] Reference Figure 1 , which is a flow chart of a dual-camera precision positioning method in a bonding system proposed by the present invention, comprising the following steps:

[0057] S1, respectively capturing and continuously monitoring a bonding tool and a bonding interface using a first camera and a second camera to obtain a bonding image, wherein the bonding image includes an image of the bonding tool and an image of the bonding interface; wherein the first camera focuses on the bonding tool and the second camera focuses on the bonding interface, and the first camera and the second camera synchronously capture image information corresponding to the same moment based on a timestamp;

[0058] S2, performing image processing on the bonding image and extracting features to obtain the bonding tool position, posture and geometric features of the bonding area;

[0059] S3, based on the position and posture of the bonding tool and the geometric features of the bonding area, obtain the current movement path of the bonding tool through a pre-built bonding path planning model, and output instructions for controlling the bonding tool to perform chip bonding according to the current movement path. The bonding path planning model is based on data on various bonding tool positions, postures, and geometric features of the bonding area in various historical bonding operations, and is trained using a machine learning algorithm. The input is the data related to the bonding tool position, posture, and geometric features of the bonding area, and the output is the movement path, movement speed, and movement force of the bonding tool when performing chip bonding;

[0060] S4, executing the chip bonding instruction through the bonding tool, and using the first camera and the second camera to monitor in real time the bonding process of the bonding tool performing the bonding operation according to the current moving path.

[0061] As described in step S1 above, the bonding tool and the bonding interface are respectively captured and continuously monitored by the first camera and the second camera to obtain a bonding image. Wherein, the first camera is used to monitor the bonding tool, and the bonding tool refers to a device or apparatus for connecting two or more components (such as a chip and a substrate) together in the fields of semiconductor manufacturing and microelectronic packaging. The second camera is used to monitor the bonding interface, and the bonding interface refers to the contact area between the two components under the action of the bonding tool. The image information of the bonding tool and the bonding interface is captured and recorded in real time and continuously by the two cameras, and the image information of the two cameras is ensured to be captured at the same time based on the timestamp.

[0062] As described in above-mentioned step S2, by image denoising, enhancement algorithm, described bonding image is carried out image processing and feature extraction, obtain bonding tool position, posture and the geometrical feature of bonding area.In data acquisition process, be affected by various noises (such as environmental noise, camera noise etc.), cause the image of quality degradation to carry out denoising, reduce or eliminate these noises, to improve the clarity of image, and carry out image enhancement, improve the visual effect of image, make image more suitable for subsequent image analysis and feature extraction, including adjusting the attributes such as contrast, brightness, sharpness of image, so that the feature of bonding tool and bonding interface is more obvious. The image feature extracted can be the edge in image, shape, texture etc. and can represent the key information in image. By image processing algorithm, by detecting the edge or specific mark of bonding tool, determine the specific position of bonding tool in image, extract the posture information of bonding tool simultaneously, such as rotation angle, tilt degree etc., for understanding the motion state of bonding tool in bonding process, extract the geometrical feature of bonding interface or bonding area, such as area, perimeter, shape etc., these features can reflect the state and quality of bonding interface.

[0063] As described in step S3 above, based on the bonding tool position, posture and geometric features of the bonding area, the current movement path of the bonding tool is obtained through a pre-built bonding path planning model, and an instruction for controlling the bonding tool to perform chip bonding according to the current movement path is output. When new bonding tool position, posture and bonding area geometric feature data are input, the model will output an optimal movement path. This path is learned by the model based on historical data, and is intended to ensure that the bonding tool can accurately and efficiently move to the correct position and perform chip bonding at an appropriate speed and force. When new bonding tool position, posture and bonding area geometric feature data are input, the model will output an optimal current movement path. This path is learned by the model based on historical data, and is intended to ensure that the bonding tool can accurately and efficiently move to the correct position and perform chip bonding at an appropriate speed and force. In addition, the model output also includes movement speed and movement force, thereby achieving efficient and accurate chip bonding.

[0064] As described in the above step S4, the chip bonding instruction is executed by the bonding tool, and the first camera and the second camera are used to monitor in real time the bonding process of the bonding tool performing the bonding operation according to the current moving path. According to the chip bonding instruction calculated and converted in step S3, the bonding tool is driven to start moving along the predetermined current moving path, thereby implementing the chip bonding operation. During the movement of the bonding tool and the actual execution of the bonding operation, the first camera and the second camera are continuously used for real-time monitoring. The first camera focuses on the bonding tool and captures the position, posture, etc. of the bonding tool in real time, such as whether the key parts of the bonding tool are in the expected spatial position and whether its angle posture meets the requirements; the second camera focuses on the bonding interface and grasps the status of the bonding area in real time, such as whether there are abnormal changes in the bonding interface between the chip to be synthesized and the substrate. The dual cameras are used to jointly ensure all-round monitoring of the key elements of the entire bonding process from different perspectives. Based on the pre-planned movement path and the data obtained from real-time monitoring, the bonding path planning model is used to calculate the position and posture deviations. According to the calculated position deviation and posture deviation results, the bonding path planning model is used to make corresponding adjustments to the movement path of the bonding tool or the bonding operation.

[0065] In one embodiment, the step S1 of acquiring and continuously monitoring the bonding tool and the bonding interface by the first camera and the second camera respectively to obtain the bonding image includes:

[0066] S11, calibrating the first camera and the second camera to determine internal parameters of the first camera and the second camera and a relative positional relationship between the first camera and the second camera, and performing image registration and calibration;

[0067] S12, collecting an image of the bonding tool in real time and continuously monitoring the first camera to obtain an image of the bonding tool, wherein the image of the bonding tool includes at least the shape and size of a key part when performing a bonding action;

[0068] S13, adjusting the focal length and depth of field of the second camera according to the size and depth of the bonding area;

[0069] S14, while the first camera captures images of the bonding tool, the second camera captures and continuously monitors the bonding interface between the chip to be synthesized and the substrate to obtain an image of the bonding area.

[0070] As described in step S11 above, by taking images of standard objects of known size and shape, the internal parameters of the camera (such as focal length, principal point position, distortion coefficient, etc.) are calculated using these images, and the relative position relationship between the two cameras is determined using a stereo calibration method, including a rotation matrix and a translation vector. Based on the above parameters, the images taken by the two cameras are registered and calibrated to ensure that the two cameras are aligned in the same coordinate system and can accurately reflect the three-dimensional information of the real world. As described in steps S12-S14 above, the first camera is set to focus on the bonding tool and collect its image in real time. Through the image processing algorithm, the shape and size information of the key parts of the bonding tool during bonding (such as the bonding tool head) are extracted from the collected image. The second camera is set to focus on the bonding interface and collect its image in real time. According to the size of the bonding area, the focal length of the second camera is adjusted to obtain the best image clarity, and according to the depth of the bonding interface, the depth of field of the camera is adjusted to ensure that the entire bonding area is within the range of clear imaging. Through the image processing algorithm, the state of the bonding interface (such as alignment, gap size, etc.) is continuously monitored and relevant geometric feature information is extracted. The above steps provide accurate and reliable image data for subsequent image processing, feature extraction, and bonding path planning through camera calibration, focus and depth of field adjustment, and real-time image acquisition and monitoring. Together, they form the basis of the dual-camera precision positioning method in the bonding system.

[0071] In one embodiment, the step S2 of performing image processing on the bonding image and extracting features by image denoising and enhancement algorithms to obtain the position and posture of the bonding tool and the geometric features of the bonding area includes:

[0072] S21, performing image preprocessing on the acquired bonded image;

[0073] S22, performing feature extraction on the preprocessed bonding image through dynamic range adjustment and a pre-trained temperature compensation model, and determining feature points of the bonding tool and the bonding area respectively;

[0074] S23, determining the position and posture of the bonding tool and the geometric features of the bonding area based on the result of feature extraction.

[0075] As described in step S21 above, the preprocessing operation includes but is not limited to removing common noise types such as salt and pepper noise and Gaussian noise from the image, and adjusting the contrast and brightness of the image through methods such as histogram equalization to make the key information in the image clearer and more discernible. For example, the collected bonding tool image contains some randomly distributed bright spots (noise). These bright spots are removed by a suitable denoising algorithm to make the true shape and outline of the bonding tool more clearly visible. As described in step S22 above, the bonding image may have large differences in brightness and contrast between different areas of the image due to factors such as the settings of the acquisition device and the ambient temperature. Some areas have uneven brightness, making some details difficult to see clearly. Dynamic range adjustment is used to adjust the brightness and contrast of the image to an appropriate range so that all details in the image can be presented as clearly as possible, facilitating the subsequent accurate extraction of feature points. For example, some subtle structures of the bonding tool are unclear in the original image because they are in a darker area. By adjusting the dynamic range, the brightness of the area is increased while maintaining a reasonable display of other areas, making these subtle structures visible and facilitating the subsequent determination of their feature points. Temperature changes can cause slight variations in the physical dimensions of the bonding tool and bonding interface, which can also affect image acquisition quality. A pre-trained temperature compensation model compensates the bonding image based on real-time temperature information, ensuring that the image accurately reflects the actual state of the bonding tool and bonding area, without being affected by factors such as thermal expansion caused by temperature changes. For example, in a high-temperature environment, the bonding tool may expand slightly, causing the shape and size of the bonding tool in the captured image to differ from that at room temperature. The temperature compensation model can correct for these variations. The pre-trained temperature compensation model is trained based on various image data related to the bonding process, which are labeled with temperature and correspond to correct feature points. During the chip bonding process, temperature changes can affect the physical dimensions of the bonding tool and bonding interface, thereby affecting image acquisition quality. The pre-trained temperature compensation model is designed to compensate for these image variations caused by temperature changes, ensuring that the extracted feature points accurately reflect the actual state of the bonding tool and bonding area, rather than false feature points that are biased by temperature.

[0076] As described in step S23 above, the feature points of the bonding tool determined by step S22 are combined with the image coordinate system and some known reference points (such as the optical center of the camera, etc.) to determine the position coordinates of the bonding tool in the image through geometric calculation methods, and then infer its position in the actual space (combined with the calibration parameters of the camera, etc. for further spatial conversion). According to the relative position relationship between the multiple feature points of the bonding tool and their changes in the continuous images, the posture of the bonding tool (such as horizontal placement, tilting at a certain angle, etc.) is analyzed. For example, if the ordinate of a corner point of the bonding tool gradually decreases in several consecutive frames of images, while the abscissa remains basically unchanged, it means that the bonding tool is gradually tilting downward. Based on the feature points of the bonding area, the geometric features of the bonding area are determined by analyzing the geometric parameters such as the shape, area, and perimeter surrounded by these feature points. For example, if the bonding area is a rectangle, its length, width, area and other geometric properties can be accurately calculated through the feature points. These geometric features are used for subsequent planning of operations such as the movement path of the bonding tool.

[0077] In one embodiment, the step S22 of extracting features from the preprocessed bonding image by using dynamic range adjustment and a pretrained temperature compensation model to determine feature points of the bonding tool and the bonding area respectively includes:

[0078] S221, based on the temperature in the current bonding environment acquired in real time by the temperature sensor, using a set image processing algorithm to adjust the dynamic range of the bonding image to enhance image details, and simultaneously using a pre-trained temperature compensation model to compensate for thermal expansion of the bonding image;

[0079] S222, based on the dynamic range adjustment and thermal expansion compensation results of the bonding image, performing feature extraction on the bonding image to obtain feature points of the bonding tool and the bonding area respectively, wherein the feature extraction includes edge detection, shape recognition, and corner detection;

[0080] S223 , continuously tracking the bonding tool feature points and the bonding area feature points by the first camera and the second camera, and monitoring the movement trajectory and posture changes of the bonding tool and the morphological changes of the bonding interface.

[0081] As described in step S221 above, based on the temperature in the current bonding environment acquired in real time by the temperature sensor, the dynamic range of the bonding image is adjusted using a set image processing algorithm (such as histogram stretching, logarithmic transformation, etc.) to expand the brightness range of the image, making the dark and bright details of the image clearer, and providing high-quality image input for subsequent feature extraction. At the same time, a pre-trained temperature compensation model is used to compensate for thermal expansion of the bonding image. Since temperature changes cause the material to expand and contract, thereby affecting the size and shape of the objects in the image, the pre-trained temperature compensation model predicts and compensates for this thermal expansion effect based on the current temperature to ensure that the size and shape of the objects in the image are consistent with the actual size. As described in step S222 above, an edge detection algorithm (such as Canny edge detection) is used to identify the edge information of the bonding tool and bonding area in the image to determine the outline and shape; based on the results of the edge detection, the shape features of the objects in the image are further identified through template matching and shape context; and a corner detection algorithm (such as Harris corner detection) is used to determine the corner features in the image. Corner points are important local features in the image, i.e., feature points. Through the above-mentioned feature extraction process, the feature points of the bonding tool and the bonding area are respectively obtained for subsequent position, posture and morphology monitoring. As described in the above step S223, the image data captured synchronously and in real time by two cameras are used in combination with a feature point matching algorithm (such as the least squares method) to continuously track the feature points of the bonding tool and the bonding area. By tracking the motion trajectory of the feature points, the movement trajectory and posture changes of the bonding tool are monitored in real time to ensure that the bonding operation is carried out according to the predetermined path and posture. At the same time, by monitoring the morphological changes (such as area, perimeter, shape, etc.) of the feature points in the bonding area, potential bonding problems (such as dislocation, breakage, etc.) are discovered and handled in a timely manner.

[0082] In one embodiment, step S3 of obtaining a current movement path of the bonding tool based on the position and posture of the bonding tool and the geometric features of the bonding area through a pre-built bonding path planning model, and outputting an instruction for controlling the bonding tool to perform chip bonding according to the current movement path, includes:

[0083] S31, integrating and formatting the data of the bonding tool position and posture and the data of the geometric features of the bonding area;

[0084] S32, obtaining the current temperature in the bonding environment in real time through the temperature sensor and performing error compensation;

[0085] S33, inputting the integrated and formatted data and the current temperature into a pre-trained bonding path planning model to obtain a required moving position and moving posture of the bonding tool relative to the bonding area;

[0086] S34, determining a current moving path required for the bonding tool to perform chip bonding based on an output result of the bonding path planning model;

[0087] S35, converting an executable chip bonding instruction based on the calculated current moving path;

[0088] S36: Send the chip bonding instruction to the execution component of the bonding tool.

[0089] As described in step S31 above, the position (such as X, Y, Z coordinates) and posture (such as rotation angle, inclination, etc.) data of the bonding tool, as well as the geometric features of the bonding area, such as shape, size, position, etc., are obtained by the camera sensor (first and second cameras). These data are integrated together and converted into the input format required by the model, such as a specific data structure. As described in step S32 above, a temperature sensor is used to monitor the temperature in the bonding environment in real time. Based on the temperature data, a pre-established temperature compensation model is used to calculate the position or posture error caused by temperature changes, and the error is compensated. As described in steps S33-S36 above, the integrated data (including position, posture and geometric features) and the current temperature are used as input to a pre-trained bonding path planning model to calculate the movement path and posture required for the bonding tool from the current position to the target position. The movement path information, including path points, speed, acceleration, direction, etc., is extracted from the output of the model. Based on this information, a complete movement path of the bonding tool from the starting position to the target position is constructed. The movement path information is converted into control instructions, such as the number of steps for the stepper motor and the position setting for the servo system, ensuring that the instruction format is compatible with the interface of the bonding tool's actuator. The instructions are then sent to the bonding tool's actuator via a communication interface. Based on the instructions, the actuator controls the bonding tool for precise movement and bonding operations. By integrating data, compensating for temperature errors, calculating and determining the movement path using a bonding path planning model, converting instructions, and sending them, these steps enable precise planning and control of the bonding tool's movement path, thereby ensuring the accuracy and reliability of chip bonding.

[0090] In one embodiment, the step of training the bonding path planning model includes:

[0091] Collect bonding image data covering bonding tools and bonding interfaces in various positions, postures, and temperature environments. Also collect parameter data on speed and force recorded during each bonding operation, and pre-process the collected data. The data is sourced from operational records of various chip bonding equipment models in actual production and data collected from specially designed bonding experiments.

[0092] According to the physical model and kinematic principles of the bonding system in this embodiment, a large amount of combined data of the bonding tool position, posture, and geometric features of the bonding area are generated using computer simulation software. These simulation data cover various possible bonding scenarios, including different chip sizes and shapes, the starting position and posture of the bonding tool, etc. For example, in the simulation, under different three-dimensional spatial coordinates, the bonding tool approaches the bonding area with different geometric shapes (such as rectangle, circle, etc.) at different tilt angles and rotation directions. For each simulated combined data, the known ideal bonding path (predetermined based on physical rules and process requirements) is used as the corresponding label data, that is, the path information of how the bonding tool should move to accurately complete the bonding, and the above data is organized into a bonding system data set.

[0093] Extracting key features for characterizing relevant morphological information of the bonding tool and the bonding interface from the pre-processed bonding image data, wherein the key features include edges, shapes, and corners;

[0094] Annotating the extracted key features and corresponding parameter data of speed and force, wherein the annotation content includes the position, posture, geometric features of the bonding area, and corresponding speed and force values of the bonding tool, and constructing a bonding image training dataset, wherein the bonding image training dataset includes images of multi-dimensional features and their corresponding position, posture, geometric features, speed, and force annotations;

[0095] The deep neural network algorithm model is trained using the bonding image training data set and the bonding system data set. Through training and optimization, a bonding path planning model is obtained that calculates the movement path, speed, and force of the bonding tool when performing chip bonding based on the feature points of the bonding tool and the bonding area.

[0096] In one embodiment, the step S4 of executing the chip bonding instruction by the bonding tool and using the first camera and the second camera to monitor in real time the bonding process of the bonding tool performing the bonding operation according to the current movement path includes:

[0097] S41, based on the chip bonding instruction, driving the bonding tool to move along the current movement path to perform a chip bonding operation;

[0098] S42, during the process of the bonding tool moving and performing the bonding operation, continuously monitoring the position, posture, and state of the bonding area of the bonding tool in real time by using the first camera and the second camera;

[0099] S43, calculating position and posture deviations through a bonding path planning model based on the current movement path and real-time monitoring data;

[0100] S44, establishing a correspondence between the expected feature point coordinates of the bonding tool on the current moving path output by the bonding path planning model and the actual feature point coordinates monitored in real time by the first camera, and determining a positional deviation between the bonding tool and the expected feature point coordinates using a coordinate transformation method;

[0101] S45, determining a posture deviation of the bonding tool from the expected posture based on the expected posture of the bonding tool at each node in the movement path obtained from the bonding path planning model and the current actual posture of the prime bonding tool calculated by a posture estimation algorithm using the first camera to monitor the bonding tool in real time;

[0102] S46 , adjusting the moving path or bonding operation of the bonding tool through a bonding path planning model based on the position deviation and the posture deviation results.

[0103] As described in step S41 above, the control system receives the chip bonding instruction, and the control system controls the bonding tool to move along the current moving path through the driving mechanism (such as a stepping motor, a servo motor, etc.) according to the instruction. As described in steps S42-S43 above, the bonding tool and the bonding area are respectively imaged by the first camera and the second camera, and the image data collected are transmitted to the image processing system in real time to extract the position, posture information of the bonding tool and the state information of the bonding area. The position and posture data of the bonding tool monitored in real time are compared with the expected data output by the bonding path planning model, and the deviation value of the position and posture is calculated using an algorithm. As described in steps S44-S46 above, according to the current moving path output by the bonding path planning model, a series of key feature points and their coordinates are preset, the actual feature points on the bonding tool are monitored in real time by the first camera, and their coordinates are extracted. The actual feature point coordinates are compared with the expected feature point coordinates by the coordinate transformation method to calculate the position deviation. Based on the expected posture of each node in the current moving path output by the bonding path planning model, the actual posture of the bonding tool is monitored in real time with the first camera, and the current actual posture is calculated using the posture estimation algorithm. The expected posture and actual posture are compared to calculate the posture deviation of the bonding tool. The calculated position deviation and posture deviation are input into the bonding path planning model. The model calculates the adjusted moving path or bonding operation parameters based on the deviation value. The control system adjusts the moving path or bonding operation of the bonding tool based on the adjusted parameters to reduce the deviation and improve the bonding quality. The above steps achieve precise control and optimization of the bonding process by real-time monitoring of the bonding process, calculating the position and posture deviation, determining the deviation value, and adjusting the moving path or bonding operation using the bonding path planning model, thereby improving the accuracy and reliability of chip bonding.

[0104] Reference Figure 2, is a structural block diagram of a dual-camera precision positioning device in a bonding system according to one embodiment of the present invention, comprising:

[0105] a data acquisition module, configured to acquire and continuously monitor the bonding tool and the bonding interface through the first camera and the second camera respectively, to obtain a bonding image;

[0106] a data processing module, configured to perform image processing on the bonding image and extract features to obtain the bonding tool position, posture, and geometric features of the bonding area;

[0107] An information output module is configured to obtain a current movement path of the bonding tool based on the position and posture of the bonding tool and the geometric features of the bonding area through a pre-built bonding path planning model, and output an instruction for controlling the bonding tool to perform chip bonding according to the current movement path;

[0108] The information monitoring module is used to execute the chip bonding instruction through the bonding tool, and use the first camera and the second camera to monitor in real time the bonding process of the bonding tool performing the bonding operation according to the current moving path.

[0109] For the specific implementation of each device in the above device example, please refer to the above method embodiment, which will not be repeated here.

[0110] Reference Figure 3 In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3 As shown. The computer device includes a processor, memory, display screen, input device, network interface and database connected via a system bus. The processor of the computer design is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the above method is implemented.

[0111] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0112] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the above-described method when executed by a processor. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0113] To summarize, the steps of the dual-camera precision positioning method in the bonding system include: respectively collecting and continuously monitoring the bonding tool and the bonding interface through the first camera and the second camera to obtain a bonding image; performing image processing on the bonding image and extracting features to obtain the bonding tool position, posture and geometric features of the bonding area; based on the bonding tool position, posture and geometric features of the bonding area, obtaining the current movement path of the bonding tool through a pre-built bonding path planning model, and outputting instructions for controlling the bonding tool to perform chip bonding according to the current movement path; executing the chip bonding instruction through the bonding tool, and using the first camera and the second camera to monitor in real time the bonding process of the bonding tool performing the bonding operation according to the current movement path, so as to achieve high-precision and high-efficiency positioning through dual cameras in a bonding system with a complex temperature environment and complete the purpose of chip bonding.

[0114] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Among them, any reference to memory, storage, database, or other media provided by the present invention and used in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM.

[0115] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, apparatus, article, or method comprising the element.

[0116] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A dual-camera precision positioning method in a bonding system, characterized in that: The following steps are involved: Capturing and continuously monitoring a bonding tool and a bonding interface using a first camera and a second camera, respectively, to obtain a bonding image, the bonding image including an image of the bonding tool and an image of the bonding interface; wherein the first camera focuses on the bonding tool and the second camera focuses on the bonding interface, and the first camera and the second camera synchronously capture image information corresponding to the same moment based on a timestamp; Performing image processing on the bonding image and extracting features to obtain the bonding tool position, posture, and geometric features of the bonding area; Based on the position and posture of the bonding tool and the geometric features of the bonding area, a pre-built bonding path planning model is used to obtain the current movement path of the bonding tool, and an instruction for controlling the bonding tool to perform chip bonding according to the current movement path is output. The bonding path planning model is based on data on the positions and postures of various bonding tools and the geometric features of the bonding area in various historical bonding operations, and is trained using a machine learning algorithm. The input is the data related to the position and posture of the bonding tool and the geometric features of the bonding area, and the output is the movement path, movement speed, and movement force of the bonding tool when performing chip bonding; The chip bonding instruction is executed by the bonding tool, and the first camera and the second camera are used to monitor in real time the bonding process of the bonding tool performing the bonding operation according to the current moving path.

2. The dual-camera precision positioning method in a bonding system according to claim 1, characterized in that: The step of respectively capturing and continuously monitoring the bonding tool and the bonding interface by the first camera and the second camera to obtain a bonding image includes: Calibrate the first camera and the second camera to determine internal parameters of the first camera and the second camera and the relative position relationship between the first camera and the second camera, and perform image registration and calibration; The first camera collects an image of the bonding tool in real time and continuously monitors the image to obtain an image of the bonding tool, wherein the image of the bonding tool includes at least the shape and size of a key part when performing the bonding action; adjusting the focal length and depth of field of the second camera according to the size and depth of the bonding area; While the first camera is capturing images of the bonding tool, the second camera is used to capture and continuously monitor the bonding interface between the chip to be synthesized and the substrate to obtain an image of the bonding area.

3. The dual-camera precision positioning method in a bonding system according to claim 1, characterized in that: The step of performing image processing on the bonding image and extracting features to obtain the bonding tool position, posture and geometric features of the bonding area includes: Performing image preprocessing on the acquired bonded image; Performing feature extraction on the pre-processed bonding image through dynamic range adjustment and a pre-trained temperature compensation model to determine feature points of the bonding tool and the bonding area respectively; Based on the result of feature extraction, the position and posture of the bonding tool and the geometric features of the bonding area are determined.

4. The dual-camera precision positioning method in a bonding system according to claim 3, characterized in that: The step of extracting features from the pre-processed bonding image by using a dynamic range adjustment and a pre-trained temperature compensation model to determine feature points of the bonding tool and the bonding area respectively includes: Based on the temperature in the current bonding environment acquired in real time by the temperature sensor, a set image processing algorithm is used to adjust the dynamic range of the bonding image to enhance image details, and a pre-trained temperature compensation model is used to compensate for thermal expansion of the bonding image; Based on the dynamic range adjustment and thermal expansion compensation results of the bonding image, feature extraction is performed on the bonding image to obtain feature points of the bonding tool and the bonding area respectively, wherein the feature extraction includes edge detection, shape recognition, and corner detection; The first camera and the second camera continuously track the characteristic points of the bonding tool and the characteristic points of the bonding area to monitor the movement trajectory and posture changes of the bonding tool and the morphological changes of the bonding interface.

5. The dual-camera precision positioning method in a bonding system according to claim 1, characterized in that: The step of obtaining the current movement path of the bonding tool through a pre-built bonding path planning model based on the position and posture of the bonding tool and the geometric features of the bonding area, and outputting an instruction for controlling the bonding tool to perform chip bonding according to the current movement path, includes: Integrating and formatting the data of the bonding tool position and posture and the data of the geometric features of the bonding area; The current temperature in the bonding environment is acquired in real time through the temperature sensor, and error compensation is performed; Inputting the integrated and formatted data and the current temperature into a pre-trained bonding path planning model to obtain a desired movement position and movement posture of the bonding tool relative to the bonding area; Determining a current moving path of the bonding tool required for chip bonding based on an output result of the bonding path planning model; Based on the calculated current moving path, converting executable chip bonding instructions; The chip bonding instruction is sent to the execution component of the bonding tool.

6. The dual-camera precision positioning method in a bonding system according to claim 5, characterized in that: The training steps of the bonding path planning model include: Collect bonding image data covering bonding tools and bonding interfaces in various positions, postures, and temperature environments. Also collect parameter data on speed and force recorded during each bonding operation, and pre-process the collected data. The data is sourced from operational records of various chip bonding equipment models in actual production and data collected from specially designed bonding experiments. Extracting key features for characterizing relevant morphological information of the bonding tool and the bonding interface from the pre-processed bonding image data, wherein the key features include edges, shapes, and corners; Annotating the extracted key features and corresponding parameter data of speed and force, wherein the annotation content includes the position, posture, geometric features of the bonding area, and corresponding speed and force values of the bonding tool, and constructing a bonding image training dataset, wherein the bonding image training dataset includes images of multi-dimensional features and their corresponding position, posture, geometric features, speed, and force annotations; The bonding image training data set is input into a deep neural network algorithm model, and through training and optimization, a bonding path planning model is obtained that calculates the movement path, speed and force of the bonding tool during chip bonding based on the feature points of the bonding tool and the bonding area.

7. The dual-camera precision positioning method in a bonding system according to claim 1, characterized in that: The step of executing the chip bonding instruction by the bonding tool and using the first camera and the second camera to monitor in real time the bonding process of the bonding tool performing the bonding operation according to the current moving path includes: Based on the chip bonding instruction, driving the bonding tool to move along the current moving path to perform a chip bonding operation; During the process of the bonding tool moving and performing the bonding operation, continuously monitoring the position and posture of the bonding tool and the state of the bonding area in real time by the first camera and the second camera; Calculating position and posture deviations using a bonding path planning model based on the current movement path and real-time monitoring data; Establishing a correspondence between the expected feature point coordinates of the bonding tool on the current moving path output by the bonding path planning model and the actual feature point coordinates monitored in real time by the first camera, and determining the position deviation between the bonding tool and the expected feature point coordinates using a coordinate transformation method; Determining a posture deviation of the bonding tool from the expected posture based on an expected posture of the bonding tool at each node obtained in the movement path output by the bonding path planning model and a current actual posture of the prime bonding tool calculated by a posture estimation algorithm using real-time monitoring of the bonding tool using the first camera; Based on the position deviation and the posture deviation results, the moving path or bonding operation of the bonding tool is adjusted through a bonding path planning model.

8. A dual-camera precision positioning device in a bonding system, characterized in that: The dual-camera precision positioning device in the bonding system is used to perform the dual-camera precision positioning method in the bonding system according to any one of claims 1 to 7, and the device includes: a data acquisition module, configured to acquire and continuously monitor the bonding tool and the bonding interface through the first camera and the second camera respectively, to obtain a bonding image; a data processing module, configured to perform image processing on the bonding image and extract features to obtain the bonding tool position, posture, and geometric features of the bonding area; An information output module is configured to obtain a current movement path of the bonding tool based on the position and posture of the bonding tool and the geometric features of the bonding area through a pre-built bonding path planning model, and output an instruction for controlling the bonding tool to perform chip bonding according to the current movement path; The information monitoring module is used to execute the chip bonding instruction through the bonding tool, and use the first camera and the second camera to monitor in real time the bonding process of the bonding tool performing the bonding operation according to the current moving path.

9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the dual-camera precision positioning method in the bonding system according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the dual-camera precision positioning method in a bonding system according to any one of claims 1 to 7 are implemented.

Images