Method and system for accurately taking bonded silver wires based on computer vision

Abstract

The invention discloses a bonding silver wire accurate material taking method and system based on computer vision, and relates to the technical field of semiconductor packaging, and the method comprises the steps: collecting image features between a bonding silver wire and a material taking device through a sensor module; based on the image features, the position and morphological features of the bonding silver wire are judged; establishing a virtual material taking characteristic table to judge the real-time state of the bonding silver wire; and the material taking type is judged according to the real-time state of the bonding silver wire, and accurate material taking is carried out. According to the invention, image features are collected through the sensor module, and high-precision real-time monitoring of the bonding silver wire is realized; an advanced computer vision algorithm and a deep learning technology are utilized to accurately judge the position and morphological characteristics of the bonding silver wire; a virtual material taking feature table is established, dynamic tracking and prediction of the state of the bonded silver wire are achieved, and the stability and controllability of the material taking process are improved; and according to the real-time state of the bonding silver wire, a grading processing mechanism is adopted for accurate material taking, so that the material taking precision is improved, and the response capability of the system to abnormal conditions is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of semiconductor packaging technology, and in particular to a method and system for accurately picking up bonding silver wires based on computer vision. Background Art

[0002] With the rapid development of the microelectronics manufacturing industry, silver bonding wire plays a key role in the field of semiconductor packaging as an important interconnect material. Traditional silver bonding wire picking technology mainly relies on manual operation or simple mechanical automation systems, and these methods have obvious limitations in terms of accuracy and efficiency. In recent years, with the continuous advancement of computer vision technology, its application in the field of industrial automation has become increasingly extensive. However, there are still many challenges in applying computer vision technology to the precise picking of silver bonding wires. First, the tiny size and high reflectivity of silver bonding wires make image acquisition and processing complicated; second, silver bonding wires may deform or displace during the picking process, making real-time positioning difficult; third, existing visual algorithms often have delays and insufficient accuracy when processing high-speed dynamic scenes. In addition, due to the particularity of silver bonding wire materials, its surface condition and tension changes have a significant impact on the quality of picking, but it is difficult for existing technologies to accurately monitor and control these parameters.

[0003] In order to overcome the above technical difficulties, researchers have proposed various improvement plans. For example, multispectral imaging technology is used to improve image quality, deep learning algorithms are introduced to optimize target recognition and tracking, and high-speed cameras and parallel processing technology are used to reduce system delays. However, these methods still have some shortcomings in practical applications. For example, most systems lack the ability to comprehensively evaluate the state of the bonding silver wire, and it is difficult to dynamically adjust the material picking strategy according to real-time conditions; at the same time, the existing technology has limited adaptive capabilities in dealing with abnormal situations, and often requires manual intervention, which affects production efficiency. In addition, most current vision systems focus on 2D image processing and fail to make full use of 3D information to improve material picking accuracy.

[0004] In addition, the existing methods still have room for improvement in integrating multiple sensor data to achieve more accurate state judgment and control. Therefore, developing an intelligent system based on advanced computer vision technology that can achieve accurate material collection of bonding silver wires is of great significance to improving the automation level of semiconductor packaging processes and product quality. Summary of the invention

[0005] In view of the problems existing in the existing bonding silver wire material extraction technology, the present invention is proposed.

[0006] Therefore, the problem to be solved by the present invention is how to achieve accurate material collection of bonding silver wire based on computer vision to improve the material collection accuracy, efficiency and adaptability.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] In the first aspect, an embodiment of the present invention provides a method for accurately picking up bonding silver wires based on computer vision, which includes collecting image features between the bonding silver wires and a picking device through a sensor module; judging the position and morphological features of the bonding silver wires based on the image features; establishing a virtual picking feature table to judge the real-time status of the bonding silver wires; and judging the picking type according to the real-time status of the bonding silver wires for accurate picking.

[0009] As a preferred solution of the method for accurately sampling bonding silver wires using computer vision described in the present invention, the image features include the shape, surface condition, texture, color, diameter, length and curvature of the bonding silver wire; the image feature collection process includes the following steps: capturing the surface condition of the bonding silver wire through a high-resolution image sensor to generate a 2D image; based on the 2D image, measuring the diameter and length of the bonding silver wire through a depth sensor to form a 3D model of the bonding silver wire, and combining the data to form image features to determine the position and morphological characteristics of the bonding silver wire.

[0010] As a preferred solution of the method for precise material extraction of bonding silver wires using computer vision described in the present invention, the method comprises: judging the position and morphological characteristics of the bonding silver wire comprises, based on the image characteristics, using a computer vision algorithm, starting a multimodal sensor array, collecting and analyzing the data of the 2D image and 3D model of the bonding silver wire; preprocessing the data of the 2D image and the 3D model, and processing the preprocessed RGB image using a deep learning semantic segmentation technique to identify the edge of the bonding silver wire; combining the depth information obtained by the laser radar to obtain the actual distance between the bonding silver wire and the material extraction device; calculating the 3D coordinates of the bonding silver wire and the 3D coordinates of the material extraction device using a computational geometry method based on the actual distance and the 2D coordinates of the edge of the bonding silver wire; determining the actual position and shape of the bonding silver wire using a regression model based on machine learning based on the 3D coordinates of the bonding silver wire and the 3D coordinates of the material extraction device; the preprocessing comprises denoising, filtering and brightness adjustment.

[0011] As a preferred solution of the method for precise material collection of bonding silver wires by computer vision described in the present invention, the process of establishing the virtual material collection feature table includes the following steps: calculating the offset a of the bonding silver wire according to the actual position and expected position of the bonding silver wire; collecting the offset a, diameter, length, surface condition and bending degree of the bonding silver wire at the same time, and calculating the instantaneous volume of the bonding silver wire; judging whether the material collection point is the optimal position based on the instantaneous volume, and recording all the collected and calculated information in the virtual material collection feature table; judging the real-time status of the bonding silver wire according to the information in the virtual material collection feature table; and judging the material collection type according to the real-time status.

[0012] As a preferred embodiment of the method for accurately picking up bonding silver wires by computer vision of the present invention, the picking type is judged as follows: when the offset a>0.1mm, the bonding silver wire state is detected; if the bonding silver wire diameter deviation exceeds 0.01mm, and the bonding silver wire length deviation exceeds 0.1mm, the bonding silver wire state is judged to be B1 level; when 0.05mm≤offset a≤0.1mm, if the surface condition and bending degree of the bonding silver wire are both in abnormal trends, the bonding silver wire state is judged to be B2 level; when 0.05mm≤offset a≤0.1mm, if the bonding silver wire diameter deviation is ≤0.01mm, and the bonding silver wire length deviation is ≤0.1mm, the bonding silver wire state is judged to be B3 level; if the bonding silver wire state is B1 level, the picking operation is stopped immediately, the abnormal area is automatically calibrated, and the operator is guided to perform Adjust, and during the adjustment process, monitor the adjustment progress in real time; if the state of the bonding silver wire is level B2, judge the state of the bonding silver wire according to the real-time information of the virtual material picking feature table, adjust the position of the material picking device and the actual speed v of the material picking device, and adjust the actual angle θ of the material picking device; if the surface condition and bending degree of the bonding silver wire cannot be stopped after adjustment and are in an abnormal trend at the same time, the state of the bonding silver wire will be transferred to level B1 for processing; if the state of the bonding silver wire is level B3, continue to monitor the state of the bonding silver wire, adjust the material picking parameters in real time, and prevent material picking problems caused by changes in the state of the bonding silver wire; if the offset a, the diameter of the bonding silver wire or the length of the bonding silver wire L shows an abnormal trend in future tests, the state of the bonding silver wire will be transferred to level B2 for processing; the abnormal trend includes abnormal results in any of the last five consecutive tests, which are judged to be in an abnormal trend.

[0013] As a preferred solution of the computer vision bonding silver wire precise material picking method of the present invention, wherein: the state of the bonding silver wire is judged based on the real-time information of the virtual material picking feature table, by calculating the offset a, diameter, length, surface condition and bending degree of the bonding silver wire; the specific formula of the offset a is as follows:

[0014]

[0015] Wherein, a is the offset of the bonding silver wire, x is the actual x-coordinate of the bonding silver wire, x0 is the expected x-coordinate of the bonding silver wire, y is the actual y-coordinate of the bonding silver wire, y0 is the expected y-coordinate of the bonding silver wire, z is the actual z-coordinate of the bonding silver wire, z0 is the expected z-coordinate of the bonding silver wire, α is the angle deviation weight coefficient, θ is the actual angle of the material picking device, θ0 is the expected angle of the material picking device, v is the actual speed of the material picking device, v0 is the expected speed of the material picking device, β is the speed deviation weight coefficient, v max is the maximum speed of the material taking device, and L is the length of the bonding silver wire.

[0016] As a preferred solution of the method for accurately picking up bonding silver wires by computer vision described in the present invention, wherein: the judging the state of the bonding silver wire also includes, when the state of the bonding silver wire is in a static picking state, accurately measuring and controlling the position of the picking point by a pre-picking posture device; if the position of the picking point deviates from a preset range, adjusting the posture of the picking device and correcting the position of the picking point; if the adjusted position still deviates from the preset range, real-time monitoring and adjusting the tension of the bonding silver wire by a bonding silver wire tension monitor; if the tension exceeds a preset threshold, adjusting the picking point position and the picking device posture; if the picking point position and the bonding silver wire tension still cannot be restored to the preset range, starting an alarm machine The material picking operation is suspended for manual inspection and processing; when the bonding silver wire is in a dynamic material picking state, the moving speed of the bonding silver wire is monitored in real time by the bonding silver wire speed monitor; if the moving speed exceeds the preset range, the speed of the material picking device is adjusted; if the adjusted speed still exceeds the preset speed, the moving trajectory of the bonding silver wire is monitored in real time by the bonding silver wire trajectory monitor; if the trajectory deviates from the preset range, the actual speed v of the material picking device is adjusted by the material picking speed control system according to the bonding silver wire speed and trajectory conditions; if the bonding silver wire speed and trajectory still cannot be restored to the preset range, the system will activate the alarm mechanism and suspend the material picking operation for manual inspection and processing.

[0017] In the second aspect, an embodiment of the present invention provides a system for precise material picking of bonding silver wires based on computer vision, which includes: a sensor module, used to collect image features between the bonding silver wire and a picking device through the sensor module; an image processing module, used to determine the position and morphological features of the bonding silver wire based on the image features; a state judgment module, used to establish a virtual picking feature table to determine the real-time state of the bonding silver wire; and a picking control module, used to determine the material type according to the real-time state of the bonding silver wire for precise material picking.

[0018] In a third aspect, an embodiment of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program instructions are executed by the processor, the steps of the method for precise material extraction of bonding silver wire using computer vision as described in the first aspect of the present invention are implemented.

[0019] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program instructions are executed by a processor, the steps of the method for accurately picking up bonding silver wire using computer vision as described in the first aspect of the present invention are implemented.

[0020] The beneficial effects of the present invention are as follows: the present invention collects image features through the sensor module to realize high-precision real-time monitoring of the bonding silver wire; uses advanced computer vision algorithms and deep learning technology to accurately judge the position and morphological characteristics of the bonding silver wire; establishes a virtual material collection feature table to realize dynamic tracking and prediction of the state of the bonding silver wire, and improves the stability and controllability of the material collection process; according to the real-time state of the bonding silver wire, a hierarchical processing mechanism is used for accurate material collection, which not only improves the material collection accuracy, but also enhances the system's ability to respond to abnormal situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0022] Figure 1 This is a flow chart of the method for accurately picking up bonding silver wire using computer vision in Example 1.

[0023] Figure 2 This is the overall system structure diagram of the method for accurately picking up bonding silver wire using computer vision in Example 2. DETAILED DESCRIPTION

[0024] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0025] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0026] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0027] Example 1

[0028] Reference Figure 1 , which is the first embodiment of the present invention, provides a method for accurately picking bonding silver wire based on computer vision, including:

[0029] S1: Collect image features between the bonding silver wire and the material removal device through the sensor module.

[0030] Specifically, the image features include the shape, surface condition, texture, color, diameter, length and curvature of the bonding silver wire.

[0031] Furthermore, the image feature collection process includes the following steps: capturing the surface condition of the bonding silver wire through a high-resolution image sensor to generate a 2D image; based on the 2D image, measuring the diameter and length of the bonding silver wire through a depth sensor to form a 3D model of the bonding silver wire, and combining the data to form image features to determine the position and morphological characteristics of the bonding silver wire.

[0032] Furthermore, the high-resolution image sensor captures the surface details of the silver bonding wire at 4K resolution (3840x2160 pixels) to generate a 2D image. The depth sensor uses structured light technology with an accuracy of 0.01mm to measure the distance information of the silver bonding wire and form 3D point cloud data. The lidar uses time-of-flight (ToF) technology with an accuracy of 0.1mm to obtain the precise depth information of the silver bonding wire.

[0033] S2: Based on the image features, determine the position and morphological features of the bonding silver wire.

[0034] Specifically, determining the position and morphological features of the bonding silver wire includes: based on the image features, using a computer vision algorithm, starting a multimodal sensor array to collect and analyze the 2D image and 3D model data of the bonding silver wire; preprocessing the 2D image and 3D model data, and using deep learning semantic segmentation technology to process the preprocessed RGB image to identify the edge of the bonding silver wire; combining the depth information obtained by the lidar to obtain the actual distance between the bonding silver wire and the material picking device; using computational geometry methods to calculate the 3D coordinates of the bonding silver wire and the 3D coordinates of the material picking device based on the actual distance and the 2D coordinates of the edge of the bonding silver wire; and using a regression model based on machine learning to determine the actual position and shape of the bonding silver wire based on the 3D coordinates of the bonding silver wire and the 3D coordinates of the material picking device.

[0035] Furthermore, the 3D model of the bonding silver wire is reconstructed using the triangulation principle in combination with the 2D image segmentation results and depth information. The iterative closest point (ICP) algorithm is used to align the point cloud data to improve the 3D reconstruction accuracy. The RANSAC algorithm is used to fit the center line of the bonding silver wire and calculate its precise position and posture in 3D space.

[0036] It should be noted that the preprocessing includes denoising, filtering and brightness adjustment; Gaussian filtering is applied to remove image noise, adaptive histogram equalization method is used to adjust image brightness and contrast, and a deep learning model based on U-Net architecture is used for semantic segmentation to accurately identify the edges of bonding silver wires. This model has been trained on a large number of bonding silver wire images, and the recognition accuracy has reached 99.5%.

[0037] S3: Establish a virtual material feature table to determine the real-time status of the bonding silver wire.

[0038] Specifically, the process of establishing the virtual material picking feature table includes the following steps: calculating the offset a of the bonding silver wire according to the actual position and expected position of the bonding silver wire; collecting the offset a, diameter, length, surface condition and bending degree of the bonding silver wire at the same time, and calculating the instantaneous volume of the bonding silver wire; judging whether the material picking point is the optimal position based on the instantaneous volume, and recording all the collected and calculated information in the virtual material picking feature table; judging the real-time status of the bonding silver wire according to the information in the virtual material picking feature table; and judging the material picking type according to the real-time status.

[0039] Furthermore, the specific formula of the offset a is as follows:

[0040]

[0041] Wherein, a is the offset of the bonding silver wire, x is the actual x-coordinate of the bonding silver wire, x0 is the expected x-coordinate of the bonding silver wire, y is the actual y-coordinate of the bonding silver wire, y0 is the expected y-coordinate of the bonding silver wire, z is the actual z-coordinate of the bonding silver wire, z0 is the expected z-coordinate of the bonding silver wire, α is the angle deviation weight coefficient, θ is the actual angle of the material picking device, θ0 is the expected angle of the material picking device, v is the actual speed of the material picking device, v0 is the expected speed of the material picking device, β is the speed deviation weight coefficient, v max is the maximum speed of the material taking device, and L is the length of the bonding silver wire.

[0042] Furthermore, the material type is judged as follows: when the offset a>0.1mm, the state of the bonding silver wire is detected; if the diameter deviation of the bonding silver wire exceeds 0.01mm, and the length deviation of the bonding silver wire exceeds 0.1mm, the state of the bonding silver wire is judged to be B1; when 0.05mm≤offset a≤0.1mm, if the surface condition and bending degree of the bonding silver wire are both in an abnormal trend, the state of the bonding silver wire is judged to be B2; when 0.05mm≤offset a≤0.1mm, if the diameter deviation of the bonding silver wire is ≤0.01mm, and the length deviation of the bonding silver wire is ≤0.1mm, the state of the bonding silver wire is judged to be B3.

[0043] Specifically, if the state of the bonding silver wire is at level B1, the material picking operation is stopped immediately, the abnormal area is automatically calibrated, the operator is guided to make adjustments, and during the adjustment process, the adjustment progress is monitored in real time; if the state of the bonding silver wire is at level B2, the state of the bonding silver wire is judged according to the real-time information of the virtual material picking feature table, the position of the material picking device and the actual speed v of the material picking device are adjusted, and the actual angle θ of the material picking device is adjusted; if the bonding silver wire cannot be stopped after adjustment and the surface condition and bending degree of the bonding silver wire are in an abnormal trend at the same time, the state of the bonding silver wire is transferred to level B1 processing; if the state of the bonding silver wire is at level B3, the state of the bonding silver wire is continuously monitored, and the material picking parameters are adjusted in real time to prevent material picking problems caused by changes in the state of the bonding silver wire; if the offset a, the diameter of the bonding silver wire or the length L of the bonding silver wire show an abnormal trend in future detection, the state of the bonding silver wire is transferred to level B2 processing.

[0044] Furthermore, as shown in Table 1, it can be seen that over time, the various indicators of the silver wire gradually deteriorated: the offset increased from 0.03mm to 0.11mm, the diameter and length increased slightly, the surface condition score dropped from 98 to 90, the bending degree increased, and the instantaneous volume increased. The material type also gradually dropped from the best B3 level to the B1 level that needs to be stopped immediately. This fast and accurate state tracking demonstrates the efficient monitoring capability of the system, which can promptly detect and respond to abnormal changes in the material retrieving process, providing an important basis for ensuring material retrieving quality and process optimization.

[0045] Table 1 Virtual material extraction characteristics table

[0046]

[0047] It should be noted that the abnormal trend includes abnormal results in any of the last five consecutive tests, which is judged to be in an abnormal trend.

[0048] S4: Determine the material type based on the real-time status of the bonding silver wire and take the material accurately.

[0049] Specifically, the state of the bonding silver wire is judged based on the real-time information of the virtual material picking feature table, by calculating the offset a, diameter, length, surface condition and bending degree of the bonding silver wire; judging the state of the bonding silver wire also includes, when the state of the bonding silver wire is in a static material picking state, accurately measuring and controlling the position of the material picking point through the pre-material picking posture device; if the position of the material picking point deviates from the preset range, adjusting the posture of the material picking device and correcting the position of the material picking point; if the adjusted position still deviates from the preset range, real-time monitoring and adjusting the tension of the bonding silver wire through the bonding silver wire tension monitor; if the tension exceeds the preset threshold, adjusting the position of the material picking point and the posture of the material picking device; if the position of the material picking point and the tension of the bonding silver wire still cannot be restored to the preset range, the alarm mechanism is activated to suspend the material picking operation for manual inspection and processing.

[0050] It should be noted that the preset threshold is to ensure that the bonding silver wire maintains appropriate tension during the material picking process, neither too loose to cause inaccurate material picking, nor too tight to cause damage or deformation of the silver wire. It is used to determine whether the current material picking operation is safe and accurate.

[0051] Furthermore, when the bonding silver wire is in a dynamic material-picking state, the moving speed of the bonding silver wire is monitored in real time by a bonding silver wire speed monitor; if the moving speed exceeds a preset range, the speed of the material-picking device is adjusted; if the adjusted speed still exceeds the preset speed, the moving trajectory of the bonding silver wire is monitored in real time by a bonding silver wire trajectory monitor; if the trajectory deviates from the preset range, the actual speed v of the material-picking device is adjusted by a material-picking speed control system according to the bonding silver wire speed and trajectory conditions; if the bonding silver wire speed and trajectory still cannot be restored to the preset range, the system will activate an alarm mechanism and suspend the material-picking operation for manual inspection and processing.

[0052] Example 2

[0053] Reference Figure 2 , is an embodiment of the present invention, and provides a system for accurately picking up bonding silver wires based on computer vision, including: a sensor module, used to collect image features between the bonding silver wire and a picking device through the sensor module; an image processing module, used to determine the position and morphological features of the bonding silver wire based on the image features; a state judgment module, used to establish a virtual picking feature table to determine the real-time state of the bonding silver wire; and a picking control module, used to determine the picking type according to the real-time state of the bonding silver wire for accurate picking.

[0054] This embodiment also provides a computer device, which is suitable for the method of accurately picking up bonding silver wires using computer vision, and includes a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the method of accurately picking up bonding silver wires using computer vision as proposed in the above embodiment.

[0055] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0056] The present embodiment also provides a storage medium having a computer program stored thereon, which implements the following steps when executed by a processor: collecting image features between the bonding silver wire and the material picking device through a sensor module; judging the position and morphological features of the bonding silver wire based on the image features; establishing a virtual material picking feature table to judge the real-time status of the bonding silver wire; judging the material picking type according to the real-time status of the bonding silver wire for precise material picking.

[0057] In summary, the present invention collects image features through the sensor module to achieve high-precision real-time monitoring of the bonding silver wire; uses advanced computer vision algorithms and deep learning technology to accurately determine the position and morphological characteristics of the bonding silver wire; establishes a virtual material collection feature table to achieve dynamic tracking and prediction of the state of the bonding silver wire, and improves the stability and controllability of the material collection process; according to the real-time state of the bonding silver wire, a hierarchical processing mechanism is used for accurate material collection, which not only improves the material collection accuracy, but also enhances the system's ability to respond to abnormal situations.

[0058] Example 3

[0059] Referring to Table 2, which is the second embodiment of the present invention, this embodiment provides a method for accurately picking up bonding silver wires based on computer vision. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0060] Specifically, the experimental device includes a high-precision bonding machine, a multimodal sensor array (including high-resolution image sensors, depth sensors and laser radars), a computer vision processing system and a precision material handling device. The bonding silver wire uses high-purity silver wire with a diameter of 0.025mm and a length of 10mm.

[0061] Furthermore, as shown in Table 2, the material collection success rate: under normal conditions, the material collection success rate of the method of the present invention reaches 99.2%, which is 6.7 percentage points higher than that of the traditional method. Under interference conditions, the method of the present invention still maintains a high success rate of 97.8%, while the traditional method drops to 78.3%. This fully demonstrates the stability and reliability of the method of the present invention in complex environments. In the average material collection time, the average material collection time of the method of the present invention under normal conditions is 2.1 seconds, which is 34.4% faster than the traditional method. Even under interference conditions, the material collection time of the method of the present invention is only increased to 2.4 seconds, which is still faster than the 3.2 seconds of the traditional method under normal conditions. This shows that the method of the present invention not only improves the accuracy, but also significantly improves the efficiency.

[0062] Table 2 Experimental data table

[0063]

[0064] Furthermore, position accuracy: the method of the present invention achieves a high accuracy of 5μm under normal conditions, which is three times the 15μm accuracy of the traditional method. Under interference conditions, the method of the present invention still maintains an accuracy of 8μm, which is much better than the performance of the traditional method under normal conditions. This is due to the high-precision multimodal sensor array and advanced computer vision algorithms used in the present invention. Surface damage rate: The method of the present invention controls the surface damage rate at 0.5% (normal conditions) and 0.8% (interference conditions), which are 82.1% and 84.6% lower than the traditional method, respectively. This greatly improves product quality and yield rate, and can bring considerable economic benefits to the enterprise.

[0065] Specifically, the abnormality recognition rate: the method of the present invention maintains a high abnormality recognition rate of more than 96% under both normal and interference conditions, while the traditional method only has 65% and 55%. This shows that the method of the present invention can detect and deal with potential problems in a timely manner, greatly reducing production risks.

[0066] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A method for accurately picking bonding silver wire based on computer vision, characterized in that: include, The image features between the bonding silver wire and the material taking device are collected by the sensor module; Based on the image features, determining the position and morphological features of the bonding silver wire; Establish a virtual material feature table to determine the real-time status of the bonding silver wire; Determine the material type based on the real-time status of the bonding silver wire and take the material accurately.

2. The computer vision bonding silver wire accurate material extraction method as claimed in claim 1, characterized in that: The image features include the shape, surface condition, texture, color, diameter, length and curvature of the bonding silver wire; and the process of collecting the image features includes the following steps: The surface condition of the silver bonding wire is captured by a high-resolution image sensor to generate a 2D image. Based on the 2D image, the diameter and length of the bonding silver wire are measured by a depth sensor to form a 3D model of the bonding silver wire, and the data are combined to form image features to determine the position and morphological features of the bonding silver wire.

3. The computer vision bonding silver wire accurate material extraction method as claimed in claim 2, characterized in that: The determination of the position and morphological characteristics of the bonding silver wire includes: Based on the image features, a multimodal sensor array is activated using a computer vision algorithm to collect and analyze data of a 2D image and a 3D model of the bonding silver wire; Preprocessing the data of the 2D image and the 3D model, and processing the preprocessed RGB image using a semantic segmentation technique of deep learning to identify the edge of the bonding silver wire; Combined with the depth information obtained by the laser radar, the actual distance between the bonding silver wire and the material removal device is obtained; Calculating the 3D coordinates of the bonding silver wire and the 3D coordinates of the material removal device by using a computational geometry method according to the actual distance and the 2D coordinates of the edge of the bonding silver wire; According to the 3D coordinates of the silver bonding wire and the 3D coordinates of the material removal device, the actual position and shape of the silver bonding wire are determined using a regression model based on machine learning; The preprocessing includes denoising, filtering and brightness adjustment.

4. The computer vision bonding silver wire accurate material extraction method as claimed in claim 1, characterized in that: The process of establishing the virtual material extraction feature table includes the following steps: According to the actual position and expected position of the bonding silver wire, the offset a of the bonding silver wire is calculated; At the same time, the offset a, diameter, length, surface condition and bending degree of the bonding silver wire are collected to calculate the instantaneous volume of the bonding silver wire; Based on the instantaneous volume, determine whether the material taking point is the optimal position, and record all the collected and calculated information in a virtual material taking feature table; According to the information in the virtual material extraction feature table, the real-time status of the bonding silver wire is determined; At the same time, the material type is determined according to the real-time status.

5. The computer vision-based bonding silver wire accurate material extraction method as claimed in claim 4, characterized in that: The determination of the material type comprises: When the offset a>0.1mm, the state of the bonding silver wire is detected; if the diameter deviation of the bonding silver wire exceeds 0.01mm, and the length deviation of the bonding silver wire exceeds 0.1mm, the state of the bonding silver wire is judged to be B1; when 0.05mm≤offset a≤0.1mm, if the surface condition and bending degree of the bonding silver wire are both in an abnormal trend, the state of the bonding silver wire is judged to be B2; when 0.05mm≤offset a≤0.1mm, if the diameter deviation of the bonding silver wire is ≤0.01mm, and the length deviation of the bonding silver wire is ≤0.1mm, the state of the bonding silver wire is judged to be B3; If the bonding silver wire status is B1, the material removal operation will be stopped immediately, the abnormal area will be automatically calibrated, the operator will be guided to make adjustments, and the adjustment progress will be monitored in real time during the adjustment process; If the state of the bonding silver wire is level B2, the state of the bonding silver wire is judged according to the real-time information of the virtual material taking characteristic table, and the position of the material taking device and the actual speed v of the material taking device are adjusted, and the actual angle θ of the material taking device is adjusted at the same time; If the surface condition and bending degree of the silver bonding wire cannot be stopped after adjustment and are in an abnormal trend at the same time, the silver bonding wire state will be transferred to B1 level treatment; If the state of the silver bonding wire is B3, the state of the silver bonding wire is continuously monitored, and the material collection parameters are adjusted in real time to prevent material collection problems caused by changes in the state of the silver bonding wire; if the offset a, the diameter of the silver bonding wire or the length of the silver bonding wire L shows an abnormal trend in future tests, the state of the silver bonding wire is transferred to B2 level processing; The abnormal trend includes showing an abnormal result in any of the last five consecutive tests, and being judged as being in an abnormal trend.

6. The computer vision-based bonding silver wire accurate material extraction method as claimed in claim 5, characterized in that: The state of the bonding silver wire is determined by calculating the offset a, diameter, length, surface condition and bending degree of the bonding silver wire according to the real-time information of the virtual material feature table; the specific formula of the offset a is as follows: Wherein, a is the offset of the bonding silver wire, x is the actual x-coordinate of the bonding silver wire, x0 is the expected x-coordinate of the bonding silver wire, y is the actual y-coordinate of the bonding silver wire, y0 is the expected y-coordinate of the bonding silver wire, z is the actual z-coordinate of the bonding silver wire, z0 is the expected z-coordinate of the bonding silver wire, α is the angle deviation weight coefficient, θ is the actual angle of the material picking device, θ0 is the expected angle of the material picking device, v is the actual speed of the material picking device, v0 is the expected speed of the material picking device, β is the speed deviation weight coefficient, v max is the maximum speed of the material taking device, and L is the length of the bonding silver wire.

7. The method for accurately picking bonding silver wires by computer vision as claimed in claim 6, characterized in that: The determining of the state of the bonding silver wire further comprises: When the bonding silver wire is in a static feeding state, the position of the feeding point is accurately measured and controlled by the pre-feeding posture device; if the feeding point position deviates from the preset range, the posture of the feeding device is adjusted to correct the feeding point position; If the adjusted position still deviates from the preset range, the tension of the bonding wire is monitored and adjusted in real time through the bonding wire tension monitor; if the tension exceeds the preset threshold, the position of the material taking point and the posture of the material taking device are adjusted; If the position of the material taking point and the tension of the bonding silver wire still cannot be restored to the preset range, the alarm mechanism will be activated to suspend the material taking operation for manual inspection and processing; When the bonding silver wire is in a dynamic material taking state, the moving speed of the bonding silver wire is monitored in real time by a bonding silver wire speed monitor; if the moving speed exceeds a preset range, the speed of the material taking device is adjusted; if the adjusted speed still exceeds the preset speed, the moving trajectory of the bonding silver wire is monitored in real time by a bonding silver wire trajectory monitor; if the trajectory deviates from the preset range, the actual speed v of the material taking device is adjusted by a material taking speed control system according to the bonding silver wire speed and trajectory conditions; If the speed and trajectory of the bonding wire still cannot be restored to the preset range, the system will activate the alarm mechanism and suspend the material removal operation for manual inspection and processing.

8. A system for accurately picking bonding silver wires based on computer vision, and a method for accurately picking bonding silver wires based on computer vision according to any one of claims 1 to 7, characterized in that: include, A sensor module, used for collecting image features between the bonding silver wire and the material taking device through the sensor module; An image processing module, used for determining the position and morphological features of the bonding silver wire based on the image features; A state judgment module is used to establish a virtual material extraction feature table to judge the real-time state of the bonding silver wire; The material picking control module is used to determine the material picking type according to the real-time status of the bonding silver wire and to pick the material accurately.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for accurately picking bonding silver wires using computer vision as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for accurately picking bonding silver wires using computer vision as described in any one of claims 1 to 7 are implemented.

Images