Real-time Image Segmentation Method for Gold Wire Bonding

Abstract

The present invention relates to the field of image processing technology, specifically a real-time image segmentation method for wire bonding, which includes the following steps: input a sequence of wire images, perform Fourier transform on the sequence of wire images, extract frequency domain amplitude and phase information, calculate and superimpose the phase difference between adjacent frames to obtain the wire edge spectrum; enhance the gradient of the wire edge spectrum to generate an enhanced edge feature map. In the present invention, the gradient enhancement strategy strengthens the gradient change characteristics of the edge region, making the wire region more distinguishable under complex backgrounds. The calculation of the brightness gradient combined with the adaptive threshold adjustment can dynamically adapt to the characteristic changes of the wire region under different lighting conditions and improve the stability of classification. The super-resolution reconstruction uses spatial high-frequency information to improve the image resolution, making the edge contour sharper and reducing the structural blur problem caused by low resolution.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a real-time image segmentation method for wire bonding. Background Art

[0002] Image processing is a research field based on computer vision and digital signal processing technologies, involving operations such as image data acquisition, analysis, transformation, enhancement, segmentation, and recognition. The real-time image segmentation method for wire bonding belongs to a part of image processing technology, mainly used for detecting and analyzing the position, shape, and motion trajectory of wires during the electronic packaging process.

[0003] However, in the prior art, the displacement calculation method in the spatial domain fails to fully consider the temporal characteristics, resulting in a weak ability to capture dynamic change information, and the wire motion trajectory is easily interfered by background noise. In the case of low image resolution, the loss of edge information reduces the accuracy of motion feature extraction, affecting the subsequent segmentation accuracy. Therefore, improvements are needed. Summary of the Invention

[0004] The purpose of the present invention is to solve the drawbacks existing in the prior art, and a real-time image segmentation method for wire bonding is proposed.

[0005] To achieve the above purpose, the present invention adopts the following technical solutions. The real-time image segmentation method for wire bonding includes the following steps:

[0006] Input a sequence of wire images, perform Fourier transform on the sequence of wire images, extract frequency domain amplitude and phase information, calculate and superimpose the phase difference between adjacent frames to obtain the wire edge spectrum; enhance the gradient of the wire edge spectrum to generate an enhanced edge feature map;

[0007] Based on the enhanced edge feature map, obtain the brightness gradient value, set an adaptive threshold for image block classification to obtain the wire region distribution map; perform super-resolution reconstruction on the wire region distribution map to generate a wire super-resolution map;

[0008] Based on the wire super-resolution map, calculate the displacement vector of wire region pixel points between adjacent frames to establish the wire motion feature; perform temporal prediction on the wire motion feature to generate the wire motion trajectory;

[0009] Based on the wire motion trajectory, calculate the set of wire region boundary points, and combine the edge gradient information in the wire super-resolution map to perform noise suppression to generate the wire segmentation boundary.

[0010] Preferably, the step of obtaining the wire edge spectrum is as follows:

[0011] Input the gold wire image sequence, convert each frame of the image into the frequency domain through Fourier transform, extract the amplitude and phase information of each frame, and obtain the frequency domain feature data;

[0012] According to the frequency domain feature data, perform a difference on the phase information between every two frames, calculate the phase change amount, and the calculation formula is:

[0013] ;

[0014] Wherein, is the phase change amount, is the phase information of the th frame, is the total number of frames, is the influence constant;

[0015] Based on the phase change amount, apply high-pass filtering to enhance the gold wire edge feature and obtain the gold wire edge spectrum.

[0016] Preferably, the steps for obtaining the enhanced edge feature map are as follows:

[0017] Receive the gold wire edge spectrum, calculate the horizontal gradient component and the vertical gradient component respectively, calculate the gradient value of each pixel point through convolution operation using the gradient operator, and synthesize the overall gradient amplitude to obtain the original edge intensity map;

[0018] Based on the original edge intensity map, calculate the edge enhancement value, and the calculation formula is:

[0019] ;

[0020] Wherein, is the edge enhancement value, is the horizontal gradient component of the pixel point , is the vertical gradient component of the pixel point , is the constant to avoid division by zero error;

[0021] Based on the edge enhancement value, suppress noise through non-linear filtering and generate an enhanced edge feature map.

[0022] Preferably, the steps for obtaining the gold wire region distribution map are as follows:

[0023] Receive the enhanced edge feature map, obtain the brightness gradient value of each pixel, extract the gradient components in the horizontal and vertical directions, and obtain the brightness gradient value map;

[0024] Based on the brightness gradient value map, calculate the adaptive threshold, and the calculation formula is:

[0025] ;

[0026] where T is the adaptive threshold, is the luminance gradient value of the pixel point , and are the number of rows and columns of the image respectively, is a constant to avoid division by zero;

[0027] Based on the adaptive threshold, perform image block classification on the luminance gradient value map, remove isolated noise blocks, determine the distribution of the gold wire area, and generate a gold wire area distribution map.

[0028] Preferably, the steps for obtaining the gold wire super-resolution map are as follows:

[0029] Receive the gold wire area distribution map, analyze the edge detail information of the local area, and obtain a preliminary super-resolution image through image interpolation correction;

[0030] Based on the preliminary super-resolution image, perform noise removal and contrast equalization to generate a gold wire super-resolution map.

[0031] Preferably, the steps for obtaining the gold wire motion characteristics are as follows:

[0032] Receive the gold wire super-resolution map, locate the boundary pixel points of the gold wire area, analyze the pixel point displacement between adjacent frames, obtain the pixel point trajectory, and get the gold wire area pixel point trajectory set;

[0033] Based on the gold wire area pixel point trajectory set, obtain the pixel point displacement vector and generate the gold wire motion characteristics.

[0034] Preferably, the steps for obtaining the gold wire motion trajectory are as follows:

[0035] Receive the gold wire motion characteristics, extract the displacement vector data in the time series, construct the time-dependent relationship, and obtain the motion state sequence;

[0036] Based on the motion state sequence, calculate the gold wire position, and the expression is:

[0037] ;

[0038] where is the gold wire position at time , is the velocity at the previous moment, is the acceleration at the previous moment, is the time interval, is the random perturbation term;

[0039] Based on the gold wire position, perform trajectory reconstruction on the time series data, and smooth the motion trajectory through curve fitting to generate the gold wire motion trajectory.

[0040] Preferably, the steps for obtaining the gold wire segmentation boundary are as follows:

[0041] Receive the gold wire movement trajectory and gold wire movement characteristics, extract the outer contour pixel points of the gold wire area in all frames, and construct a set of gold wire area boundary points;

[0042] Based on the set of gold wire area boundary points, calculate the boundary point noise suppression factor, and the calculation formula is:

[0043] ;

[0044] Wherein, is the boundary point noise suppression factor, respectively represent the gradient values in the horizontal and vertical directions of the boundary point, respectively represent the edge gradient values of the corresponding pixels in the gold wire super-resolution map, respectively represent the second-order derivatives of the edge gradient in the horizontal and vertical directions, is a constant to prevent division by zero;

[0045] Based on the boundary point noise suppression factor, eliminate noise points, correct the boundary curve, and generate the gold wire segmentation boundary.

[0046] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0047] In the present invention, the gradient enhancement strategy strengthens the gradient change characteristics of the edge region, making the gold wire area more distinguishable under complex backgrounds. The combination of brightness gradient calculation and adaptive threshold adjustment can dynamically adapt to the characteristic changes of the gold wire area under different illumination conditions, improving the stability of classification. The super-resolution reconstruction uses spatial high-frequency information to improve the image resolution, making the edge contour sharper and reducing the structural blurring problem caused by low resolution. The pixel point displacement vector calculation method combined with the gold wire super-resolution map data makes the description of the dynamic changes of the gold wire area more accurate, avoiding the information loss caused by single-frame analysis. The time series prediction strategy introduces a motion trend adjustment mechanism, improving the continuity of the motion trajectory and making the gold wire movement characteristics remain stable in different scenarios. The boundary point set calculation method combined with the time series change characteristics effectively suppresses edge noise, improves the accuracy of the final segmentation boundary, and reduces the interference of non-target areas on the segmentation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a schematic diagram of the steps of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0049] To make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not used to limit the present invention.

[0050] Please refer to Figure 1 , the present invention provides a technical solution, a real-time image segmentation method for wire bonding, including the following steps:

[0051] Input a sequence of wire images, perform Fourier transform on the sequence of wire images, extract frequency-domain amplitude and phase information, calculate and superimpose the phase difference between adjacent frames to obtain the wire edge spectrum; enhance the gradient of the wire edge spectrum to generate an enhanced edge feature map;

[0052] Based on the enhanced edge feature map, obtain the brightness gradient value, set an adaptive threshold for image block classification to obtain the wire region distribution map; perform super-resolution reconstruction on the wire region distribution map to generate a super-resolved wire map;

[0053] Based on the super-resolved wire map, calculate the displacement vector of the pixel points in the wire region between adjacent frames to establish the wire motion feature; perform temporal prediction on the wire motion feature to generate the wire motion trajectory;

[0054] Based on the wire motion trajectory, calculate the set of boundary points of the wire region, and combine the edge gradient information in the super-resolved wire map to suppress noise and generate the wire segmentation boundary.

[0055] The steps for obtaining the wire edge spectrum are as follows:

[0056] Input a sequence of wire images, convert each frame of the image to the frequency domain through Fourier transform, and extract the amplitude and phase information of each frame to obtain frequency-domain feature data;

[0057] According to the frequency-domain feature data, perform differentiation on the phase information between every two frames to calculate the phase change amount, and the calculation formula is:

[0058] ;

[0059] Where is the phase change amount, is the phase information of the th frame, is the total number of frames, is the influence constant;

[0060] Based on the phase change amount, apply high-pass filtering to enhance the wire edge feature to obtain the wire edge spectrum.

[0061] Specifically, for the input gold wire image sequence, first set up an industrial camera with 2 million pixels at the acquisition end to obtain continuous images at a frequency of 25 frames per second. Number all the obtained image frames in sequence, and read the grayscale value of each pixel point with the help of a basic image processing library. If the grayscale value is lower than 10, it is determined as noise and stored in a list to be confirmed. At the same time, check for highlight spots through a fixed threshold of 20 and record them separately in another list for subsequent comparison. Then, perform a fast Fourier transform on the pixel information between the current frame and the next frame. In this process, the image needs to be input into the Fourier operation function in the form of a two-dimensional array. This operation function maps the pixel grayscale distribution in the spatial domain to the frequency domain, thereby obtaining data on both the amplitude and phase parts. Since each pixel corresponds to a complex number representation in the frequency domain, the amplitude information and phase information of all pixels are extracted separately. To ensure the credibility of the amplitude and phase data, the abnormal parts with amplitudes lower than 0.001 and higher than 10000 are excluded during extraction. This is identified by the upper and lower threshold values determined according to the dynamic range of the camera used on-site and industry detection standards. At the same time, for the case where the phase information has a volatility exceeding 5 degrees, compare it with the phase difference of the surrounding pixels in the same frame, and mark the points that significantly do not conform to the difference law of the surrounding pixels as abnormal phase points. This operation utilizes the characteristic of local consistency of the image. Finally, integrate the remaining amplitude and phase information after excluding the anomalies and rearrange and express them according to the pixel coordinates, so as to obtain the corresponding frequency domain feature data.

[0062] The benefit of the formula is that by adding an influence constant to the phase difference in the frequency domain , it can resist the drastic phase fluctuations caused by some image noise and provide a stable quantitative basis for subsequent analysis based on the phase change amount, thereby helping to accurately extract the overall phase change trend in the dynamic image sequence;

[0063] The steps for obtaining the parameters are as follows: In a frame of image, perform frequency domain conversion on the complex form of each pixel and extract the argument of the complex number. Consider this argument value as the phase value. In on-site detection, in the interval where the camera exposure time is from 1 / 2000 second to 1 / 500 second, capture the gold wire samples frame by frame. Perform a fast Fourier transform on the grayscale matrix obtained from each capture, and combine it with the brightness distribution data of the same frame to eliminate overly bright or overly dark abnormal pixel points, so as to reduce noise interference when extracting the argument. At the same time, through statistical analysis of multiple batches of data, the phase value distributions of a total of 500 frames are recorded. Among them, the phases of the vast majority of pixels are concentrated between -3 degrees and 3 degrees. The extremely small number of pixels exceeding this range are marked as invalid values during the aforementioned abnormal elimination process. For example, under a relatively stable imaging condition, the statistical result of the phase range detection of a certain frame of image shows that the phase values of more than 90% of the pixels fall between -2.5 degrees and 2.8 degrees, and can be directly used for subsequent calculations;

[0064] The steps for obtaining the parameters are as follows: In the continuously acquired image sequence, count the number of frames. When the shooting task of a certain duration is completed, the total number of frames for analysis can be accurately obtained. This number of frames is closely related to the phase difference calculation range. By linking with the actual production efficiency and detection duration on-site, set each monitoring cycle to be from 2 seconds to 5 seconds. During this cycle, the industrial camera captures from 50 frames to 125 frames, thus corresponding to the numerical range of 50 to 125. If it is necessary to improve the detection granularity on-site, the shooting frame rate can be appropriately increased and corrected according to the effective number of frames obtained from multiple tests to determine the final one used for calculation For example, in the process of capturing for 2 seconds in a certain case, 60 frames are obtained, then it can be directly substituted into subsequent calculations;

[0065] The steps for obtaining the parameters are as follows: When analyzing the phase noise, record the phase difference and the low-light interference situation in the imaging environment, as well as the frequency characteristics of mechanical vibration by comparing frame by frame. According to the amplitude data fed back by the vibration sensors installed around the processing device on-site, convert it into the jitter range that may affect the phase detection. After statistics, in the operating states of multiple workstations, the vibration amplitudes captured by the vibration sensors are between 0.001g and 0.008g, where g represents the gravitational acceleration of 9.8 m / s². Then, combined with the distribution evaluation of the camera imaging noise, introduce the final influence constant The value is between 0.01 and 0.03. If the vibration is large, a relatively high value is taken; if the vibration is low, a relatively low value is taken. For example, in the detection link of a semi-automatic production line, the vibration is relatively stable, and the amplitude of 0.0025g recorded by the vibration measurement is used for quantification. After comparing the phase noise distribution map generated in the imaging, finally is used as the influence constant in this production line;

[0066] Calculation process:

[0067] In the first step, on the premise of determining , , read the phase data of frames 1 to 60, and let represent the phase values obtained in sequence,

[0068] In the second step, calculate the phase difference of adjacent frames item by item, substitute the obtained phase difference into and add , take the square root of the sum, and then accumulate and sum until ,

[0069] In the third step, divide the above cumulative result by , so as to obtain ;

[0070] For example, in a certain calculation, 59 adjacent frame differences obtained in sequence are [0.5, 2.1, 1.9,...], and the specific calculation is as follows:

[0071] ;

[0072] Record the sum of all 59 items of results as , then:

[0073] ;

[0074] This result indicates that in the 60-frame image sequence being monitored, the phase change amount is approximately 1.231 degrees. By performing subsequent analysis on , the phase fluctuation condition of the overall gold wire at this time period can be known, and it can provide a reference basis for subsequent high-pass filtering processing and edge enhancement.

[0075] Based on the phase change, high-pass filtering is applied to enhance the edge features of the gold wire and obtain the gold wire edge spectrum. In the operation, the fluctuation information list obtained by statistics in the previous step is first read, and the list is compared with the original phase matrix pixel by pixel. When it is found that some phase fluctuations are at a high level, the emphasis on high-frequency components is appropriately increased when calculating the high-pass filter kernel. At this time, a high-frequency offset coefficient can be set with a value range of 1.2 to 2.5, which is obtained from the statistics of the local phase intense area. For example, by detecting 90 pixels around a gold wire solder point, the energy distribution of its frequency domain characteristics is quantified, and it is found that about 70% of the pixels have a high-frequency component proportion of nearly 40%. Based on this proportion, the offset coefficient in the high-pass filter kernel is derived to be approximately 1.8, and then apply the coefficient to the convolution operation of the current frame to amplify the frequency domain changes between adjacent pixels, and then superimpose the noise information extracted in the illumination interference analysis stage, mark the abnormal pixels suspected to be caused by sudden illumination changes during the filtering process, and do not perform independent amplification after marking. And by processing the pixels with obviously abnormal high-frequency amplitude separately, calculate the root mean square of the difference between its surrounding pixels and its memory level. If the result is greater than the preset critical value of 5, this pixel will be taken as a key investigation target and automatically added to the abnormal list. In this way, after traversing all frames, the corresponding high-pass filtering result sequence is obtained, and finally the high-frequency component information in the sequence is integrated and output as a gold wire edge spectrum for further segmentation in the next stage.

[0076] The steps for obtaining the enhanced edge feature map are:

[0077] Receive the gold wire edge spectrum, calculate the horizontal gradient component and the vertical gradient component respectively, calculate the gradient value of each pixel point by applying the gradient operator through convolution operation, and synthesize the overall gradient amplitude to obtain the original edge intensity map;

[0078] Based on the original edge intensity map, the edge enhancement value is calculated using the following formula:

[0079] ;

[0080] in, is the edge enhancement value, Pixel The horizontal gradient component of Pixel The vertical gradient component of A constant to avoid division by zero errors;

[0081] Based on the edge enhancement value, noise is suppressed through nonlinear filtering to generate an enhanced edge feature map.

[0082] Specifically, after receiving the gold wire edge spectrum obtained previously, it is first necessary to confirm the pixel coordinate information and amplitude distribution data corresponding to the edge spectrum, and record the initial gradient measurement environment of all pixel points. After obtaining this basic information, the gradient operators suitable for horizontal and vertical convolution can be extracted and matched row by row and column by column with reference to the previously statistically pixel intensity change range (for example, the peak gray level observed by a 2-million-pixel industrial camera on site is about 255, and the lowest gray level is about 0) and the spatial position of the pixel distribution. In practice, the operator usually selects a discrete template with a size of 3×3 or 5×5. The specific selection needs to be determined through actual comparison. After referring to the edge sharpness and noise distribution of the adjacent parts of the gold wire, a trial calculation can be carried out in a local area. When the cumulative number of noise points in a certain area exceeds the preset quantity threshold (for example, 15 noise points or more), it is necessary to detect the original edge texture of this area. If the texture density is too low, the size of the gradient operator template should be reduced to avoid amplifying noise. If the texture density is high, the 3×3 or 5×5 template can remain unchanged. Then, the horizontal gradient component and vertical gradient component of each pixel point are calculated. In this step, by performing convolution processing on the brightness values between each pixel and its adjacent pixels, the brightness change amount in the horizontal direction is extracted and denoted as , and the brightness change amount in the vertical direction is denoted as . And the absolute value operation is performed on these two components to avoid negative value interference. At the same time, specific too-high or too-low gradient values need to be classified and managed. Specifically, the high and low gradient intervals can be determined according to the gradient distribution range extracted from multiple batches of gold wire images in the early stage (for example, the high gradient interval is located in the range of 40 to 255, and the low gradient interval is located in the range of 0 to 10). When it is found that the gradient value at a certain pixel exceeds the normal range, further comparison is made on its neighboring pixels. If the neighboring pixels also contain continuous high or low value gradients, the position of this pixel is marked as a suspected high-noise point or low-contrast area, and the corresponding compensation is matched according to the measurement result of the light intensity in the acquisition environment (the range is about between 200 and 600 lux). This can provide a more reliable reference when integrating the horizontal gradient component and vertical gradient component of a single pixel in the subsequent process. Then, the overall gradient amplitude is obtained by synthesizing these two gradient components in a vector manner. The gradient amplitude of each pixel is uniformly written into the gradient map and other pixels are continuously traversed until all processing of the image is completed. Finally, the obtained gradient amplitude information is summarized and an original edge intensity map represented by a two-dimensional matrix is drawn.

[0083] The advantage of the formula is that by performing cubic processing on the horizontal gradient component and vertical gradient component respectively, a more sensitive response can be obtained in the edge area. At the same time, The structure can maintain the stability of the operation under extremely small gradients, so as to better highlight the local significant changes when actually detecting the gold wire image containing fine edge features.

[0084] Parameter Obtaining steps:

[0085] This parameter represents the horizontal gradient component at the pixel point position, which is obtained by performing differential convolution in the horizontal direction on the previously obtained original edge intensity map. Its numerical range is usually determined by the brightness dynamic range of the on-site image and the measured scene. For example, when using an industrial camera with a resolution of 1920×1080 to capture the gold wire, in a workshop environment with stable lighting, after multiple batches of acquisition and calibration, it is found that the brightness change of the pixel usually falls within the range of 0 to 255, and the horizontal gradient value also fluctuates between -255 and 255 accordingly. In order to obtain more accurate gradient results, in actual implementation, it is necessary to perform refined analysis on the edge area. For example, in an image with a resolution of 600×800, all rows can be traversed, the gray difference between adjacent pixels can be recorded in a temporary list, and methods such as the Sobel operator or the Prewitt operator can be used for calculation When it is detected that the absolute value of the horizontal gradient value of at least 100 pixels is higher than 200, its distribution is statistically analyzed to confirm whether the occurrence of this high value is related to excessive lighting or camera exposure delay. Then, combined with the illuminance measurement of the actual production line (for example, the average illuminance is measured to be 300 lux under incandescent lighting and 500 lux under LED lighting), these illuminance information is quantified into a correction coefficient of 0.2 to 0.5. Finally, the correction coefficient is multiplied into the adjacent pixel difference result and updated The actual value of The pixel gray value at the position is 180, The pixel gray value at the position is 90, and the correction coefficient is set to 0.3. Then , and thus the horizontal gradient component at this pixel point can be obtained.

[0086] Parameter Obtaining steps:

[0087] This parameter represents the vertical gradient component at the pixel point position. The acquisition method is similar to that of , except that during traversal, it needs to be done column by column, reading and The gray-scale difference between them can also use traditional gradient operators such as the Sobel operator to complete the basic operations, and the results need to be corrected according to the illuminance data or noise distribution measured on-site. For example, in the same 600×800 image, when it is detected that there are more than ten consecutive rows in the vertical direction showing a large brightness difference (such as the absolute value exceeding 220), it is necessary to first verify whether this area is within the range of high-brightness reflection or shadow influence, and use a correction coefficient in the range of 0.25 to 0.6 to make corresponding adjustments to the current gradient value. For example, for the pixels and calculate the gray-scale difference between them. If the gray-scale difference is -150 and the observed on-site illuminance value is 400 lux, it can be mapped to the correction coefficient range of 0.4, so as to , and then take the absolute value and substitute it into the formula during subsequent operations. Through the same type of operation, all pixels can be traversed to obtain the complete vertical gradient component.

[0088] Parameter acquisition steps:

[0089] This parameter is a constant introduced to avoid division-by-zero errors, and its value can be comprehensively determined according to factors such as the minimum gradient amplitude in the edge region and the dark current noise level of the shooting environment. During the process of an industrial camera shooting gold wires, weak noise will be generated. Through the evaluation of the camera's dark current under laboratory conditions (temperature 25°C, humidity 50%), it is found that the random noise measured under the minimum exposure condition causes the pixel brightness to fluctuate by 2 to 5 gray levels. Combining the analysis of 100 images, the average minimum gradient amplitude is calculated to be about 1.5. Therefore, select in the range of 1.0 to 2.0. For example, in an actual workshop with a temperature of 30°C, the slightly higher dark current noise makes the measured minimum gradient amplitude about 1.9, then can be set, and this value can be directly substituted into subsequent calculations. If higher noise is encountered in local detection, can be appropriately increased to around 2.0. Here is a specific example: in the previously counted image, the minimum gradient amplitude is 1.7. According to the camera noise record, select , and write it into the denominator part of the formula operation.

[0090] Calculation process:

[0091] In the first step, first obtain the and at the same pixel, then take the absolute value and cube them respectively to get and ,

[0092] In the second step, in the denominator, add and , and perform a square root operation with the sum of to form ,

[0093] In the third step, divide the sum of the results of the first step by the result of the second step to obtain ,

[0094] Given a practical example, when , , , the operation is as follows:

[0095]

[0096] Denominator part:

[0097] ;

[0098] Take the square root ;

[0099] Therefore,

[0100] ;

[0101] This result indicates that this pixel point has large gradients in both the horizontal and vertical directions, indicating significant local edge features. The higher the value, the more obvious the edge is usually. If the value is lower than 50, it usually indicates that the gradient change of neighboring pixels is small. In the subsequent process of non-linear filtering to suppress noise, this value can be used as a reference to determine whether to perform stronger smoothing or preservation processing on this point.

[0102] According to the edge enhancement value matrix obtained previously, it is necessary to summarize the numerical distribution of all pixels and compare them within the same range. When it is found that the enhancement value of a certain pixel is extremely high or low, record the difference between this enhancement value and the surrounding pixels. Here, a preliminary threshold range can be set. For example, based on the statistical results of the previous collection of gold wire images under different lighting conditions, extract the maximum and minimum enhancement values in the past about 300 frames, and use the average value of them to float up or down by 10% as the segmentation interval. Then compare the pixel enhancement value with this interval, and combine the additional fine-tuning amount (such as an offset coefficient between 0.05 and 0.15) to determine whether this pixel belongs to a high outlier or a low outlier. If it is regarded as a high outlier, move it from the temporary queue to the queue to be specially inspected, and grab the enhancement values of other pixels within a certain range (such as a 3×3 or 5×5 neighborhood) around it in the queue to be specially inspected for repeated comparison. In this way, a check dimension of the local noise ratio can be constructed, and then associate the local noise ratio with the previously detected lighting change amplitude or mechanical vibration frequency. The specific method is to extract the lux value at a certain moment from the previously recorded actual lighting intensity measurement in the workshop, or extract the vibration acceleration in a certain one-minute segment from the historical readings of the vibration sensor, and perform a quantitative mapping on these two parameters. For example, a lighting lux value of 400 can be mapped to 0.4, and a vibration acceleration of 0.003g can be mapped to 0.2. Multiply them item by item with the local noise ratio. If it is found that the product exceeds the predetermined threshold (such as set to 0.08), then further confirm whether the current pixel has a repeated record in the high-noise list. When the number of repeated records is greater than three, a larger smoothing coefficient needs to be used in the non-linear filtering process to make it weaken the fluctuation amplitude of these abnormal points faster. In addition, similar operations are also performed on low outliers, but when comparing the enhancement value with the current threshold interval, it needs to be matched in the lower limit area. Once the determination condition is met, mark that this pixel is in the range of weak gradient and may be affected by dark current or other low-brightness interferences, and then perform filtering row by row or column by column on these marked points. During the process, perform multiple differential measurements on the enhancement value of each pixel and its neighboring pixels, and statistically calculate the mean square value of the differences. If the mean square value is lower than a previously set smoothing judgment line (such as 10), then regard it as an overall smooth area. Finally, summarize the pixel information that has been determined, merge the parallel processing results and output them as the final enhanced edge feature map.

[0103] The steps for obtaining the gold wire area distribution map are as follows:

[0104] Receive the enhanced edge feature map, obtain the brightness gradient value of each pixel, extract the gradient components in the horizontal and vertical directions, and obtain the brightness gradient value map;

[0105] Based on the brightness gradient value map, calculate the adaptive threshold, and the calculation formula is:

[0106] ;

[0107] where T is the adaptive threshold, is the brightness gradient value of pixel point , and are the number of rows and columns of the image respectively, is a constant to avoid division by zero;

[0108] Based on the adaptive threshold, classify the image blocks of the brightness gradient value map, remove the isolated noise blocks, determine the distribution of the gold wire area, and generate the gold wire area distribution map.

[0109] Specifically, receive the enhanced edge feature map obtained previously. First, scan the pixel coordinates of all rows and columns in the memory and establish a pixel index. Calculate the horizontal brightness gradient by taking the difference of the gray values of adjacent pixels in each row. When performing the difference, refer to the previously measured pixel gray value range (0 to 255). When the absolute value of the difference value for three consecutive times exceeds 50, record the high-gradient characteristic of the pixel in this row and column, and mark it with the value 1. If it does not exceed 50, mark it as 0. The accumulated difference in brightness between this mark and adjacent pixels is stored in a row vector. After all columns are traversed, the horizontal gradient set can be obtained. Subsequently, perform the same operation on the column direction to obtain the vertical gradient. Analyze the difference of the gray values of adjacent pixels in each column. When the absolute value of the difference in several consecutive rows exceeds 60, mark the current pixel as having a high-gradient characteristic and assign the value 2 to the vertical gradient set. Then, compare the gradient information in the horizontal direction and the vertical direction pixel by pixel. When the horizontal gradient mark and the vertical gradient mark of a certain pixel are both greater than 0, mark it as 3 in the merged matrix to indicate that there are significant brightness changes in both the horizontal and vertical directions of the pixel. At this time, traverse the merged matrix again to check if there are any unupdated blank pixels. If so, calculate the gradient difference between it and the surrounding pixels within the range of 3×3 or 5×5 according to the neighborhood method. Once the calculation result is higher than the previously set threshold of 40 for three consecutive times, mark it as a high-brightness gradient pixel. This can label both the edge area and the transition area. Finally, convert the merged matrix into a unified brightness gradient value map and output it.

[0110] The benefit of the formula is that through the interactive calculation of the cube root and the numerator and denominator, it can comprehensively evaluate the degree of change in pixel brightness gradient in the row and column directions of the image. When multiplying the difference square amounts in the horizontal and vertical directions in the numerator part, it not only highlights the large-gradient area but also takes into account the overall balance. And in the absolute value cumulative summation of the denominator, it combines the constant to offset the potential division-by-zero problem, and can generate more sensitive thresholds suitable for different lighting and noise environments when used in multiple batches of gold wire images.

[0111] parameter Steps for obtaining

[0112] This parameter represents the pixel point brightness gradient value, which is integrated from the pixel gradient values extracted from the aforementioned brightness gradient map. After adjusting the brightness of the original image obtained by the industrial camera, it combines the gradient information in the horizontal and vertical directions. In order to accurately represent the brightness change, when recording , it is necessary to distinguish between the gray component and the noise component. For example, areas with a gray level greater than 220 in the picture are marked as high-brightness areas, and areas with a gray level less than 30 are marked as low-brightness areas. The corresponding noise correction amount (this noise correction amount comes from the dark current test of the industrial camera at 25°C, and statistics indicate that it will bring an average error of 1 to 4 gray levels) needs to be added or subtracted from the gradient value. Then, the gradient values of most pixels after correction are cropped to prevent excessive extreme values. In actual on-site detection, if an image of 600×800 is collected, then it is between 1 and 600, it is between 1 and 800. In multiple tests, it has also been observed that the brightness gradient value usually remains in the range of 0 to 300. Sometimes higher values may appear near the solder joints. By comparing the on-site illumination and the shooting distance, the overly soaring gradient values are reviewed again. Finally, they are unified into a two-dimensional matrix for subsequent statistical calculations. For example, in a single sampling, statistics show that 80% of the pixel gradient values are concentrated in the range of 10 to 120. At this time, local extreme value detection can be intuitively established to further ensure the credibility.

[0113] Parameter Steps for obtaining

[0114] This parameter represents the number of rows of the image, which is directly determined by the height of the image collected by the industrial camera. In actual production lines, it usually takes values between 600 and 2000. The specific number varies with the camera resolution and shooting configuration. For example, in a common 1920×1080 resolution scenario, , in specific situations, if the upper and lower edges of the image are cropped, then it will be correspondingly reduced to 800 or other numbers. Generally, in multiple repeated detections, the complete row and column parameters will be recorded first and written into the image label information for storage. For example, in the local database, is bound to the current image as a key attribute to avoid row and column mismatches in subsequent operations. Given an example: in a detection task, the camera resolution is set to 1440×1080, then , and then the row index is constructed based on this value to retrieve the brightness gradient distribution row by row.

[0115] Parameter Steps for obtaining

[0116] This parameter represents the number of columns of the image, and like it is also determined during the acquisition process and usually fluctuates between 800 and 3000. For example, for an image with a resolution of 1920×1080, it can make , when the on-site equipment uses the automatic cropping method to only retain the region of interest, the left and right parts of the image will be cut. At this time it may be reduced to 1500 or less. To ensure sufficient observation of the gold wire area, the column number setting is not randomly changed during the entire production process. For example, in a certain detection, the camera directly outputs an image with a resolution of 600×800, then , this value is recorded in the database and participates in subsequent calculations together with the corresponding to ensure that the row and column dimensions match and the actual imaging size during acquisition can be traced.

[0117] Parameter acquisition steps:

[0118] This parameter is a constant to prevent division-by-zero errors and is used to provide stability when the absolute value accumulation result in the denominator is too small or zero. Extreme cases may occur in the image row and column differences measured on-site. For example, in an extremely uniform area, the brightness gradients of dozens of consecutive pixels are almost 0. At this time, there is a risk that the denominator will be close to zero. Therefore, after observing several batches of images, a positive number less than 3 is taken as , usually it can be selected between 0.5 and 1.5. If the overall noise level of the image is slightly high in a specific test, can be increased to above 1.0 to keep the formula calculation process stable. For example, in a welding detection scenario, the local gold wire brightness distribution collected is extremely smooth, and the denominator sum is only about 5 after accumulation. After introducing division-by-zero is avoided and the result is ensured to have a smooth transition.

[0119] Calculation process:

[0120] First step, according to the previously recorded number of rows and the number of columns , traverse the pixel coordinates in turn, take out and its adjacent pixels , and calculate the brightness gradient difference, then multiply and and add the obtained product to the numerator.

[0121] Second step, calculate in the denominator and accumulate the results of all pixels. After completing the traversal, add this accumulated value to the constant .

[0122] In the third step, divide the numerator by the denominator and take the cube root to obtain the final adaptive threshold. ,

[0123] Given a numerical example: Let , , , in actual statistics, for the numerator part , for the denominator part , so the denominator is , and then:

[0124] ;

[0125] ;

[0126] This result indicates that when the numerator and denominator are at this order of magnitude, the calculated adaptive threshold is approximately 2.884. In the subsequent image block classification, pixels or regions with gradient values higher than 2.884 can be regarded as candidate high-gradient regions, while those lower than 2.884 are classified into ordinary regions. This threshold will be used as the segmentation basis in subsequent operations to detect and locate the main distribution range of the gold wires.

[0127] Based on the adaptive threshold obtained in the previous step, perform image block classification on the luminance gradient value map. During the execution, first divide the luminance gradient value map into several small blocks according to the row and column dimensions, and each block contains an equal number of pixels (for example, 16×16 or 32×32 per block). When traversing these small blocks, first count the number of pixels in the block whose luminance gradient value is greater than the previously obtained threshold. If this number exceeds 300 or more in a certain block, register it as a high-gradient block in the block-level marker table, and record whether there is also a high-gradient phenomenon in adjacent blocks. In this way, the continuous or discrete high-gradient region distribution can be gradually pieced together. When encountering an isolated block (for example, all surrounding blocks do not belong to high-gradient, only itself is marked as high-gradient), regard it as a noise block, register it in a local exclusion list, and then check the overall luminance information of the pixels in the isolated block. If the luminance mostly falls between 10 and 30 or is higher than 230, it indicates that there may be an abnormal distribution caused by being too dark or too bright. For such high-gradient noise blocks, they will be excluded from the subsequent region distribution map. After all blocks are classified, number and record the positions of the high-gradient blocks, and then regard the high-gradient blocks with consecutive numbers as the same gold wire region candidate area. When there are several consecutive blocks marked as high-gradient around the gold wire solder joints or along the extension of the gold wires, merge these candidate blocks into a gold wire region strip, and then perform simple connection or truncation checks on these strips in combination with the gold wire shape reference parameters (such as length, direction, etc.) collected in advance on-site. Finally, summarize to form a coordinate set of the gold wire region distribution, and obtain the gold wire region distribution map through re-encapsulation and merging.

[0128] The steps for obtaining the gold wire super-resolution image are as follows:

[0129] Receive the gold wire area distribution map, analyze the edge detail information of the local area, and obtain the preliminary super-resolution image through image interpolation correction.

[0130] Based on the preliminary super-resolution image, perform noise removal and contrast equalization to generate the gold wire super-resolution image.

[0131] Specifically, receive the gold wire area distribution map obtained previously. First, record the pixel coordinate ranges of each local block in the image and identify the rows and columns of edge pixels where texture transitions may exist. To analyze these edge details, the brightness change data can be read row by row and column by column. Pixel points that meet the edge determination criteria are grouped as refinement targets. For example, when the threshold counted in the previous step is between 30 and 50, if the brightness difference exceeds this range, it can be listed as a position that needs interpolation. Then, perform interpolation magnification according to the pre-set frequency doubling coefficient, such as taking 2 times or 3 times. During interpolation, take the average or weighted distribution of the brightness distribution of the left and right pixels and the upper and lower pixels, and record the distance and brightness difference between adjacent pixels and the current pixel in the weight setting. If the distance is too close and the brightness difference is higher than 10, a slightly lower weight is assigned. When the distance is relatively far and the brightness difference is stable in the range of 5 to 10, a higher weight is assigned. This approach is defined numerically by the user in the interpolation configuration file. After interpolation, perform a point-by-point verification of the peripheral gray gradient of the result pixels. If there are three consecutive rows or columns with inconsistent transitions in the same interpolation area, adjust the current interpolation multiple to other values such as 1.5 or 2.5 and re-calculate. When interpolation compensation is completed for all positions where local jumps may occur, compare the global edge details again. Record the edge anomaly points that accumulate more than the specified value of 30 during the comparison process, and then call the interpolation correction logic to process these anomaly points. After all steps are completed, splice them back into the pixel matrix of the overall image, and finally output the preliminary super-resolution image externally.

[0132] Based on the preliminary super-resolution image obtained previously, first, separate the luminance information of all pixels within the target range and perform noise removal. When performing noise removal, it is necessary to first record the data detected on-site. For example, in the noise distribution curve collected under the condition of an illuminance of 500 lux, if it is found that the luminance fluctuates randomly between 0 and 5 gray levels, the fluctuations below 5 can be set as the normal range. If the fluctuation amount of a certain block of continuous pixels exceeds this set range, it is marked as a high-noise point. For such high-noise points, the three-point or five-point adjacent mean method can be used for smoothing processing. Then, combined with the edge position list accumulated during the interpolation stage before, check the cross situation between the noise point distribution and the edge position. When the fluctuation value is always higher than 10 and continuously spans four rows or more, record this area as a strong noise segment and increase the smoothing coefficient to weaken the random peak. Then, perform a global traversal of the luminance distribution of the overall image, count the number of high-brightness pixels and low-brightness pixels. If the number of high-brightness pixels is greater than 800 and the number of low-brightness pixels is less than 50, or the difference between the two exceeds the specified ratio of 2 to 1, perform contrast equalization according to the recorded luminance difference distribution. Compress the high-brightness part downward and expand the low-brightness part upward through linear stretching. The stretching coefficient is set by the user between 1.2 and 2.0 in the configuration and compared in multiple test scenarios. After the luminance of all pixels has been smoothed and equalized, output the gold wire super-resolution image.

[0133] The steps to obtain the motion characteristics of the gold wire are as follows:

[0134] Receive the gold wire super-resolution image, locate the boundary pixel points of the gold wire area, analyze the pixel displacements between adjacent frames, obtain the pixel trajectories, and get the pixel trajectory set of the gold wire area;

[0135] Based on the pixel trajectory set of the gold wire area, obtain the pixel displacement vectors and generate the motion characteristics of the gold wire.

[0136] Specifically, for the obtained super-resolution image of the gold wire, it is first necessary to retrieve all suspected edge positions row by row and column by column in the image. During this process, the brightness distribution will be compared, and pixels with a brightness gradient or color difference exceeding a set standard (for example, based on the statistical results of multiple batches of on-site samples, the average gradient at most real edges is about 15, so areas exceeding 15 are regarded as potential boundaries) will be recorded as candidates. Then, local verification will be performed on these candidate edge pixels. For example, it can be checked whether the pixels above, below, left, and right also exhibit similar gradient characteristics. If three or more consecutive rows meet the similar gradient phenomenon and are concentrated in a certain direction, it is regarded as the real edge position and the coordinates are confirmed. For the pixel points with confirmed coordinates, inter-frame comparison needs to be performed in subsequent steps. When comparing, the specific coordinates of the boundary pixels in different frames will be read in the order of the image frame numbers, and then these coordinates will be subjected to successive difference operations to obtain the displacement. When processing the numerical value of the displacement, if it is found that there is a situation where the value exceeds a preset threshold (such as 5 or more), the displacement trends of several frames before and after it will be checked to distinguish whether it is a jitter error or a normal movement. Here, the threshold is obtained through the joint observation of mechanical vibration and shooting rate in the workshop environment. For example, after counting 30 frames of images, it is found that most normal movement amounts are between 3 and 5, and situations where the value exceeds 6 are mostly related to mechanical vibration. Based on these records, high-displacement rejection is performed to make the final trajectory more accurate. When all frames have been compared, the coordinates of each pixel point in each frame are connected to form a trajectory. If there are multiple pixel point trajectories in the same edge area, these trajectories will be integrated and merged into a gold wire edge trajectory group. After all trajectories have been processed, the overall pixel point trajectory set of the gold wire area is obtained.

[0137] Based on the obtained pixel point trajectory set of the gold wire area, first, the movement direction and amplitude between adjacent frames of each trajectory are statistically analyzed, and the amplitude value is compared with a preset reference value. This reference value is determined by observing the robotic arm motion model or actual working conditions. For example, in a welding production line, a numerical range between 3 and 6 is often set as the reference range. If the movement amplitude of a certain trajectory continuously appears to be larger than the upper limit of this range, it is recorded in a summary list to exclude the interference of extreme values when calculating the displacement vector in the next step. Then, the coordinate differences of the remaining normal trajectories are taken for component decomposition, and the horizontal displacement and vertical displacement are respectively subjected to numerical operations. If at the same moment in a certain area, multiple trajectories share similar displacement characteristics, cross-checking will be performed to confirm that they are continuous movements generated from the same gold wire edge area. The displacement data of this part will be merged and marked as the same vector group. Then, mean or median filtering is performed on all vector groups to exclude random fluctuations. When filtering, a threshold is usually set. For example, if the amplitude is lower than 2 and only appears in one frame, it is merged as zero movement, and if it is higher than 7, it is marked as an abnormal deviation and recorded separately. After these data processes, a relatively stable final displacement vector set is obtained, that is, the gold wire motion characteristics are generated.

[0138] The steps for obtaining the gold wire movement trajectory are as follows:

[0139] Receive the gold wire movement characteristics, extract the displacement vector data in the time series, construct the time-dependent relationship, and obtain the movement state sequence;

[0140] Based on the movement state sequence, calculate the gold wire position, and the expression is:

[0141] ;

[0142] where, is the gold wire position at time , is the velocity at the previous moment, is the acceleration at the previous moment, is the time interval, is the random perturbation term;

[0143] Based on the gold wire position, perform trajectory reconstruction on the time series data, smooth the movement trajectory through curve fitting, and generate the gold wire movement trajectory.

[0144] Specifically, after receiving the gold wire motion characteristics obtained previously, it is necessary to select the displacement vector data between adjacent frames in the recorded time series, perform index matching on these data for position, speed, and acceleration. When selecting the index, first sort them in the order of frame numbers. For example, read the displacement vector values one by one from frame 1 to frame N, and then form a two-dimensional table in the form of row and column coordinates. Each row represents the pixel point position corresponding to a certain frame, and each column corresponds to a specific coordinate component. After the table is constructed, traverse all the data in the displacement vector, start counting the numerical size of the continuous displacement difference from the adjacent frame switching point. When it is found that the displacement difference exceeds the preset threshold (such as 6 or more), record it in the list and exclude it from the sample to reduce the influence of vibration interference on the subsequent time-dependent relationship. This preset threshold can be determined through a large number of experiments. For example, under normal gold wire motion, the displacement difference mostly concentrates between 2 and 5. For cases where it exceeds 6, it often needs to be recorded separately and excluded when constructing the subsequent time-dependent relationship. Then, according to the filtered displacement vectors, trace the continuous changes that occur in the time series one by one. If the displacement within the adjacent five frames is monotonically increasing or decreasing, it is recorded as a section of stable movement. If fluctuations or jitters occur, it is marked as a dynamic fluctuation section in the attached file. At the same time, record the data of the robotic arm or vibration source according to the working mode during the operation of the workshop equipment. For example, within one hour, the motion frequency of the robotic arm is in the range of 3 to 5 Hz, and the vibration sensor data is in the range of 0.002 g to 0.008 g. Then map these values into displacement correction coefficients (such as 0.3 to 0.7) and incorporate them into the final displacement vector table. In this way, a time-dependent relationship closely corresponding to the time sequence can be constructed. Finally, integrate all the corresponding sets of time and displacement vectors into a motion state sequence.

[0145] The advantage of the formula is that it takes into account the influence of the speed and acceleration at the previous moment on the current position, and adds a random perturbation term based on the position difference between the previous moment and the moment before the previous moment at the end, which is more flexible when dealing with local jitters or minute jumps of the gold wire.

[0146] Parameter obtaining steps:

[0147] This parameter represents the speed at the previous moment, and its value can be obtained by combining the position change between adjacent frames and the actual frame rate. In an industrial production line, if a camera captures 60 frames per second, then the time interval between two frames is about 0.0167 seconds. For frame and frame Subtract the position of the gold wire and divide by the time value to obtain the speed magnitude. Then, quantify it in pixels per second or millimeters per second according to the unit. This quantification process requires prior recording of data such as camera focal length, shooting distance, and pixel size. If the gold wire moves 5 pixels in one second, it can be converted to 0.1 millimeters on the image, which can also meet the accuracy requirements of most gold wire detections. For example, in a batch of measurements, the actual movement of the gold wire is mostly 2 to 8 pixels, then It is usually between 0.2 millimeters per second and 0.4 millimeters per second. In the final application, summarize the frame rate, spatial size conversion factor, and the previous position difference for calculation to obtain the .

[0148] parameter acquisition steps:

[0149] This parameter represents the acceleration at the previous moment, which is obtained from the difference of speeds. For example, first calculate the difference between two sets of adjacent speeds and , and then divide by the same time interval . If the sampling frequency is set at 60 frames per second, then is about 0.0167 seconds. The unit of acceleration can be characterized as pixels per square second or millimeters per square second. In on-site detection, if the speed of the gold wire fluctuates between about 0.3 millimeters per second and 0.5 millimeters per second, the acceleration may fall within the range of 0.01 millimeters per square second to 0.05 millimeters per square second. It is necessary to verify the jitter records exceeding this interval. For example, when it is found that the acceleration is continuously greater than 0.07 millimeters per square second and lasts for three frames, record it in the annotation file to check for unexpected large movements of the robotic arm or gold wire soldering point equipment. Subsequently, only retain the data that is probably true to obtain a relatively stable value. Here is an example: If the speed of frame is 0.4 millimeters per second and the speed of frame is 0.35 millimeters per second, seconds, then .

[0150] parameter acquisition steps:

[0151] This parameter is the time interval, which depends on the image acquisition frame rate. In industrial applications, it is usually set between 30 frames per second and 120 frames per second, and is specifically selected according to production efficiency and detection accuracy. When the frame rate is fixed, is 1 / frame rate. For example, when the frame rate is 60 frames per second, is 1 / 60 seconds. During actual detection, the stability of camera acquisition must be maintained, and it is necessary to check for frame loss or skipped frames. If skipped frames are found, time stamps need to be supplemented in the record table, and For a numerical value, if a certain test record indicates that there is an interval of about 1.5 times between frame 10 and frame 11, then this section can be revised in subsequent calculations to improve the accuracy of timing analysis.

[0152] Parameter acquisition steps:

[0153] As a random perturbation term, this parameter needs to be quantified in combination with the vibration and light changes or mechanical jitter statistical values in the actual production environment. When monitoring the gold wire welding process, the corresponding amplitudes can be obtained through vibration sensors and environmental light measuring instruments, and then mapped to a random range of 0.1 to 0.5 to act as the perturbation size. For example, if the vibration acceleration is between 0.002g and 0.006g, it can be converted into a random perturbation weight of 0.15 to 0.3. If the light changes suddenly (for example, the light suddenly increases from 300 lux to 600 lux in a short time), the perturbation component can be pushed up to 0.4 or even larger. After all the monitoring records are completed, for the random distribution formed by the vibration values and light values of each frame, a numerical value will be extracted from it and assigned to each time an iteration is performed to simulate the influence of random fluctuations on the position of the gold wire. For example, in an actual measurement, through a two-hour data acquisition, it is found that the average value recorded by the vibration sensor is about 0.003g. After mapping, can be set near 0.2 and have a small amount of up and down fluctuations during the process.

[0154] Calculation process:

[0155] In the first step, substitute the position at the previous moment into the initial value and obtain the velocity and acceleration at the previous moment. At the same time, take the square root of the position difference between the previous frame and the frame before the previous frame and multiply it by the random perturbation .

[0156] In the second step, obtain according to the frame rate, multiply by and in turn and sum them up, and then add the aforementioned perturbation value.

[0157] In the third step, record the above result as , and then the position of the gold wire at time can be obtained.

[0158] Given a practical example, take (unit: pixel), (unit: pixel / second), (unit: pixel / second ) second , random perturbation , then the calculation is as follows:

[0159] ;

[0160] ;

[0161] ;

[0162] ;

[0163] ;

[0164] The result shows that under the given speed, acceleration and random perturbation conditions, at time , the position of the gold wire is approximately 10.365 pixel coordinate values. The larger the value, the greater the corresponding displacement of the gold wire in the image during this time interval. If the positions of consecutive multiple frames exceed a certain range later, it means the gold wire is moving rapidly. If the position remains stable or changes slightly, it indicates that the movement trend slows down or tends to be stable. The above values will be recorded one by one in the subsequent trajectory calculation and brought into the calculation process of the next frame.

[0165] Based on the position of the gold wire obtained previously, trajectory reconstruction is carried out on the time series data. First, all the values in the records of all frames will be called and arranged in a sequence, and then the adjacent positions will be compared by difference. Local extrema will be searched within the time periods when the differences continuously show an upward or downward trend. For these extreme points, curve segments can be constructed. When the local extrema exceed the set benchmark (for example, the average position increment of five consecutive frames exceeds 4), this part will be marked as high fluctuation in the annotation table. Then, polynomial fitting or spline fitting methods will be further selected for curve splicing. When fitting, the positions of adjacent several frames will be sampled point by point, mapped to two-dimensional coordinates and connected by lines. When the fitting residual exceeds the predetermined standard (such as 0.5 pixels), a new round of fitting will be carried out and the previously highly outlying points will be skipped. All time points will be traversed step by step according to this method. Finally, the continuous curves after fitting will be integrated into a complete smooth motion trajectory and output as the gold wire motion trajectory.

[0166] The steps to obtain the gold wire segmentation boundary are as follows:

[0167] Receive the gold wire motion trajectory and gold wire motion characteristics, extract the outer contour pixel points of the gold wire area in all frames, and construct a set of gold wire area boundary points;

[0168] Based on the set of gold wire area boundary points, calculate the boundary point noise suppression factor, and the calculation formula is:

[0169] ;

[0170] Among them, is the boundary point noise suppression factor, respectively represent the gradient values of the boundary point in the horizontal and vertical directions, respectively represent the edge gradient values of the corresponding pixels in the gold wire super-resolution map, respectively represent the second-order derivatives of the edge gradient in the horizontal and vertical directions, is a constant to prevent division by zero;

[0171] Based on the boundary point noise suppression factor, noise points are removed, the boundary curve is corrected, and the gold wire segmentation boundary is generated.

[0172] Specifically, after receiving the gold wire motion trajectory and gold wire motion characteristics obtained previously, and after reading the determined gold wire areas in each frame, it is necessary to summarize all the pixel coordinates that can represent the outer contour row by row and column by column. When processing, the pixels in the area can be first mapped to a two-dimensional coordinate table according to the frame number, the coordinate distribution is scanned, and the edge pixel positions with a color or brightness gradient higher than a pre-set threshold (for example, between 15 and 20) are found and locally inspected. This kind of threshold can be selected after obtaining the statistical distribution by taking pictures of gold wire samples of the same specification on site multiple times. If the continuous numerical difference in the brightness of the surrounding pixels in a certain area is greater than the upper limit of this interval, it is marked as a high edge possibility, and the pixel is added to the outer contour candidate list. Then, similar gradient differences are repeatedly detected among adjacent pixels to confirm that it is connected to the previously calibrated candidate pixels to form a closed or semi-closed boundary. When all frames are completely traversed and an outer contour list is generated for each frame, the outer contour lists of all frames are integrated and stored in segments in the order of frame numbers. If the outer contour marks appear repeatedly at the same position in multiple adjacent frames, a statistical comparison is made on whether it is a fixed noise point or a real boundary at that place. When there are pixels with consistent edge attributes in the statistical results for more than a specified number of times (for example, more than four consecutive frames), they are regarded as real outer contour pixel points. If they only appear in individual frames or intermittent frames, they are recorded as discrete points or noise points. Then, check whether there are large areas of distributed noise points in the merged outer contour list. If a large area of chaotic distribution is found and the difference between the surrounding brightness and the center is too large, this area can be marked as a noise area and removed from the final boundary point candidate set. After all outer contour candidate pixels are confirmed, they are merged according to the frame number and pixel coordinates to construct a set of gold wire area boundary points, so as to retain the accurate outer contour marks formed in each valid frame.

[0173] The benefit of the formula is that by synthesizing the superposition effect of the edge gradient and its second-order derivative in the numerator part, it can more sensitively identify the mutation values brought by local noise, and introduce To balance the gradient magnitudes in different edge directions, for pixel points with obvious sharp fluctuations, this formula will give a relatively higher noise suppression factor, thus helping to locate and exclude pixels that do not have the meaning of real boundaries.

[0174] The parameter is obtained as follows:

[0175] This parameter represents the horizontal gradient value at the boundary point, which is obtained by scanning the brightness changes of adjacent pixels in the gold wire super-resolution map. In industrial inspection, it is necessary to first perform brightness projection on visible light grayscale data or multi-channel images, and then use differential or traditional gradient operators (such as Sobel) to traverse in the horizontal direction and accumulate to obtain , which is usually between -255 and 255 after acquisition. For the absolute value processing in subsequent operations, only positive values will be recorded in actual use, or negative values will be mapped to the positive domain for convenient unified operation. At the same time, investigate the distribution that exceeds the actual gold wire brightness range (such as the occurrence of anomalies with absolute values greater than 300), and confirm whether it is abnormally invalid after multi-frame accumulation. In some cases, the imaging of fine textures of gold wires will result in slightly larger gradients, so it is necessary to perform normalization processing in combination with on-site statistics and then write it into the list. For example, for the statistics of 200 images collected at a certain workstation, the average horizontal gradient is about 30, and the highest reaches 240. If 300 is detected, it is very likely a strong reflection caused by a light spot, and this data can be marked as abnormal at this time.

[0176] The parameter is obtained as follows:

[0177] This parameter is the gradient value in the vertical direction, which is similar to the calculation method of , except that the brightness difference is performed between rows, and the pixel gray-scale changes of different row coordinates are taken and accumulated successively to obtain , which is closely related to the sharpness of the vertical edge of the gold wire. If it is detected that the vertical gradient at a certain place in the image fluctuates greatly in a short time (such as the absolute value surges above 180), it will be marked, and during the statistical stage, check whether it is a real high-contrast boundary or local noise, and finally take the absolute value of the confirmed retained vertical gradient value and write it into , if this position shows fluctuations in multiple frames, the cause can be inferred by combining vibration sensor and light change data and different degrees of correction coefficients can be assigned. In actual measurement, most typical values (such as 10 to 150) can be found in a large number of pictures (hundreds of frames). If it exceeds this interval too much, consider excluding it or making special marks and then calculating.

[0178] The parameter is obtained as follows:

[0179] They respectively represent the horizontal and vertical edge gradient values of the same boundary pixel in the gold wire super-resolution image. It is necessary to first locate the coordinates of this pixel on the gold wire super-resolution image, and then perform local convolution or difference on the surrounding area of this pixel to obtain the gradient values in the horizontal and vertical directions, which are usually comparable It can better reflect the fine structure of the gold wire itself. Since super-resolution processing will magnify edge details, in the actual production line, multi-scale or multi-direction operators can be used to layer the gold wire image, and then evaluate the gradient magnitude at the sub-layers. If it is found that the gradient value differs too much from the original image (such as doubling or more), it is necessary to combine the shooting conditions and the imaging magnification factor, and extract the real distance on-site for comparison. For example, at a certain solder joint When it reaches 200, it is necessary to first check whether there is cable jitter during the industrial camera shooting. If it is confirmed that there is no abnormality, this value is included in the final statistics and brought into the formula.

[0180] parameter and The acquisition steps are as follows:

[0181] These two second-order derivatives illustrate the practice of performing a second difference on the edge gradient to evaluate the curvature or degree of mutation. Based on the already obtained , perform a difference in a larger neighborhood range in the horizontal and vertical directions respectively. For example, traverse 3 to 5 pixel ranges in the row or column direction respectively, average the gradient differences before and after each pixel, and then perform a further gradient. In this way, the second-order derivative is obtained. In the industrial field, usually a small range (such as a 3×3 pixel neighborhood) is selected for approximate calculation. If the difference between a certain pixel and its adjacent pixels is too large (for example, the absolute difference in gradient is greater than 50), it can be marked in the next second-order derivative calculation to make the result more stable. Then, record the obtained second-order derivative in a temporary array and compare it with other frames or the neighborhood of the same frame. For example, in a data analysis, when up and down and left and right.

[0182] parameter The acquisition steps are as follows:

[0183] This parameter is a constant to prevent division by zero, used to avoid the denominator from having too small a value, which may lead to unstable operations. In gold wire detection, it is necessary to first perform statistics on the horizontal and vertical gradient distributions of a large number of images, especially in areas with extremely low brightness or uniform distribution and may be very close to 0. In order to prevent the denominator from becoming 0 or close to 0 in such an approximately pure color background, it is necessary to introduce And it is set within a small range, for example, between 0.1 and 1.5. During the testing of some production lines, it is found that values exceeding 1.0 are prone to excessive smoothing. Therefore, the range of 0.5 to 1.0 is generally selected. If the image resolution is increased or the illumination is increased again, corresponding fine-tuning can be done. Given an example, if the smallest statistical mean is about 0.2 in a certain highlight test, then can be adjusted to ensure that the denominator is always greater than 0.3.

[0184] Calculation process:

[0185] First step, read the horizontal gradient and vertical gradient of the current boundary pixel, as well as the edge gradient value of the corresponding gold wire super-resolution map , and perform a second-order derivative operation on the latter to obtain and .

[0186] Second step, multiply in the numerator by and by , add the two and take the absolute value.

[0187] Third step, calculate in the denominator, divide the numerator by the denominator to obtain .

[0188] Given a numerical example: Let , , , , the second-order derivative , , , then the operation is as follows:

[0189] ;

[0190] For the numerator part ,

[0191] For the denominator part around,

[0192] ;

[0193] This result indicates that the noise suppression factor of this boundary pixel is relatively high and can usually be regarded as an interference point. It can be deleted or smoothed in the final boundary marking. If subsequent multi-frame comparisons also show that this point has a high noise suppression factor, it will not be retained when correcting the boundary curve.

[0194] Based on the boundary point noise suppression factor results obtained previously, the investigation can be carried out in the way of each boundary point. First, associate the boundary coordinates and the corresponding factor values in each frame, and record the pixels whose factors exceed the specified standard (such as 8 or higher). This standard can be set by the operator after statistical comparison in multiple actual detection scenarios. If the record shows that the same pixel point has a large factor in adjacent frames, it is judged as a noise point and eliminated. If a large factor appears only in individual frames, it can be temporarily stored in a temporary list to check whether the specific occurrence times reach a certain threshold. When all frames are recorded, all suspected noise points are summarized, and then the gradient distributions between these suspected points and the surrounding real boundary points are compared. If the difference is significant, the elimination operation is performed. After elimination, curve interpolation is performed again to fill in the missing part and make the overall boundary more coherent. When performing curve interpolation, local fitting is used to smooth the nearby coordinates. For gaps with a small span, adjacent boundary pixels are interpolated and connected. If the span is large, this section is marked as an incomplete boundary and may need to compare multiple frames of information again later. When all the paragraph boundary curves are corrected, all the corrected curves are merged to form the final gold wire segmentation boundary.

[0195] The above is only the preferred embodiment of the present invention, and it is not intended to limit the present invention in other forms. Any person skilled in the art may use the technical content disclosed above to make changes or modifications into equivalent embodiments with equivalent changes and apply them to other fields. However, as long as it does not depart from the technical solution content of the present invention, any simple modification, equivalent change, and modification made to the above embodiments based on the technical essence of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. A real-time image segmentation method for wire bonding, characterized in that, It includes the following steps: Input the gold wire image sequence, perform Fourier transform on the gold wire image sequence, extract the frequency domain amplitude and phase information, calculate and superimpose the phase difference between adjacent frames to obtain the gold wire edge spectrum; enhance the gradient of the gold wire edge spectrum to generate an enhanced edge feature map; Based on the enhanced edge feature map, obtain the luminance gradient value, set an adaptive threshold for image block classification to obtain the gold wire area distribution map; Perform super-resolution reconstruction on the gold wire area distribution map to generate a gold wire super-resolution map; Based on the gold wire super-resolution map, calculate the displacement vector of pixel points in the gold wire area between adjacent frames, establish the gold wire motion feature; perform temporal prediction on the gold wire motion feature to generate the gold wire motion trajectory; Based on the gold wire motion trajectory, calculate the set of boundary points of the gold wire area, and combine the edge gradient information in the gold wire super-resolution map for noise suppression to generate the gold wire segmentation boundary; The steps for obtaining the gold wire area distribution map are as follows: Receive the enhanced edge feature map, obtain the luminance gradient value of each pixel, extract the gradient components in the horizontal and vertical directions to obtain the luminance gradient value map; Based on the luminance gradient value map, calculate the adaptive threshold, and the calculation formula is: ； where T is the adaptive threshold, is the pixel point brightness gradient value, and are the number of rows and columns of the image respectively, is a constant to avoid division by zero; Based on the adaptive threshold, perform image block classification on the luminance gradient value map, remove isolated noise blocks, determine the gold wire area distribution, and generate the gold wire area distribution map.

2. The real-time image segmentation method for gold wire bonding according to claim 1, wherein The steps for obtaining the gold wire edge spectrum are as follows: Input the gold wire image sequence, convert each frame of the image to the frequency domain through Fourier transform, extract the amplitude and phase information of each frame to obtain the frequency domain feature data; According to the frequency domain feature data, perform difference on the phase information between every two frames, calculate the phase change amount, and the calculation formula is: ； in, is the phase change, For the The phase information of the frame, is the total number of frames, is the influence constant; Based on the phase change amount, apply high-pass filtering to enhance the gold wire edge feature to obtain the gold wire edge spectrum.

3. The real-time image segmentation method for gold wire bonding according to claim 1, wherein The steps for obtaining the enhanced edge feature map are as follows: Receive the gold wire edge spectrum, calculate the horizontal gradient component and the vertical gradient component respectively, calculate the gradient value of each pixel point by applying the gradient operator through convolution operation, and synthesize the overall gradient amplitude to obtain the original edge intensity map; Based on the original edge intensity map, calculate the edge enhancement value, and the calculation formula is: ； Among them, is the edge enhancement value, is the horizontal gradient component of the pixel , is the vertical gradient component of the pixel , is a constant to avoid division-by-zero errors; Based on the edge enhancement value, suppress noise through non-linear filtering to generate the enhanced edge feature map.

4. The real-time image segmentation method for gold wire bonding according to claim 1, characterized in that The steps for obtaining the gold wire super-resolution map are as follows: Receive the gold wire area distribution map, analyze the edge detail information of the local area, and obtain the preliminary super-resolution image through image interpolation correction; Based on the preliminary super-resolution image, perform noise removal and contrast equalization to generate the gold wire super-resolution map.

5. The real-time image segmentation method for wire bonding according to claim 1, characterized in that, The steps for obtaining the gold wire motion feature are as follows: Receive the gold wire super-resolution map, locate the boundary pixel points of the gold wire area, analyze the pixel point displacement between adjacent frames, obtain the pixel point trajectory, and obtain the gold wire area pixel point trajectory set; Based on the gold wire area pixel point trajectory set, obtain the pixel point displacement vector to generate the gold wire motion feature.

6. The real-time image segmentation method for wire bonding according to claim 1, wherein The steps for obtaining the gold wire motion trajectory are as follows: Receive the gold wire motion feature, extract the displacement vector data in the time series, construct the time-dependent relationship to obtain the motion state sequence; Based on the motion state sequence, calculate the gold wire position, and the expression is: ； Among them, is the position of the gold wire at time , is the velocity at the previous moment, is the acceleration at the previous moment, is the time interval, is the random disturbance term; Based on the position of the gold wire, perform trajectory reconstruction on the time series data, smooth the motion trajectory through curve fitting, and generate the gold wire motion trajectory.

7. The real-time image segmentation method for gold wire bonding according to claim 1, wherein The steps for obtaining the gold wire segmentation boundary are as follows: Receive the gold wire motion trajectory and the gold wire motion characteristics, extract the outer contour pixel points of the gold wire region in all frames, and construct a set of boundary points of the gold wire region; Based on the set of boundary points of the gold wire region, calculate the boundary point noise suppression factor, and the calculation formula is: ； Among them, is the boundary point noise suppression factor, respectively represent the gradient values of the boundary point in the horizontal and vertical directions, respectively represent the edge gradient values of the corresponding pixels in the gold wire super-resolution image, respectively represent the second-order derivatives of the edge gradient in the horizontal and vertical directions, is a constant to prevent division by zero; Based on the boundary point noise suppression factor, remove the noise points, correct the boundary curve, and generate the gold wire segmentation boundary.

Images