Gold wire bonding real-time image segmentation method

Abstract

The invention relates to the technical field of image processing, in particular to a gold wire bonding real-time image segmentation method, which comprises the following steps of: inputting a gold wire image sequence, performing Fourier transform on the gold wire image sequence, extracting frequency domain amplitude and phase information, and obtaining a gold wire edge frequency spectrum through phase difference calculation and superposition operation between adjacent frames; and performing gradient enhancement on the gold wire edge spectrum to generate an enhanced edge feature map. According to the invention, the gradient enhancement strategy enhances the gradient change characteristics of the edge region, so that the gold wire region has higher distinguishability under a complex background. Brightness gradient calculation is combined with adaptive threshold adjustment, so that the method can dynamically adapt to gold wire region feature changes under different illumination conditions, and the classification stability is improved. The super-resolution reconstruction utilizes spatial high-frequency information to improve the image resolution, so that the edge contour is sharper, and the problem of structure blurring caused by low resolution is reduced.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a real-time image segmentation method for gold wire bonding. Background Art

[0002] Image processing is a research field based on computer vision and digital signal processing technology, involving operations such as image data acquisition, analysis, transformation, enhancement, segmentation and recognition. The real-time image segmentation method of gold wire bonding is part of image processing technology, which is mainly used for the detection and analysis of the position, shape and movement trajectory of gold wires in the electronic packaging process.

[0003] However, in the prior art, the displacement calculation method in the spatial domain fails to fully consider the temporal characteristics, resulting in a weak ability to capture dynamic change information, and the motion trajectory of the gold wire is easily disturbed by background noise. When the image resolution is low, the loss of edge information reduces the accuracy of motion feature extraction, affecting the subsequent segmentation accuracy. Therefore, improvements are needed. Summary of the invention

[0004] The purpose of the present invention is to solve the shortcomings in the prior art and to propose a real-time image segmentation method for gold wire bonding.

[0005] In order to achieve the above object, the present invention adopts the following technical solution, a real-time image segmentation method of gold wire bonding, comprising the following steps: Input a gold wire image sequence, perform Fourier transform on the gold wire image sequence, extract frequency domain amplitude and phase information, and obtain a gold wire edge spectrum by calculating the phase difference between adjacent frames and performing superposition operations; perform gradient enhancement on the gold wire edge spectrum to generate an enhanced edge feature map; Based on the enhanced edge feature map, a brightness gradient value is obtained, and an adaptive threshold is set to classify image blocks to obtain a gold wire area distribution map; super-resolution reconstruction is performed on the gold wire area distribution map to generate a gold wire super-resolution map; Based on the gold thread super-resolution image, the displacement vector of the pixels in the gold thread area between adjacent frames is calculated to establish the gold thread motion feature; the time series prediction of the gold thread motion feature is performed to generate the gold thread motion trajectory; Based on the gold wire motion trajectory, a set of gold wire region boundary points is calculated, and noise suppression is performed in combination with edge gradient information in the gold wire super-resolution image to generate a gold wire segmentation boundary.

[0006] Preferably, the steps of acquiring the gold wire edge spectrum are: Input the gold wire image sequence, convert each frame of the image into the frequency domain through Fourier transform, extract the amplitude and phase information of each frame, and obtain the frequency domain feature data; According to the frequency domain feature data, the phase information between every two frames is differentiated to calculate the phase change. The calculation formula is: ; in, is the phase change, For the The phase information of the frame, is the total number of frames, is the influence constant; Based on the phase change, high-pass filtering is applied to enhance the gold wire edge feature to obtain the gold wire edge spectrum.

[0007] Preferably, the step of acquiring the enhanced edge feature map is: The gold wire edge spectrum is received, the horizontal gradient component and the vertical gradient component are calculated respectively, the gradient value of each pixel is calculated by applying the gradient operator through convolution operation, and the overall gradient amplitude is synthesized to obtain the original edge intensity map; Based on the original edge strength map, the edge enhancement value is calculated using the following formula: ; in, is the edge enhancement value, Pixel The horizontal gradient component of Pixel The vertical gradient component of A constant to avoid division by zero errors; Based on the edge enhancement value, noise is suppressed by nonlinear filtering to generate an enhanced edge feature map.

[0008] Preferably, the steps of obtaining the gold wire area distribution map are: Receive the enhanced edge feature map, obtain the brightness gradient value of each pixel, extract the gradient components in the horizontal direction and the vertical direction, and obtain a brightness gradient value map; Based on the brightness gradient value map, the adaptive threshold is calculated using the following formula: ; Where T is the adaptive threshold, Pixel The brightness gradient value, and are the number of rows and columns of the image, respectively. To avoid division by zero; Based on the adaptive threshold, the brightness gradient value map is classified into image blocks, isolated noise blocks are removed, the gold wire area distribution is determined, and a gold wire area distribution map is generated.

[0009] Preferably, the steps of obtaining the gold wire super-resolution map are: Receiving the gold wire area distribution map, analyzing edge detail information of the local area, and obtaining a preliminary super-resolution image through image interpolation correction; Based on the preliminary super-resolution image, noise removal and contrast equalization are performed to generate a gold wire super-resolution image.

[0010] Preferably, the steps of acquiring the gold wire motion characteristics are: Receive the gold thread super-resolution image, locate the boundary pixel points of the gold thread area, analyze the pixel point displacement between adjacent frames, obtain the pixel point trajectory, and obtain the pixel point trajectory set of the gold thread area; Based on the gold wire area pixel point trajectory set, a pixel point displacement vector is obtained to generate a gold wire motion feature.

[0011] Preferably, the steps of obtaining the gold wire motion trajectory are: Receiving the gold wire motion feature, extracting displacement vector data in the time series, constructing a time dependency relationship, and obtaining a motion state sequence; Based on the motion state sequence, the position of the gold wire is calculated, and the expression is: ; in, For the moment The position of the gold thread, is the velocity at the previous moment, is the acceleration at the previous moment, is the time interval, is the random disturbance term; Based on the position of the gold wire, the trajectory of the time series data is reconstructed, and the motion trajectory of the gold wire is generated by smoothing the motion trajectory through curve fitting.

[0012] Preferably, the step of obtaining the gold wire segmentation boundary is: Receiving the gold wire motion trajectory and the gold wire motion feature, extracting the outer contour pixel points of the gold wire area in all frames, and constructing a gold wire area boundary point set; Based on the set of boundary points in the gold wire area, the boundary point noise suppression factor is calculated, and the calculation formula is: ; in, is the boundary point noise suppression factor, Respectively represent the horizontal and vertical gradient values ​​of the boundary points, They represent the edge gradient values ​​of the corresponding pixels in the gold wire super-resolution image, Represent the second-order derivatives of the edge gradient in the horizontal and vertical directions, respectively. A constant to prevent division by zero; Based on the boundary point noise suppression factor, noise points are removed, the boundary curve is corrected, and a gold wire segmentation boundary is generated.

[0013] Compared with the prior art, the advantages and positive effects of the present invention are: In the present invention, the gradient enhancement strategy strengthens the gradient change characteristics of the edge area, making the gold wire area more distinguishable under complex backgrounds. The brightness gradient calculation combined with the adaptive threshold adjustment can dynamically adapt to the characteristic changes of the gold wire area under different lighting conditions, and improve the stability of classification. Super-resolution reconstruction uses spatial high-frequency information to improve the image resolution, making the edge contour sharper and reducing the structural blur problem caused by low resolution. The pixel point displacement vector calculation method is combined with the gold wire super-resolution image data to make the dynamic change description of the gold wire area more accurate and avoid the information loss caused by single-frame analysis. The timing prediction strategy introduces a motion trend adjustment mechanism to improve the continuity of the motion trajectory, so that the gold wire motion characteristics remain stable in different scenarios. The boundary point set calculation method combined with the timing change characteristics can effectively suppress the edge noise, improve the accuracy of the final segmentation boundary, and reduce the interference of non-target areas on the segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of the steps of the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0016] See also Figure 1 The present invention provides a technical solution, a real-time image segmentation method of gold wire bonding, comprising the following steps: Input the gold wire image sequence, perform Fourier transform on the gold wire image sequence, extract the frequency domain amplitude and phase information, and obtain the gold wire edge spectrum by calculating the phase difference between adjacent frames and superposition operation; perform gradient enhancement on the gold wire edge spectrum to generate an enhanced edge feature map; Based on the enhanced edge feature map, the brightness gradient value is obtained, and the adaptive threshold is set to classify the image blocks to obtain the gold wire area distribution map; the gold wire area distribution map is super-resolved to generate a gold wire super-resolution map; Based on the gold thread super-resolution image, the displacement vector of the gold thread area pixels between adjacent frames is calculated to establish the gold thread motion features; the gold thread motion features are time-series predicted to generate the gold thread motion trajectory; Based on the gold wire motion trajectory, the gold wire area boundary point set is calculated, and the noise is suppressed by combining the edge gradient information in the gold wire super-resolution image to generate the gold wire segmentation boundary.

[0017] The steps to obtain the gold edge spectrum are: Input the gold wire image sequence, convert each frame of the image into the frequency domain through Fourier transform, extract the amplitude and phase information of each frame, and obtain the frequency domain feature data; According to the frequency domain feature data, the phase information between every two frames is differentiated and the phase change is calculated. The calculation formula is: ; in, is the phase change, For the The phase information of the frame, is the total number of frames, is the influence constant; Based on the phase change, high-pass filtering is applied to enhance the gold wire edge features and obtain the gold wire edge spectrum.

[0018] Specifically, input the golden wire image sequence, first set a 2-megapixel industrial camera at the acquisition end, acquire continuous images at a frequency of 25 frames per second, number all the acquired image frames in sequence, and use the basic image processing library to read the grayscale value of each pixel. If the grayscale value is lower than 10, it is judged as a noise point and stored in a pending confirmation list. At the same time, a fixed threshold of 20 is used to check the highlight spots, which are recorded separately in another list for subsequent comparison. Then, a fast Fourier transform is performed on the pixel information between the current frame and the next frame. In this process, the image needs to be input into the Fourier operation function in the form of a two-dimensional array. The operation function will map the pixel grayscale distribution in the spatial domain to the frequency domain, thereby obtaining the amplitude and phase data. Since each A pixel corresponds to a complex representation in the frequency domain, so the amplitude information and phase information of all pixels are extracted separately. In order to ensure the credibility of the amplitude and phase data, the abnormal parts with amplitudes lower than 0.001 and higher than 10000 are eliminated during extraction. This is identified by the upper and lower thresholds determined according to the dynamic range of the camera used on site and the industry detection standards. At the same time, for the situation where the fluctuation rate of the phase information exceeds 5 degrees, the phase difference of the surrounding pixels in the same frame is compared, and the points that obviously do not conform to the law of surrounding pixel differences are marked as abnormal phase points. This operation utilizes the characteristics of local consistency of the image. Finally, the remaining amplitude and phase information after eliminating the anomalies are integrated and rearranged according to the pixel coordinates, so that the corresponding frequency domain feature data can be obtained.

[0019] The formula is useful because it adds the influencing constant to the frequency domain phase difference , which can resist the violent phase fluctuation caused by some image noise and provide a stable quantitative basis for subsequent analysis based on phase change, thus helping to accurately extract the overall phase change trend in dynamic image sequences; The parameter acquisition step is to perform frequency domain conversion on the complex form of each pixel in a frame of image and extract the complex angular distribution, and regard the angular distribution value as the phase value. In the on-site detection, the gold wire sample is photographed frame by frame with the camera exposure time ranging from 1 / 2000 second to 1 / 500 second, and the grayscale matrix obtained from each shot is fast Fourier transformed. In addition, the abnormal pixel points that are too bright or too dark are eliminated in conjunction with the brightness distribution data of the same frame, so that noise interference can be reduced when extracting the angular distribution. At the same time, by statistically analyzing multiple batches of data, the phase value distribution of a total of 500 frames is recorded, among which the phases of most pixels are concentrated between -3 degrees and 3 degrees, and the very few pixels that exceed this range are marked as invalid values ​​in the aforementioned abnormal elimination process. For example, under a relatively stable imaging condition, the statistical results of the phase range detection of a frame of image show that more than 90% of the pixel phase values ​​fall between -2.5 degrees and 2.8 degrees, which can be directly used for Subsequent calculations; The parameter acquisition step is to count the number of frames in the continuously acquired image sequence. After completing the shooting task for a certain period of time, the total number of frames used for analysis can be accurately obtained. This number of frames is closely related to the phase difference calculation range. By linking it with the actual production efficiency and detection time on site, each monitoring cycle is set to 2 seconds to 5 seconds. During this period, the industrial camera collects 50 to 125 frames, corresponding to In the range of 50 to 125, if the detection granularity needs to be improved on site, the shooting frame rate can be appropriately increased and corrected. , the final number of frames used for calculation is determined based on the number of valid frames obtained from multiple tests For example, in a 2-second acquisition process, 60 frames are obtained. , which can be directly brought into subsequent calculations; The steps for obtaining the parameters are as follows: when analyzing phase noise, the phase difference value and the low-light interference in the imaging environment, as well as the frequency characteristics of mechanical vibration, are recorded frame by frame by frame comparison. The amplitude data fed back by the vibration sensors installed on site around the processing device are converted into the jitter range that may affect phase detection. According to statistics, the vibration amplitude captured by the vibration sensor is between 0.001g and 0.008g under the operation status of multiple workstations. G represents the acceleration of gravity of 9.8m / s². Combined with the distribution evaluation of the camera imaging noise, the final influencing constant is introduced. The value is between 0.01 and 0.03. If the vibration is large, a higher value is taken, and if the vibration is low, a lower value is taken. For example, in the detection link of a semi-automatic production line, the vibration is relatively stable. The amplitude of 0.0025g recorded by the vibration measurement is used for quantification. After comparing the phase noise distribution diagram generated in the imaging, the phase noise distribution diagram is finally Used as the influence constant for this production line; Calculation process: The first step is to determine , Under the premise of reading the phase data of frame 1 to frame 60, let represents the phase values ​​obtained in sequence, The second step is to calculate the phase difference between adjacent frames one by one , substituting the obtained phase difference into And add , take the square root of the sum, and then accumulate the sum until , The third step is to divide the above accumulated result by , thus obtaining ; For example, in a certain calculation, the differences of 59 adjacent frames are obtained in sequence, which are [0.5, 2.1, 1.9, ...] respectively. The specific calculation is as follows: ; The sum of all 59 results is recorded as ,but: ; The results show that in the 60-frame image sequence monitored, the phase change is about 1.231 degrees. Subsequent analysis can reveal the phase fluctuation of the entire gold wire during this period, and provide a reference for subsequent high-pass filtering and edge enhancement.

[0020] Based on the phase change, high-pass filtering is applied to enhance the edge features of the gold wire and obtain the gold wire edge spectrum. In the operation, the fluctuation information list obtained by statistics in the previous step is first read, and the list is compared with the original phase matrix pixel by pixel. When it is found that some phase fluctuations are at a high level, the emphasis on high-frequency components is appropriately increased when calculating the high-pass filter kernel. At this time, a high-frequency offset coefficient can be set with a value range of 1.2 to 2.5, which is obtained from the statistics of the local phase intense area. For example, by detecting 90 pixels around a gold wire solder point, the energy distribution of its frequency domain characteristics is quantified, and it is found that about 70% of the pixels have a high-frequency component proportion of nearly 40%. Based on this proportion, the offset coefficient in the high-pass filter kernel is derived to be approximately 1.8, and then apply the coefficient to the convolution operation of the current frame to amplify the frequency domain changes between adjacent pixels, and then superimpose the noise information extracted in the illumination interference analysis stage, mark the abnormal pixels suspected to be caused by sudden illumination changes during the filtering process, and do not perform independent amplification after marking. And by processing the pixels with obviously abnormal high-frequency amplitude separately, calculate the root mean square of the difference between its surrounding pixels and its memory level. If the result is greater than the preset critical value of 5, this pixel will be taken as a key investigation target and automatically added to the abnormal list. In this way, after traversing all frames, the corresponding high-pass filtering result sequence is obtained, and finally the high-frequency component information in the sequence is integrated and output as a gold wire edge spectrum for further segmentation in the next stage.

[0021] The steps for obtaining the enhanced edge feature map are: Receive the gold wire edge spectrum, calculate the horizontal gradient component and the vertical gradient component respectively, calculate the gradient value of each pixel point by applying the gradient operator through convolution operation, and synthesize the overall gradient amplitude to obtain the original edge intensity map; Based on the original edge intensity map, the edge enhancement value is calculated using the following formula: ; in, is the edge enhancement value, Pixel The horizontal gradient component of Pixel The vertical gradient component of A constant to avoid division by zero errors; Based on the edge enhancement value, noise is suppressed through nonlinear filtering to generate an enhanced edge feature map.

[0022] Specifically, after receiving the gold wire edge spectrum obtained above, we first need to confirm the pixel coordinate information and amplitude distribution data corresponding to the edge spectrum, and record the initial gradient measurement environment of all pixel points. After obtaining these basic data, we can refer to the previously statistical pixel intensity variation range (for example, the peak grayscale observed on-site using a 2-megapixel industrial camera is about 255, and the lowest grayscale is about 0) and the spatial position of the pixel distribution, and extract and match the gradient operator suitable for horizontal and vertical convolution row by row and column by column. In practice, this operator usually uses a discrete template of size 3×3 or 5×5. The specific selection should be determined through actual comparison. After referring to the edge sharpness of the adjacent parts of the gold wire and the distribution of noise points, a trial calculation can be performed in the local area. When the accumulated noise points in a certain area exceed the preset threshold (for example, 15 noise points or more), it is necessary to detect the original edge texture of the area. If the texture density is too low, the size of the gradient operator template should be reduced to avoid amplifying the noise. If the texture density is high, the 3×3 or 5×5 template can be kept unchanged. Then, the horizontal gradient component and vertical gradient component of each pixel are calculated. In this step, the brightness value between each pixel and its adjacent pixels is convolved to extract the brightness change in the horizontal direction and record it as , the brightness change in the vertical direction is recorded as , and perform absolute value operations on these two components to avoid negative value interference. At the same time, it is necessary to classify and manage specific gradient values ​​that are too high or too low. Specifically, the high and low gradient intervals can be determined according to the gradient distribution range extracted from multiple batches of gold wire images in the early stage (for example, the high gradient interval is located in the range of 40 to 255, and the low gradient interval is located in the range of 0 to 10). When it is found that the gradient value at a certain pixel exceeds the normal range, its neighboring pixels will be further compared. If the neighborhood also contains continuous high or low value gradients, the location of the pixel is marked as a suspected high noise point or low contrast area, and the corresponding compensation is matched according to the light intensity measurement results of the acquisition environment (ranging from about 200 to 600 lux). This can provide a more reliable reference when integrating the horizontal gradient component and the vertical gradient component of a single pixel in the subsequent process. Then, the overall gradient amplitude is obtained by synthesizing these two gradient components in a vector manner, and the gradient amplitude of each pixel is uniformly written into the gradient map and other pixels are continuously traversed until the entire image processing is completed. Finally, the obtained gradient amplitude information is summarized and a two-dimensional matrix is ​​used to draw the original edge intensity map.

[0023] The benefit of the formula is that by performing cubic processing on the horizontal gradient component and the vertical gradient component respectively, a more sensitive response can be obtained in the edge area, and at the same time, The structure can maintain the stability of the operation under extremely small gradient conditions, so as to better highlight the local significant changes when actually detecting the gold wire image containing fine edge features.

[0024] parameter Steps to obtain: This parameter represents the pixel The horizontal gradient component at the position is obtained by performing horizontal differential convolution on the original edge intensity map obtained previously. Its value range is usually determined by the brightness dynamic range of the on-site image and the actual measurement scene. For example, when using an industrial camera with a resolution of 1920×1080 to shoot gold wire, in a workshop environment with stable lighting, after multiple batches of acquisition and calibration, it is found that the brightness change of pixels usually falls in the range of 0 to 255, and the horizontal gradient value also fluctuates between -255 and 255. In order to obtain more accurate gradient results, it is necessary to perform detailed analysis on the edge area in actual implementation. For example, in an image with a resolution of 600×800, all rows can be traversed, the grayscale difference between adjacent pixels can be recorded as a temporary list, and the Sobel operator or Prewitt operator can be used to calculate When the absolute value of the horizontal gradient value of at least 100 pixels is detected to be higher than 200, its distribution is counted to confirm whether the high value is related to excessive illumination or camera exposure delay. Then, combined with the illumination measurement of the real production line (for example, the average illumination is 300 lux under incandescent lighting and the average illumination is 500 lux under LED lighting), the illumination information is quantified into a correction coefficient of 0.2 to 0.5, and finally the correction coefficient is multiplied by the adjacent pixel difference result and updated. The actual value of , gives a calculation example: The pixel grayscale is 180, The pixel grayscale is 90, and the correction coefficient is set to 0.3. , thus the horizontal gradient component at the pixel point can be obtained.

[0025] parameter Steps to obtain: This parameter represents the pixel The vertical gradient component at the position, the acquisition method and Similar, but different in that it needs to be traversed column by column, reading and The grayscale difference between the two pixels can also be calculated by using traditional gradient operators such as the Sobel operator to complete basic operations, and the results need to be corrected by the illumination data or noise distribution measured in the field. For example, in the same 600×800 image, when more than ten consecutive lines in the vertical direction are detected to have a large brightness difference (such as an absolute value exceeding 220), it is possible to first verify whether the area is within the range of highlight reflection or shadow influence, and use a correction coefficient in the range of 0.25 to 0.6 to make corresponding adjustments to the current gradient value, such as for pixels and If the grayscale difference is -150 and the illumination value of the observation scene is 400 lux, it can be mapped to the correction coefficient range of 0.4, thus , and then take the absolute value in the subsequent calculation and put it into the formula. Through the same type of operation, all pixels can be traversed to obtain the complete vertical gradient component.

[0026] parameter Steps to obtain: This parameter is a constant introduced to avoid zero division error. The value can be determined based on the minimum value of the gradient amplitude in the edge area, the dark current noise level of the shooting environment and other factors. When shooting gold wire with an industrial camera, weak noise will be generated. Through the evaluation of the camera dark current under laboratory conditions (temperature 25°C, humidity 50%), it was found that the random noise measured under the minimum exposure condition caused the pixel brightness to fluctuate by 2 to 5 gray levels. Combined with the analysis of 100 images, the average minimum gradient amplitude was calculated to be about 1.5, so it was selected. In the range of 1.0 to 2.0, for example, in an actual workshop with a temperature of 30°C, slightly higher dark current noise makes the minimum gradient amplitude measured to be about 1.9, then you can set , and directly substitute this value in subsequent calculations. If higher noise is encountered in local detection, the To about 2.0, here is a specific example: in the image previously counted, the minimum gradient amplitude is 1.7. According to the camera noise record, select , and write it into the denominator of the formula.

[0027] Calculation process: The first step is to obtain the and , and then take the absolute value and cube it, we get and , The second step is to replace and Add and The sum is squared to form , In the third step, divide the sum of the results of the first step by the result of the second step to get , Given a practical example, when , , When , the operation is as follows:

[0028] Denominator: ; Take the square root ; therefore, ; The result shows that this pixel has a large gradient in both the horizontal and vertical directions, indicating that the local edge feature is significant. A higher value often indicates a more obvious edge. If the value is lower than 50, it usually indicates that the gradient of the neighboring pixels changes little. In the subsequent nonlinear filtering process to suppress noise, this value can be used as a reference to determine whether to perform stronger smoothing or hold processing on the point.

[0029] According to the edge enhancement value matrix obtained above, the numerical distribution of all pixels needs to be summarized and compared within the same range. When a pixel enhancement value is found to be extremely high or low, the difference between the enhancement value and the surrounding pixels is recorded. Here, a preliminary threshold range can be set. For example, based on the statistical results of the gold wire image collected under different lighting conditions, the maximum and minimum enhancement values ​​in the past 300 frames are extracted, and their average values ​​are used as the segmentation interval with a fluctuation of 10% up or down. Then, the pixel enhancement value is compared with the interval, and then combined with the additional fine-tuning amount (for example, 0.05 to 0. 15) to determine whether the pixel is classified as a high outlier or a low outlier. If it is regarded as a high outlier, it will be moved from the temporary queue to the queue for special inspection, and other pixel enhancement values ​​within a certain range above, below, left, and right (for example, a 3×3 or 5×5 neighborhood) in the queue for special inspection will be captured for repeated comparison. In this way, an inspection dimension of local noise proportion can be constructed, and then the local noise proportion can be associated with the previously detected illumination change amplitude or mechanical vibration frequency. The specific method is to extract the lux value at a certain moment in the actual workshop illumination intensity measurement recorded in advance, or in the vibration. The vibration acceleration of a certain minute period is extracted from the historical readings of the motion sensor, and the two parameters are quantified and mapped. For example, the light lux value of 400 can be mapped to 0.4, and the vibration acceleration of 0.003g can be mapped to 0.2. They are multiplied with the local noise ratio item by item. If it is found that the product exceeds the predetermined threshold (for example, set to 0.08), it is further confirmed whether the current pixel has repeated records in the high noise list. When the number of repeated records is greater than three, the smoothing coefficient needs to be increased during the nonlinear filtering process so that it can reduce the fluctuation amplitude of these abnormal points more quickly. In addition, the same method is also performed on low outliers. The operation is similar to that of the previous one, except that the enhanced value is matched with the current threshold interval in the lower limit area. Once the judgment condition is met, the pixel is marked as being in a range with a weak gradient and may be interfered by dark current or other low brightness. Then, these marked points are filtered row by row or column by column. In the process, the enhanced value of each pixel is differentially measured with the adjacent pixels for multiple times, and the mean square value of the difference is calculated. If the mean square value is lower than a pre-established smooth judgment line (such as 10), it is regarded as an overall smooth area. Finally, the pixel information that has been judged is summarized, the parallel processing results are merged and output as the final enhanced edge feature map.

[0030] The steps to obtain the gold wire area distribution map are: Receive the enhanced edge feature map, obtain the brightness gradient value of each pixel, extract the gradient components in the horizontal direction and the vertical direction, and obtain a brightness gradient value map; Based on the brightness gradient value map, the adaptive threshold is calculated using the following formula: ; Where T is the adaptive threshold, Pixel The brightness gradient value, and are the number of rows and columns of the image, respectively. To avoid division by zero; Based on the adaptive threshold, the brightness gradient value map is classified into image blocks, isolated noise blocks are removed, the gold wire area distribution is determined, and the gold wire area distribution map is generated.

[0031] Specifically, the enhanced edge feature map obtained above is received, and the pixel coordinates of all rows and columns are first scanned in the memory and the pixel index is established. The grayscale values ​​of adjacent pixels in each row are differentiated to obtain the horizontal brightness gradient. When performing the differentiation, the previously measured pixel grayscale range (0 to 255) is referred to. When the absolute value of the differential value exceeds 50 for three consecutive times, the high gradient characteristics of the pixel in this row and column are recorded and marked with a value of 1. If it does not exceed 50, it is marked as 0. The brightness difference of the mark and the adjacent pixels is accumulated and stored in a row vector. After all columns are traversed, the horizontal gradient set can be obtained. Then, the column direction is operated in the same way to obtain the vertical gradient. The grayscale values ​​of adjacent pixels in each column are differentially analyzed. When the absolute value of the difference in several consecutive rows exceeds 60, it is marked. The current pixel has a high gradient characteristic and is assigned a value of 2 and stored in the vertical gradient set. Then, the horizontal and vertical gradient information are compared pixel by pixel. When the horizontal gradient flag and vertical gradient flag of a pixel are both greater than 0, it is marked as 3 in the merged matrix to indicate that the pixel has significant brightness changes in both the horizontal and vertical directions. At this time, the merged matrix is ​​repeatedly traversed to check whether there are blank pixels that have not been updated. If so, the gradient difference between it and the surrounding pixels is calculated within the range of 3×3 or 5×5 according to the neighborhood method. Once the calculation results for three consecutive times are higher than the previously established threshold of 40, it is marked as a high-brightness gradient pixel. In this way, both the edge area and the transition area can be marked. Finally, the merged matrix is ​​converted into a unified brightness gradient value map and output.

[0032] The benefit of the formula is that through the interactive calculation of the cube root and the numerator and denominator, the degree of pixel brightness gradient change in the row and column directions of the image can be comprehensively evaluated, so that when the numerator multiplies the square of the horizontal and vertical differences, it can highlight the large gradient area while taking into account the overall balance, and combine the constant in the accumulation of the absolute value of the denominator. By offsetting the potential zero division problem, the threshold suitable for different lighting and noise environments can be generated more sensitively when used in batches of golden wire images.

[0033] parameter Steps to obtain: This parameter represents the pixel The brightness gradient value is obtained by extracting the gradient value of each pixel from the brightness gradient map mentioned above, and combining the gradient information in the horizontal and vertical directions after adjusting the brightness of the original image obtained by the industrial camera. In order to accurately represent the brightness change, When performing the grayscale correction, it is necessary to distinguish between the grayscale component and the noise component. For example, the area with a grayscale greater than 220 in the picture is marked as a highlight area, and the area with a grayscale less than 30 is marked as a low-brightness area. The corresponding noise correction amount needs to be added or subtracted from the gradient value (this noise correction amount comes from the dark current test of the industrial camera at 25°C. Statistics show that it will cause an error of 1 to 4 grayscale levels on average). Then, most of the corrected pixel gradient values ​​are clipped to prevent the extreme values ​​from being too large. In actual on-site detection, if a 600×800 image is collected, the corresponding Between 1 and 600, The brightness gradient value is between 1 and 800. In multiple tests, it is also observed that the brightness gradient value is usually maintained in the range of 0 to 300. Sometimes higher values ​​may appear near the solder joints. By comparing the on-site illumination and shooting distance, the excessively soaring gradient values ​​are reviewed again and finally Unified into a two-dimensional matrix, it is convenient for subsequent statistical calculations. For example, in a sampling, statistics show that 80% of the pixel gradient values ​​are concentrated in the range of 10 to 120. At this time, local extreme value detection can be intuitively established to further ensure credibility.

[0034] parameter Steps to obtain: This parameter indicates the number of rows in the image. It is directly determined by the image height collected by the industrial camera. In actual production lines, it is usually between 600 and 2000. The specific number varies with the camera resolution and shooting configuration. For example, the commonly used 1920×1080 resolution scene can use , in certain cases, if the upper and lower edges of the image are cropped, then It will be reduced to 800 or other numbers accordingly. Generally, in multiple repeated detections, the complete row and column parameters will be recorded first and written into the image tag information for storage. For example, As a key attribute bound to the current image, to avoid mismatching of rows and columns in subsequent operations, given an example: in a detection task, the camera resolution is set to 1440×1080, then , and then construct a row index based on this value to retrieve the brightness gradient distribution row by row.

[0035] parameter Steps to obtain: This parameter indicates the number of columns of the image. This is also determined during the acquisition phase, usually between 800 and 3000. For example, for a 1920×1080 resolution image, When the on-site equipment uses automatic cropping to retain only the area of ​​interest, the left and right parts of the image will be cut. It may be reduced to 1500 or less. To ensure sufficient observation of the gold wire area, the number of columns is not changed arbitrarily during the entire production process. For example, in a certain inspection, the camera output image resolution is directly 600×800. , record this value in the database and They participate in subsequent calculations together to ensure that the row and column dimensions match and that the imaging size during actual acquisition can be traced back.

[0036] parameter Steps to obtain: This parameter is a constant to prevent division by zero errors. It is used to provide stability when the absolute value of the accumulated result is too small or zero in the denominator. The row and column differences of the images measured on site may have extreme cases. For example, in an extremely uniform area, the brightness gradient of dozens of pixels in a row is close to 0. At this time, the denominator is at risk of approaching zero. Therefore, after observing several batches of images, a positive number less than 3 is taken as , which can usually be selected between 0.5 and 1.5. In specific tests, if the overall noise level of the image is slightly higher, it can be increased To 1.0 or above, so that the calculation process of the formula remains stable. For example, in a welding inspection scene, the collected local gold wire brightness distribution is extremely smooth, and the accumulated denominator is only about 5. This avoids division by zero and ensures a smooth transition of the result.

[0037] Calculation process: The first step is to use the number of rows recorded previously With the number of columns , traverse the pixel coordinates in sequence ,take out Its neighboring pixels , And calculate the brightness gradient difference, then and Multiply them and add the resulting product to the numerator. The second step is to calculate the denominator And accumulate all pixel results, and after completing the traversal, add the accumulated value to the constant Add, The third step is to perform a ratio operation on the numerator and denominator and take the cube root to obtain the final adaptive threshold. , Given a numerical example: Assume , , , in actual statistics, the numerator , the denominator , so the denominator is ,and then: ; ; The result shows that when the numerator and denominator are at this order of magnitude, the adaptive threshold is approximately 2.884. In the subsequent image block classification, pixels or regions with gradient values ​​higher than 2.884 can be regarded as candidate high gradient areas, while those lower than 2.884 are classified as ordinary areas. In subsequent operations, this threshold will be used as the basis for segmentation to detect and locate the main distribution range of the gold wire.

[0038] Based on the adaptive threshold obtained in the previous step, the brightness gradient value map is classified into image blocks. During the execution process, the brightness gradient value map is first divided into several small blocks according to the row and column dimensions, and each block contains an equal number of pixels (for example, 16×16 or 32×32 per block). When traversing these small blocks, the number of pixels in the block whose brightness gradient values ​​are greater than the threshold obtained previously is first counted. If this number exceeds 300 or more in a certain block, it is registered as a high-gradient block in the block-level marking table, and whether the high-gradient phenomenon also occurs in the adjacent blocks is recorded. In this way, continuous or discrete high-gradient area distributions can be gradually spliced ​​out. When an isolated block is encountered (for example, all surrounding blocks do not belong to high gradient, and only the block itself is marked as high gradient), it is regarded as a noise block, and a local check list is established to register it, and then check The overall brightness information of pixels in isolated blocks, if the brightness mostly falls between 10 and 30 or is higher than 230, it means that it may be an abnormal distribution caused by being too dark or too bright. For such high-gradient noise blocks, they will be excluded from the subsequent regional distribution map. After all blocks are classified, the high-gradient blocks are numbered and their positions are recorded, and then the high-gradient blocks with connected numbers are regarded as the same candidate area for the gold wire area. When several consecutive blocks marked as high gradient appear around the gold wire welding point or the gold wire extension, these candidate blocks are merged into a gold wire area strip, and then these strips are simply connected or truncated in combination with the gold wire shape reference parameters (such as length, direction, etc.) collected in advance on site. Finally, the coordinate set of the gold wire area distribution is summarized, and the gold wire area distribution map is obtained by re-packaging and merging.

[0039] The steps to obtain the gold wire super-resolution map are: Receive the gold wire area distribution map, analyze the edge detail information of the local area, and obtain a preliminary super-resolution image through image interpolation correction; Based on the preliminary super-resolution image, noise removal and contrast equalization are performed to generate a gold wire super-resolution image.

[0040] Specifically, the gold wire area distribution map obtained above is received, and first, the pixel coordinate range of each local block in the image is recorded and the edge pixel rows and columns where texture transition may exist are identified. To analyze these edge details, the brightness change data can be read row by row and column by column, and the pixel points that meet the edge judgment criteria are concentrated as refinement targets. For example, when the threshold value counted in the previous step is between 30 and 50, if the brightness difference exceeds this range, it can be listed as a position that needs to be interpolated, and then interpolated and amplified according to a pre-established multiplication factor, such as 2 or 3 times. During interpolation, the brightness distribution of the left and right pixels and the upper and lower pixels is averaged or weighted, and the distance and brightness difference between the adjacent pixels and the current pixel are recorded in the weight setting. If the distance is too close and the brightness difference is higher than 10, it is assigned A slightly lower weight is given when the distance is relatively far and the brightness difference is stable in the range of 5 to 10. This practice is defined by the user in the interpolation configuration file in numerical form. After interpolation, the surrounding grayscale gradient of the result pixel is checked point by point. If there is a transition discontinuity in three consecutive rows or columns in the same interpolation area, the current interpolation multiple is adjusted to 1.5 or 2.5 or other values ​​for recalculation. When all positions where local jumps may occur have completed interpolation compensation, the global edge details are compared again. During the comparison process, the edge abnormal points that accumulate more than the specified value of 30 are recorded in a list, and then the interpolation correction logic is called to process these abnormal points. After all steps are completed, they are spliced ​​back to the pixel matrix of the overall image, and finally a preliminary super-resolution image is output.

[0041] Based on the preliminary super-resolution image obtained above, we first separate the brightness information of all pixels in the target range and perform noise removal. When performing noise removal, we need to first record the data from on-site detection. For example, in the noise distribution curve collected under an illumination of 500 lux, if it is found that the brightness fluctuates randomly between 0 and 5 gray levels, then the fluctuations below 5 can be set as the normal range. If the fluctuation amount of a block of continuous pixels exceeds this set range, it will be marked as a high noise point. For this type of high noise point, the three-point or five-point adjacent mean method can be used for smoothing. Combined with the edge position list accumulated in the interpolation stage, the intersection of the noise point distribution and the edge position is compared. When a fluctuation occurs, When the dynamic value is always higher than 10 and continuously spans four lines or more, this area is recorded as a strong noise segment, and the smoothing coefficient is increased to weaken the random peak. Then, the brightness distribution of the entire image is globally traversed, and the number of highlight pixels and low-brightness pixels is counted. If the number of highlight pixels is greater than 800 and the number of low-brightness pixels is less than 50, or the difference between the two exceeds the specified ratio of 2 to 1, the contrast is balanced according to the recorded brightness difference distribution, and the highlight part is compressed downward by linear scaling, and the low-brightness part is expanded upward. The scaling coefficient is set by the user in the configuration between 1.2 and 2.0 and compared in multiple test scenarios. After the brightness of all pixels is smoothed and balanced, the gold wire super-resolution image is output.

[0042] The steps to obtain the gold wire motion features are: Receive the gold thread super-resolution image, locate the boundary pixels of the gold thread area, analyze the pixel displacement between adjacent frames, obtain the pixel trajectory, and obtain the pixel trajectory set of the gold thread area; Based on the gold wire area pixel point trajectory set, the pixel point displacement vector is obtained to generate the gold wire motion feature.

[0043] Specifically, after receiving the golden wire super-resolution image obtained above, it is first necessary to retrieve all suspected edge positions in the image row by row and column by column. In this process, the brightness distribution will be compared and pixels whose brightness gradient or color difference exceeds the set standard (for example, based on statistics of multiple batches of samples on site, the gradient mean of most real edges is about 15, so the area exceeding 15 is regarded as a potential boundary) are recorded as candidates. Then, these candidate edge pixels are locally verified. For example, it can be checked whether several pixels above, below, left and right also show similar gradient characteristics. If three or more consecutive rows meet the similar gradient phenomenon and are concentrated in a certain direction, they are regarded as real edge positions and their coordinates are confirmed. For the pixels with confirmed coordinates, adjacent frames need to be compared in subsequent steps. During the comparison, the specific coordinates of the boundary pixels in different frames are read in the order of image frame numbers. , and then perform difference calculation on these coordinates to obtain the displacement. When processing the displacement value, if it is found that it exceeds the preset threshold (such as 5 or above), check the displacement trend of several frames before and after it to distinguish whether it is a jitter error or normal movement. The threshold here is obtained by the joint observation of mechanical vibration and shooting rate in the workshop environment. For example, after counting 30 frames of images, it is found that most normal movements are between 3 and 5, and the situation of exceeding 6 is mostly related to mechanical vibration. Based on these records, high displacement is eliminated to make the final trajectory more accurate. After all frames are compared, the coordinate connection line of each pixel point in each frame is recorded as a trajectory. If multiple consecutive pixel point trajectories appear in the same edge area, these trajectories are integrated and merged into a gold wire edge trajectory group. After all trajectories are processed, the overall gold wire area pixel point trajectory set is obtained.

[0044] Based on the previously obtained gold wire area pixel point trajectory set, the moving direction and amplitude of each trajectory between adjacent frames are first counted, and the amplitude value is compared with the pre-set reference value. This reference value is determined by observing the robot arm motion model or actual working conditions. For example, in the welding production line, a numerical range of 3 to 6 is often set as the reference range. If the movement amplitude of a certain trajectory continuously appears to be larger than the upper limit of the range, it is recorded in a summary list to exclude extreme value interference when calculating the displacement vector in the next step. Then, the coordinate difference of the remaining normal trajectory is taken for component decomposition, and the horizontal displacement and vertical displacement are calculated. If there are multiple trajectories sharing similar displacement characteristics in a certain area at the same time, they are cross-checked to confirm that they are continuous movements generated by the same gold wire edge area. The displacement data of this part are merged and marked as the same vector group, and then all vector groups are subjected to mean or median filtering to eliminate random fluctuations. A threshold is usually set during filtering. If the amplitude is lower than 2 and only appears in one frame, it is merged into zero movement. If it is higher than 7, it is marked as an abnormal deviation and recorded separately. After these data are processed, a relatively stable final displacement vector set is obtained, that is, the gold wire motion feature is generated.

[0045] The steps to obtain the gold wire motion trajectory are: Receive the motion features of the gold wire, extract the displacement vector data in the time series, build the time dependency, and obtain the motion state sequence; Based on the motion state sequence, the position of the gold wire is calculated, and the expression is: ; in, For the moment The position of the gold thread, is the velocity at the previous moment, is the acceleration at the previous moment, is the time interval, is the random disturbance term; Based on the position of the gold wire, the trajectory of the time series data is reconstructed, and the motion trajectory of the gold wire is generated by smoothing the motion trajectory through curve fitting.

[0046] Specifically, after receiving the previously obtained golden wire motion features, it is necessary to select the displacement vector data between adjacent frames in the recorded time series, and perform index matching of position, velocity and acceleration on these data. When selecting the index, first sort them in the order of frame numbers, for example, read the displacement vector values ​​one by one from frame 1 to frame N, and then make a two-dimensional table in the form of row and column coordinates, where each row represents the pixel position corresponding to a certain frame, and each column corresponds to a specific coordinate component. When the table is constructed, all data in the displacement vector are traversed, and the numerical values ​​of continuous displacement differences are counted starting from the switching points of adjacent frames. When it is found that the displacement difference exceeds a preset threshold (for example, 6 or more), the record is recorded in the list and excluded from the sample to reduce the influence of vibration interference on subsequent time dependence. This preset threshold can be determined through a large number of experiments. For example, under normal golden wire motion, most of the displacement differences are concentrated in 2 To 5, the situation that exceeds 6 is often recorded separately and eliminated when constructing the time dependency relationship later. Then, according to the screened displacement vector, the continuous changes in the time series are tracked one by one. If the displacement within five adjacent frames is monotonically increasing or decreasing, it is recorded as a smooth movement. If there is fluctuation or jitter, it is marked as a dynamic fluctuation segment in the additional file. At the same time, the data of the robot arm or vibration source is recorded according to the working mode of the workshop equipment during operation. For example, the movement frequency of the robot arm is in the range of 3 to 5 Hz within one hour, and the vibration sensor data is in the range of 0.002g to 0.008g. These values ​​are then mapped into displacement correction coefficients (such as 0.3 to 0.7) and integrated into the final displacement vector table. In this way, a time dependency relationship that closely corresponds to the time sequence can be constructed. Finally, the corresponding sets of all time and displacement vectors are integrated into a motion state sequence.

[0047] The benefit of the formula is that it takes into account the impact of the velocity and acceleration of the previous moment on the current position, and adds a random disturbance term based on the position difference between the previous moment and the previous two moments at the end, which makes it more flexible in dealing with local jitter or tiny jumps of the gold wire.

[0048] parameter Steps to obtain: This parameter indicates the speed at the previous moment. The value can be obtained by combining the position change of adjacent frames with the actual frame rate. In an industrial production line, if the camera collects at 60 frames per second, the time interval between two frames is about 0.0167 seconds. With frame The speed can be obtained by subtracting the position of the gold wire and dividing it by the time value. Then, pixels per second or millimeters per second are selected for quantification according to the unit. This quantification process requires recording the camera focal length, shooting distance, pixel size and other data in advance. If the gold wire moves 5 pixels in one second, it can be converted into 0.1 mm on the image. This can also meet most gold wire detection accuracy requirements. For example, in a batch of measurements, the actual movement of the gold wire is mostly 2 to 8 pixels, so Usually it is between 0.2 mm / s and 0.4 mm / s. In the final application, the frame rate, spatial size conversion factor, and the last position difference are all calculated to obtain the value that can be put into the formula. .

[0049] parameter Steps to obtain: This parameter represents the acceleration at the previous moment, which is obtained by the difference of velocity. For example, first find two sets of adjacent velocities and The difference, divided by the same time interval , if the sampling frequency is set to 60 frames per second, then About 0.0167 seconds. The unit of acceleration can be expressed as pixels per square second or millimeters per square second. In on-site testing, if the speed of the gold wire fluctuates between about 0.3 mm per second and 0.5 mm per second, the acceleration may fall within the range of 0.01 mm per square second to 0.05 mm per square second. It is necessary to verify the jitter records exceeding this range. For example, when it is found that the acceleration is continuously greater than 0.07 mm per square second for three frames, it is recorded in the annotation file to check for unexpected large-scale movements of the robot arm or the gold wire welding point equipment. Only the data with a high probability of being true is retained later to obtain a more stable Value, give an example: if the frame The speed is 0.4 mm per second, frame The speed is 0.35 mm per second. seconds, then .

[0050] parameter Steps to obtain: This parameter is the time interval, which depends on the image acquisition frame rate. In industrial applications, it is usually set between 30 frames per second and 120 frames per second. It is selected based on production efficiency and detection accuracy. When the frame rate is fixed, That is 1 / frame rate. For example, when the frame rate is 60 frames per second, In actual detection, the camera acquisition stability must be maintained to check whether there is frame loss or frame skipping. If frame skipping is found, the timestamp needs to be added to the record table and corrected by comparing the image timestamp. For example, a test record indicates that there is a 1.5-fold interval between frame 10 and frame 11. In the subsequent calculation, this interval can be Made revisions to improve the accuracy of timing analysis.

[0051] parameter Steps to obtain: As a random disturbance item, this parameter needs to be quantified in combination with the vibration and illumination changes or mechanical jitter statistics in the actual production environment. When monitoring the gold wire welding process, the corresponding amplitude can be obtained through vibration sensors and ambient light meters, and then mapped to a random range of 0.1 to 0.5 to serve as the disturbance size. For example, if the vibration acceleration is between 0.002g and 0.006g, it can be converted into a random disturbance weight of 0.15 to 0.3. If the illumination changes drastically (for example, the illumination increases from 300 lux to 600 lux in a short period of time), the disturbance component can be pushed up to 0.4 or even larger. After all monitoring records are completed, a value is extracted from the random distribution formed by the vibration value and illumination value of each frame and assigned to it in each iteration. , which is used to simulate the influence of random floating on the position of the gold wire. For example, in an actual measurement, after a two-hour data collection, it was found that the average value recorded by the vibration sensor was about 0.003g. It is set around 0.2 and fluctuates slightly up and down during the process.

[0052] Calculation process: The first step is to change the last moment position Substitute the initial value and get the speed at the previous moment With acceleration , and the position difference between the previous frame and the previous two frames After doing square root operation and random perturbation Multiply, The second step is to get the frame rate ,Will and and Multiply and sum them in sequence, and then add the aforementioned disturbance value, The third step is to record the above results as , we can get the time The position of the gold thread, Given a practical example, take (Unit: pixel), (Unit: pixels / second), (Unit: pixels / second ), Second, , random perturbation , then the calculation is as follows: ; ; ; ; ; This result shows that under given velocity, acceleration and random disturbance conditions, The position of the gold wire is approximately 10.365 pixel coordinates. The larger the value, the corresponding displacement of the gold wire in the image occurs within the time interval. If the position of subsequent consecutive frames exceeds a certain range, it means that the gold wire is moving quickly. If the position remains stable or changes slightly, it means that the movement trend is slowing down or stabilizing. The above values ​​will be recorded one by one in the subsequent trajectory calculation and brought into the calculation process of the next frame.

[0053] Based on the previously obtained gold wire position, the trajectory of the time series data is reconstructed. First, the The values ​​are arranged side by side into a sequence, and then the adjacent positions are compared by difference. The local extreme value is searched in the time period when the difference continuously rises or falls. For these extreme value points, a segment of curve fragments can be constructed. When the local extreme value exceeds the set benchmark (for example, the average position increment of five consecutive frames exceeds 4), the part is recorded as high fluctuation in the annotation table, and then polynomial fitting or spline fitting is further selected for curve splicing. When fitting, the adjacent frames are sampled point by point. Position, map it to two-dimensional coordinates and connect them. When the fitting residual exceeds the predetermined standard (such as 0.5 pixels), a new round of fitting is performed and the previous high outlier points are skipped. All time points are traversed step by step according to this method. Finally, the continuous curves completed by fitting are integrated into a complete smooth motion trajectory and output as a golden wire motion trajectory.

[0054] The steps to obtain the golden wire segmentation boundary are: Receive the gold wire motion trajectory and gold wire motion features, extract the outer contour pixel points of the gold wire area in all frames, and construct a set of gold wire area boundary points; Based on the set of boundary points in the gold wire area, the boundary point noise suppression factor is calculated. The calculation formula is: ; in, is the boundary point noise suppression factor, Respectively represent the horizontal and vertical gradient values ​​of the boundary points, They represent the edge gradient values ​​of the corresponding pixels in the gold wire super-resolution image, Represent the second-order derivatives of the edge gradient in the horizontal and vertical directions, respectively. A constant to prevent division by zero; Based on the noise suppression factor of the boundary points, the noise points are removed, the boundary curve is corrected, and the golden wire segmentation boundary is generated.

[0055] Specifically, after receiving the previously obtained golden wire motion trajectory and golden wire motion features, after reading the determined golden wire area in each frame, it is necessary to summarize all pixel coordinates that can represent the outer contour row by row and column by column. When processing, the pixels in the area can be mapped to a two-dimensional coordinate table according to the frame number, and the coordinate distribution is scanned to find the edge pixel position whose color or brightness gradient is higher than the preset threshold (for example, between 15 and 20) and perform a local inspection on it. This threshold can be selected after obtaining the statistical distribution by taking multiple shots of golden wire samples of the same specification on site. If the brightness difference of the surrounding pixels in a certain area continues to be greater than the upper limit of the interval, it is marked as a high edge possibility, and the pixel is added to the outer contour candidate list, and then similar gradient differences are repeatedly detected in adjacent pixels to confirm that it is connected to the previously calibrated candidate pixels to form a closed or semi-closed boundary. When all frames are completely traversed and their respective outer contour columns are generated in each frame, After the outer contour list is formed, the outer contour lists of all frames are integrated and stored in segments according to the frame number sequence. If the outer contour mark appears repeatedly at the same position in multiple adjacent frames, a statistical comparison is performed to determine whether it is a fixed noise point or a real boundary. When pixels with consistent edge attributes appear in the statistical results for more than a specified number of times (for example, more than four consecutive frames), they are regarded as real outer contour pixels. If they only appear in individual frames or intermittent frames, they are recorded as discrete points or noise points. Then, the merged outer contour list is searched for large-scale noise points. If a large-scale cluttered distribution is found and the difference between the peripheral brightness and the center is too large, the area can be marked as a noise area and removed from the final boundary point candidate set. After all outer contour candidate pixels are confirmed, they are merged according to the frame number and pixel coordinates to construct a golden wire area boundary point set to retain the accurate outer contour marks formed in each valid frame.

[0056] The benefit of the formula is that by combining the superposition effect of the edge gradient and its second-order derivative in the numerator, it can more sensitively identify the sudden changes caused by local noise and introduce To balance the gradient scales in different edge directions, this formula will give a relatively higher noise suppression factor for pixels with obvious sharp fluctuations, thereby helping to locate and exclude pixels that do not have real boundary significance.

[0057] parameter The steps to obtain are: This parameter represents the horizontal gradient value at the boundary point, which is obtained by scanning the brightness changes of adjacent pixels in the gold wire super-resolution image. In industrial detection, it is necessary to first perform brightness projection on the visible light grayscale data or multi-channel image, and then use the difference or traditional gradient operator (such as Sobel) to traverse in the horizontal direction and accumulate it. , after being obtained, it is usually between -255 and 255. In order to process the absolute value in the subsequent calculation, in actual use, only positive values ​​will be recorded or negative values ​​will be mapped to the positive domain to facilitate unified operation. At the same time, the distribution beyond the actual gold wire brightness range (such as anomalies with an absolute value greater than 300) will be investigated. After accumulating multiple frames, confirm whether it is abnormal and invalid. In some cases, the imaging of the fine texture of the gold wire will lead to a slightly larger gradient, so it needs to be normalized in combination with on-site statistics before writing. For example, for statistics of 200 images collected at a certain workstation, the average horizontal gradient is about 30, and the highest reaches 240. If 300 is detected, it is likely to be a strong reflection caused by the light spot. In this case, the data can be marked as abnormal.

[0058] parameter The steps to obtain are: This parameter is the vertical gradient value. The calculation method is similar, except that the brightness difference is made between rows, and the grayscale changes of pixels at different row coordinates are taken and accumulated successively to obtain , which is closely related to the clarity of the vertical edge of the gold wire. If a vertical gradient in a certain place in the image is detected to fluctuate greatly in a short period of time (such as an absolute value that surges to more than 180), it will be marked. In the statistical stage, it will be checked whether it is a real high-contrast boundary or a local noise point. Finally, the absolute value of the confirmed and retained vertical gradient value is written If the position fluctuates in multiple frames, the cause can be inferred by combining the vibration sensor and light change data and assigning correction coefficients of different degrees. In actual measurements, most of the fluctuations can be found in a large number of pictures (hundreds of frames). Typical values ​​(e.g. 10 to 150) are considered to be excluded or specially marked before calculation if they exceed this range.

[0059] parameter The steps to obtain are: They represent the horizontal and vertical edge gradient values ​​of the same boundary pixel in the golden wire super-resolution image. It is necessary to locate the coordinates of the pixel on the golden wire super-resolution image first, and then perform local convolution or difference around the pixel to obtain the gradient values ​​in the horizontal and vertical directions. They are usually comparable. It can better reflect the fine structure of the gold wire itself. Because super-resolution processing will magnify the edge details, in the actual production line, the gold wire image can be layered using multi-scale or multi-directional operators, and then the gradient size can be evaluated at the subdivision level. If the gradient value is found to be too different from the original image (such as doubled or more), it is necessary to combine the shooting conditions and imaging magnification factor to extract the real distance on site for comparison. For example, at a certain solder joint, When it is as high as 200, you need to first check whether there is cable jitter when the industrial camera is shooting. If there is no abnormality, include the value in the final statistics and enter it into the formula.

[0060] parameter and The steps to obtain are: These two second-order derivatives illustrate the practice of re-differentiating the edge gradient to assess the degree of curvature or abruptness, which can be obtained from On this basis, a difference is made in more neighborhood ranges in the horizontal and vertical directions respectively, such as traversing 3 to 5 pixel ranges in the row or column direction respectively, and the gradient difference before and after each pixel is averaged and then further gradient is made, so that the second-order derivative is obtained. In industrial sites, a small range (such as a 3×3 pixel neighborhood) is usually selected for approximate calculation. If the difference between a pixel and its adjacent pixels is too large (for example, the absolute gradient difference is greater than 50), it can be marked in the next second-order derivative calculation to make the result more stable. Then the obtained second-order derivative is recorded in a temporary array and compared with other frames or neighborhoods in the same frame. For example, in a data analysis, when Up and down about.

[0061] parameter The steps to obtain are: This parameter is a constant to prevent division by zero, used to avoid the denominator Too small values ​​lead to unstable calculations. It is necessary to first make statistics on the horizontal and vertical gradient distributions of a large number of images in gold wire detection, especially in areas with extremely low brightness or uniform distribution. and It may be very close to 0. In order to prevent the denominator from becoming 0 or close to 0 in this nearly pure color background, it is necessary to introduce And set it in a small range, such as between 0.1 and 1.5. During testing on some production lines, it was found that values ​​exceeding 1.0 are prone to excessive smoothing, so the range of 0.5 to 1.0 is generally selected. If the image resolution is increased or the lighting is increased again, corresponding fine-tuning can be done. For example, if the smallest The statistical mean is about 0.2, then , to ensure that the denominator is always greater than 0.3.

[0062] Calculation process: The first step is to read the horizontal gradient of the current boundary pixel With vertical gradient , and the corresponding edge gradient value of the gold wire super-resolution image , and taking the second-order derivative of the latter, we get and , The second step is to convert the molecules Multiply and Multiply , add the two together and take the absolute value, The third step is to calculate the denominator , dividing the numerator by the denominator gives , Given a numerical example: Let , , , , the second-order derivative , , , then the operation is as follows: ; Molecular part , Denominator about, ; This result indicates that the noise suppression factor of the boundary pixel is high, and it can usually be regarded as an interference point, which can be deleted or smoothed in the final boundary marker. If the subsequent multi-frame comparison also shows that the point has a high noise suppression factor, it will not be retained when correcting the boundary curve.

[0063] Based on the boundary point noise suppression factor results obtained above, the check can be performed one by one. First, the boundary coordinates in each frame are associated with the corresponding factor values, and the pixels whose factors exceed the specified standard (for example, 8 or higher) are recorded. The standard can be set by the operator after statistical comparison in multiple actual detection scenarios. If the record shows that the same pixel has a large factor in adjacent frames, it is judged as a noise point and removed. If a large factor only appears in individual frames, it can be temporarily stored in a temporary list to check whether its specific number of occurrences reaches a certain threshold. When all frames are completed, After recording, all suspected noise points are summarized, and then the gradient distribution between these suspected points and the surrounding real boundary points is compared. If the difference is significant, the elimination operation is performed. After elimination, the curve interpolation is performed again to fill the missing parts and make the overall boundary more coherent. When the curve is interpolated, the nearby coordinates are smoothed by local fitting. For gaps with small spans, adjacent boundary pixels are interpolated to connect. If the span is large, the segment is marked as an incomplete boundary and it may be necessary to compare multiple frames of information again in the future. When all segment boundary curves are corrected, all corrected curves are merged to form the final golden wire segmentation boundary.

[0064] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A real-time image segmentation method for gold wire bonding, characterized in that: The following steps are involved: Input a gold wire image sequence, perform Fourier transform on the gold wire image sequence, extract frequency domain amplitude and phase information, and obtain a gold wire edge spectrum by calculating the phase difference between adjacent frames and performing superposition operations; perform gradient enhancement on the gold wire edge spectrum to generate an enhanced edge feature map; Based on the enhanced edge feature map, a brightness gradient value is obtained, an adaptive threshold is set to classify image blocks, and a gold wire area distribution map is obtained; Performing super-resolution reconstruction on the gold wire area distribution map to generate a gold wire super-resolution map; Based on the gold thread super-resolution image, the displacement vector of the pixels in the gold thread area between adjacent frames is calculated to establish the gold thread motion feature; the time series prediction of the gold thread motion feature is performed to generate the gold thread motion trajectory; Based on the gold wire motion trajectory, a set of gold wire region boundary points is calculated, and noise suppression is performed in combination with edge gradient information in the gold wire super-resolution image to generate a gold wire segmentation boundary.

2. The real-time image segmentation method of gold wire bonding according to claim 1, characterized in that: The steps for obtaining the gold wire edge spectrum are as follows: Input the gold wire image sequence, convert each frame of the image into the frequency domain through Fourier transform, extract the amplitude and phase information of each frame, and obtain the frequency domain feature data; According to the frequency domain feature data, the phase information between every two frames is differentiated to calculate the phase change. The calculation formula is: ； in, is the phase change, For the The phase information of the frame, is the total number of frames, is the influence constant; Based on the phase change, high-pass filtering is applied to enhance the gold wire edge feature to obtain the gold wire edge spectrum.

3. The real-time image segmentation method of gold wire bonding according to claim 1, characterized in that: The steps of obtaining the enhanced edge feature map are: The gold wire edge spectrum is received, the horizontal gradient component and the vertical gradient component are calculated respectively, the gradient value of each pixel is calculated by applying the gradient operator through convolution operation, and the overall gradient amplitude is synthesized to obtain the original edge intensity map; Based on the original edge strength map, the edge enhancement value is calculated using the following formula: ； in, is the edge enhancement value, Pixel The horizontal gradient component of Pixel The vertical gradient component of A constant to avoid division by zero errors; Based on the edge enhancement value, noise is suppressed by nonlinear filtering to generate an enhanced edge feature map.

4. The real-time image segmentation method of gold wire bonding according to claim 1, characterized in that: The steps for obtaining the gold wire area distribution map are: Receive the enhanced edge feature map, obtain the brightness gradient value of each pixel, extract the gradient components in the horizontal direction and the vertical direction, and obtain a brightness gradient value map; Based on the brightness gradient value map, the adaptive threshold is calculated using the following formula: ； Where T is the adaptive threshold, Pixel The brightness gradient value, and are the number of rows and columns of the image, respectively. To avoid division by zero; Based on the adaptive threshold, the brightness gradient value map is classified into image blocks, isolated noise blocks are removed, the gold wire area distribution is determined, and a gold wire area distribution map is generated.

5. The real-time image segmentation method of gold wire bonding according to claim 1, characterized in that: The steps for obtaining the gold wire super-resolution map are as follows: Receiving the gold wire area distribution map, analyzing edge detail information of the local area, and obtaining a preliminary super-resolution image through image interpolation correction; Based on the preliminary super-resolution image, noise removal and contrast equalization are performed to generate a gold wire super-resolution image.

6. The real-time image segmentation method of gold wire bonding according to claim 1, characterized in that: The steps for obtaining the gold wire motion feature are as follows: Receive the gold thread super-resolution image, locate the boundary pixel points of the gold thread area, analyze the pixel point displacement between adjacent frames, obtain the pixel point trajectory, and obtain the pixel point trajectory set of the gold thread area; Based on the gold wire area pixel point trajectory set, a pixel point displacement vector is obtained to generate a gold wire motion feature.

7. The real-time image segmentation method of gold wire bonding according to claim 1, characterized in that: The steps for obtaining the gold wire motion trajectory are as follows: Receiving the gold wire motion feature, extracting displacement vector data in the time series, constructing a time dependency relationship, and obtaining a motion state sequence; Based on the motion state sequence, the position of the gold wire is calculated, and the expression is: ； in, For the moment The position of the gold thread, is the velocity at the previous moment, is the acceleration at the previous moment, is the time interval, is the random disturbance term; Based on the position of the gold wire, the trajectory of the time series data is reconstructed, and the motion trajectory of the gold wire is generated by smoothing the motion trajectory through curve fitting.

8. The real-time image segmentation method of gold wire bonding according to claim 1, characterized in that: The steps for obtaining the gold wire segmentation boundary are as follows: Receiving the gold wire motion trajectory and the gold wire motion feature, extracting the outer contour pixel points of the gold wire area in all frames, and constructing a gold wire area boundary point set; Based on the set of boundary points in the gold wire area, the boundary point noise suppression factor is calculated, and the calculation formula is: ； in, is the boundary point noise suppression factor, Respectively represent the horizontal and vertical gradient values ​​of the boundary points, They represent the edge gradient values ​​of the corresponding pixels in the gold wire super-resolution image, Represent the second-order derivatives of the edge gradient in the horizontal and vertical directions, respectively. A constant to prevent division by zero; Based on the boundary point noise suppression factor, noise points are removed, the boundary curve is corrected, and a gold wire segmentation boundary is generated.

Images