Ultramicro bonding copper wire winding abnormal state detection method, device, equipment and product

Abstract

The invention discloses a winding abnormal state detection method, device and equipment of an ultramicro bonding copper wire and a product, and relates to the field of copper alloy wire quality control. The method comprises the following steps: firstly, acquiring a dynamic reflection light spot real-time image of the ultramicro bonding copper wire; the dynamic reflection light spot real-time image is obtained by irradiating, shooting and capturing the ultramicro bonding copper wire by using a strip-shaped array light source which is sequentially lightened according to columns; capturing a front frame image and a rear frame image from the dynamic reflection light spot real-time image and carrying out image preprocessing; feature point extraction is carried out on the preprocessed light spot images of the front frame and the back frame; on the basis of the preprocessed light spot images and the light spot feature points of the front and rear frames, light stream estimation of the front and rear frames is calculated, and time-space information of the front and rear frames is obtained; the track consistency is judged based on the time-space information of the front frame and the rear frame, the current state detection result of the ultramicro bonding copper wire is determined, whether abnormity is prompted or not and the breakpoint area is prompted, and the detection efficiency and precision of the winding abnormal state of the ultramicro bonding copper wire can be remarkably improved.

Description

Technical Field

[0001] This application relates to the technical field of quality control of copper alloy wires, and particularly to a method, device, equipment and product for detecting abnormal winding states of ultra-fine bonding copper wires. Background Art

[0002] Bonding wires are parts that connect pins and silicon wafers to convey electrical signals and are indispensable core connection materials in semiconductor production. Ultra-fine bonding copper wires are high-precision connection materials with a micron-level diameter processed from copper alloy materials. Compared with ordinary bonding wires, ultra-fine bonding copper wires have the characteristics of high strength, ultra-precision and high temperature resistance, and can achieve higher-density circuit interconnections, thereby improving the performance and reliability of devices. During the production process of ultra-fine bonding copper wires, due to reasons such as rapid bending and swaying, the phenomenon of copper wire breakage often occurs, and workers need to visually detect the breakage position and manually process it in a high-temperature environment, resulting in slow production speed, low energy efficiency and high defect rate. How to achieve abnormal detection such as wire breakage through automated technical means has become one of the pain points and key technical challenges in the industry. Summary of the Invention

[0003] The purpose of this application is to provide a method, device, equipment and product for detecting abnormal winding states of ultra-fine bonding copper wires to improve the efficiency and accuracy of detecting abnormal winding states of ultra-fine bonding copper wires.

[0004] To achieve the above purpose, the following solutions are provided in this application.

[0005] In the first aspect, this application provides a method for detecting abnormal winding states of ultra-fine bonding copper wires, including:

[0006] Obtain a real-time image of the dynamic reflection light spot of the ultra-fine bonding copper wire; the real-time image of the dynamic reflection light spot is captured by irradiating and photographing the ultra-fine bonding copper wire with a bar-shaped array light source that lights up column by column;

[0007] Grab two consecutive frames of images from the real-time image of the dynamic reflection light spot and perform image preprocessing to obtain preprocessed light spot images of the two consecutive frames;

[0008] Extract feature points from the preprocessed light spot images of the two consecutive frames to obtain light spot feature points of the two consecutive frames;

[0009] Based on the preprocessed light spot images and light spot feature points of the two consecutive frames, calculate the optical flow estimations of the two consecutive frames respectively to obtain the spatio-temporal information of the two consecutive frames representing the current state;

[0010] Judge the trajectory consistency based on the spatio-temporal information of the two consecutive frames to determine the current state detection result of the ultra-fine bonding copper wire; the current state detection result includes a prompt of whether it is abnormal and a prompt of the break point area.

[0011] Optionally, before obtaining the real-time image of the dynamic reflection spot of the ultra-fine bonded copper wire, the following steps are further included:

[0012] Arrange a strip array light source and a high-definition camera above the ultra-fine bonded copper wire; both the strip array light source and the high-definition camera are connected to an industrial control computer;

[0013] By controlling the frequency and light intensity, the strip array light source is turned on column by column in sequence to irradiate the ultra-fine bonded copper wire with a conical space structure, and the high-definition camera continuously takes pictures to obtain a real-time image of the dynamic reflection spot, and transmits it to the industrial control computer through a connected network cable for detection and processing.

[0014] Optionally, the specific steps of grabbing two consecutive frames of images from the real-time image of the dynamic reflection spot and performing image preprocessing to obtain preprocessed spot images of the two consecutive frames are as follows:

[0015] Grab the latest two frames of images from the real-time image of the dynamic reflection spot in reverse order according to the shooting time as the two consecutive frames of images, which are respectively called the front frame image and the rear frame image;

[0016] Convert the two consecutive frames of images into grayscale images and perform smoothing processing using Gaussian blur to obtain the two consecutive frames of images after smoothing processing;

[0017] Use the method of threshold segmentation for the two consecutive frames of images after smoothing processing to filter out the regions with pixel brightness lower than the segmentation threshold, and obtain the two consecutive frames of images after threshold segmentation;

[0018] Perform a dilation operation on the two consecutive frames of images after threshold segmentation to connect adjacent bright points, thereby obtaining the preprocessed spot images of the two consecutive frames.

[0019] Optionally, the specific steps of extracting feature points from the preprocessed spot images of the two consecutive frames to obtain the spot feature points of the two consecutive frames are as follows:

[0020] Use the Shi-Tomasi corner detection algorithm for the preprocessed spot images of the two consecutive frames respectively to obtain a set of feature points, and select the feature points with pixel brightness higher than the screening threshold in each set of feature points, thereby obtaining the spot feature points of the two consecutive frames.

[0021] Optionally, based on the preprocessed spot images and spot feature points of the two consecutive frames, calculate the optical flow estimation of the two consecutive frames respectively to obtain the spatio-temporal information of the two consecutive frames representing the current state. The specific steps are as follows:

[0022] Construct image pyramids for the preprocessed spot images of the two consecutive frames respectively, and gradually reduce the resolution from the bottom layer to the top layer;

[0023] Based on the constructed image pyramid, starting from the top layer, the Lucas-Kanade method is used to estimate the optical flow of the spot feature points, and the optical flow estimation of the top layer is calculated.

[0024] Based on the optical flow estimation of the top layer, the optical flow estimation of the next layer is calculated, and iterative solution is performed. The output of the previous layer is used as the input of the next layer until the bottom layer.

[0025] Taking the optical flow estimation of the bottom layer as the final optical flow estimation, the position and velocity information of the preprocessed spot images of two consecutive frames are obtained, and the corresponding time information is combined to jointly serve as the spatio-temporal information of the two consecutive frames.

[0026] Optionally, before determining the current state detection result of the ultra-fine bonded copper wire based on the spatio-temporal information of two consecutive frames and judging the trajectory consistency, the following steps are further included:

[0027] At the first detection, when each column of the bar array light source is lit once, a trajectory database is constructed based on the spatio-temporal information of all two consecutive frames; where as each column of the light source lights up, the spot leaves corresponding trajectory information P(x, y, v x , v y , t); where (x, y) is the position information of the spot feature point; (v x , v y ) is the velocity information of the spot feature point; t is the time information of the spot feature point.

[0028] Optionally, determining the current state detection result of the ultra-fine bonded copper wire based on the spatio-temporal information of two consecutive frames and judging the trajectory consistency specifically includes:

[0029] For the current image of the dynamic reflection spot of the currently detected ultra-fine bonded copper wire, its corresponding spatio-temporal information is extracted as the current trajectory information.

[0030] Based on the current trajectory information, a comparison search is performed in the trajectory database, and the closest trajectory information is searched out as the reference trajectory information.

[0031] The trajectory consistency between the current trajectory information and the reference trajectory information is calculated.

[0032] When the overall deviation of the trajectory consistency exceeds the overall deviation threshold, an abnormality is prompted and the break point area is prompted based on the reference trajectory information.

[0033] In a second aspect, the present application provides a detection device for abnormal winding states of ultra-fine bonding copper wires, comprising: a strip array light source, a high-definition camera, and an industrial control computer; the strip array light source and the high-definition camera are arranged above the ultra-fine bonding copper wires; both the strip array light source and the high-definition camera are connected to the industrial control computer; by controlling the frequency and light intensity, the strip array light source is turned on column by column in sequence to irradiate the ultra-fine bonding copper wires with a conical spatial structure, and the high-definition camera is used to continuously take pictures to obtain real-time images of dynamic reflected light spots, which are transmitted to the industrial control computer through a connected network cable for detection and processing; the industrial control computer is used to implement the method for detecting abnormal winding states of the ultra-fine bonding copper wires as described in claim 1.

[0034] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the method for detecting abnormal winding states of the ultra-fine bonding copper wires.

[0035] In a fourth aspect, the present application provides a computer program product, comprising a computer program which, when executed by a processor, implements the method for detecting abnormal winding states of the ultra-fine bonding copper wires.

[0036] According to the specific embodiments provided by the present application, the following technical effects are disclosed.

[0037] A method, device, equipment and product for detecting abnormal winding states of ultra-fine bonding copper wires provided by the present application uses the method of turning on the strip array light source in sequence to obtain dynamic reflected light spots of the copper wires, and based on the improved KLT (Kanade-Lucas-Tomasi) pyramid optical flow method, solves the detection difficulties of difficult to take clear images and difficult to fully obtain image features in the scenario of the ultra-fine bonding copper wires with a conical spatial structure, effectively detects whether the ultra-fine bonding copper wires are in an abnormal state and can prompt the specific break point area, greatly improves the detection efficiency and detection accuracy, and at the same time improves the automation level of quality control in the production process of the ultra-fine bonding copper wires. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the following described drawings are only some embodiments of the present application, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.

[0039] Figure 1 It is a schematic flowchart of a method for detecting abnormal winding states of an ultra-fine bonding copper wire according to the present application;

[0040] Figure 2Schematic diagram of the arrangement of a bar-shaped array light source and a high-definition camera;

[0041] Figure 3 Schematic diagram of the movement of dynamic reflection light spots;

[0042] Figure 4 Schematic diagram of the constructed image pyramid;

[0043] Figure 5 Schematic diagram of the normal image and abnormal image of the light spot at a certain moment. Specific implementation manner

[0044] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0045] During the production process of ultra-fine bonded copper wires, there are many scenarios where a large number of copper wires have a conical spatial structure, and the break point cannot be prompted, requiring workers to judge with the naked eye in a high-temperature environment. To solve the above problems, the present application proposes a method, device, equipment and product for detecting the abnormal winding state of ultra-fine bonded copper wires, which can effectively detect whether the ultra-fine bonded copper wires are abnormal and prompt the break point in the detection scenario of the conical spatial structure, improve the detection efficiency and accuracy of the abnormal state, and at the same time improve the automation level of quality control in the production process of ultra-fine bonded copper wires.

[0046] To make the above objects, features and advantages of the present application more obvious and understandable, the present application will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0047] In an exemplary embodiment, as Figure 1 shown, a method for detecting the abnormal winding state of ultra-fine bonded copper wires is provided, including the following steps 1 to 5.

[0048] Step 1: Obtain a real-time image of the dynamic reflection light spot of the ultra-fine bonded copper wire.

[0049] The real-time image of the dynamic reflection light spot used in the present application is captured by irradiating and photographing the ultra-fine bonded copper wire with a bar-shaped array light source that lights up column by column. As Figure 2As shown in the figure, a strip array light source (referred to as the light source) 2 and a high-definition camera (referred to as the camera) 3 are arranged above the ultra-fine bonded copper wire (referred to as the copper wire) 1 with a conical space structure. Both the strip array light source 2 and the high-definition camera 3 are connected to the industrial control computer. By controlling the frequency and light intensity, the strip array light source 2 is lit column by column in sequence to irradiate the ultra-fine bonded copper wire 1 with a conical space structure, and the high-definition camera 3 is used to continuously take pictures to obtain a real-time image of the dynamic reflection spot 4, which is transmitted to the industrial control computer through the connected network cable for detection and processing.

[0050] Specifically, an STC8H1K08 chip can be built into the strip array light source 2, communicate with the STC serial port software of the industrial control computer through the DMX512 protocol, and be connected with a programming cable for programming and burning to control the light emission of the strip array light source 2. For example, by adjusting the duty cycle of the PWM signal to change the ratio of the high level to the low level, the control of the light intensity can be achieved. A loop is carried out within the timer function of STC, and the frequency is set by spacing a fixed time. The strip array light source 2 can adopt a high-brightness strip array light source, which is divided into 12 columns, and each column corresponds to one address. In the loop, every once in a while, the previous address is emptied and the next address is assigned a value, so that the light of the previous column is turned off and the light of the next column is turned on, making the high-brightness strip array light source light up column by column in sequence. The strip array light source 2 starts to work according to the established logic when powered on.

[0051] The high-brightness strip array light source 2 and the high-definition camera 3 are supported by a fixed bracket and are located above the ultra-fine bonded copper wire 1. In an exemplary embodiment, the high-brightness strip array light source 2 irradiates the copper wire with a conical space structure directly above, and at the same time, the high-definition camera 3 continuously takes pictures at a rate of dozens of frames per second. As the light source lights up in sequence, the reflection spot of the copper wire also moves correspondingly, as Figure 3 shown in parts (a), (b), and (c) of the figure, so as to obtain a real-time image of the dynamic reflection spot of the copper wire, which is called the real-time image of the dynamic reflection spot. Multiple frames of real-time images of the dynamic reflection spot are stored in the database, and at the same frequency as the light source lights up, the current latest real-time image of the dynamic reflection spot of the copper wire is regularly obtained from the database for detection.

[0052] Step 2: Grab two consecutive frames of images from the real-time image of the dynamic reflection spot and perform image preprocessing to obtain the preprocessed spot images of the two consecutive frames.

[0053] The specific steps of step 2 include:

[0054] Step 2.1: Grab the latest two frames of images from the real-time image of the dynamic reflection spot in reverse order according to the shooting time as the two consecutive frames of images, which are respectively called the previous frame image and the next frame image.

[0055] The real-time image of the dynamic reflection light spot is stored in the database, and then the latest two frames of images are captured in reverse chronological order as the front and rear frames of images. Specifically, a thread of the industrial control computer uses the SDK interface function of the camera to capture the video stream and continuously extracts pictures (also known as images) and stores them in the Redis database. Another thread of the industrial control computer queries the latest two frames of pictures in the Redis database in reverse chronological order as the front and rear frames of images.

[0056] Step 2.2: Convert the front and rear frames of images into grayscale images and perform smoothing processing using Gaussian blur to obtain the front and rear frames of images after smoothing processing.

[0057] The above front and rear frames of images are grayscaled to convert them into grayscale images. The GaussianBlur function is called to perform smoothing processing on the grayscale images to make the contour of the light spot smoother and reduce the interference of noise.

[0058] Step 2.3: Use the method of threshold segmentation on the front and rear frames of images after smoothing processing to filter out the areas where the pixel brightness is lower than the segmentation threshold, and obtain the front and rear frames of images after threshold segmentation.

[0059] The above front and rear frames of images after smoothing processing are subjected to binary processing of threshold segmentation to filter out the areas where the pixel brightness is lower than the segmentation threshold, thereby shielding the interference of low-brightness pixels on subsequent image analysis.

[0060] Step 2.4: Perform dilation operations on the front and rear frames of images after threshold segmentation to connect adjacent bright points, thereby obtaining the preprocessed light spot images of the front and rear frames.

[0061] The dilate dilation function is used to perform dilation on the front and rear frames of images after threshold segmentation respectively to connect adjacent bright points and make the bright point area more obvious, thereby obtaining the preprocessed light spot images of the front and rear frames.

[0062] Step 3: Extract feature points from the preprocessed light spot images of the front and rear frames respectively to obtain the light spot feature points of the front and rear frames.

[0063] The Shi-Tomasi corner detection algorithm is used for each of the preprocessed light spot images of the front and rear frames to obtain a set of feature points respectively, and the feature points with pixel brightness higher than the screening threshold in each set of feature points are selected, thereby obtaining the light spot feature points of the front and rear frames.

[0064] Step 4: Based on the preprocessed light spot images and light spot feature points of the front and rear frames, calculate the optical flow estimation of the front and rear frames respectively to obtain the spatio-temporal information of the front and rear frames representing the current state.

[0065] The specific steps of step 4 include:

[0066] Step 4.1: Construct image pyramids for the preprocessed spot images of two consecutive frames, and gradually reduce the resolution from the bottom layer to the top layer.

[0067] Based on the preprocessed spot images of two consecutive frames, use them as the original images with the highest resolution. Specifically, for an image Image with a size of n x *n t The pyramid structure is established by the following method. As Figure 4 shown, let I 0 = Image represent the image of the 0th layer, that is, the original image, whose width and height are the same as those of the original image. Then, build the pyramid recursively: Calculate I 0 based on I 1 , calculate I 1 based on I 2 … Let L = 1, 2… be one of the layers of the pyramid, and let I L-1 represent the image of the (L - 1)th layer. As Figure 4 shown, by convolving I L-1 with a low-pass filter of [0.25 0.25 0.25 0.25], obtain the image of the I L layer. From the bottom layer to the top layer, gradually reduce the resolution.

[0068] Step 4.2: Based on the constructed image pyramids, start from the top layer and use the Lucas-Kanade method to estimate the optical flow of the spot feature points, and calculate the optical flow estimation of the top layer.

[0069] Set point a as any spot feature point in the target optical flow area. Assume that the gray value of point a = (x, y) at time t is I(x, y, t), where I is the gray value function, x is the abscissa in the horizontal direction, y is the ordinate in the vertical direction, and t is the time. According to the assumption that the image gray value remains unchanged before and after motion, derive the optical flow constraint equation I x v x + I y v y + I t = 0, and its vector form is where, I x is the partial derivative of I with respect to x, v x is the velocity in the x direction, I y is the partial derivative of I with respect to y, v y is the velocity in the y direction, and I t is the partial derivative of I with respect to t. is the gradient of the image at a, and v a = (v x , v y ) is the optical flow estimation of point a.

[0070] Under the condition that the optical flow of each point in a small area centered at point a is the same, search for the displacement amount to minimize the matching error of the neighborhood related to the corresponding point, that is, define a function on the local neighborhood of a and minimize it: where Ω represents the neighborhood centered at point a. W(x, y) is the weight function, α is a small positive number to avoid division by zero. Optimally solving it gives: v a =[A T W 2 A] -1 A T W 2 b. Where x n represents the coordinates of the nth point in the neighborhood Ω centered at point a. W is the abbreviation of the weight function W(x, y). b is the difference between the gray value of the current frame and the gray value of the previous frame in the neighborhood Ω.

[0071] Thus, based on the above image pyramid and the above spot feature points, the optical flow estimation v a at the top layer of these spot feature points a can be calculated, that is, the velocity information v = (v x , v y ) of each spot feature point is obtained. When not specifying a particular point, the optical flow estimation is uniformly represented as v.

[0072] Step 4.3: Calculate the optical flow estimation of the next layer based on the optical flow estimation of the top layer, and solve it iteratively. The output of the previous layer is used as the input of the next layer until the bottom layer.

[0073] Based on the above optical flow estimation of the top layer, use it as the starting point for the input of the next layer of the pyramid, that is, by scaling up the v of the previous layer according to the scaling ratio of the pyramid, as the initial search position of the optical flow estimation v in the next layer of the pyramid, and calculate the optical flow estimation of the next layer according to the formula v = [A T W 2 A] -1 A T W 2 b. Repeat this process in a loop, from coarse to fine, until the bottom layer. Use the optical flow estimation calculated at the bottom layer as the final optical flow estimation (v x , v y ), and obtain the position information (x, y) and velocity information (v x , v y ) of the preprocessed spot images of the two consecutive frames, that is, the (x, y, v x , v y ) corresponding to the two consecutive frames of pictures respectively.

[0074] Step 4.4: Use the optical flow estimation of the bottom layer as the final optical flow estimation to obtain the position and velocity information of the preprocessed spot images of the two consecutive frames, and combine the corresponding time information as the spatio-temporal information of the two consecutive frames.

[0075] Combine the position and velocity information (x, y, v x , v y ) of the preprocessed spot images of the two consecutive frames with the corresponding time information t as the spatio-temporal information P(x, y, v x , v y , t) of the two consecutive frames, which can also be called the trajectory information of the spot.

[0076] Step 5: Judge the trajectory consistency based on the spatio-temporal information of the two consecutive frames to determine the current state detection result of the ultra-fine bonded copper wire; the current state detection result includes a prompt of whether it is abnormal and a prompt of the break point area.

[0077] During the first detection, after each column of the bar array light source is lit once, it is necessary to construct a trajectory database based on the spatio-temporal information of all the two consecutive frames to facilitate subsequent judgment of trajectory consistency. Among them, as each column of the light source lights up, the spot leaves the corresponding trajectory information P(x, y, v x , v y , t); where (x, y) is the position information of the spot feature point; (v x , v y ) is the velocity information of the spot feature point; t is the time information of the spot feature point.

[0078] The trajectory database stores the spot trajectories P(x, y, v x , v y , t) at all times under the irradiation of all columns of the light source, and the real-time image of the dynamic reflection spot of the currently detected ultra-fine bonded copper wire is called the current image of the dynamic reflection spot (abbreviated as the current image). Based on the position and velocity information of the current image, that is, (x, y, v x , v y ) at a certain moment t, detect whether there are points close to or consistent with it in the trajectory database. If it is the first detection, there is no corresponding record in the trajectory database. If all columns of the light source have been lit once, the second detection will start to detect the corresponding record, so as to judge whether the trajectory database has been constructed.

[0079] If the trajectory database has not been constructed yet, then record the trajectory information P(x, y, v x , v y , t) of the current spot, and continue to execute the next detection action. Among them, x is the horizontal coordinate, y is the vertical coordinate, v x is the horizontal velocity, v yis the vertical velocity, and t is the lighting time of a certain column of light sources.

[0080] As the array light sources light up in sequence, the dynamic reflection spot of the copper wire moves accordingly. With the lighting of each column of light sources, the spot leaves corresponding trajectories P(x, y, v x , v y , t). When each column of light sources in the bar-shaped array light source has been lit once, all the trajectories corresponding to the spots are generated, that is, the trajectory database is constructed, and the subsequent detection enters the next step 5.

[0081] The specific steps of step 5 are as follows:

[0082] Step 5.1: For the current image of the dynamic reflection spot of the ultra-fine bonding copper wire being detected currently, extract the spatio-temporal information corresponding to the current image as the current trajectory information, denoted as P'(x, y, v x , v y , t).

[0083] Step 5.2: Based on the current trajectory information, conduct a comparison search in the trajectory database, and search for the closest trajectory information as the reference trajectory information.

[0084] Conduct a comparison search in the above-mentioned trajectory database of the spots. When the copper wire is being wound, under normal circumstances, as the light sources light up in sequence, the reflection spot moves in the x direction with a constant speed and a constant position; in the y direction, the speed is 0 or there is a small speed, and the position remains unchanged or there is a small displacement. When the swing is severe and it is very easy to break the wire, the movement of the reflection spot in the x direction is basically not affected and is the same as the normal situation, while there are larger speeds and displacements in the y direction. In the case of wire breakage, the reflection spot disappears directly, and the x and y coordinates of the current trajectory information P' are all 0, v x , v y is huge. That is, under normal circumstances, the (x, y, v x , v y , t) of each spot is basically constant. After identifying the (x, y, v x , v y ) in the current state, the moment t can be determined, as well as the spot trajectory that should correspond to each subsequent moment. In the case of an abnormality, it means that the previous frame is in a normal state and the next frame has an abnormality, and the (x, y, v x , v y ) of some spots will mutate. Therefore, by calculating the Euclidean distance of the x and y coordinates of the current trajectory information P' and taking the minimum value, the trajectory information corresponding to the spot in the current state (that is, the current trajectory information P') can be searched, which is called the reference trajectory information P''(x, y, v x , v y , t).

[0085] Step 5.3: Calculate the trajectory consistency between the current trajectory information and the reference trajectory information.

[0086] According to the formula Calculate the trajectory consistency. Where m is the overall deviation and Δt is the time interval between two frames. and Substitute the (x, y, v) corresponding to the current trajectory information P' and the reference trajectory information P'' respectively. x ,v y ). When the overall deviation m is greater than the overall deviation threshold M, it is judged that the light spot P' is abnormal and does not appear at the predetermined position P(x,y), and a breakpoint is prompted and an alarm is issued, otherwise the loop is entered to continue the detection. Among them, the overall deviation threshold M is a hyperparameter, which is selected based on experience and is mainly adjusted according to the allowable fluctuation of the copper wire in the y direction.

[0087] Step 5.4: When the overall deviation of the trajectory consistency exceeds the overall deviation threshold, an abnormality is prompted and a breakpoint area is prompted based on the reference trajectory information.

[0088] In a specific embodiment, Figure 5 Part (a) shows a normal image diagram of the light spot at a certain moment. Figure 5 Part (b) is a schematic diagram of when an abnormality is detected in the light spot at a certain moment. According to step 5.3, when the light spot does not appear at the predetermined position P(x, y) at a certain moment, it is determined to be abnormal, and a frame is drawn with P(x, y) as the center point. The size of the frame refers to the empirical value of the light spot size, and it only needs to be eye-catching enough to mark the abnormal position (breakpoint area) and prompt an alarm.

[0089] This application detects the current state of the copper wire through the dynamic reflection spot of the copper wire and the optical flow algorithm. In the detection scenario of the conical space structure, it can effectively detect whether the ultra-fine bonding copper wire is abnormal and prompt the breakpoint area, thereby improving the efficiency and accuracy of abnormality detection, and at the same time improving the automation level of quality control in the production process of ultra-fine bonding copper wire.

[0090] In an exemplary embodiment, the present application further provides a detection device for abnormal winding states of ultra-fine bonding copper wires, including: a strip array light source, a high-definition camera, a fixed bracket, a connection network cable, and an industrial control computer. Among them, the strip array light source and the high-definition camera are arranged above the ultra-fine bonding copper wires, and the fixed bracket is used to support the high-brightness strip array light source and the high-definition camera. The strip array light source and the high-definition camera are both connected to the industrial control computer. By controlling the frequency and light intensity, the strip array light source is turned on column by column in sequence to irradiate the ultra-fine bonding copper wires with a conical spatial structure, and the high-definition camera is used to continuously take pictures to obtain real-time images of dynamic reflected light spots, which are transmitted to the industrial control computer through the connection network cable for detection and processing. The industrial control computer is used to implement the detection method for abnormal winding states of the ultra-fine bonding copper wires.

[0091] Specifically, the STC8H1K08 chip built in the light source and the STC serial port software of the industrial control computer communicate based on the DMX512 protocol and are connected with a programming cable, so as to program and burn the program to control the light source to turn on column by column in sequence and emit light according to the established program when powered on. The high-definition camera is controlled based on the open API of the camera to capture image frames from the real-time video stream. The real-time images of the dynamic reflected light spots collected are displayed on the screen of the industrial control computer. The industrial control computer is used to implement the detection method for abnormal winding states of the ultra-fine bonding copper wires. By obtaining the real-time images of the dynamic reflected light spots of the copper wires; based on the previous and next frames of the real-time images of the dynamic reflected light spots, image preprocessing and feature point extraction are performed to obtain the light spot feature points of the previous and next frames of images; based on the light spot feature points of the previous and next frames of images, the optical flow estimation of the previous and next frames of images is calculated respectively to obtain the position and velocity information of the latter frame of picture, that is, the position and velocity information of the current state picture; based on the position and velocity information of the current state picture, the trajectory consistency is judged to determine the current state of the copper wire. The present application adopts an intelligent method to effectively detect whether the ultra-fine bonding copper wires are abnormal and can prompt the specific break point area in the detection scenario of the conical spatial structure.

[0092] The ultra-fine bonding copper wires themselves have the characteristics of being ultra-fine and easy to reflect light, and are distributed in a conical spatial structure, with high and low disorder and unevenness. Under ordinary light sources, it is difficult to obtain clear high-quality images directly by using a camera, which seriously affects the accuracy of conventional image detection methods. Starting from the idea of optical flow calculation, the present application uses the method of turning on the strip array light source in sequence to obtain the dynamic reflected light spots of the copper wires. Based on the improved KLT pyramid optical flow method, it solves the detection difficulty that it is difficult to take clear pictures to fully obtain image features in the scenario of the conical spatial structure of the copper wires, effectively detects whether the ultra-fine bonding copper wires are abnormal and prompts the break point area, improves the detection efficiency and accuracy, and at the same time improves the automation level of quality control in the production process of ultra-fine bonding copper wires.

[0093] The gray-scale invariance assumption is one of the prerequisite conditions for the KLT pyramid optical flow method. Due to the influence and interference of various factors such as illumination and large displacement, the gray-scale invariance assumption may fail. In response to this, the present application preprocesses the real-time image to reduce the interference of noise, and tracks the latest two consecutive frames of the real-time image to narrow the difference between the pictures, thereby strengthening the conditions for the gray-scale invariance assumption to hold and further improving the accuracy of the current state detection result.

[0094] In an exemplary embodiment, the present application further provides a computer device, which may be a server or a terminal (such as an industrial control computer). The computer device includes a processor, a memory, an input / output interface, and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through a network connection. When the computer program is executed by the processor, it implements the method for detecting the abnormal winding state of the ultra-fine bonding copper wire.

[0095] In an exemplary embodiment, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the method for detecting the abnormal winding state of the ultra-fine bonding copper wire.

[0096] In an exemplary embodiment, the present application further provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the method for detecting the abnormal winding state of the ultra-fine bonding copper wire.

[0097] Those of ordinary skill in the art will understand that all or part of the processes in the above-described embodiment methods can be completed by hardware related to computer program instructions. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiment methods as described above. Among them, any reference to a memory or other medium provided in the embodiments of the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0098] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data that have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of the relevant data need to comply with the relevant regulations.

[0099] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0100] In this article, specific examples are used to elaborate on the principles and implementation manners of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A method for detecting abnormal winding state of ultra-fine bonding copper wire, characterized in that: include: Acquire a real-time image of a dynamic reflection spot of an ultra-fine bonding copper wire; the real-time image of the dynamic reflection spot is captured by irradiating the ultra-fine bonding copper wire with a strip array light source that lights up in columns in sequence and photographing it; Capture the front and rear two frames of images from the dynamic reflection spot real-time image and perform image preprocessing to obtain the front and rear two frames of preprocessed spot images; Feature points are extracted from the pre-processed spot images of the two frames respectively to obtain the spot feature points of the two frames; Based on the pre-processed spot images and spot feature points of the previous and next two frames, the optical flow estimation of the previous and next two frames is calculated respectively to obtain the spatiotemporal information of the previous and next two frames representing the current state; The trajectory consistency is judged based on the spatiotemporal information of the previous and next two frames, and the current state detection result of the ultra-fine bonding copper wire is determined; the current state detection result includes a prompt of whether it is abnormal and a prompt of the breakpoint area.

2. The method for detecting abnormal winding state of ultra-fine bonding copper wire according to claim 1, characterized in that: Before obtaining the real-time image of the dynamic reflection spot of the ultra-fine bonding copper wire, the method further includes: Arrange a strip array light source and a high-definition camera above the ultra-fine bonding copper wire; the strip array light source and the high-definition camera are both connected to an industrial computer; By controlling the frequency and light intensity, the strip array light sources light up in columns one by one to illuminate the ultra-fine bonding copper wires with a conical spatial structure. A high-definition camera is used to continuously shoot to obtain real-time images of dynamic reflected light spots, which are then transmitted to an industrial computer for detection and processing via a network cable.

3. The method for detecting abnormal winding state of ultra-fine bonding copper wire according to claim 1, characterized in that: The method of capturing two frames of images from the real-time image of the dynamic reflected light spot and performing image preprocessing to obtain the preprocessed light spot images of the two frames specifically includes: The two latest frames of images are captured in reverse order from the real-time image of the dynamic reflected light spot according to the shooting time as the front and back frames of images, which are called the front frame image and the back frame image respectively; The two frames of images are converted into grayscale images, and Gaussian blur is used to perform smoothing to obtain the smoothed two frames of images; The threshold segmentation method is used for the smoothed front and back two frames of images to filter out the areas where the pixel brightness is lower than the segmentation threshold, and the front and back two frames of images after threshold segmentation are obtained; The two frames of images after threshold segmentation are expanded to connect adjacent bright spots, thereby obtaining pre-processed spot images of the two frames.

4. The method for detecting abnormal winding state of ultra-fine bonding copper wire according to claim 3, characterized in that: The extracting of feature points from the pre-processed spot images of the two frames respectively to obtain the spot feature points of the two frames specifically includes: The Shi-Tomasi corner detection algorithm is used for the preprocessed spot images of the front and back two frames to obtain a set of feature points respectively, and the feature points with pixel brightness higher than the screening threshold in each set of feature points are selected to obtain the spot feature points of the front and back two frames.

5. The method for detecting abnormal winding state of ultra-fine bonding copper wire according to claim 4, characterized in that: The optical flow estimation of the two frames before and after is calculated based on the pre-processed spot images and spot feature points of the two frames before and after, respectively, to obtain the spatiotemporal information of the two frames before and after representing the current state, specifically including: Image pyramids are constructed for the pre-processed spot images of the previous and next two frames, and the resolution is gradually reduced from the bottom to the top. Based on the constructed image pyramid, the Lucas-Kanade method is used to estimate the optical flow of the feature points of the light spot from the top layer, and the optical flow estimation of the top layer is calculated; Based on the optical flow estimation of the top layer, the optical flow estimation of the next layer is calculated, and the solution is iterative. The output of the previous layer is used as the input of the next layer, until the bottom layer; The bottom-level optical flow estimation is used as the final optical flow estimation to obtain the position and speed information of the two pre-processed spot images, and the corresponding time information is combined as the spatiotemporal information of the two frames.

6. The method for detecting abnormal winding state of ultra-fine bonding copper wire according to claim 5, characterized in that: Before judging the trajectory consistency based on the spatiotemporal information of the two previous and next frames and determining the current state detection result of the ultra-fine bonding copper wire, the method further includes: During the first detection, after each column of the strip array light source is lit up, a trajectory database is constructed based on the spatiotemporal information of all the previous and next two frames; as each column of the light source is lit up, the light spot leaves corresponding trajectory information P (x, y, v x ,v y ,t); (x,y) is the location information of the feature point of the light spot; (v x ,v y ) is the speed information of the light spot feature point; t is the time information of the light spot feature point.

7. The method for detecting abnormal winding state of ultra-fine bonding copper wire according to claim 6, characterized in that: The step of judging the trajectory consistency based on the spatiotemporal information of the two frames before and after and determining the current state detection result of the ultra-fine bonding copper wire specifically includes: For the current image of the dynamic reflection spot of the ultra-fine bonding copper wire currently being detected, the corresponding time and space information is extracted as the current trajectory information; Based on the current trajectory information, a comparison search is performed in the trajectory database to search for the closest trajectory information as the reference trajectory information; Calculate the trajectory consistency between the current trajectory information and the reference trajectory information; When the overall deviation of trajectory consistency exceeds the overall deviation threshold, an abnormality is prompted and a breakpoint area is prompted based on the reference trajectory information.

8. A device for detecting abnormal winding state of ultra-fine bonding copper wire, characterized in that: include: A strip array light source, a high-definition camera and an industrial computer; the strip array light source and the high-definition camera are arranged above the ultra-fine bonding copper wire; The strip array light source and the high-definition camera are both connected to an industrial computer; by controlling the frequency and light intensity, the strip array light source is lit up in sequence in columns to illuminate the ultra-fine bonding copper wire with a conical spatial structure, and the high-definition camera is used to continuously shoot to obtain a real-time image of the dynamic reflection light spot, which is transmitted to the industrial computer for detection and processing via a connecting network cable; the industrial computer is used to implement the abnormal winding state detection method of the ultra-fine bonding copper wire described in claim 1.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for detecting abnormal winding state of ultra-fine bonding copper wire according to claim 1.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for detecting abnormal winding state of ultra-fine bonding copper wire according to claim 1 is implemented.

Images