Automatic counting method and device for chip devices

Abstract

The invention provides an automatic counting method and device for chip devices, and the method comprises the steps: S1, placing a to-be-detected chip device in a darkroom, placing the to-be-detected chip device on a parallel light source, and capturing an image through a camera located on the plane of the to-be-detected chip device; s2, identifying and analyzing features of the chip device in the image, and distinguishing individual individuals and adhesion areas; s3, feature extraction and target detection are carried out on the adhesion areas through a deep learning algorithm, and the matching number in the adhesion areas is obtained; and S4, adding the number of the individual individuals and the matching number in the adhesion area to obtain the number of the chip devices to be detected. The scheme of combining traditional image processing and deep learning is adopted, operation is convenient, calculation efficiency is high, manpower resources can be effectively saved, and the method can adapt to counting of chip devices of various sizes.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic equipment, and in particular to a method and device for automatically counting chip devices. Background Art

[0002] Component screening tasks are widely present in the measurement and testing industry. The current component screening relies on manual counting, and its time cost and manual error are immeasurable. Some automated counting methods are targeted at devices with specific specifications, and need to implement corresponding identification methods in a targeted manner and switch them when used, which is not enough to meet the counting requirements of devices of different sizes. At the same time, existing methods rely on detailed full-image template matching or feature descriptor matching methods, which is beneficial for unstacked device detection. However, for a large number of more dispersed individuals in actual use, the time cost is huge. And for some stacked devices, the recognition error rate is high, which is intolerable for precision devices. The counter is an automatic counting device based on image processing and computer vision technology with automatic component detection function, which is mainly used for automatic detection of the number of devices. It does not require manual intervention and is suitable for counting various types of chip components. It has the characteristics of convenient use, high anti-interference and high accuracy. Therefore, the counter needs to solve two main problems: faster processing and more accurate recognition. Summary of the invention

[0003] The present invention provides a chip device automatic counting method and device to solve the technical problems of huge workload and large error in the existing manual counting method.

[0004] In order to solve the above technical problems, the present invention provides a chip device automatic counting method, comprising the following steps:

[0005] Step S1: placing the chip device to be tested in a dark room and above a parallel light source, and capturing an image through a camera in the plane of the device to be tested;

[0006] Step S2: Identify and analyze the chip device features in the image to distinguish between individual components and adhesion areas;

[0007] Step S3: performing feature extraction and target detection on the adhesion area through a deep learning algorithm to obtain the number of matches in the adhesion area;

[0008] Step S4: Add the number of the individual components and the number of matches in the adhesion area to obtain the number of chip components to be tested.

[0009] Preferably, before step S2, a binarization operation is performed on the captured image to distinguish the foreground, and an invalid foreground is removed by a closing operation.

[0010] Preferably, step S2 comprises:

[0011] Step S21: for the pixel point at the coordinate position (x, y) in the image, determine whether the surrounding eight points (x-1, y), (x-1, y+1), (x, y+1), (x+1, y+1), (x+1, y), (x+1, y-1), (x, y-1), (x-1, y-1) have foreground representations. If so, they are considered connected and the range of the connected domain is recorded;

[0012] Step S22: Performing statistical analysis on the areas of all connected domains using the histogram method, and taking the samples corresponding to the area values ​​with the highest occurrence frequency as the majority samples;

[0013] Step S23: Based on the clustering algorithm, the cluster centers of the majority of samples are obtained as the area reference values ​​of the device to be tested, and the individual components and the adhesion areas are distinguished by the interval range frequency and the empirical value.

[0014] Preferably, step S3 comprises:

[0015] Step S31: Adjust the image of the adhesion area to a size of 640*640 without distortion;

[0016] Step S32: extracting feature maps;

[0017] Step S33: predicting the target position based on coordinate regression analysis;

[0018] Step S34: Perform template matching to obtain the number of matches in the adhesion area.

[0019] 5. The method for automatic counting of chip devices according to claim 4 is characterized in that: in step S33, redundant detection frames are removed by non-maximum suppression: after sorting the candidate frames from large to small according to confidence, the candidate frame with the highest current confidence is selected as the reference frame, and the remaining candidate frames are traversed in turn to filter out the candidate frames whose IOU with the reference frame is higher than a threshold of 0.3.

[0020] Preferably, in step S33, redundant detection frames are removed by distance filtering: if the center distance between two candidate frames is less than the minimum side length of the candidate frames, ie, min(width, length), one of the candidate frames is discarded.

[0021] Preferably, step S34 includes:

[0022] Step S341: According to the predefined template image, based on the horizontal area matching, the center position and length and width of the template are obtained;

[0023] Step S342: Create a three-layer image pyramid with a convolution kernel size of 3*3 and a step size of (sr, sc) for the template image and the adhesion area image, and perform 360° shape matching from low to high levels to obtain the center position (row, col), matching angle θ and matching score score;

[0024] Step S343: The level and position with the highest matching score are used as candidate matching points, and are back-calculated to the original image. If the image matching point is at the position (ri, ci) of the i-th level, the original image position is (sr*i+ri, sc*i+ci);

[0025] Step S344: Search with the matching score list [0.9, 0.8, 0.7, 0.6, 0.5], calculate the center point distance and IOU overlap of the matching position results at each score and the existing matching results, and discard them if the center point distance is less than the minimum side length min(width, length) of the candidate box or the overlap is greater than 0.3; eliminate the device positions that meet the overlap check results according to the length and width of the template image;

[0026] Step S344: If no device can be matched under a certain score, a connected area with an area larger than half of the original template area is searched according to the connected amount analysis, and the center position is added to the matching list;

[0027] Step S345: Add the matching numbers obtained in step S343 and step S344 to obtain the matching number of the adhesion area.

[0028] Preferably, in step S3, the deep learning algorithm uses the following method to expand the training set during training:

[0029] Step S351: obtaining a real device image org;

[0030] Step S352: Rotate org by a random angle as a new img;

[0031] Step S353: randomly generating an image img1 containing n synthetic devices;

[0032] Step S354: rotate img1 by a random angle to obtain new img2;

[0033] Step S355: randomly determine the stacking direction; randomly determine the stacking position;

[0034] Step S356: According to the parameters obtained in step S355, img2 is stacked on img and assigned to img; its value is img = img⊙(img2<THRESH)+img2⊙(img2> =THRESH), THRESH is a randomly generated display pixel threshold;

[0035] Step S357: repeat steps S354 to S356, looping n-1 times;

[0036] Step S358: Output new pictures as training sets.

[0037] The present invention also provides a chip device automatic counting device, which is suitable for the above chip device automatic counting method, and comprises a light source and a darkroom;

[0038] The light source is placed at the bottom of the darkroom, and a material box is provided on the light source;

[0039] The upper part of the darkroom is provided with a shooting port, and the chip devices in the material box are photographed and counted through the shooting port by a lens on an industrial camera.

[0040] Preferably, the industrial camera is driven to shoot through the pyplon library.

[0041] The beneficial effects of the present invention include at least: the present invention is based on a multi-stage, progressive target detection method, and adopts a solution combining traditional image processing and deep learning. First, the individual and the adhesion area are distinguished, and then the deep learning algorithm is used to monitor and identify the target in the adhesion area. The detection of a single device is accelerated without losing the correctness of the stacked device identification. The operation is convenient, the calculation efficiency is high, human resources can be effectively saved, and the counting of chip devices of various sizes can be adapted. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of a method flow of an embodiment of the present invention;

[0043] Figure 2 A photograph of the device to be counted taken according to an embodiment of the present invention;

[0044] Figure 3 A schematic diagram of template matching according to an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of some examples of training data sets according to an embodiment of the present invention;

[0046] Figure 5 is a schematic diagram of the structure of a device according to an embodiment of the present invention;

[0047] Figure 6 is a schematic diagram of the parallel light source structure of an embodiment of the present invention;

[0048] Figure 7 A schematic diagram of a darkroom structure according to an embodiment of the present invention;

[0049] Figure 8 The figure is a schematic diagram of the structure of an industrial camera according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0051] like Figure 1 As shown, an embodiment of the present invention provides a chip device automatic counting method, comprising the following steps:

[0052] Step S1: placing the chip device to be tested in a dark room and above a parallel light source, and capturing an image through a camera in the plane of the device to be tested;

[0053] Step S2: Identify and analyze the chip device features in the image to distinguish between individual components and adhesion areas;

[0054] Step S3: performing feature extraction and target detection on the adhesion area through a deep learning algorithm to obtain the number of matches in the adhesion area;

[0055] Step S4: Add the number of individual components and the number of matches in the adhesion area to obtain the number of chip components to be tested.

[0056] Specifically, when the embodiment of the present invention is used, the overall process is as follows.

[0057] 1) Call the program: Call the customized application UI.

[0058] 2) Loading: Load the devices to be tested into the box and ensure that the devices are not stacked.

[0059] 3) Place the material box with the device in the darkroom and close the darkroom door.

[0060] 4) The computer drives the camera to take pictures.

[0061] Specifically, capture camera / acquisition: the computer communicates with the camera through the network port, drives the camera to run based on the pyplon library, captures the frame image, and the resolution size is 5472*3648. Let the image at this time be X, such as Figure 2 shown.

[0062] 5) Analyze the images and perform simple individual counting.

[0063] It mainly includes two parts: coarse recognition and fine recognition. Coarse recognition is to extract and filter out isolated and easily identifiable devices through connected domain analysis and histogram statistics to accelerate the recognition process. The fine recognition process integrates template matching with yolov7s target detection to identify the position of each device. At the same time, a device stacking generation algorithm is designed to construct a data set to train the yolov7s model and verify the recognition effect of the algorithm. After recognition, invalid recognition devices are filtered out through adjacent position distance filtering and IOU overlap calculation.

[0064] Specifically, in order to distinguish independent chip device units, for the pixel point located at the X coordinate position (x, y) of the image, determine whether the surrounding 8 points (x-1, y), (x-1, y+1), (x, y+1), (x+1, y+1), (x+1, y), (x+1, y-1), (x, y-1), (x-1, y-1) have foreground representation. If so, they are considered connected and the range of the connected area is recorded. The mask representing the connected area is set to M, whose size is the same as the image, and its element value is the index of the connected domain to which the current pixel point belongs.

[0065] Then, based on the prior assumption that most samples are independent individuals, area histogram statistics are used to statistically analyze the areas of all connected domains through the histogram method. The clustering center of most samples is obtained based on the clustering algorithm as the reference value of the current device area, and the single individuals and the adhered individuals are distinguished through the interval range frequency and empirical value.

[0066] In the embodiment of the present invention, illustratively, a binarization operation is performed on the captured image X to distinguish the foreground, and an invalid foreground is removed by a closing operation.

[0067] Due to the diffraction of light between the gaps of chip devices, the color of the gaps between devices will tend to be consistent with the color of the device. In order to facilitate counting, it is necessary to separate the independent individuals. In this embodiment, since the device color tends to be black and the background color tends to be white, a binary operation is used to distinguish the foreground and background to obtain X_thresh=THRESHOLD(X), and a closing operation is further used to remove the invalid foreground to obtain X_morph=MERPH_CLOSE(X_thresh).

[0068] 6) Deep model inference of complex model areas to determine the number of chip devices.

[0069] Specifically, for the adhesion area, let the current adhesion area index be i, the circumscribed rectangular mask of the current area be Mr, and the current abnormal area be exported as an image Xe=(X⊙Mr)⊙(M==i), and this part of the image will be further processed.

[0070] Preprocessing

[0071] Resize the image Xe to 640*640 without distortion. Resize the longest side to 640, resize the shortest side to the corresponding size in the same proportion, and pad the image size to 640 with the RGB background color (128,128,128) on both sides.

[0072] Feature map extraction

[0073] Based on Darknet-50 as the baseline model, the target feature map is extracted.

[0074] Pre-selected box recommendation

[0075] Based on coordinate regression analysis, predict the target position: Assume that the feature network predicts coordinate information: width and height wh, coordinate information xy. The prior width and height at the current scale are scaled_anchors, and the coordinate grid is grid_xy. Then the actual width and height offset of the predicted box is wh*scaled_anchors. The center coordinate is xy+grid_xy.

[0076] Non-maximum suppression

[0077] In order to remove redundant detection boxes, the most credible candidate boxes are retained. After sorting the candidate boxes from large to small according to their confidence, the candidate boxes are traversed one by one, and other boxes whose IOU with the current candidate box is higher than the threshold of 0.3 are filtered out.

[0078] filter

[0079] The obtained candidate box center is filtered by distance. If the distance is less than the minimum side length of the candidate box, that is, min(width,length), it is discarded.

[0080] Template matching, such as Figure 3 shown.

[0081] Step S341: According to the predefined template image Xt, based on the horizontal area matching, the template center position and length and width are obtained.

[0082] Step S342: Create a three-layer image pyramid with a convolution kernel size of 3*3 and a step size of (sr, sc) for the template image and the adhesion area image, and perform 360° shape matching from low to high levels to obtain the center position (row, col), matching angle θ and matching score score.

[0083] Step S343: The level and position with the highest matching score are used as candidate matching points, and are back-calculated to the original image. If the image matching point is at the position (ri, ci) of the i-th level, the original image position is (sr*i+ri, sc*i+ci).

[0084] Step S344: Search with the matching score list [0.9, 0.8, 0.7, 0.6, 0.5], calculate the center point distance and IOU overlap of the matching position results at each score and the existing matching results. If the center point distance is less than the minimum side length min(width, length) of the candidate box or the overlap is greater than 0.3, it will be discarded; the device position that meets the overlap check result will be eliminated according to the length and width of the template image.

[0085] Step S344: If no device can be matched under a certain score, a connected area with an area larger than half of the original template area is searched according to the connected amount analysis, and the center position is added to the matching list.

[0086] Step S345: Add the matching numbers obtained in step S343 and step S344 to obtain the matching number of the adhesion area.

[0087] In the present invention, since the pixel values ​​of the original image devices tend to 0, in order to facilitate model learning, this method reverses the original pixel values. In order to realize the recognition of devices of various types and specifications, the original devices are scaled at multiple scales. And due to the lack of prior data, the embodiment of the present invention uses a data expansion strategy to achieve data expansion. For example, Figure 4 The expansion method is shown below.

[0088] Step S351: obtaining a real device image org;

[0089] Step S352: Rotate org by a random angle as a new img;

[0090] Step S353: randomly generating an image img1 containing n synthetic devices;

[0091] Step S354: rotate img1 by a random angle to obtain new img2;

[0092] Step S355: randomly determine the stacking direction; randomly determine the stacking position;

[0093] Step S356: According to the parameters obtained in step S355, img2 is stacked on img and assigned to img; its value is img = img⊙(img2<THRESH)+img2⊙(img2> =THRESH), THRESH is a randomly generated display pixel threshold;

[0094] Step S357: repeat steps S354 to S356, looping n-1 times;

[0095] Step S358: Output new pictures as training sets.

[0096] During training, the loss is represented by coordinate position loss and confidence loss. Since the task does not require a clear category, the category loss is uniformly set to 0.

[0097] like Figures 5 to 8 As shown, the present invention also provides a chip device automatic counting device, including a parallel light source 1, a material box 2, a darkroom 3, a support frame 4, a lens 5, an industrial camera 6 and a computer;

[0098] The computer is connected to the display and the industrial camera 6 through communication cables, and the computer controls the output of the industrial camera 6 through a signal controller. The material box 2 is placed at the bottom of the darkroom and above the parallel light source 1, in a vertical direction to the industrial camera 6. The lens 5 on the industrial camera 6 shoots and counts the chip devices in the material box 2 through the shooting port above the darkroom 3.

[0099] The chip component automatic counting device is based on computer vision and image recognition principles, and can obtain the number of chip components in the material box without manual intervention. Through the chip component automatic counting method, it can count 2,000 components in just 1 second.

[0100] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. Only the preferred embodiments of the present invention are expressed. The description is more specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. As long as there is no contradiction in the combination of these technical features, they should be considered as the scope recorded in this specification.

[0101] It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these modifications and improvements all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A chip device automatic counting method, characterized in that: The following steps are involved: Step S1: placing the chip device to be tested in a dark room and above a parallel light source, and capturing an image through a camera in the plane of the device to be tested; Step S2: Identify and analyze the chip device features in the image to distinguish between individual components and adhesion areas; Step S3: performing feature extraction and target detection on the adhesion area through a deep learning algorithm to obtain the number of matches in the adhesion area; Step S4: Add the number of the individual components and the number of matches in the adhesion area to obtain the number of chip components to be tested.

2. The chip device automatic counting method according to claim 1, characterized in that: Before step S2, the captured image is binarized to distinguish the foreground, and the invalid foreground is removed by a closing operation.

3. The chip device automatic counting method according to claim 1, characterized in that: Step S2 includes: Step S21: for the pixel point at the coordinate position (x, y) in the image, determine whether the surrounding eight points (x-1, y), (x-1, y+1), (x, y+1), (x+1, y+1), (x+1, y), (x+1, y-1), (x, y-1), (x-1, y-1) have foreground representations. If so, they are considered connected and the range of the connected domain is recorded; Step S22: Performing statistical analysis on the areas of all connected domains using the histogram method, and taking the samples corresponding to the area values ​​with the highest occurrence frequency as the majority samples; Step S23: Based on the clustering algorithm, the cluster centers of the majority of samples are obtained as the area reference values ​​of the device to be tested, and the individual components and the adhesion areas are distinguished by the interval range frequency and the empirical value.

4. The chip device automatic counting method according to claim 1, characterized in that: Step S3 includes: Step S31: Adjust the image of the adhesion area to a size of 640*640 without distortion; Step S32: extracting feature maps; Step S33: predicting the target position based on coordinate regression analysis; Step S34: Perform template matching to obtain the number of matches in the adhesion area.

5. The chip device automatic counting method according to claim 4, characterized in that: In step S33, redundant detection frames are removed by non-maximum suppression: after sorting the candidate frames from large to small according to confidence, the candidate frame with the highest current confidence is selected as the reference frame, and the remaining candidate frames are traversed in turn to filter out the candidate frames whose IOU with the reference frame is higher than the threshold of 0.

3.

6. The chip device automatic counting method according to claim 4, characterized in that: In step S33, redundant detection frames are removed by distance filtering: if the center distance between two candidate frames is less than the minimum side length of the candidate frames, that is, min(width,length), one of the candidate frames is discarded.

7. The chip device automatic counting method according to claim 4, characterized in that: Step S34 includes: Step S341: According to the predefined template image, based on the horizontal area matching, the center position and length and width of the template are obtained; Step S342: Create a three-layer image pyramid with a convolution kernel size of 3*3 and a step size of (sr, sc) for the template image and the adhesion area image, and perform 360° shape matching from low to high levels to obtain the center position (row, col), matching angle θ and matching score score; Step S343: The level and position with the highest matching score are taken as candidate matching points, and are back-calculated to the original image. If the image matching point is at the position (ri, ci) of the i-th level, the original image position is (sr*i+ri, sc*i+ci); Step S344: Search with the matching score list [0.9, 0.8, 0.7, 0.6, 0.5], calculate the center point distance and IOU overlap of the matching position results at each score and the existing matching results, and discard them if the center point distance is less than the minimum side length min(width, length) of the candidate box or the overlap is greater than 0.3; eliminate the device positions that meet the overlap check results according to the length and width of the template image; Step S344: If no device can be matched under a certain score, a connected area with an area larger than half of the original template area is searched according to the connected amount analysis, and the center position is added to the matching list; Step S345: Add the matching numbers obtained in step S343 and step S344 to obtain the matching number of the adhesion area.

8. The chip device automatic counting method according to claim 1, characterized in that: In step S3, the deep learning algorithm uses the following method to expand the training set during training: Step S351: obtaining a real device image org; Step S352: Rotate org by a random angle as a new img; Step S353: randomly generate an image img1 containing n synthetic devices; Step S354: rotate img1 by a random angle to obtain new img2; Step S355: randomly determine the stacking direction; randomly determine the stacking position; Step S356: stack img2 onto img according to the parameters obtained in step S355, and assign the value to img; Its value is img=img⊙(img2<THRESH)+img2⊙(img2> =THRESH), THRESH is a randomly generated display pixel threshold; Step S357: repeat steps S354 to S356, looping n-1 times; Step S358: Output new pictures as training sets.

9. An automatic chip device counting device, applicable to the automatic chip device counting method according to any one of claims 1 to 8, characterized in that: It comprises a parallel light source (1) and a dark room (3); The parallel light source (1) is placed at the bottom of the dark room (3), and a material box (2) is arranged on the parallel light source (1); The upper portion of the darkroom (3) is provided with a photographing port, and the lens (5) on the industrial camera (6) is used to photograph and count the chip components in the material box (2) through the photographing port.

10. The chip device automatic counting device according to claim 9, characterized in that: The industrial camera (6) is driven to shoot via the pyplon library.

Images