A method and apparatus for detecting weld quality

Abstract

The embodiment of the present application provides a kind of detection welding quality method and device, the method comprises: from the second image in which each chip is located is determined from first image;First image is obtained by visual system from the back of substrate and is shot;The front of substrate is provided with the chip installed by flip process;Visual system is configured with shooting device and the light source coaxial with shooting device;Second image is carried out regional segmentation, and at least one region is obtained;Wherein, each pixel point in the same region has similarity;For each region, the feature vector of region is determined, and the feature vector of region is input into multilayer perceptron model, and the welding detection result of region is obtained;Multilayer perceptron model is obtained by pre-training.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to a method and device for detecting welding quality. BACKGROUND

[0002] With the continuous development of economy, electronic products have become an essential part of people's lives, and MiniLED (micro LED array) panels are also applied to more and more electronic products. In the production process of MiniLED panels, one of the steps is to weld the chip on the pad. In the welding process, problems such as empty welding and false welding may occur.

[0003] Currently, X-ray imaging is generally used for empty welding and false welding detection, but X-rays can affect the health of workers and cannot be used on a large scale on the production line.

[0004] In summary, how to accurately determine the welding condition of the chip without harming the health of the employees is a technical problem that needs to be solved at present. SUMMARY

[0005] The embodiments of the present application provide a method and device for detecting welding quality to solve the problem that X-ray imaging is used to detect the welding condition in the prior art, which can affect the health of workers.

[0006] In a first aspect, the embodiments of the present application provide a method for detecting welding quality, which comprises: determining a second image in which a chip is located from a first image; the first image is obtained by a visual system from the back of a substrate; the front of the substrate is provided with a chip installed by flip technology; the visual system is configured with a shooting device and a light source coaxial with the shooting device; region segmentation is performed on the second image to obtain at least one region; wherein each pixel point in the same region has similarity; for each region, a feature vector of the region is determined, and the feature vector of the region is input into a multilayer perceptron model to obtain a welding detection result of the region; the multilayer perceptron model is obtained by pre-training.

[0007] In the above technical solution, the visual system is used to shoot from the back of the substrate, so that the welding condition of the chip can be shot in the flip condition; the second image is determined from the first image according to the position of the chip, which can reduce the image area that needs to be processed subsequently, thereby speeding up the detection, and then the feature vector of the segmented region is input into the multilayer perceptron model, which can improve the accuracy of the multilayer perceptron model identification.

[0008] Optionally, the determining the second image in which the chip is located from the first image comprises: determining an initial position from the first image by a shape matching algorithm based on the boundary shape of the chip pad on the substrate; and dividing the image in the set region centered on the initial position as the second image in which the chip is located.

[0009] In the technical solution, the second image in which the chip is located is found from the first image based on the boundary shape of the chip pad on the substrate, so that the region in which the chip is located can be quickly positioned, the amount of image data to be processed subsequently is reduced, and the speed of detecting the welding quality is improved.

[0010] Optionally, before the region segmentation of the second image, the method further comprises: performing brightness equalization processing on the second image; and performing gray scale stretching processing on the second image after the brightness equalization processing.

[0011] In the technical solution, the brightness equalization processing is performed on the second image first, and then the gray scale stretching processing is performed on the second image, so that the darker region in the second image is darker and the brighter region is brighter, the brightness of the second image can be clearly seen, the second image can be segmented into initial regions subsequently, and the result of detecting the welding quality can be accurately obtained.

[0012] Optionally, the brightness equalization processing on the second image comprises: determining an over-bright region and an over-dark region in the second image; the over-bright region is a region in which the brightness value of a pixel is higher than a first threshold value; the over-dark region is a region in which the brightness value of a pixel is lower than a second threshold value; setting the brightness value of each pixel in the over-bright region and the brightness value of each pixel in the over-dark region according to the average brightness of each pixel in an equalization region; the equalization region is a region in the second image except the over-bright region and the over-dark region; and performing brightness adjustment on each pixel in the second image, so that the average brightness of the second image is a set value.

[0013] In the technical solution, the average brightness of the equalization region is assigned to the over-bright region and the over-dark region, so that the over-bright region and the over-dark region can avoid interfering with the effect of the subsequent gray scale stretching processing, the average brightness of the second image is normalized before the gray scale stretching, the effect of the gray scale stretching is better, and the result of detecting the welding quality can be accurately obtained.

[0014] Optionally, the second image is segmented into regions to obtain at least one region, including: performing region segmentation on the second image through a region growing algorithm to obtain initial regions; for any initial region, if there is an overly dark region in the initial region, deleting the overly dark region to obtain at least one region; the overly dark region is a region in which the brightness value of pixels in the second image is lower than a second threshold value.

[0015] In the above technical solution, since dark areas indicate damaged pads, the second image is segmented into initial regions using a region growing algorithm, and the dark areas in each region are removed. This can improve the accuracy of determining empty and cold solder joints, thereby achieving more accurate results in detecting solder joint quality.

[0016] Optionally, the welding detection result of the area is a cold joint or a false joint. After obtaining the welding detection result of the area, it also includes: determining whether the welding detection result of the chip is a cold joint based on the area of ​​each area belonging to the false joint detection result; or, determining whether the welding detection result of the chip is a cold joint based on the area of ​​each area belonging to the cold joint detection result; or, determining whether the welding detection result of the chip is a false fault based on the area of ​​each area of ​​the cold joint detection result, the area of ​​each area of ​​the false joint detection result and the area of ​​the dark area in the second picture.

[0017] In the above technical solution, the welding inspection result of the second image is determined based on the areas of the empty and cold joints in each region. Because the second image corresponds to the image of the pad, the chip welding result can be accurately obtained based on the areas of the empty and cold joints in each region. Not only is the chip inspected for empty, cold joints, and damage, but the areas of the cold, empty, and damaged joints in each region are also combined. These four conditions can make a more comprehensive and accurate judgment on the chip welding condition.

[0018] Optionally, the method further includes: determining whether the welding detection result of the chip is a damaged pad based on the area of ​​the too dark area in the second image; the too dark area is an area where the brightness value of the pixel in the second image is lower than a second threshold.

[0019] In the above technical solution, the chip welding detection result can be determined according to the area of ​​the dark area, and the subsequent combination of the areas of false solder joints, empty solder joints and damage in each area can be used to more accurately detect the chip welding condition.

[0020] Optionally, the characteristic vector of the region includes: energy, autocorrelation, homogeneity, contrast, entropy, anisotropy, grayscale mean, grayscale standard deviation, difference between grayscale maximum and minimum, approximate image plane grayscale value deviation, fuzzy entropy, and fuzzy perimeter.

[0021] In the above technical solution, based on the determined characteristic vector, the welding condition of the area can be comprehensively analyzed, and the characteristic vector is subsequently input into the multi-layer perceptron model to obtain the chip welding detection result.

[0022] In the second aspect, an embodiment of the present invention also provides a device for detecting welding quality, including a determination unit for determining a second image where a chip is located from a first image; the first image is obtained by photographing from the back of a substrate through a visual system; a chip installed through a flip-chip process is provided on the front of the substrate; the visual system is configured with a shooting device and a light source coaxial with the shooting device; a processing unit for performing regional segmentation on the second image to obtain at least one region; wherein each pixel point in the same region has similarity; for each region, determining the feature vector of the region, and inputting the feature vector of the region into a multi-layer perceptron model to obtain a welding detection result of the region; the multi-layer perceptron model is obtained through pre-training.

[0023] Optionally, the determination unit is specifically used to determine the initial position from the first image by a shape matching algorithm based on the boundary shape of the chip pad on the substrate; and with the initial position as the center, divide the image within the set area into the second image where the chip is located.

[0024] Optionally, the processing unit is specifically configured to perform brightness equalization processing on the second image; and perform grayscale stretching processing on the second image after the brightness equalization processing.

[0025] Optionally, the processing unit is specifically used to determine the overbright area and the overdark area in the second image; the overbright area is the area where the brightness value of the pixel is higher than the first threshold; the overdark area is the area where the brightness value of the pixel is lower than the second threshold; according to the average brightness of each pixel in the balanced area, the brightness value of each pixel in the overbright area and the brightness value of each pixel in the overdark area are set; the balanced area is the area in the second image excluding the overbright area and the overdark area; the brightness of each pixel in the second image is adjusted so that the average brightness of the second image is the set value.

[0026] Optionally, the processing unit is specifically used to perform region segmentation on the second image through a region growing algorithm to obtain initial regions; for any initial region, if there is an overly dark region in the initial region, the overly dark region is deleted to obtain at least one region; the overly dark region is a region in which the brightness value of the pixel in the second image is lower than a second threshold.

[0027] Optionally, the processing unit is specifically configured to determine whether the soldering detection result of the chip is a false solder according to the area of each region belonging to the false solder detection result, or determine whether the soldering detection result of the chip is an empty solder according to the area of each region belonging to the empty solder detection result, or determine whether the soldering detection result of the chip is a failure according to the area of each region of the false solder detection result, the area of each region of the empty solder detection result and the area of the dark region in each second picture.

[0028] Optionally, the processing unit is specifically configured to determine whether the soldering detection result of the chip is a pad damage according to the area of the dark region in each second picture, and the dark region is a region in which the brightness value of a pixel in the second image is lower than a second threshold.

[0029] Optionally, the processing unit is specifically configured to energy, autocorrelation, homogeneity, contrast, entropy, anisotropy, mean gray value, standard deviation of gray value, difference between maximum and minimum gray value, approximated image plane gray value deviation, fuzzy entropy and fuzzy perimeter.

[0030] In a third aspect, an embodiment of the present application further provides a computing device, including at least one processor and at least one memory, wherein the memory stores a computer program, and when the program is executed by the processor, the processor executes any of the methods for detecting soldering quality in the first aspect.

[0031] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, and the storage medium stores a program, and when the program is run on a computer, the computer implements any of the methods for detecting soldering quality in the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0033] Figure 1 A possible application scenario provided by the embodiment of the present application is shown in the following figure;

[0034] Figure 2 A flow chart of a method for detecting soldering quality provided by the embodiment of the present application is shown in the following figure;

[0035] Figure 3 A structure schematic diagram of a visual system provided by the embodiment of the present application is shown in the following figure;

[0036] Figure 4A schematic diagram of a first image provided by an embodiment of the present invention;

[0037] Figure 5 Schematic diagram of the structure of the boundary shapes of six chip pads on the substrate provided by the embodiment of the present invention;

[0038] Figure 6 A schematic diagram of a second image provided by an embodiment of the present invention;

[0039] Figure 7 A schematic diagram showing a comparison between a second image and the second image after brightness equalization processing provided by an embodiment of the present invention;

[0040] Figure 8 Schematic diagram of comparison between the second image and the segmented second image in an embodiment of the present invention;

[0041] Figure 9 A schematic diagram of a sample image provided by an embodiment of the present invention;

[0042] Figure 10 A schematic structural diagram of a device for detecting welding quality is provided in accordance with an embodiment of the present invention;

[0043] Figure 11 A schematic structural diagram of a computing device is provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0044] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the embodiments described herein are merely some, rather than all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0045] like Figure 1 The figure shows a possible application scenario provided by an embodiment of the present invention. This application scenario can be taken as an example of a MiniLED panel. Figure 1The MiniLED panel 100 and the chip 101 are included. Generally, in the process of producing the MiniLED panel 100, there is a step of welding the chip 101 on the pad. In the welding process, the chip 101 may be welded poorly. In a serious case of poor welding, the MiniLED panel 100 will be checked during appearance inspection and panel lighting inspection before leaving the factory, and will be classified as a defective product. Such a defective product needs to be repaired or scrapped. However, some cases of empty welding or virtual welding are not easy to be checked out. The cases of empty welding or virtual welding belong to slight poor welding. During appearance inspection and panel lighting inspection before leaving the factory, the MiniLED panel 100 can be temporarily lighted and will be shipped as a qualified product. However, after leaving the factory, the MiniLED panel 100 with the cases of empty welding or virtual welding may be detached due to vibration during packaging, transportation, unpacking and reaching the customer, so that the MiniLED panel 100 becomes a defective product and needs to be repaired or even scrapped. In a possible case, the MiniLED panel 100 not only undergoes appearance inspection and panel lighting inspection before leaving the factory, but also undergoes infrared imaging detection. According to the infrared imaging, whether the MiniLED panel 100 has the problems of empty welding or virtual welding can be detected. However, the infrared imaging detection method needs to heat the welding point and then continuously collect the thermal imaging of the welding point. Thus, the time for detecting infrared imaging is long and the efficiency is low. Moreover, heating the welding point may damage the chip, so that the MiniLED panel which is originally a qualified product becomes a defective product due to improper heating of the welding point.

[0046] As shown in Figure 2 A method flow chart for detecting welding quality is provided in the embodiment of the present application. The method comprises the following steps:

[0047] In step 201, the second image in which the chip is located is determined from the first image.

[0048] In the embodiment of the present application, the MiniLED panel is provided with a chip installed by a flip process. The structure of the chip installed by the flip process is that the front surface of the chip faces downward to the substrate. Thus, the welding point is below the chip during welding. Thus, the pad image of the chip cannot be directly observed by using a charge coupled device (CCD) camera. The back surface of the substrate is used to collect the pad image of the chip by using a vision system.

[0049] As shown in Figure 3As shown in the figure, a structural schematic diagram of a visual system provided by an embodiment of the present application is shown. The visual system comprises a shooting device and a light source coaxially arranged with the shooting device. Specifically, the shooting device can be a CCD camera with an industrial lens, and the light source can be a high-uniform coaxial light source. The function of the CCD camera is to convert the light signal into an electrical signal, the function of the industrial lens is to image the target on the light-sensitive surface of the image sensor, and the function of the high-uniform coaxial light source is to highlight the uneven part of the pad image of the chip, so that the interference caused by surface reflection can be overcome. Since the area of the MiniLED panel is relatively large, the visual system cannot completely and clearly shoot the MiniLED panel with one image, so the visual system can shoot the MiniLED panel according to a preset order, or can shoot the MiniLED panel according to specific circumstances, which is not limited here. For example, the visual system can take the leftmost upper part of the MiniLED panel as the starting point, assuming that the size of each shot image is 900*900, then the visual system moves from left to right, moves down a set step, and then moves from left to right again, and so on to traverse the entire MiniLED panel. In this way, the visual system can shoot at least one first image. As shown in the figure, Figure 4 As shown in the figure, a schematic diagram of a first image provided by an embodiment of the present application is shown.

[0050] Because not all positions in the first image exist chips, the area where the chip exists is screened out, so that the amount of image data that needs to be processed subsequently can be reduced, and the efficiency of subsequent image processing can be greatly improved. As shown in the figure, Figure 5 As shown in the figure, a structural schematic diagram of six boundary shapes of chip pads on a substrate provided by an embodiment of the present application is shown. Since the chip is welded on the pad, if the boundary shape of the chip pad on the substrate is taken as the reference, the position of the chip can be found from the first image through the shape matching algorithm. Specifically, the boundary shape of the chip pad matched on the substrate is taken as the center through the shape matching algorithm, and then a plurality of second images are cut out from the first image according to the preset width and height, wherein each second image includes only one chip. For example, if the size of the first image is 900*900, the shape matching algorithm takes the leftmost upper part as the starting point in the first image, the detection window is set to 30*30, and the detection window is moved according to the set step, that is, first from left to right, then down a step, then from left to right again, and so on to move until the first image is traversed. If a region with a similarity to the boundary shape of the chip pad on the substrate reaches a set threshold is detected, the region is cut from the first image according to the preset width and height to obtain a second image. As shown in the figure, Figure 6 As shown in the figure, a schematic diagram of a second image provided by an embodiment of the present application is shown.

[0051] At step 202, the second image is regionally segmented to obtain at least one region.

[0052] As can be seen from step 201, in order to completely detect the Mini LED panel, multiple first images need to be taken, and multiple chips are usually arranged in the first images. Therefore, each chip of the Mini LED panel needs to be detected, that is, each second image needs to be detected. Hereinafter, one second image is taken as an example for description.

[0053] In the embodiment of the present application, in order to more accurately segment the second image, the second image needs to be subjected to brightness equalization processing and gray scale stretching processing, so that the darker places in the second image are darker and the brighter places are brighter.

[0054] First, the second image is subjected to brightness equalization processing. Specifically, the region in which the brightness value of the pixel in the second image is lower than the second threshold value is divided into a too dark region, and the region in which the brightness value of the pixel in the second image is higher than the first threshold value is divided into a too bright region. The first threshold value and the second threshold value can be pre-set or can be determined according to specific conditions, which are not limited herein. The too dark region is generally a pad damage region, and the too bright region can be a region in which dust, hair and other interference elements exist on the chip. After identifying the too dark region, the contour of each too dark region is recorded, and the total area of the too dark region is counted. One possible implementation is that when the proportion of the area of the too dark region to the area of the second image is greater than or equal to the first area threshold value, it is directly determined that the pad is damaged, and the following detection is not performed. In this way, the efficiency of the welding detection can be improved. When the proportion of the area of the too dark region to the area of the second image is less than the first area threshold value, the subsequent detection is continued.

[0055] Since the subsequent second image needs to be subjected to gray scale stretching processing to make the darker places darker and the brighter places brighter, in order to prevent the too dark region and the too bright region from interfering with the result of the gray scale stretching processing, the average brightness of the equalization region of the second image excluding the too dark region and the too bright region is calculated, and the calculated average brightness is assigned to the too dark region and the too bright region. That is, the brightness of the too dark region of the second image is adjusted to be consistent with the average brightness of the equalization region, and the brightness of the too bright region is adjusted to be consistent with the average brightness of the equalization region, so as not to interfere with the result of the subsequent gray scale stretching processing. For example, if the second image is generally dark, the too dark region in the second image needs to be adjusted to be consistent with the average brightness of the equalization region according to the brightness equalization processing, such as Figure 7As shown, it is a contrast diagram of a second image and the second image after brightness equalization provided by the embodiment of the present application. As can be seen from the figure, the same area in the second image, the over-dark area in the second image after brightness processing is brighter than the over-dark area in the original second image, so the over-dark area in the processed second image will not interfere with the result of subsequent gray scale stretching.

[0056] Then, in order to make the effect of gray scale stretching better, it is necessary to first adjust the average value of the brightness of each pixel point in the second image to a set value, which can be pre-set or determined according to specific circumstances, and generally taking the average value of the brightness value to the median of 8-bit depth, i.e. 128 as an example. Then the second image is subjected to gray scale stretching processing. For example, the gray scale value in the specified range is stretched in advance, assuming that the specified range is [first gray scale, second gray scale], the range to be stretched is [third gray scale, fourth gray scale], and there are three cases as follows:

[0057] The first case is that if the target gray scale is less than the first gray scale, the re-assigned target gray scale is calculated according to formula 1.

[0058] The re-assigned target gray scale = target gray scale * third gray scale / first gray scale Formula 1

[0059] According to formula 1, when the target gray scale is less than the first gray scale, the re-assigned target gray scale will be less than the third gray scale. For example, if the specified range is [100, 150] and the range to be stretched is [50, 200], then if the target gray scale is 80, since 80 < 100, the re-assigned target gray scale will be less than 50, so that the originally darker area in the second image will become darker after gray scale stretching.

[0060] The second case is that when the target gray scale is greater than or equal to the first gray scale and less than or equal to the second gray scale, the re-assigned target gray scale is calculated according to formula 2.

[0061] The re-assigned target gray scale =

(target gray scale - first gray scale) * (fourth gray scale - third gray scale) / (second gray scale - first gray scale)

[0062] According to formula 2, when the target gray scale is greater than or equal to the first gray scale and less than or equal to the second gray scale, the re-assigned target gray scale will be greater than or equal to the third gray scale and less than or equal to the fourth gray scale. For example, if the specified range is [100, 150] and the range to be stretched is [50, 200], then if the target gray scale is 120, since 100 < 120 < 150, the re-assigned target gray scale is within the range of [50, 200], so that the originally slightly brighter area in the second image will become brighter after gray scale stretching.

[0063] The third case is when the target gray scale is greater than the second gray scale, the revalued target gray scale is calculated according to Formula 3.

[0064] Revalued target gray scale =

(target gray scale-second gray scale)*(255-fourth gray scale) / (255-second gray scale)

[0065] According to Formula 3, when the target gray scale is greater than the second gray scale, the revalued target gray scale will be greater than the fourth gray scale. For example, if the specified range is [100, 150] and the range to be stretched is [50, 200], then if the target gray scale is 180, since 180>150, the revalued target gray scale will be greater than 200, which will make the originally brighter area in the second image become brighter after gray scale stretching.

[0066] Through the above three formulas, the darker areas in the second image after gray scale stretching become darker, and the brighter areas become brighter.

[0067] Then, the region growing algorithm is used to divide the second image after gray scale stretching into areas with rapid changes and areas with gentle changes. The areas with rapid changes are likely to be well-welded areas, and the areas with gentle changes are likely to be poorly-welded areas. For example, a group of growing points is determined in the second image, and the determination of the growing points can be set in advance or can be determined according to specific circumstances, which is not limited here. Adjacent pixel points or areas with a luminance difference meeting a first threshold value are merged with the growing points to form new growing points, and the operation is repeated until the luminance difference does not meet the first threshold value, and the merging is stopped. Through the merging of the growing points, a region composed of many growing points can be obtained, as shown in FIG. 2. Figure 8 As shown in FIG. 2, it is a comparison diagram of the second image and the segmented second image in the embodiment of the present application. The area profile in the box is an area with gentle changes, and the remaining areas are areas with rapid changes. The area profile can be a regular shape or an irregular shape, which is not limited here. Since the dark areas in the second image are damaged areas, one way to achieve is to mark the dark areas in the second image after determining the area of the dark areas in the second image, so that the dark areas in the second image can be removed after the region segmentation of the second image according to the mark of the dark areas in the second image, to obtain the second image without dark areas, wherein the second image is divided into areas with rapid changes and areas with gentle changes.

[0068] In step 203, for each region, a feature vector of the region is determined.

[0069] In this embodiment of the present invention, a 12-dimensional feature vector is calculated for each region of the second image determined according to step 202 above. The 12-dimensional feature vector includes energy, autocorrelation, homogeneity, contrast, entropy, anisotropy, grayscale mean, grayscale standard deviation, difference between maximum and minimum grayscale values, approximate image plane grayscale value deviation, fuzzy entropy, and fuzzy perimeter. Specifically, energy is the sum of the squared grayscale values ​​of all pixels. Autocorrelation measures the correlation between a pixel and its neighbors across the entire image. Homogeneity measures the degree to which elements in the image's co-occurrence matrix are distributed diagonally. Contrast measures the brightness contrast between a pixel and its neighbors in the image. Entropy reflects the average amount of information in the image. Anisotropy is the characteristic of image pixels exhibiting grayscale differences in all directions. Grayscale mean is the average grayscale value of all pixels in the image. Grayscale standard deviation is the standard deviation of the grayscale values ​​of all pixels in the image. The difference between maximum and minimum grayscale values ​​is the difference between the grayscale values ​​of the brightest and darkest points in the image. The approximated image plane grayscale deviation is the deviation between the calculated image grayscale value and the grayscale approximation. Fuzzy entropy measures the degree to which an image approximates a pure black or pure white image. Fuzzy perimeter calculates the perimeter of the blurred region, treating the image as a collection of blurred regions.

[0070] According to the 12-dimensional feature vector, the image information of each area can be analyzed more comprehensively, so as to facilitate the subsequent determination of the welding condition of the area according to the 12-dimensional feature vector.

[0071] In step 204 , the feature vector of the region is input into a multi-layer perceptron model to obtain a welding detection result of the region.

[0072] In the embodiment of the present invention, a multilayer perceptron model is used as an example. Inputting the 12-dimensional feature vector of a region into the MLP model will yield the welding detection result for the region. Specifically, the MLP model is first trained, then the 12-dimensional feature vector of the region is input into the MLP model, and the MLP model then outputs the welding detection result for the region.

[0073] The specific steps of training the MLP model are to collect sample images in advance, such as several images of qualified products, several defective images containing empty solder joints, and several defective images containing cold solder joints. Figure 9 The figure shows a schematic diagram of a sample image provided by an embodiment of the present invention. These images are then input into the MLP model to train the MLP model, so that the MLP model can distinguish between empty solder joints, poor solder joints, and good solder joints. Because the images are marked before input, that is, the MLP model knows whether the input image is a qualified product, a poor solder joint, or an empty solder joint. This allows the MLP model to be trained on different images in a targeted manner, thereby finding the differences between empty solder joints, poor solder joints, and good solder joints.

[0074] The 12-dimensional feature vector of the region is input into the MLP model, and then the MLP model outputs the welding detection result of the region. Specifically, the 12-dimensional feature vector of the region is input into the MLP model, and the empty welding confidence and the false welding confidence are obtained.

[0075] One implementation manner is to set that there is a priority relationship among the empty welding, the false welding and the good welding, first determine whether the false welding confidence is greater than the false welding confidence threshold value, if yes, determine that the region is false welding, if no, continue to determine whether the empty welding confidence is greater than the empty welding confidence threshold value, if yes, determine that the region is empty welding, if no, determine that the region is good welding.

[0076] After determining the welding conditions of each region, the welding condition of the second image can be determined according to the welding conditions of each region. Specifically, first, the total area of the empty welding, the total area of the false welding and the total area of the pad damage in each region are counted, wherein the total area of the pad damage is determined in the above step 202 and will not be repeated here. It is determined that the welding condition of the second image has a priority relationship. The determination of the priority includes the following steps:

[0077] First, according to the above step 202, the area ratio threshold value of the pad damage in the second image is set as the first threshold value, the area ratio threshold value of the false welding in the second image is set as the second threshold value, and the area ratio threshold value of the empty welding in the second image is set as the third threshold value. The weight values of the false welding, the empty welding and the pad damage are set, which can be pre-set or determined according to the actual situation, and are not limited here. The unknown fault threshold value is set as the fourth threshold value.

[0078] Secondly, it is determined whether the total area of the pad damage is greater than or equal to the first threshold value, if yes, it is determined as pad damage, if no;

[0079] Then, it is determined whether the total area of the false welding is greater than or equal to the second threshold value, if yes, it is determined that the second image is false welding, if no;

[0080] Then, it is determined whether the total area of the empty welding is greater than or equal to the third threshold value, if yes, it is determined that the second image is empty welding, if no;

[0081] Then, it is determined whether the total area of the false welding * the first weight + the total area of the empty welding * the second weight + the pad damage * the third weight is greater than or equal to the fourth threshold value, if yes, it is determined that the second image is unknown fault. If no, it is determined that the second image is good welding.

[0082] Because each second image corresponds to a chip, the welding condition of the second image is also the welding condition of each chip, and subsequent targeted repair operations can be performed according to the welding condition of the corresponding chip.

[0083] As can be seen from the steps 201 to 204, by using the shape matching algorithm, the second image in which each chip is located can be determined from the first image, so that the subsequent processing area is reduced, and the detection speed is accelerated. After the second image is subjected to the gray scale stretching processing and the broken area is removed after the segmentation, the accuracy of the subsequent determination of the region of the empty welding or the false welding can be improved.

[0084] Based on the same technical concept, the present application also provides a device for detecting welding quality, which can execute the method in the method embodiment. The structure of the device for detecting welding quality provided by the present application can be referred to Figure 10 The device 1000 comprises a determination unit 1001 configured to determine a second image in which each chip is located from a first image; the first image is obtained by a visual system shooting from the back of a substrate; the front of the substrate is provided with chips mounted by a flip process; the visual system is configured with a shooting device and a light source coaxial with the shooting device. A processing unit 1002 is configured to perform region segmentation on the second image to obtain at least one region; wherein each pixel point in the same region has similarity; for each region, a feature vector of the region is determined, and the feature vector of the region is input into a multi-layer perceptron model to obtain a welding detection result of the region; the multi-layer perceptron model is obtained by pre-training.

[0085] Optionally, the determination unit 1001 is specifically configured to determine an initial position from the first image by a shape matching algorithm based on the boundary shape of the chip pad on the substrate; and divide the image in the set region as the second image in which the chip is located with the initial position as the center.

[0086] Optionally, the processing unit 1002 is specifically configured to perform brightness equalization processing on the second image; and perform gray scale stretching processing on the second image subjected to the brightness equalization processing.

[0087] Optionally, the processing unit 1002 is specifically configured to determine an over-bright region and an over-dark region in the second image; the over-bright region is a region in which the brightness value of a pixel is higher than a first threshold value; the over-dark region is a region in which the brightness value of a pixel is lower than a second threshold value; the brightness value of each pixel in the over-bright region and the brightness value of each pixel in the over-dark region are set according to the average brightness of each pixel in an equalization region; the equalization region is a region in the second image except the over-bright region and the over-dark region; and the brightness of each pixel point in the second image is adjusted so that the average brightness of the second image is a set value.

[0088] Optionally, the processing unit 1002 is specifically used to perform region segmentation on the second image through a region growing algorithm to obtain initial regions; for any initial region, if there is an overly dark region in the initial region, the overly dark region is deleted to obtain at least one region; the overly dark region is a region in which the brightness value of the pixel in the second image is lower than a second threshold.

[0089] Optionally, the processing unit 1002 is specifically used to determine whether the welding detection result of the chip is a cold joint based on the area of ​​each area belonging to the cold joint detection result, or; determine whether the welding detection result of the chip is a cold joint based on the area of ​​each area belonging to the empty joint detection result, or; determine whether the welding detection result of the chip is a fault based on the area of ​​each area of ​​the cold joint detection result, the area of ​​each area of ​​the empty joint detection result and the area of ​​the dark area in each second image.

[0090] Optionally, the processing unit 1002 is specifically used to determine whether the welding detection result of the chip is a damaged pad based on the area of ​​the dark area in each second image; the dark area is an area where the brightness value of the pixel in the second image is lower than the second threshold.

[0091] Optionally, the processing unit 1002 is specifically used for energy, autocorrelation, homogeneity, contrast, entropy, anisotropy, grayscale mean, grayscale standard deviation, difference between grayscale maximum and minimum values, approximate image plane grayscale value deviation, fuzzy entropy, and fuzzy perimeter.

[0092] Based on the same technical concept, the embodiment of the present application also provides a computing device, such as Figure 11 As shown, it includes at least one processor 1101 and a memory 1102 connected to the at least one processor. The specific connection medium between the processor 1101 and the memory 1102 is not limited in the embodiment of the present application. Figure 11 For example, the processor 1101 and the memory 1102 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, and the like.

[0093] In an embodiment of the present application, the memory 1102 stores instructions that can be executed by at least one processor 1101. By executing the instructions stored in the memory 1102, the at least one processor 1101 can execute the steps included in the aforementioned method for detecting welding quality.

[0094] The processor 1101 is the control center of the computing device, can connect various parts of the computing device by using various interfaces and lines, and realize data processing by running or executing instructions stored in the memory 1102 and calling data stored in the memory 1102. Optionally, the processor 1101 can include one or more processing units, and the processor 1101 can integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, an application program and the like, and the modem processor mainly processes an issued instruction. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 1101. In some embodiments, the processor 1101 and the memory 1102 can be implemented on the same chip, and in some embodiments, they can also be respectively implemented on independent chips.

[0095] The processor 1101 can be a general-purpose processor, for example, a central processing unit (CPU), a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in combination with the method embodiment for detecting the welding quality can be directly embodied as hardware processor execution or executed by a combination of hardware and software modules in the processor.

[0096] The memory 1102, as a non-volatile computer readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 1102 can include at least one type of storage medium, for example, can include flash memory, hard disk, multimedia card, card type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. The memory 1102 is any other medium capable of carrying or storing desired program codes in the form of instructions or data structures and capable of being accessed by a computer, but is not limited thereto. The memory 1102 in the embodiments of the present application can also be a circuit or any other device capable of realizing a storage function, used for storing program instructions and / or data.

[0097] Based on the same technical concept, the embodiments of the present application also provide a computer readable storage medium storing a computer program executable by a computing device, which, when running on the computing device, causes the computing device to perform the steps of the above method for detecting welding quality.

[0098] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0099] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as a combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1one or more processes and / or blocks Figure 1 an apparatus for performing the functions specified in the flowchart or multiple flows and / or blocks.

[0100] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flowchart or multiple flows and / or blocks. Figure 1 one or more processes and / or blocks Figure 1 an apparatus for performing the functions specified in the flowchart or multiple flows and / or blocks.

[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart or multiple flows and / or blocks. Figure 1 one or more processes and / or blocks Figure 1 an apparatus for performing the functions specified in the flowchart or multiple flows and / or blocks.

[0102] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those of skill in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, the appended claims are intended to encompass within their scope all such variations and modifications as being within the scope of the application.

[0103] It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application cover the modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.

Claims

1. A method of detecting the quality of a weld, characterized by, The method comprises: determining a second image in which a chip is located from a first image; the first image is obtained by a vision system shooting from the back of a substrate; the front of the substrate is provided with a chip mounted by a flip process; a shooting device and a light source coaxial with the shooting device are configured in the vision system; the second image is an image of a single chip; region segmentation is performed on the second image to obtain at least one region; wherein each pixel point in the same region has similarity; for each region, a feature vector of the region is determined, and the feature vector of the region is input into a multilayer perceptron model to obtain a solder joint detection result of the region; the multilayer perceptron model is obtained by pre-training; before the region segmentation is performed on the second image, the method further comprises: brightness equalization processing is performed on the second image; gray scale stretching processing is performed on the second image after the brightness equalization processing to obtain regions with rapid changes and regions with slow changes; the brightness equalization processing on the second image comprises: determining an over-bright region and an over-dark region in the second image; the over-bright region is a region in which the brightness value of a pixel is higher than a first threshold value; the over-dark region is a region in which the brightness value of a pixel is lower than a second threshold value; if the area ratio of the over-dark region to the area of the second image is greater than or equal to a first area threshold value, it is determined that the pad is damaged and the detection is ended; if the area ratio of the over-dark region to the area of the second image is less than the first area threshold value, the detection is continued; the brightness value of each pixel in the over-bright region and the brightness value of each pixel in the over-dark region are set according to the average brightness of each pixel in an equalization region; the equalization region is a region in the second image other than the over-bright region and the over-dark region; brightness adjustment is performed on each pixel point in the second image so that the average brightness of the second image is a set value.

2. The method of claim 1, wherein, The determination of the second image in which the chip is located from the first image comprises: the initial position is determined from the first image by a shape matching algorithm with the boundary shape of the chip pad on the substrate as a reference; the image in a set region centered on the initial position is divided into the second image in which the chip is located.

3. The method of claim 1, wherein, The region segmentation on the second image to obtain at least one region comprises: each initial region is obtained by region growing algorithm for region segmentation on the second image; for any initial region, if there is an over-dark region in the initial region, the over-dark region is deleted to obtain at least one region; the over-dark region is a region in the second image in which the brightness value of a pixel is lower than a second threshold value.

4. The method according to any one of claims 1 to 3, characterized in that, The solder joint detection result of the region is empty soldering or false soldering, and after the solder joint detection result of the region is obtained, the method further comprises: determining whether the solder joint detection result of the chip is false soldering according to the area of each region belonging to the false soldering detection result; or determining whether the solder joint detection result of the chip is empty soldering according to the area of each region belonging to the empty soldering detection result; or According to the area of each region of the virtual soldering detection result, the area of each region of the empty soldering detection result, and the area of the over-dark region in the second image, it is determined whether the soldering detection result of the chip is faulty.

5. The method according to any one of claims 1 to 3, wherein The feature vector of the region includes: energy, autocorrelation, homogeneity, contrast, entropy, anisotropy, mean gray value, standard deviation of gray value, difference between maximum and minimum gray value, approximated image plane gray value deviation, fuzzy entropy, and fuzzy perimeter.

6. An apparatus for detecting the quality of a weld, characterized by The method includes: determining a second image in which a chip is located from a first image; The first image is obtained by photographing the back of a substrate by a vision system; The front of the substrate is provided with a chip mounted by a flip process; the vision system is configured with a photographing device and a light source coaxial with the photographing device; and the second image is an image of a single chip; a processing unit configured to perform region segmentation on the second image to obtain at least one region; each pixel point in the same region has similarity; for each region, a feature vector of the region is determined, and the feature vector of the region is input into a multilayer perceptron model to obtain a soldering detection result of the region; the multilayer perceptron model is obtained by pre-training; The processing unit is further configured to perform brightness equalization processing on the second image; and perform gray scale stretching processing on the second image after the brightness equalization processing to obtain regions with abrupt changes and regions with gradual changes; The processing unit is further configured to determine an over-bright region and an over-dark region in the second image; the over-bright region is a region in which the brightness value of a pixel is higher than a first threshold value; the over-dark region is a region in which the brightness value of a pixel is lower than a second threshold value; if the area of the over-dark region accounts for more than or equal to a first area threshold value of the area of the second image, it is determined that the pad is damaged and the detection is ended; if the area of the over-dark region accounts for less than the first area threshold value of the area of the second image, the detection is continued; the brightness value of each pixel in the over-bright region and the brightness value of each pixel in the over-dark region are set according to the average brightness of each pixel in the equalization region; the equalization region is a region in the second image other than the over-bright region and the over-dark region; the brightness of each pixel in the second image is adjusted so that the average brightness of the second image is a set value.

7. A computing device, comprising: The storage medium stores a program, and when the program runs on a computer, the computer implements the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program, and when the program runs on a computer, the computer implements the method of any one of claims 1 to 5.

Images