Machine vision-based industrial product quality detection system and method

Abstract

The application discloses a machine vision-based industrial product quality detection system and method, and particularly relates to the technical field of industrial automation; the application effectively solves the technical problem that the prior art can only complete static appearance quality detection of electronic components and cannot identify hidden functional defects of the components under rated working conditions, through multi-view appearance image acquisition and defect analysis, a surface evaluation coefficient is quantitatively generated, and preliminary screening and filtration of unqualified products are completed; then, rated electrical excitation is applied to the appearance qualified products, through multi-modal machine vision imaging, multi-dimensional response images such as infrared thermal imaging and microscopic dynamic imaging of the components during electrified working are synchronously acquired, the functional characteristics of four core dimensions of heat, deformation, action response and photoelectric switch are accurately quantified, hidden defects without static appearance representation are effectively detected, the risk of unqualified products flowing into the market is greatly reduced, and the running stability and use safety of terminal equipment are guaranteed from the source.

Description

Technical Field

[0001] This invention relates to the field of industrial automation inspection technology, and more specifically, to an industrial product quality inspection system and method based on machine vision. Background Technology

[0002] Electronic components are core basic components in fields such as industrial automation, consumer electronics, and automotive electronics. Their product quality directly determines the operational stability and safety of terminal equipment. Therefore, the full-process quality inspection of components before they leave the factory is a crucial process in the electronic manufacturing process.

[0003] With the development of industrial automation technology, machine vision inspection technology, with its advantages of non-contact operation, high inspection speed, and adaptability to continuous production lines, is gradually replacing traditional manual visual inspection and becoming the mainstream technical solution for the appearance quality inspection of electronic components. However, in actual production applications, existing inspection solutions still have the following technical shortcomings: Existing technologies can only inspect the static appearance quality of components and cannot effectively identify and quantify the internal latent functional defects of components under rated electrical operating conditions. During the production process, electronic components are prone to latent defects such as drive performance degradation. These defects are not obvious in static appearance inspection and can only be seen when the components are energized. Existing technologies cannot effectively detect such defects, which can easily lead to unqualified products entering the market and causing terminal equipment malfunctions. Furthermore, existing technologies struggle to achieve multi-dimensional, synchronous, and precise testing of component performance. The performance of electronic components encompasses multiple core dimensions, including thermal characteristics, mechanical deformation characteristics, action response characteristics, and photoelectric switching characteristics. Most existing testing solutions only perform independent testing on single-dimensional indicators and cannot simultaneously acquire multi-dimensional operational response data of components under rated electrical excitation using multi-modal machine vision imaging technology. Nor can they perform standardized quantitative evaluation of performance indicators in each dimension. Consequently, the comprehensiveness and accuracy of the test results are insufficient to meet the testing requirements of high-end electronic components.

[0004] To address this, a machine vision-based industrial product quality inspection system and method have been developed. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide an industrial product quality inspection system and method based on machine vision.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The machine vision-based industrial product quality inspection system includes the following modules: The appearance visual evaluation module is used to acquire multi-view appearance images of the electronic components to be tested, perform appearance defect analysis, and output the appearance evaluation coefficient of the electronic components to be tested. The product sorting and unloading module is used to receive the surface evaluation coefficients of the electronic components to be tested, and filter out the products that are out of appearance by combining them with the set surface evaluation pass coefficients. After the products that are out of appearance are numbered, the electrical excitation signal is triggered. The electrical excitation module is used to apply the rated electrical working excitation to the apparent qualified product after the electrical excitation signal is triggered, so that the apparent qualified product enters the normal working state; The functional testing module is used to acquire multimodal working response images of apparent qualified products under electrical excitation and perform functional quality testing analysis. Based on the results of the functional quality testing analysis, it selectively determines whether the apparent qualified products are functionally unqualified or functionally qualified products, and generates a functional distribution model of functionally qualified products.

[0007] Specifically, the logic for obtaining the evaluation coefficient; Extract the sets of appearance defect parameters corresponding to the pin area and the package area from the multi-view appearance images of the electronic component to be inspected; The apparent defect parameter set of the pin area includes dimensional deviation and surface condition. The apparent defect parameter set of the packaging area includes the degree of damage and surface smoothness; The evaluation coefficient of the electronic component under test is obtained by comprehensively processing the dimensional deviation and surface condition of the pin area, the damage and defects of the package area, and the surface flatness.

[0008] Specifically, the logic for obtaining the apparent defect parameter set of the pin area and the package area; The measured dimensions of a single pin of an electronic component are compared with the nominal values ​​to calculate the absolute deviation rate. The dimensions include length, width, thickness, and pin spacing. The absolute deviation rates calculated for length, width, thickness, and pin spacing are weighted and calculated using logic to output the pin deviation degree. The highest value among the pin deviation degrees in each group is identified as the size deviation degree. Obtain the total area of ​​the oxide region of a single pin and calculate its proportion in the total area of ​​the single pin pad to obtain the oxide ratio; obtain the total area of ​​the plating damage of a single pin and calculate its proportion in the total area of ​​the single pin pad to obtain the plating damage ratio; use weighted calculation logic to calculate the oxide ratio and the plating damage ratio to output the pin status degree; identify the highest value in each group of pin status degrees as the surface status degree. The apparent defect parameter set of the packaging area includes the degree of damage and surface smoothness; The defect areas of various types within the packaging area corresponding to electronic components are statistically analyzed and summed to obtain the total area of ​​package damage; the proportion of the total area of ​​package damage in the packaging area is calculated to obtain the degree of damage. Point cloud data of the electronic component package surface is acquired and fitted to a plane to obtain a reference plane; the height deviation of all points on the package area surface relative to the reference plane is calculated; the maximum positive deviation and the maximum negative deviation are identified, and the surface flatness is obtained after difference calculation.

[0009] Specifically, the functional quality testing and analysis logic; Multimodal working response images include infrared thermal imaging sequence images, high-resolution microscopic dynamic sequence images, high-speed industrial camera dynamic response sequence images, and electroluminescence imaging sequence images; After comprehensive processing of the multimodal working response images, the thermal performance coefficient, deformation performance coefficient, response performance coefficient, and switching performance coefficient of the apparent qualified product are obtained.

[0010] Specifically, the logic for obtaining the thermal performance coefficient; Based on the infrared thermal imaging sequence of the apparent qualified product under electrical excitation, the time points when the apparent qualified product reaches thermal steady state under rated electrical working excitation are marked; the overall temperature value is extracted from the infrared image after the marked time point, and the arithmetic mean is taken to obtain the steady-state average temperature of the apparent qualified product. During the thermal steady-state phase, match and lock a single pin, extract the average temperature of the pad corresponding to each pin, and take the arithmetic mean of the average temperature of each pin as the pin average temperature; calculate the ratio between the average temperature of the pad corresponding to each pin and the pin average temperature, and take the maximum ratio as the pin consistency deviation. Starting from the application of excitation, the temperature rise sequence is acquired at a fixed frame rate until thermal steady state is reached. The time difference between the start time of excitation application and the time of reaching thermal steady state is calculated to obtain the thermal response time. The absolute deviation between the thermal response time and the pre-stored standard thermal response time is calculated to obtain the thermal response deviation. During the thermal steady-state stage, the pre-marked normal heating areas are removed, and the remaining areas are marked as non-heating areas. The highest temperature value of the non-heating areas after the thermal steady-state stage is obtained as the hot spot peak temperature. The steady-state average temperature, pin consistency deviation, thermal response deviation, and hot spot peak temperature of the apparent qualified product are comprehensively processed from the infrared thermal imaging sequence images under electrical excitation to determine the thermal performance coefficient of the apparent qualified product.

[0011] Specifically, the logic for obtaining the deformation function coefficient; Based on high-resolution microscopic dynamic sequence images of apparent qualified products under electrical excitation, the pin length is extracted from the high-resolution microscopic dynamic sequence images under thermal steady-state stage as the steady-state length. The deformation is calculated by the absolute difference between the steady-state length and the pin length before electrical excitation. The maximum value of the deformation among all pins is taken as the deformation peak. The initial position of the bond line of the apparent qualified product is obtained before electrical excitation. After electrical excitation, the maximum displacement of the arc apex of the bond line is calculated by feature point matching and used as the measured relative displacement. The deformation peak and measured relative displacement of the apparent qualified product under electrical excitation are analyzed by comprehensively processing, and the deformation functional coefficient of the apparent qualified product is determined.

[0012] Specifically, the logic for obtaining response function coefficients; Based on the dynamic response sequence images of the apparent qualified product under electrical excitation from a high-speed industrial camera, the timing starts from the moment the excitation is applied and ends at the point in time when the movable part of the apparent qualified product completes the specified action, which is used as the action response delay time. Traverse the high electroluminescence imaging sequence image, identify the first image frame where the first contact point enters the contact state, and record its corresponding timestamp as the bounce start time. Traverse backward from the bounce start time, identify the starting frame that meets the preset bounce end condition, and record its corresponding timestamp as the bounce end time. Calculate the time difference between the bounce end time and the bounce start time to obtain the bounce duration. The response function coefficient of the apparent qualified product is determined by comprehensively processing the motion response delay and bounce time of the dynamic response sequence image of the high-speed industrial camera under electrical excitation.

[0013] Specifically, the logic for obtaining the switch function coefficient; Based on the high electroluminescence imaging sequence images of the apparent qualified products under electrical excitation, the effective light-emitting area, core functional area and fixed background area are pre-divided according to the standard drawings of electronic components. After the excitation is applied, wait for the apparent qualified product to enter the luminescence stable state, extract the average gray value and gray standard deviation of the effective luminescence area after the luminescence stable state, divide the gray standard deviation by the average gray value to obtain the luminescence intensity uniformity. After entering the stable luminescence state, the pixels in the core functional area are segmented by threshold to identify all areas with gray values ​​lower than the preset gray value and mark them as dark spot areas; the area of ​​all dark spot areas is summed and then divided by the total area of ​​the core functional area to obtain the dark spot area ratio. The uniformity of luminescence intensity and the proportion of dark spot area analyzed from the electroluminescence imaging sequence images of the apparent qualified product under electrical excitation are comprehensively processed to determine the switching function coefficient of the apparent qualified product.

[0014] Specifically, it selectively determines whether an apparent qualified product is a functionally unqualified product or a functionally qualified product, and generates a functional distribution model for functionally qualified products. Extraction steps - The calculated thermal performance coefficient, deformation performance coefficient, response performance coefficient, and switching performance coefficient are input into a pre-edited coefficient threshold library for comparison. Based on the comparison results, products with unqualified functions and products with qualified functions are output. The coefficient threshold library stores the threshold coefficients corresponding to the thermal performance coefficient, deformation performance coefficient, response performance coefficient, and switching performance coefficient, respectively. If the product is functionally qualified, calculate the ratios between the thermal performance coefficient, deformation performance coefficient, response performance coefficient, and switching performance coefficient and the corresponding threshold coefficients; thus obtaining four sets of coefficient ratios. Starting from the origin of the Cartesian coordinate system, four polar coordinate rays with equal angular intervals are emitted; the length of each ray corresponds to a set of coefficient ratios, and the endpoints of adjacent rays are connected in sequence to form a closed quadrilateral shape, which serves as the functional distribution model.

[0015] Machine vision-based industrial product quality inspection methods include: Visual inspection of appearance: control the electronic components to be inspected to complete the feeding, and after pose calibration, acquire multi-view appearance images of the electronic components to be inspected. The appearance evaluation coefficient of the electronic components to be inspected is calculated based on the extracted set of appearance defect parameters. Product sorting and initial screening: The evaluation coefficient of the electronic components to be tested is compared with the preset evaluation pass coefficient to complete the classification and screening of apparent qualified products and apparent unqualified products. Appearance qualified products are uniquely numbered and bound, and electrical excitation signaling is triggered. Rated electrical excitation: After receiving the electrical excitation signal, the pre-stored excitation parameter library is called according to the model of the electronic component to be tested, and the matching rated electrical working excitation is applied to the apparent qualified product, so that the apparent qualified product enters the rated working state; Multimodal functional testing: Collect multimodal working response images of apparent qualified products under electrical excitation, extract multi-dimensional functional feature parameters based on the multimodal working response images, and calculate the functional coefficients of the corresponding dimensions. Qualification assessment and model building: The functional coefficients of each dimension are compared with the preset coefficient threshold library to determine whether the product is functionally qualified or not; for functionally qualified products, a functional distribution model is built based on the functional coefficients of each dimension, and the functional performance value is calculated. Test report generation: Summarize the test data from the entire process, fill it into a pre-built report template, and generate a product quality test report with a unique number.

[0016] The technical effects and advantages of this invention are as follows: (1) This invention effectively solves the technical problem that the existing technology can only complete the static appearance quality inspection of electronic components and cannot identify the hidden functional defects of components under rated working conditions. By acquiring and analyzing appearance images from multiple perspectives, the appearance evaluation coefficient is quantified and the initial screening of appearance non-conforming products is completed. Then, rated electrical excitation is applied to appearance qualified products. Through multi-modal machine vision imaging, multi-dimensional response images such as infrared thermal imaging and microscopic dynamic imaging are simultaneously acquired when the components are powered on. The functional characteristics of the four core dimensions of thermal, deformation, action response and photoelectric switching are accurately quantified. The hidden defects that are not characterized by static appearance are effectively detected, which greatly reduces the risk of non-conforming products entering the market and ensures the operational stability and safety of terminal equipment from the source. (2) This invention breaks through the technical limitations of existing testing schemes, which make it difficult to achieve synchronous and accurate testing of multi-dimensional performance of components and cannot be standardized and quantitatively evaluated. It constructs a standardized quantitative evaluation system for the entire process, realizes the full-dimensional quantification of testing indicators, avoids the subjective bias and blind spots of manual visual inspection and single-dimensional testing; at the same time, through the precise synchronization of multi-modal image acquisition and electrical excitation timing, it realizes the synchronous testing of multi-dimensional performance, and also constructs a polar coordinate functional distribution model for qualified products, intuitively quantifying the comprehensive performance. It not only meets the high-precision testing requirements of high-end electronic components, but also realizes full-process traceability of products, significantly improving the testing efficiency and quality control level of the production line. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the machine vision-based industrial product quality inspection system of the present invention. Figure 2 This is a flowchart of the machine vision-based industrial product quality inspection method of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1

[0020] like Figure 1 As shown, the modules of an industrial product quality inspection system based on machine vision are as follows: The appearance visual evaluation module is used to control the conveying equipment to implement the orderly feeding of electronic components to be tested and the precise conveying between the testing stations. After completing the component pose calibration, it collects multi-view appearance images of the electronic components to be tested, performs appearance defect analysis, and outputs the appearance evaluation coefficient of the electronic components to be tested. Image preprocessing: Gaussian denoising, histogram equalization, contrast enhancement, lens distortion correction, and multi-view image stitching are performed on the acquired multi-view images.

[0021] Specifically: Extract the sets of appearance defect parameters corresponding to the pin area and the package area from the multi-view appearance images of the electronic component to be inspected; Precise ROI segmentation: Through template matching, threshold segmentation, and edge detection algorithms, the three core detection areas of package body ROI, pin ROI, and silkscreen marking ROI are accurately separated, with each ROI corresponding to an independent pixel coordinate system.

[0022] The apparent defect parameter set of the pin area includes dimensional deviation and surface condition. Extract pre-stored standard drawings of electronic components from the database, compare the actual measured dimensions of a single pin of the electronic component with the nominal values ​​marked in the standard drawings, and calculate the absolute deviation rate; dimensions include length, width, thickness, and pin spacing. Verify the soldering compatibility of the pins; excessive deviation can lead to SMT mounting misalignment failure, cold solder joints, or short circuits.

[0023] Measured dimensional deviation = ; Extract the preset maximum allowable deviation for each size from the database; The absolute deviation rate is obtained by dividing the measured dimensional deviation by the corresponding preset maximum allowable deviation.

[0024] The absolute deviation rates calculated separately for length, width, thickness, and pin spacing are weighted and calculated using logic to output the pin deviation rate. That is, by using the preset size weighting coefficients corresponding to the length, width, thickness and pin spacing, the absolute deviation rate of each group is multiplied by the corresponding size weighting coefficient, and then the sum is obtained to obtain the pin deviation.

[0025] The highest value among the pin deviations in each group is identified as the dimensional deviation. Obtain the total area of ​​the oxide region of a single pin and calculate its proportion in the total area of ​​the single pin pad to obtain the oxide proportion; Obtain the total area of ​​plating damage on a single pin and calculate its proportion in the total area of ​​the single pin pad to obtain the plating damage percentage. The oxidation percentage and the coating damage percentage are weighted and calculated using logic to output the pin status. That is, by pre-setting the state weight coefficients corresponding to plating damage and oxidation, the oxidation ratio and plating damage ratio are multiplied by the corresponding state weight coefficients, and then summed to obtain the pin state degree.

[0026] The highest value among the pin status parameters of each group is identified as the surface status parameter; The apparent defect parameter set of the packaging area includes the degree of damage and surface smoothness; The area of ​​various defects within the packaging area corresponding to electronic components is statistically analyzed and summed to obtain the total area of ​​package damage. The types of defects included in the statistics include, but are not limited to: scratches, cracks, dirt, foreign matter, bubbles, pinholes, pits, and oxide spots.

[0027] Calculate the percentage of the total damaged area in the package area to obtain the damage defect level; Point cloud data of the electronic component package surface is acquired and fitted to a plane to obtain a reference plane; Calculate the height deviation of all points on the surface of the packaged area relative to the reference plane; The surface flatness is obtained by identifying the maximum positive deviation (protrusion) and the maximum negative deviation (depression) and calculating the difference. ;in For the maximum positive deviation, This represents the maximum negative deviation.

[0028] The evaluation coefficient of the electronic component under test is obtained after comprehensive processing based on the dimensional deviation and surface condition of the pin area, the damage and defects of the package area, and the surface flatness. The calculation process is as follows: Dimensional deviation, surface condition, breakage and defects, and surface flatness are respectively denoted as... , , , ; Based on current electronic component manufacturing quality standards, preset threshold values ​​for dimensional deviation, surface condition, breakage defects, and surface flatness are established. Surface state threshold Defect threshold and surface flatness threshold ; Using formula The rating coefficient was calculated. ; , , as well as These are preset weighting coefficients, and their sum is one.

[0029] The product sorting and unloading module is used to receive the surface evaluation coefficients of the electronic components to be tested, filter out the products that are out of appearance, and trigger the electrical excitation signal after numbering the outsourced products that are out of appearance. The production batch number is linked to the production time and testing equipment information to achieve full-process traceability.

[0030] By comparing the apparent evaluation coefficient of the electronic component to be tested with the set apparent evaluation pass coefficient, if the apparent evaluation coefficient is lower than the apparent evaluation pass coefficient, it is marked as an apparent qualified product, otherwise it is marked as an apparent unqualified product.

[0031] The electrical excitation module is used to call the pre-stored excitation parameter library according to the component model after the electrical excitation signal is triggered, and control the application of the matching rated electrical working excitation to the apparent qualified product, so that the apparent qualified product enters the normal working state. The functional testing module is used to acquire multimodal working response images of apparent qualified products under electrical excitation and perform functional quality testing analysis. Based on the results of the functional quality testing analysis, it selectively determines whether the apparent qualified products are functionally unqualified or functionally qualified products, and generates a functional distribution model of functionally qualified products. Multimodal working response images include infrared thermal imaging sequence images, high-resolution microscopic dynamic sequence images, high-speed industrial camera dynamic response sequence images, and electroluminescence imaging sequence images; In addition, the acquisition timing of all multimodal images is strictly synchronized with the application / disconnection timing of electrical excitation, with a timestamp alignment accuracy of ≤1ms, to ensure that the acquired response images are completely matched with the excitation state of the components.

[0032] Specifically: Based on the infrared thermal imaging sequence of the apparent qualified product under electrical excitation, the time point when the apparent qualified product reaches thermal steady state under rated electrical working excitation is marked; the judgment condition for thermal steady state is: applying rated excitation until the temperature change rate is ≤0.5℃ / min (reaching thermal steady state), where 0.5℃ / min is a preset value and can be dynamically adjusted later; The overall temperature value is extracted from the infrared image after the marked time point, and the steady-state average temperature of the apparent qualified product is obtained by taking the arithmetic mean. Under rated electrical excitation and thermal steady-state conditions, the rated power consumption, internal thermal resistance, and heat dissipation characteristics of qualified components of the same model are fixed. The steady-state average temperature will be relatively stable. The higher the steady-state average temperature, the higher the overall heat generation of the component under rated operating conditions, which corresponds to functional defects such as excessive overall power consumption, abnormal leakage current of the entire chip, overall short circuit of the power domain, and overall performance degradation of the wafer.

[0033] In the thermal steady state stage, match and lock a single pin, extract the average temperature of the pad corresponding to each pin, and take the arithmetic mean of the average temperature of each pin as the pin average temperature. For each pin, calculate the ratio between the average temperature of the corresponding pad and the average temperature of the pin, and take the maximum ratio as the pin consistency deviation; The ratio is calculated using the average temperature as the numerator and the pin average temperature as the denominator.

[0034] The greater the pin consistency deviation, the higher the heat generation of a certain pin is compared to the average level of all pins. This corresponds to functional defects such as poor soldering, poor pad contact, pin bonding failure, and open circuit. These defects will cause an abnormal increase in pin contact resistance, generating excessive Joule heat under rated current, and causing the pin temperature to deviate significantly from the average level.

[0035] Starting from the application of excitation, the temperature rise sequence is acquired at a fixed frame rate until thermal steady state is reached. The time difference between the start time of excitation application and the time of reaching thermal steady state is calculated to obtain the thermal response time. The absolute deviation between the thermal response time and the pre-stored standard thermal response time is calculated to obtain the thermal response deviation. Thermal response deviation ,in and These represent the thermal response time and the pre-stored standard thermal response time, respectively.

[0036] The greater the thermal response deviation, the greater the degree of deviation between the thermal response characteristics of the component and the qualified standard product of the same model. If the actual thermal response time is much greater than the standard value (large deviation): it indicates that the internal thermal resistance of the component is too large, the packaging interface is delaminated, the capacitor charging and discharging is ineffective, the leakage current is abnormal, resulting in a slow temperature rise rate. If the actual thermal response time is much smaller than the standard value (the deviation is also large): it indicates that there is a local short circuit inside the component, the power consumption is abnormally high, and the temperature rise rate is too fast.

[0037] During the thermal steady-state phase, pre-marked normal heat-generating areas (such as power supplies or chips, pre-marked using standard electronic component drawings) are removed. After removal, the remaining area is marked as a non-heating area, and the highest temperature value of the non-heating area after the thermal steady state stage is obtained as the hot spot peak temperature. The higher the hot spot peak temperature, the more likely it is that an unexpected abnormal local high temperature has appeared in the non-heating area. This corresponds to hidden functional defects such as local short circuits inside the components and local die failures in the wafer. The higher the value, the more severe the local defect.

[0038] The steady-state average temperature, pin consistency deviation, thermal response deviation, and hot spot peak temperature of the apparent qualified products are comprehensively processed to determine the thermal performance coefficient of the apparent qualified products. The specific calculation process is as follows: Based on the production quality standards of electronic components, a preset steady-state average temperature is established. Pin consistency deviation Thermal response deviation and hotspot peak temperature The corresponding steady-state average allowable temperature Pin consistency tolerance Allowable deviation of thermal response And the allowed peak temperature for hotspots; Using formula The thermal functional coefficient was calculated. ; , , as well as These are preset weighting coefficients, and their sum is one.

[0039] Based on high-resolution microscopic dynamic sequence images of apparent qualified products under electrical excitation, the pin length is extracted from the high-resolution microscopic dynamic sequence images under thermal steady-state stage as the steady-state length. The deformation is calculated by the absolute difference between the steady-state length and the pin length before electrical excitation. The maximum value of the deformation among all pins is taken as the deformation peak. The larger the deformation peak, the more severe the abnormal pin contact resistance, the higher the degree of poor soldering / pad peeling defects, and the greater the risk of pin detachment, electrical open circuit, and contact failure in subsequent use.

[0040] Obtain the initial position of the bonding wire (gold or copper wire connecting the chip and the pin) of the apparent qualified product before electrical excitation. After electrical excitation, calculate the maximum displacement of the bonding wire arc apex through feature point matching, and use it as the measured relative displacement. The greater the measured relative displacement, the worse the bonding wire fixation, and the more severe the defects such as bonding failure, internal electrical open circuit, and chip packaging delamination.

[0041] The deformation peak and measured relative displacement analyzed from high-resolution microscopic dynamic sequence images of apparent qualified products under electrical excitation are comprehensively processed to determine the deformation functional coefficient of apparent qualified products. The specific calculation process is as follows: Based on the production quality standards of electronic components, the deformation peak value is preset. and measured relative displacement The corresponding pin deformation allowances and relative allowable displacement ; Using formula The deformation work coefficient was calculated. ; , These are preset weighting coefficients, and their sum is one.

[0042] Based on the dynamic response sequence images of the apparent qualified product under electrical excitation from a high-speed industrial camera, the timing starts from the moment the excitation is applied and ends at the point in time when the movable part of the apparent qualified product completes the specified action, which is used as the action response delay time. Additional explanation: The determination of whether a specified action is completed is as follows: for example, if the real-time outline of a movable part overlaps with the standard action completion template by ≥95%, the action is considered complete.

[0043] The longer the delay, the more likely the components have defects such as insufficient driving capability, performance degradation of coil / piezoelectric elements, contact jamming, and internal circuit response delay, resulting in sluggish movement of movable parts and failure to meet the rated response requirements.

[0044] Traverse the high-electroluminescence imaging sequence, identify the first image frame where the contact point enters the contact state, and record its corresponding timestamp as the bounce start time. Starting from the bounce start time, traverse backwards to identify the starting frame that meets the preset bounce end condition, and record its corresponding timestamp as the bounce end time. The preset bounce end condition is, for example, 10 or more consecutive frames in a stable contact state. Calculate the time difference between the bounce end time and the bounce start time to obtain the bounce duration. The longer the bounce time, the longer the unstable process of "contact-disconnect-re-contact" during the contact engagement process, which corresponds to defects such as contact erosion, insufficient flatness of the contact surface, loss of elasticity of the spring, and insufficient engagement force.

[0045] The response function coefficient of the apparent qualified product is determined by comprehensively processing the motion response delay and bounce time of the dynamic response sequence image of the high-speed industrial camera under electrical excitation. The specific calculation process is as follows: Based on the production quality standards of electronic components, preset action response delay time and the time taken for jumping The corresponding standard action response time and the standard duration of the jump ; Using formula Calculate the response function coefficient ; , These are preset weighting coefficients, and their sum is one.

[0046] Based on the high electroluminescence imaging sequence images of the apparent qualified products under electrical excitation, the effective light-emitting area, core functional area and fixed background area are pre-divided according to the standard drawings of electronic components. For example, the effective light-emitting area is the complete light-emitting area of ​​the wafer / PN junction, excluding invalid areas such as edge packaging blockage and gold wire reflection; Core functional area: Key areas that determine core functions, such as the chip core and the active region of the PN junction; Fixed background area: The area where the components are enclosed and do not emit light.

[0047] After the excitation is applied, wait for the apparent qualified product to enter the luminous stability state; judgment condition: average gray value fluctuation of the effective luminous area for 5 consecutive frames ≤2%; extract the average gray value and gray value standard deviation of the effective luminous area after the luminous stability state, divide the gray value standard deviation by the average gray value to obtain the luminous intensity uniformity.

[0048] A higher uniformity of luminous intensity indicates greater fluctuations in luminous intensity and more significant differences.

[0049] After entering the stable luminescence state, the pixels in the core functional area are segmented by threshold to identify all areas with gray values ​​lower than the preset gray value and mark them as dark spot areas; the area of ​​all dark spot areas is summed and then divided by the total area of ​​the core functional area to obtain the dark spot area ratio. The larger the proportion of dark spots, the larger the failure area.

[0050] The uniformity of luminescence intensity and the proportion of dark spot area analyzed from the electroluminescence imaging sequence images of the apparent qualified products under electrical excitation are comprehensively processed to determine the switching function coefficient of the apparent qualified products. The specific calculation process is as follows: Based on the production quality standards of electronic components, the luminous intensity uniformity and dark spot area ratio are preset to correspond to the luminous intensity pass uniformity and dark spot area pass ratio, respectively. Using formula Calculate the switching function coefficient ; , These are preset weighting coefficients, and their sum is one.

[0051] Extraction steps - The calculated thermal performance coefficient, deformation performance coefficient, response performance coefficient, and switching performance coefficient are input into a pre-edited coefficient threshold library for comparison. The coefficient threshold library stores the threshold coefficients corresponding to the thermal performance coefficient, deformation performance coefficient, response performance coefficient, and switching performance coefficient, respectively. If any performance coefficient of an apparent qualified product is higher than the corresponding stored threshold coefficient, the apparent qualified product is determined to be a functionally unqualified product. If all performance coefficients of an apparent qualified product are lower than the corresponding stored threshold coefficients, the apparent qualified product is determined to be a functionally qualified product. If the product is functionally qualified, calculate the ratios between the thermal performance coefficient, deformation performance coefficient, response performance coefficient, and switching performance coefficient and their corresponding threshold coefficients; thus obtaining four sets of coefficient ratios. The ratio calculation uses the thermal energy coefficient, deformation energy coefficient, response energy coefficient, and switching energy coefficient as the denominator, and the corresponding threshold coefficient as the numerator.

[0052] Starting from the origin of the Cartesian coordinate system, four polar coordinate rays are emitted at equal angular intervals; the polar angle interval between adjacent rays is fixed at 90°. The length of each ray corresponds to a set of coefficient ratios. The end points of adjacent rays are connected in sequence to form a closed quadrilateral shape, which serves as a functional distribution model. The area of ​​the model is used as the functional performance value of a functionally qualified product. The evaluation coefficients of all apparent nonconforming and apparent conforming products, the thermal function coefficients, deformation function coefficients, response function coefficients, and switching function coefficients of functionally nonconforming and functionally conforming products, and the functional distribution model of functionally conforming products are summarized, numbered, and filled into the pre-built report template to form a quality inspection report.

[0053] Example 2

[0054] Please see Figure 2 As shown, based on the machine vision-based industrial product quality inspection system provided in Embodiment 1 of this application, Embodiment 2 of this application proposes a machine vision-based industrial product quality inspection method. Embodiment 2 is merely a preferred embodiment of Embodiment 1, and its implementation will not affect the individual implementation of Embodiment 1.

[0055] Specifically, the machine vision-based industrial product quality inspection method provided in Embodiment 2 of this application includes: Visual inspection of appearance: control the electronic components to be inspected to complete the feeding, and after pose calibration, acquire multi-view appearance images of the electronic components to be inspected. The appearance evaluation coefficient of the electronic components to be inspected is calculated based on the extracted set of appearance defect parameters. Product sorting and initial screening: The evaluation coefficient of the electronic components to be tested is compared with the preset evaluation pass coefficient to complete the classification and screening of apparent qualified products and apparent unqualified products. Appearance qualified products are uniquely numbered and bound, and electrical excitation signaling is triggered. Rated electrical excitation: After receiving the electrical excitation signal, the pre-stored excitation parameter library is called according to the model of the electronic component to be tested, and the matching rated electrical working excitation is applied to the apparent qualified product, so that the apparent qualified product enters the rated working state; Multimodal functional testing: Collect multimodal working response images of apparent qualified products under electrical excitation, extract multi-dimensional functional feature parameters based on the multimodal working response images, and calculate the functional coefficients of the corresponding dimensions. Qualification assessment and model building: The functional coefficients of each dimension are compared with the preset coefficient threshold library to determine whether the product is functionally qualified or not; for functionally qualified products, a functional distribution model is built based on the functional coefficients of each dimension, and the functional performance value is calculated. Test report generation: Summarize the test data from the entire process, fill it into a pre-built report template, and generate a product quality test report with a unique number.

[0056] The above formulas are all dimensionless calculations. Dimensionless calculations can be performed using various methods such as standardization, which will not be elaborated here. The formulas are derived from software simulations based on a large amount of collected data, and the preset parameters in the formulas can be set by those skilled in the art according to the actual situation.

[0057] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, ATA hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state ATA hard disk.

[0058] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0059] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0060] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0061] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0062] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0063] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable ATA hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0064] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A machine vision-based industrial product quality inspection system, characterized in that, Includes the following modules: The appearance visual evaluation module is used to acquire multi-view appearance images of the electronic components to be inspected, extract appearance defect parameters from the pin area and package area, and output the appearance evaluation coefficient of the electronic components to be inspected. The apparent defect parameter set of the pin area includes dimensional deviation and surface condition. The apparent defect parameter set of the packaging area includes the degree of damage and surface smoothness; The product sorting and unloading module is used to receive the surface evaluation coefficients of the electronic components to be tested, and filter out the products that are out of appearance by combining them with the set surface evaluation pass coefficients. After the products that are out of appearance are numbered, the electrical excitation signal is triggered. The electrical excitation module is used to apply the rated electrical working excitation to the apparent qualified product after the electrical excitation signal is triggered, so that the apparent qualified product enters the normal working state; The functional testing module is used to acquire multimodal working response images of apparent qualified products under electrical excitation and perform functional quality testing analysis. Based on the results of the functional quality testing analysis, it selectively determines whether the apparent qualified products are functionally unqualified or functionally qualified products, and generates a functional distribution model of functionally qualified products. Multimodal working response images include infrared thermal imaging sequence images, high-resolution microscopic dynamic sequence images, high-speed industrial camera dynamic response sequence images, and electroluminescence imaging sequence images; The acquisition timing of all multimodal images is strictly synchronized with the application / disconnection timing of electrical excitation, with a timestamp alignment accuracy of ≤1ms, ensuring that the acquired response images are completely matched with the excitation state of the components; After comprehensive processing of the multimodal working response images, the thermal performance coefficient, deformation performance coefficient, response performance coefficient, and switching performance coefficient of the apparent qualified product are obtained. The steady-state average temperature, pin consistency deviation, thermal response deviation, and hot spot peak temperature of non-heating areas after removing pre-marked normal heating areas are extracted from infrared thermal imaging sequence images to form the thermal performance coefficient. The deformation peak and measured relative displacement relative to the excitation stage in the thermal steady-state stage are extracted from high-resolution microscopic dynamic sequence images to form the deformation functional coefficient. The response function coefficient is formed by extracting the motion response delay time from the application of excitation to the completion of the specified action by the movable part and the time taken for the contact to bounce from the first contact to meet the continuous and stable contact conditions from the dynamic response sequence image of the high-speed industrial camera. From the electroluminescence imaging sequence images, after the apparent qualified product enters the stable luminescence state, the uniformity of luminescence intensity of the effective luminescence area and the proportion of dark spot area of ​​the core functional area are extracted to form the switching function coefficient. The thermal performance coefficient, deformation performance coefficient, response performance coefficient, and switching performance coefficient are compared with their corresponding threshold coefficients. If any performance coefficient exceeds the corresponding allowable range, the product is judged as functionally unqualified. If all performance coefficients are within the corresponding allowable range, the product is judged as functionally qualified. If the product is functionally qualified, the ratios between the thermal performance coefficient, deformation performance coefficient, response performance coefficient, and switching performance coefficient and their corresponding threshold coefficients are calculated. Four sets of coefficient ratios were obtained; Starting from the origin of the Cartesian coordinate system, four polar coordinate rays with equal angular intervals are emitted; the length of each ray corresponds to a set of coefficient ratios, and the endpoints of adjacent rays are connected in sequence to form a closed quadrilateral shape, which serves as the functional distribution model.

2. The industrial product quality inspection system based on machine vision according to claim 1, characterized in that: The logic for obtaining the evaluation coefficient; Extract the sets of appearance defect parameters corresponding to the pin area and the package area from the multi-view appearance images of the electronic component to be inspected; The evaluation coefficient of the electronic component under test is obtained by comprehensively processing the dimensional deviation and surface condition of the pin area, the damage and defects of the package area, and the surface flatness.

3. The machine vision-based industrial product quality inspection system according to claim 2, characterized in that: Logic for obtaining the apparent defect parameter set of the pin area and package area; The measured dimensions of a single pin of an electronic component are compared with the nominal values ​​to calculate the absolute deviation rate. The dimensions include length, width, thickness, and pin spacing. The absolute deviation rates calculated for length, width, thickness, and pin spacing are weighted and calculated using logic to output the pin deviation degree. The highest value among the pin deviation degrees in each group is identified as the size deviation degree. Obtain the total area of ​​the oxide region of a single pin and calculate its proportion in the total area of ​​the single pin pad to obtain the oxide ratio; obtain the total area of ​​the plating damage of a single pin and calculate its proportion in the total area of ​​the single pin pad to obtain the plating damage ratio; use weighted calculation logic to calculate the oxide ratio and the plating damage ratio to output the pin status degree; identify the highest value in each group of pin status degrees as the surface status degree. The apparent defect parameter set of the packaging area includes the degree of damage and surface smoothness; The area of ​​various defects within the packaging area corresponding to electronic components is statistically analyzed and summed to obtain the total area of ​​packaging damage. Calculate the percentage of the total damaged area in the package area to obtain the damage defect level; Point cloud data of the electronic component package surface is acquired and fitted to a plane to obtain a reference plane; Calculate the height deviation of all points on the surface of the packaged area relative to the reference plane; identify the maximum positive deviation and the maximum negative deviation, and calculate the difference to obtain the surface flatness.

4. The industrial product quality inspection system based on machine vision according to claim 1, characterized in that: Functional quality testing and analysis logic; Multimodal working response images include infrared thermal imaging sequence images, high-resolution microscopic dynamic sequence images, high-speed industrial camera dynamic response sequence images, and electroluminescence imaging sequence images; After comprehensive processing of the multimodal working response images, the thermal performance coefficient, deformation performance coefficient, response performance coefficient, and switching performance coefficient of the apparent qualified product are obtained.

5. The machine vision-based industrial product quality inspection system according to claim 4, characterized in that: Logic for obtaining thermal performance coefficient; Based on the infrared thermal imaging sequence of the apparent qualified product under electrical excitation, the time points when the apparent qualified product reaches thermal steady state under rated electrical working excitation are marked; the overall temperature value is extracted from the infrared image after the marked time point, and the arithmetic mean is taken to obtain the steady-state average temperature of the apparent qualified product. In the thermal steady state stage, match and lock a single pin, extract the average temperature of the pad corresponding to each pin, and take the arithmetic mean of the average temperature of each pin as the pin average temperature. For each pin, calculate the ratio between the average temperature of the corresponding pad and the average temperature of the pin, and take the maximum ratio as the pin consistency deviation; Starting from the application of excitation, the temperature rise sequence is acquired at a fixed frame rate until thermal steady state is reached. The time difference between the start time of excitation application and the time of reaching thermal steady state is calculated to obtain the thermal response time. The absolute deviation between the thermal response time and the pre-stored standard thermal response time is calculated to obtain the thermal response deviation. During the thermal steady-state phase, the pre-marked normal heating areas are removed, and the remaining areas are marked as non-heating areas. The highest temperature value of the non-heating areas after the thermal steady-state phase is obtained as the hot spot peak temperature.

6. The industrial product quality inspection system based on machine vision according to claim 5, characterized in that: Logic for obtaining deformation function coefficients; Based on high-resolution microscopic dynamic sequence images of apparent qualified products under electrical excitation, the pin length is extracted from the high-resolution microscopic dynamic sequence images under thermal steady-state stage as the steady-state length. The deformation is calculated by the absolute difference between the steady-state length and the pin length before electrical excitation. The maximum value of the deformation among all pins is taken as the deformation peak. The initial position of the bond line of the apparent qualified product is obtained before electrical excitation. After electrical excitation, the maximum displacement of the arc apex of the bond line is calculated by feature point matching and used as the measured relative displacement.

7. The machine vision-based industrial product quality inspection system according to claim 6, characterized in that: Logic for obtaining response function coefficients; Based on the dynamic response sequence images of the apparent qualified product under electrical excitation from a high-speed industrial camera, the timing starts from the moment the excitation is applied and ends at the point in time when the movable part of the apparent qualified product completes the specified action, which is used as the action response delay time. Traverse the high electroluminescence imaging sequence images, identify the first image frame in which the contact point enters the contact state, and record its corresponding timestamp as the bounce start time. Starting from the bounce start time, traverse backwards to identify the starting frame that meets the preset bounce end condition, and record its corresponding timestamp as the bounce end time. Calculate the time difference between the bounce end time and the bounce start time to obtain the bounce duration.

8. The industrial product quality inspection system based on machine vision according to claim 7, characterized in that: Logic for obtaining switch function coefficients; Based on the high electroluminescence imaging sequence images of the apparent qualified products under electrical excitation, the effective light-emitting area, core functional area and fixed background area are pre-divided according to the standard drawings of electronic components. After the excitation is applied, wait for the apparent qualified product to enter the luminescence stable state, extract the average gray value and gray standard deviation of the effective luminescence area after the luminescence stable state, divide the gray standard deviation by the average gray value to obtain the luminescence intensity uniformity. After entering the stable luminescence state, the pixels in the core functional area are segmented by thresholding to identify all areas with gray values ​​lower than the preset gray value and mark them as dark spot areas; The percentage of dark spot area is obtained by summing the areas of all dark spot regions and dividing by the total area of ​​the core functional area.

9. The industrial product quality inspection system based on machine vision according to claim 4, characterized in that: Selectively determine whether an apparently qualified product is a functionally unqualified product or a functionally qualified product, and generate a functional distribution model of functionally qualified products; Extraction steps - The calculated thermal performance coefficient, deformation performance coefficient, response performance coefficient, and switching performance coefficient are input into a pre-edited coefficient threshold library for comparison. Based on the comparison results, products with unqualified functions and products with qualified functions are output. The coefficient threshold library stores the threshold coefficients corresponding to the thermal function coefficient, deformation function coefficient, response function coefficient, and switching function coefficient, respectively.

10. A machine vision-based industrial product quality inspection method, applied to any one of the machine vision-based industrial product quality inspection systems described in claims 1-9, characterized in that, include: Visual inspection of appearance: control the electronic components to be inspected to complete the feeding, and after pose calibration, acquire multi-view appearance images of the electronic components to be inspected. The appearance evaluation coefficient of the electronic components to be inspected is calculated based on the extracted set of appearance defect parameters. Product sorting and initial screening: The evaluation coefficient of the electronic components to be tested is compared with the preset evaluation pass coefficient to complete the classification and screening of apparent qualified products and apparent unqualified products. Appearance qualified products are uniquely numbered and bound, and electrical excitation signaling is triggered. Rated electrical excitation: After receiving the electrical excitation signal, the pre-stored excitation parameter library is called according to the model of the electronic component to be tested, and the matching rated electrical working excitation is applied to the apparent qualified product, so that the apparent qualified product enters the rated working state; Multimodal functional testing: Collect multimodal working response images of apparent qualified products under electrical excitation, extract multi-dimensional functional feature parameters based on the multimodal working response images, and calculate the functional coefficients of the corresponding dimensions. Qualification assessment and model building: The functional coefficients of each dimension are compared with the preset coefficient threshold library to determine whether the product is functionally qualified or not; for functionally qualified products, a functional distribution model is built based on the functional coefficients of each dimension, and the functional performance value is calculated. Test report generation: Summarize the test data from the entire process, fill it into a pre-built report template, and generate a product quality test report with a unique number.

Images