Image processing method and electronic equipment

Abstract

The invention relates to the technical field of computer vision, and provides an image processing method and electronic equipment, which are used for PCBA defect detection. For an electronic element in the PCBA single-side image, rough extraction of element corner points is realized through target detection, fine extraction of the element corner points is realized through clustering and Hough straight line detection, and screening of the element corner points is realized through at least one preset corner point condition, so that the extraction accuracy of the element corner points is effectively improved through a three-level corner point processing mode, and the extraction efficiency of the element corner points is improved. A deep learning algorithm is combined with a traditional geometric algorithm, the robustness of element corner extraction is improved, in this way, when corner matching is conducted through accurate and robust element corners, the precision and reliability of image alignment can be improved, then the precision of subsequent electronic element detection is improved, and in the corner matching process, the accuracy and reliability of element corner matching are improved. On the basis of the pin-type element corner pairs, matching of element corners of patch-type electronic elements is carried out, and the precision and reliability of image alignment are further improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and provides an image processing method and an electronic device. Background Technology

[0002] In the industrial production of printed circuit board assemblies (PCBAs), in order to ensure product quality, more than a dozen manual inspections are conducted using manual instruments. However, manual inspection is not only inefficient and costly, but also prone to missed or over-inspected items due to its strong subjectivity and inconsistent standards. This makes it difficult to meet the strict requirements of modern manufacturing for quality consistency and traceability, and greatly affects production efficiency and capacity improvement.

[0003] With the development of computer vision technology, Automated Optical Inspection (AOI) has enabled the automated detection of defects in electronic components on PCBAs. During the AOI process, accurately aligning the chip image with the template is crucial for ensuring correct assembly and avoiding misjudgments. However, the positioning features of the front and back images of a PCBA are highly similar, easily leading to template mismatch. Furthermore, traditional image feature extraction algorithms (such as corner and spot detection) rely on the apparent saliency of the image. In actual production lines, when dealing with complex interferences such as electronic component occlusion, foreign object adhesion, surface contamination, and process defects, feature extraction is impaired, resulting in missing, disordered, or noisy feature points, severely reducing the accuracy and robustness of feature matching. Incorrect feature matching can lead to image alignment inaccuracies, which in turn affect the inspection accuracy of the PCBA and ultimately product quality.

[0004] Therefore, improving the alignment accuracy of chip images is of great significance for PCBA production. Summary of the Invention

[0005] This application provides an image processing method and electronic device for improving image alignment accuracy, thereby improving image detection accuracy.

[0006] In a first aspect, embodiments of this application provide an image processing method, including: Target detection is performed on a single-sided image of a printed circuit board assembly to obtain each electronic component in the single-sided image and the component category of each electronic component, wherein the component category includes pin-type and surface-mount-type; Based on the layout information of each electronic component, the electronic components are clustered to obtain multiple sets of initial component corner points for each electronic component. For each set of initial component corner points, the positions of multiple component corner points in the initial set of component corner points are fitted with straight lines to obtain the corresponding set of target component corner points. From each set of target component corner points, remove component corner points that do not meet at least one preset corner point condition; For each set of target component corner points, in the single-sided image and the template image corresponding to the single-sided image, find the first component corner point pair corresponding to the pin-type electronic component, and in the preset search range of each pair of first component corner point pairs, find the second component corner point pair corresponding to the patch-type electronic component. Based on the found first element corner point pairs and second element corner point pairs, align the single-sided image and the template image.

[0007] The beneficial effects of the above technical solution are as follows: Target detection is performed on a single-sided image of a printed circuit board assembly to obtain each electronic component. This allows for coarse extraction of the electronic component corner points. Clustering is then performed according to the layout information of each electronic component to obtain multiple initial sets of component corner points reflecting the overall layout. Linear fitting is then performed on each initial set of corner points to achieve fine extraction of the electronic component corner points. Furthermore, at least one preset corner point condition is used to filter the electronic component corner points, reducing noise interference from occlusion, dirt, and foreign objects. This three-level corner point processing method—coarse extraction, fine extraction, and filtering—effectively improves the accuracy of electronic component corner point extraction. The coarse and fine extraction processes combine deep learning target detection algorithms with traditional geometric algorithms, effectively improving the robustness of electronic component corner point extraction. Thus, when performing corner point matching with accurate and robust component corner points, the accuracy and reliability of image alignment can be improved, thereby enhancing the accuracy of subsequent electronic component detection.

[0008] On the other hand, in the corner matching process, the corners of electronic components with the component category of pin are matched first, and then the corners of surface mount electronic components are matched based on the corner pairs of pin components, thereby improving the accuracy of corner matching and further improving the precision and reliability of image alignment.

[0009] Optionally, the layout information includes at least the center point coordinates, angle, and size of the electronic components. The step of clustering the electronic components according to their layout information to obtain multiple initial sets of component corner points for each electronic component includes: For each electronic component, the horizontal and vertical directions of the electronic component are determined based on the coordinates and angle of its center point. Within the preset clustering ranges in the horizontal and vertical directions of the electronic components, electronic components that meet the preset size conditions are searched according to their dimensions. Electronic components that meet the preset size conditions are grouped into a set of electronic components, and the corner points of the group of electronic components are extracted to obtain an initial set of component corner points.

[0010] The beneficial effects of the above technical solution are as follows: by using the layout information of each electronic component on the printed circuit board assembly, clustering can be performed in both horizontal and vertical directions to obtain the set of component corner points of electronic components in a row or column, providing a data basis for subsequent line fitting to extract the fine corner points of electronic components.

[0011] Optionally, when no electronic component meeting the preset size requirements is found, the method further includes: Obtain the component category of the electronic component; When the component category is the surface mount type, delete the electronic component; When the component category is the pin category, the component corner points of the electronic component are used as an initial set of component corner points.

[0012] The beneficial effects of the above technical solution are as follows: considering that the size of surface mount electronic components on printed circuit assemblies is small and the independent distribution is rare, when no electronic component that meets the preset size condition is found, the electronic component is an outlier and can be deleted. On the other hand, the size of pin-type electronic components is larger and the independent distribution is more common, so the electronic components can be retained. In this way, by targeting outlier electronic components according to their component categories, it helps to improve the extraction accuracy of corner points of electronic components.

[0013] Optionally, for each initial set of component corner points, performing linear fitting on the positions of multiple component corner points in the initial set of component corner points to obtain the corresponding target set of component corner points includes: Perform Hough line detection on the corner points of the initial component corner point set to obtain the line corresponding to the initial component corner point set; Calculate the distances from the corner points of each of the multiple components to the straight line, and determine a maximum average distance based on the multiple distances; When the maximum average distance is greater than a preset distance threshold, calculate the projection points of a batch of component corner points associated with the maximum average distance on the straight line, and replace the corresponding component corner points with the projection points to obtain the target component corner point set; When the average maximum distance is not greater than the preset distance threshold, the multiple component corner points are directly used as the target component corner point set.

[0014] The beneficial effects of the above technical solution are as follows: by performing Hough line detection on the corner points of electronic components, fine extraction of component corner points can be achieved. During the fine extraction process, the corner points of electronic components are corrected by projection points to ensure the number of subsequent matching point pairs and improve the accuracy of image alignment.

[0015] Optionally, removing component corner points from each set of target component corner points that do not meet at least one preset corner point condition includes: For each corner point in each set of target component corner points, perform at least one of the following filtering operations: Calculate the area of ​​the outer rectangle of the electronic component corresponding to the corner point of the component. When the area of ​​the outer rectangle is lower than a preset area threshold, the corner point of the component is removed. Obtain the confidence level of the detection frame of the electronic component corresponding to the corner point of the component; when the confidence level is lower than a preset confidence threshold, the corner point of the component is removed. When the electronic component corresponding to the corner point of the component has a defect, the corner point of the component is removed. Calculate the average distance from the corner point of the component to its k nearest neighboring corner points. When the difference between the average distance and the global average distance is greater than a preset outlier distance, the corner point of the component is removed.

[0016] The beneficial effects of the above technical solution are as follows: When the size of the electronic component is small, the corner points of the component are easily affected by imaging and noise, and should be removed. When the confidence level of the detection frame of the electronic component is low, there may be foreign objects such as cotton wool or optical fibers or surface dirt near the electronic component, which have a significant impact on the corner points of the component and should be removed. When the average distance between the electronic component and its neighboring electronic components deviates from the global average level, the electronic component is an outlier in spatial distribution and should be deleted. By screening the corner points of electronic components from multiple levels such as size, confidence level, and spatial distribution, the accuracy of electronic component corner point extraction is effectively improved, and the impact of inaccurate feature point matching on image alignment is reduced.

[0017] Optionally, for each set of target component corner points, finding the first component corner point pair corresponding to the pin-type electronic component in the single-sided image and the template image corresponding to the single-sided image includes: The component corner points of the electronic components of the pin type in each set of target component corner points are taken as the first component corner points, and the component corner points of the electronic components of the pin type in the template image corresponding to the single-sided image are taken as the second component corner points. For each corner point of the first element, a first similarity is calculated between the electronic components of the pin class based on the layout information of the first element corner point and the layout information of each second element corner point. The second element corner point corresponding to the largest first similarity and the first element corner point are taken as a first element corner point pair of the electronic components of the pin class.

[0018] The beneficial effects of the above technical solution are as follows: Since the electronic components with pins on the printed circuit board assembly are large in size, few in number, and highly distinguishable, the corner points of the electronic components with pins can be matched first to ensure matching accuracy.

[0019] Optionally, the step of searching for the second component corner pair corresponding to the patch-type electronic component within a preset search range of each pair of the first component corner points includes: For each pair of first component corner points, the component corner points of the patch-type electronic components within a preset search range centered on the first component corner point in the first component corner point pair are taken as third component corner points, and the component corner points of the patch-type electronic components within a preset search range centered on the second component corner point in the first component corner point pair are taken as fourth component corner points. Calculate a first spatial parameter between each corner point of the third element and the corner point of the first element, and calculate a second spatial parameter between each corner point of the fourth element and the corner point of the second element; For each of the third element corner points, a second similarity between the patch-type electronic components is calculated based on the layout information of the third element corner point and the layout information of each of the fourth element corner points. Combining the corresponding first spatial parameters and each of the second spatial parameters, a fourth element corner point and a third element corner point are selected as a second element corner point pair for the patch-type electronic components.

[0020] The beneficial effects of the above technical solution are as follows: within a preset search range centered on the component corner of the matched pin-type electronic component, the component corner of the small, numerous, and low-discrimination patch-type electronic component is matched. Compared with global matching, this helps to improve the matching accuracy and thus improve the image alignment accuracy.

[0021] Optionally, aligning the single-sided image and the template image based on the found first element corner point pairs and second element corner point pairs includes: Based on the matching point pairs in each first element corner point pair and each second element corner point pair, a random sampling consistency algorithm is used to calculate the transformation matrix between the single-sided image and the template image. The single-sided image is transformed into the coordinate system of the template image using a transformation matrix, and pixel interpolation is performed on the default area in the transformed single-sided image.

[0022] The beneficial effects of the above technical solution are: by calculating the transformation matrix through accurate and robust matching point pairs, the single-sided image can be accurately transformed into the coordinate system of the template image, thereby achieving alignment with the template image.

[0023] Optionally, after aligning the single-sided image and the template image, the method further includes: Based on the local coordinate information of each electronic component in the template image, a single component image is cropped from the single-sided image, and a standard component image is cropped from the template image; Based on the comparison between each individual component image and the corresponding standard component image, the defect detection result of the individual component image is obtained.

[0024] The beneficial effects of the above technical solution are as follows: after image alignment, individual component images can be accurately cropped out, so that defect detection can be performed on a unit of individual component images, reducing interference from other image areas and improving detection accuracy.

[0025] Secondly, embodiments of this application provide an electronic device, including a processor, a memory, and a communication interface, wherein the communication interface, the memory, and the processor are connected via a bus; The communication interface is used for sending and receiving data; The memory stores a computer program, and the processor executes the steps of the above-described image processing method according to the computer program.

[0026] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions that, when executed, can implement the steps of the above-described image processing method.

[0027] The technical effects of any of the implementation methods in the second to third aspects can be found in the technical effects of the corresponding implementation methods in the first aspect, and will not be repeated here. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of noise during the PCBA manufacturing process provided in an embodiment of this application; Figure 2 This is a schematic diagram of image misalignment provided in an embodiment of this application; Figure 3 A schematic diagram illustrating the method of acquiring PCBA images provided in an embodiment of this application; Figure 4A schematic diagram of surface-mount and pin-type electronic components on a PCBA provided in this application embodiment; Figure 5 This is a schematic diagram illustrating the method for creating a template image for a PCBA provided in this application embodiment; Figure 6 A flowchart of the image processing method provided in the embodiments of this application; Figure 7 This is a schematic diagram of the target detection effect provided in the embodiments of this application; Figure 8 A flowchart illustrating the clustering of electronic components provided in the embodiments of this application; Figure 9 A schematic diagram of the corner extraction results of electronic components provided in the embodiments of this application; Figure 10 A schematic diagram of the Hough line detection principle provided in the embodiments of this application; Figure 11 The Hough line detection effect diagram provided in the embodiment of this application; Figure 12A and Figure 12B This is a diagram illustrating the corner extraction effect provided in an embodiment of this application. Figure 12C A flowchart illustrating the three-level corner extraction method provided in this application embodiment; Figure 13 A flowchart of corner matching provided for embodiments of this application; Figure 14 A schematic diagram of a pin-type electronic component provided in an embodiment of this application; Figure 15 This is a schematic diagram illustrating the alignment of a single-sided image with a template image. Figure 16 This is a schematic diagram of the PCBA defect detection process; Figure 17 This is a schematic diagram of defects in electronic components; Figure 18 This is a schematic diagram of the defect review interface for electronic components. Figure 19 This is a system architecture diagram for PCBA testing; Figure 20 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0030] Based on the exemplary embodiments shown in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can constitute a complete technical solution on its own.

[0031] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0032] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related products or entities and do not necessarily imply a specific order or sequence, unless otherwise indicated. It should be understood that such terms can be used interchangeably where appropriate, for example, in situations where implementation can proceed in a sequence other than those given in the embodiments illustrated or described in this application.

[0033] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a series of components is not necessarily limited to those that are explicitly listed, but may include other components that are not explicitly listed or that are inherent to such product or device.

[0034] In AOI equipment for PCB inspection, the PCBA image to be inspected needs to be aligned with the correct template image. Then, according to the coordinates of the standard components in the template image, the electronic components in the image to be inspected are cropped one by one to obtain a component image containing only the electronic components, and the individual component images are inspected.

[0035] However, the features of the localization regions in the front and back images of PCAB often exhibit high similarity or symmetrical geometric features, which can easily lead to mismatches between the image to be detected and the wrong template image. For example, the image that should match the front template image might match the back template image instead. Furthermore, while traditional image feature extraction algorithms (such as those focusing on the detection of corners, edge intersections, or spots) can effectively identify ideal features with significant uniqueness and high contrast, their reliance on image appearance information limits their ability to handle complex conditions commonly found in production line environments, such as electronic component obstruction, foreign matter such as lint or optical fibers, surface dirt, and various process defects. These limitations severely interfere with or damage the stability and repeatability of the features, resulting in incomplete, inconsistent, or noisy feature point sets extracted by traditional image feature extraction methods. Figure 1 As shown, this results in poor performance in real-world industrial environments, ultimately leading to a sharp decline in matching reliability. Furthermore, incorrect feature matching can cause the image to be detected to fail to align with the correct template image, such as... Figure 2 As shown, this affects the detection accuracy of electronic components, and consequently, product quality.

[0036] In view of this, this application provides an image processing method that utilizes the overall layout information of electronic components on a PCBA to propose a three-level processing approach for electronic components (component corner points): "coarse extraction - fine extraction - screening". It combines deep learning object detection with traditional geometric algorithms (such as Hough transform, cluster analysis, etc.) to improve the robustness of feature extraction. Furthermore, during the screening process, a multi-rule corner point screening mechanism based on spatial distribution, confidence level, and size is introduced to effectively resist noise interference such as occlusion, dirt, and foreign objects, thereby improving the accuracy of feature extraction. In this way, when performing corner point matching with accurate and robust component images, the accuracy and reliability of image alignment can be improved, thereby improving the accuracy of subsequent electronic component detection. Furthermore, during feature matching, priority is given to matching the corner points of pin-type electronic components with high confidence. Then, using the matched corner points of pin-type electronic components as base points, automatic matching of the corner points of surface-mount electronic components is carried out. Under complex working conditions (such as occlusion, dirt, similar background, etc.), the accuracy of matching point extraction is improved, the false matching rate is reduced, and the alignment algorithm accuracy and robustness are significantly improved. This, in turn, improves the automation and reliability of electronic component inspection in PCBA, reduces the reliance on manual re-inspection, and improves production line inspection efficiency.

[0037] When AOI equipment inspects PCBAs on the production line, a camera array is usually set up above the PCBA to capture images of the front and back of the PCBA from multiple perspectives, so as to obtain a complete picture of the electronic components on the front and back of the PCBA.

[0038] Taking a camera array containing two cameras as an example, such as Figure 3As shown, the two cameras capture images of the front and back of the PCBA from viewpoint 1 and viewpoint 2, respectively.

[0039] In some embodiments, the cameras in the camera array can be ordinary RGB cameras or high-precision industrial cameras.

[0040] In some embodiments, when a single camera can capture a full view of the electronic components on both the front and back of the PCBA (e.g., when the PCBA is small), only one camera may be used for taking pictures.

[0041] Typically, electronic components on a PCBA are categorized into surface mount and leaded components. Surface mount electronic components are characterized by their elongated shape, metal plating at both ends, similar size, and small size, including but not limited to capacitors, resistors, inductors, and diodes. Leaded electronic components are characterized by their predominantly black or metallic surface, near-square shape, and generally larger size, including but not limited to integrated circuits.

[0042] like Figure 4 The diagram shows schematics of surface mount and pin-type electronic components.

[0043] Typically, before each PCBA model is rejected, a corresponding template image library is created for subsequent processing of single-sided images captured by cameras from each viewpoint. The creation of this template image library requires functional support from the testing platform.

[0044] In some embodiments, an AI model can be used to create a template image library. The AI ​​model can quickly segment or detect all electronic components on the entire PCB, as well as the component categories and special devices of the electronic components (which can be expanded according to special needs). Then, the standard component images and component categories of all electronic components are verified and corrected by humans, and the standard component images are cropped to complete the creation of a template image library for a certain model of PCBA.

[0045] like Figure 5 The diagram illustrates the process of creating a template image library. First, the PCBA model number is loaded, and a camera array is used to capture the entire PCBA from the front and back. Then, alignment marks are set on each captured image. An AI model is used to extract standard component images of electronic components from each image. Finally, the cropped standard component images are labeled with component categories, thus completing the creation of the template image library.

[0046] Based on the image template library, the flow of the image processing method provided in this application embodiment is as follows: Figure 6 As shown, it mainly includes the following steps: S601: Perform target detection on a single-sided image of a printed circuit board assembly to obtain each electronic component in the single-sided image and the component category of each electronic component.

[0047] When the test starts, the system will first initialize, automatically match the preset parameters according to the input PCBA model, complete the algorithm configuration (such as thresholds), and load the template image library of the corresponding model (containing standard component images, component categories, sizes and other information of electronic components) to ensure that the test benchmark is consistent.

[0048] After initialization, target detection can be performed on the single-sided image of the PCBA to obtain the detection box, confidence level, and component category for each electronic component in the single-sided image, thereby detecting each electronic component contained in the single-sided image. The component categories include pin-type and surface-mount types, as described in the aforementioned embodiments, and will not be repeated here.

[0049] In some embodiments, the single-sided image can be a PCBA image acquired in real time or an offline PCBA image. The single-sided image can be a PCBA image that includes the overall view of the electronic components or a PCBA image that includes some electronic components. The single-sided image can be a back view of the PCBA or a front view of the PCBA.

[0050] like Figure 7 The diagram shows the target detection effect. The detection frame for pin-type electronic components is represented by a thick dashed line, while the detection frame for surface-mount electronic components is represented by a thick solid line. The corner points of the detection frame are the component corner points, thus achieving coarse extraction of component corner points.

[0051] In some embodiments, during the target detection process, in addition to outputting the detection box and component category, some typical component defects, such as breakage, missing parts, or dirt, can also be identified.

[0052] In some embodiments, object detection may employ deep learning algorithms, including but not limited to bounding box rotation, keypoint detection, and other algorithms such as Faster R-CNN series, YOLO series, and EfficientDet series. These algorithms are trained using labeled component images.

[0053] It should be noted that the system initialization process supports dynamic adjustment of parameters to adapt to the needs of different production batches.

[0054] In real-world detection scenarios, the corner points of the detection bounding box may suffer from noise, blurring, occlusion, and deformation, leading to inaccurate positions and numbers of the extracted corner points. To ensure image alignment accuracy, the corner points can be further refined.

[0055] S602: Based on the layout information of each electronic component, cluster each electronic component to obtain the initial set of component corner points for each of the multiple electronic components.

[0056] Target detection can obtain the center point coordinates, width, height, and angle of the detection frame for each electronic component. The center point coordinates of the detection frame are the center point coordinates of the electronic component, the width and height of the detection frame are the dimensions of the electronic component, and the angle of the detection frame is the angle of the electronic component. The center point coordinates, dimensions, and angle of the electronic component can reflect the layout information of the electronic component on the PCB. Using the layout information of each electronic component on the PCB, cluster analysis is performed on the position of each electronic component to achieve grouping of each electronic component. During the clustering process, outliers are removed to complete the initial refinement of the corner points of each group of electronic components.

[0057] In some embodiments, the clustering process of the electronic components is as follows: Figure 8 As shown, it mainly includes the following steps: S6021: For each electronic component, determine the horizontal and vertical directions of the electronic component based on the coordinates and angle of the electronic component's center point.

[0058] In cluster analysis, using the center point of an electronic component to represent it can simplify the computational workload of clustering.

[0059] S6022: Within the preset clustering ranges in the horizontal and vertical directions of the electronic components, find electronic components that meet the preset size conditions according to their dimensions.

[0060] Typically, electronic components of similar size may be neatly arranged in rows or columns on a PCB. Therefore, cluster analysis can be performed in both horizontal and vertical directions. The preset clustering range can be set according to the actual PCBA model or based on experience; this application does not impose any limiting requirements.

[0061] In some embodiments, cluster analysis can be performed by following a set step size. The step size can be set empirically or according to the spacing between electronic components.

[0062] S6023: Determine whether an electronic component that meets the preset size conditions has been found. If found, proceed to S6024; otherwise, proceed to S6025.

[0063] S6024: Based on electronic components that meet the preset size conditions, a group of electronic components is formed, and the corner points of the group of electronic components are extracted to obtain an initial set of component corner points.

[0064] When multiple electronic components of similar size are found in the same direction (horizontal or vertical), these electronic components can be grouped together. Based on the detection frame of each group of electronic components, the initial set of component corner points of that group of electronic components can be obtained.

[0065] S6025: Determine the component category of the electronic component. If the component category is surface mount, execute S6026. If the component category is pinned, execute S6027.

[0066] When no electronic components of similar size are found in the same direction, it indicates that the electronic component is an outlier. Different types of electronic components have different characteristics. For example, large-sized integrated circuit chips may be distributed in small quantities on the PCB, while small-sized resistors may be distributed in large quantities and in a concentrated manner on the PCB. Therefore, for outlier electronic components, the extraction of component corner points can be refined according to component category.

[0067] S6026: Remove electronic components.

[0068] Surface mount electronic components are rarely distributed independently, so they can be directly removed by eliminating their corner points.

[0069] S6027: Use the corner points of electronic components as a set of initial component images.

[0070] Since many electronic components with pins are distributed independently, we can retain the electronic component and treat its corner points as a separate initial set of corner points.

[0071] In some embodiments, the center point of the detection frame can be used as the corner point of the electronic component, or the four vertices of the detection frame can be used as the corner points of the electronic component.

[0072] Taking the corner of the component as the center point of the detection frame as an example, such as Figure 9 The image shows a schematic diagram of the extraction results for the corner points of the component.

[0073] In PCBA inspection scenarios, the layout information of electronic components on the printed circuit board assembly is used to cluster components in both horizontal and vertical directions, resulting in rows or columns of electronic component groups. Based on the corner points of the inspection frame for each group of electronic components, an initial set of component corner points can be obtained, providing a data foundation for subsequent fine-grained corner point localization through line fitting. Furthermore, during clustering, considering that surface-mount electronic components on the printed circuit board assembly are small in size and less likely to be independently distributed, when no electronic component meeting the preset size condition is found, it is considered an outlier and can be deleted. Conversely, pin-type electronic components are larger in size and more likely to be independently distributed, and can be retained. This targeted handling of outliers based on component category helps improve the accuracy of electronic component corner point extraction.

[0074] In some embodiments, when object detection uses a rotating frame, the angle of the electronic component can be an orientation angle; when object detection uses a key point or a rotating frame, the angle of the electronic component can be a rotation angle.

[0075] Based on the clustering results, electronic components of similar size and proximity will be on the same straight line. However, the position of the electronic components in the coarse localization of the target detection will have a certain deviation from the actual position, so fine localization is required.

[0076] S603: For each set of initial component corner points, perform linear fitting on the positions of multiple component corner points in the initial set of component corner points to obtain the corresponding target component corner point set.

[0077] By performing Hough line detection on the component corner points in the initial component corner point set for each group of electronic components, minor deviations introduced by component package drawing, coordinate extraction, or previous image processing can be eliminated, ensuring that the theoretical center position of the electronic component strictly conforms to the detected alignment structure, obtaining more accurate component corner point positions, and completing the refined extraction of component corner points.

[0078] Hough line detection is mainly used to detect straight lines in images. Its principle is as follows: Figure 10 As shown, the equation of the straight line in the image is transformed into a point (polar radius) in Hough space (polar coordinate space). polar angle The formula is as follows: The peak value of the votes is found in the Hough space, and the peak value is transformed back into the image space to determine the straight line in the image. Formula 1 in, The polar angle of the line. It is the polar radius of the straight line.

[0079] After performing Hough line detection on each set of component corner points, a straight line that effectively fits each set of electronic components can be obtained. Then, the distances from multiple component corner points in each set of component corner points to the straight line are calculated, and a maximum average distance is determined based on multiple distances. The maximum average distance is compared with a preset distance threshold. When the maximum average distance is greater than the preset distance threshold, it indicates that the batch of component corner points that generated the maximum average distance are outliers that need to be corrected in position. Therefore, the projection points of the batch of component corner points associated with the maximum average distance on the straight line are calculated, and these projection points are used to replace the corresponding component corner points to obtain the target set of component corner points. When the maximum average distance is not greater than the preset distance threshold, it indicates that the overall alignment accuracy of this initial set of component corner points has been met, and no position correction is required. Multiple component corner points in this initial set of component corner points can be directly used as the target set of component corner points.

[0080] Taking the corner of the component as the center point of the detection frame of the electronic component as an example, such as Figure 11 The figure shows a schematic diagram of linear fitting at the corner points of the component.

[0081] In PCBA inspection scenarios, Hough line detection is performed on the corners of electronic components to achieve fine extraction of component corners. During the fine extraction process, the corners of electronic components are corrected by projection points to ensure the number of subsequent matching point pairs and improve the accuracy of image alignment.

[0082] S604: Remove the corner points of components that do not meet at least one preset corner point condition from each set of target component corner points.

[0083] To obtain a highly reliable set of component corner points, multiple rules can be used to filter and optimize each set of target component corner points. This aims to eliminate low-quality feature points caused by noise, foreign objects, and defects in the component itself, while ensuring the integrity of the spatial distribution of the retained feature points.

[0084] In some embodiments, for each component corner point in each set of target component corner points, at least one of the following filtering operations is performed: Calculate the area of ​​the circumscribed rectangle of the electronic component corresponding to the corner point. When the area of ​​the circumscribed rectangle is lower than the preset area threshold, it indicates that the size of the electronic component is small and the corner point is easily affected by imaging and noise. Therefore, the corner point is removed.

[0085] Obtain the confidence level of the detection frame of the electronic component corresponding to the corner point of the component. When the confidence level is lower than the preset confidence threshold, it indicates that the electronic component is affected by random foreign objects such as cotton wool and optical fibers, as well as dirt on the surface of the component. Therefore, the corner point of the component is removed.

[0086] When the electronic component corresponding to the corner point has a defect, the corner point is removed to suppress the interference of external random noise.

[0087] Calculate the average distance from a component corner point to its k nearest neighbor corner points. If the difference between the average distance and the global average distance is greater than the preset outlier distance, the component corner point is removed.

[0088] Clustering and Hough line detection can reveal the spatial distribution characteristics of electronic components. A distance-based outlier detection algorithm can be used to calculate the average distance from a component corner to its k nearest neighbor corners. When the average distance differs significantly from the global average, the component corner can be identified as a spatial outlier lacking nearby references, and its coordinate reliability is low; therefore, it should be deleted. Here, k can be set empirically.

[0089] In PCBA inspection scenarios, by screening electronic component corner points from multiple levels such as size, confidence level, and spatial distribution, the accuracy of electronic component corner point extraction is effectively improved, and the impact of inaccurate feature point matching on image alignment is reduced.

[0090] It should be noted that the embodiments of this application do not impose restrictive requirements on the order of the screening process for component corners. For example, during the target detection process, detection frames with low confidence can be directly proposed to eliminate the corners of electronic components that cause interference.

[0091] After the above filtering operations, multiple sets of high-confidence component corner points can be obtained.

[0092] Taking the corner point of the component as the center point of the electronic component as an example, the set of target component corner points after screening is as follows: Figure 12A As shown, taking the corner points of electronic components as vertices as an example, the filtered set of target component corner points is as follows: Figure 12B As shown, although using vertices as corner points increases the computational load compared to using the center point as the corner point, it results in a greater number of corner points, which is beneficial for improving alignment accuracy.

[0093] like Figure 12C The diagram shows the three-level processing method for corner extraction of electronic components, which mainly includes the following steps: S1: Obtain a single-sided image of the PCBA.

[0094] S2: A deep learning-based object detection algorithm extracts each electronic component and its component category from a single-sided image.

[0095] S3: Extract the layout information of each electronic component and cluster the electronic components in both horizontal and vertical directions.

[0096] S4: Determine whether clustering was successful. If it fails, proceed to S5; if it succeeds, proceed to S6.

[0097] S5: Remove outlier electronic components.

[0098] S6: For each group of electronic components, the center point of the electronic component is used as the corner point for Hough line detection.

[0099] S7: Calculate the distance from the electronic component within the class to the line.

[0100] S8: Select the maximum average distance.

[0101] The maximum average distance is the average distance from the corner point of the electronic component that is furthest from the straight line in each group of electronic components to the straight line, and the average distance from the corner points of other components in the preset neighborhood to the straight line.

[0102] S9: Determine whether the average maximum distance is greater than the preset distance threshold. If yes, execute S10; otherwise, execute S11.

[0103] S10: Project the corner points of a batch of electronic components associated with the maximum distance mean onto a straight line, and replace the corresponding corner points with the projected points.

[0104] S11: Obtain the initial set of component corner points for each group of electronic components.

[0105] S12: For each initial set of component corner points, remove component corner points whose size is lower than a preset area threshold, whose confidence level is lower than a preset confidence threshold, outliers, and those with defects, to obtain the target set of component corner points.

[0106] After obtaining the target component corner point set for each group of electronic components, corner point matching can be performed based on the target component corner point set.

[0107] S605: For the component corner points in each set of target component corner points, find the first component corner point pair corresponding to the pin-type electronic component in the single-sided image and the template image corresponding to the single-sided image, and find the second component corner point pair corresponding to the surface mount electronic component within the preset search range of each pair of first component corner point pairs.

[0108] After obtaining accurate component corner points, corner matching can be performed. For each component corner point in the set of target component corner points extracted from the single-sided image, the corresponding component corner point is found in the template image. Traditional corner matching algorithms (such as SIFT, SURF, etc.) calculate floating-point descriptors and use Euclidean distance or Hamming distance to find the nearest corner point pair. These algorithms suffer from high feature point computation complexity and inability to overcome interference from repeated textures, resulting in a high matching failure rate and affecting the accuracy of image alignment.

[0109] To address the problems of traditional corner matching, this application provides a new corner matching method. Based on the component category of the electronic component output by the target detection, the corners of the larger pin-type electronic components are matched first. Then, with the matched pin-type corner pairs as a reference, the corner matching of the smaller surface-mount electronic components is expanded.

[0110] like Figure 13 The diagram shown is a flowchart of corner matching, which mainly includes the following steps: S6051: Take the component corner points of the pin-type electronic components in each set of target component corner points as the first component corner points, and take the component corner points of the pin-type electronic components in the template image corresponding to the single-sided image as the second component corner points.

[0111] like Figure 14 The diagram shows a pin-type electronic component, represented by a thick dashed line. As can be seen from the diagram, pin-type electronic components are larger in size, fewer in number, and more distinctive, making them a preferred choice for matching.

[0112] S6052: For each first element corner point, calculate the first similarity between pin-type electronic components based on the layout information of the first element corner point and the layout information of each second element corner point, and take the second element corner point corresponding to the largest first similarity and the first element corner point as a first element corner point pair of pin-type electronic components.

[0113] When matching the corner points of pin-type electronic components, the center point position and geometric features (such as width, height, and aspect ratio) of the electronic component are directly used to find the corner points of the pin-type electronic components with the highest similarity in the template image for the corner points of the pin-type electronic components in the single-sided image. The first similarity can be calculated by combining multiple attributes (such as relative position offset and geometric size difference).

[0114] Generally, the number of pin-type electronic components on a PCBA is small, so exhaustive search or nearest neighbor methods can be used to obtain the first component corner point pair that matches the pin-type electronic components.

[0115] Since pin-type electronic components on PCBAs are relatively large, few in number, and highly distinguishable, the corner points of pin-type electronic components can be prioritized for matching to ensure accuracy. Furthermore, the positions of the first and second component corner points matched with the first component corner point of the pin-type electronic component can be used to search for and match the corner points of surface-mount electronic components.

[0116] S6053: For each pair of first component corner points, the component corner points of surface mount electronic components within a preset search range centered on the first component corner point in the first component corner point pair are taken as third component corner points, and the component corner points of surface mount electronic components within a preset search range centered on the second component corner point in the first component corner point pair are taken as fourth component corner points.

[0117] In the first component corner pair, the patch-type electronic component in the single-sided image within a preset search range centered on the first component corner is designated as the third electronic component, and the patch-type electronic component in the template image within a preset search range centered on the second component corner is designated as the fourth electronic component.

[0118] S6054: Calculate the first spatial parameter between each third element corner point and the first element corner point, and calculate the second spatial parameter between each fourth element corner point and the second element corner point.

[0119] The spatial parameters include the relative positional relationship and distance information between the corner points of patch-type electronic components and the corner points of pin-type electronic components in the same image within the preset search range.

[0120] S6055: For each third element corner point, calculate the second similarity between surface mount electronic components based on the layout information of the third element corner point and the layout information of each fourth element corner point. Combine the corresponding first spatial parameters and each second spatial parameter, select a fourth element corner point and a third element corner point as a second element corner point pair for the surface mount electronic components.

[0121] When matching corner points of surface-mount electronic components, in addition to using spatial parameters between the corner points of the two types of components, the inherent properties of the surface-mount electronic components are also utilized. For the third corner point of a surface-mount electronic component in a single-sided image, the corresponding fourth corner point of the surface-mount electronic component is searched in the template image. The second similarity is calculated by combining spatial parameters and the inherent properties of the surface-mount electronic components.

[0122] Similarly, since the number of surface mount electronic components within the preset search range is relatively small, exhaustive search or nearest neighbor methods can be used to obtain the second component corner point pair that matches the surface mount electronic components.

[0123] In some embodiments, when matching the corner points of surface mount electronic components, the search range can be gradually expanded until all the corner points of surface mount electronic components in each set of target component corner points have been traversed.

[0124] For example, first match the corner points of surface mount electronic components within 10cm of the center point of the pin-type electronic components, and then match the corner points of surface mount electronic components within 10cm to 20cm of the center point of the pin-type electronic components.

[0125] The search scope for each match can be set according to actual needs, and this application embodiment does not impose any limiting requirements.

[0126] During the corner matching process, the corners of the matched pin-type electronic components are used as the center to match the corners of the small, numerous, and low-distinction patch-type electronic components within the preset search range. Compared with global matching, this helps to improve the matching accuracy of the corners of patch-type components, thereby improving the image alignment accuracy.

[0127] S606: Align the single-sided image and the template image based on the found first element corner point pairs and second element corner point pairs.

[0128] By using precisely matched first and second element corner point pairs, a random sampling consensus algorithm can be employed to resist interference from incorrectly matched point pairs (outer points) and calculate an optimal transformation matrix.

[0129] In practice, based on the matching point pairs in each first element corner point pair and each second element corner point pair, a minimum sample set (i.e., 4 element corner point pairs) is randomly selected. An initial transformation matrix is ​​calculated using these 4 point pairs. All element corner point pairs are tested using this transformation matrix. If the position error of an element corner point pair after transformation is less than a set threshold, it is marked as an "interior point"; otherwise, it is marked as an "outer point." The above steps are repeated, and the transformation matrix with the most interior points is selected as the best model, i.e., the optimal transformation matrix. Furthermore, using the final transformation matrix, the single-sided image is transformed into the coordinate system of the template image, and pixel interpolation processing is performed on the default area in the transformed single-sided image, thereby achieving accurate alignment between the single-sided image and the template image.

[0130] like Figure 15 The diagram shows the alignment of a single-sided image and a template image. After alignment, the electronic components (represented by thick dashed lines) in the single-sided image to be detected have the same coordinate positions as the electronic components (represented by thick solid lines) in the template image.

[0131] It should be noted that, Figure 15 The alignment effect is represented by an electronic component. In reality, the coordinates of the electronic component at the same position in the two images are the same.

[0132] In some embodiments, after image alignment, individual component images can be accurately cropped based on the local coordinate information of the electronic components when creating the template image. This allows for defect detection on a per-component-image basis, reducing interference from other image areas and improving detection accuracy.

[0133] like Figure 16 The diagram shows a PCBA defect detection process. After the image to be detected is aligned with the template image, the single-sided image and the template image are cropped according to the local coordinate information of the electronic component to obtain the component image and standard component image of the individual component. After the cropped component image and standard component image are converted to the size that meets the input conditions of the model, the AI ​​model is used to infer the individual component image. The inference result is compared with the corresponding standard component image, and the defect detection result of the individual component image is obtained based on the comparison.

[0134] The AI ​​model can select the optimal segmentation model based on existing requirements to obtain an image mask. When the requirements do not involve accurate measurement of area or length, an object detection model can be selected to obtain a detection box.

[0135] In some embodiments, when performing defect detection on a single component image, a detection branch can be designed for each component category.

[0136] (a) Branch 1: Testing of surface mount electronic components

[0137] 1. When a missing part is detected in a single component image of a surface-mount electronic component, while the corresponding standard component image contains the electronic component, the defect of that electronic component is determined to be a missing part; 2. Obtain Mask1, an image of a single patch component, and Mask2, an image of a standard patch component, using a segmentation model. Calculate the cross-union ratio (CUI) between Mask1 and Mask2. If the CUI is greater than a first set threshold, the defect of the electronic component is determined to be offset. Alternatively, calculate the difference in center point positions between Mask1 and Mask2. If the difference in center point positions is greater than a second set threshold, the defect of the electronic component is determined to be offset. The formula is as follows: Formula 2 Formula 3 in, Indicates intersection, union, and ratio. The cross-connection ratio threshold represents the cross-connection ratio of surface-mount electronic components. This indicates the offset of the center point position. This represents the x-axis offset threshold for surface-mount electronic components. This represents the offset threshold of the y-axis for surface-mount electronic components.

[0138] 3. When a surface mount component image is found to be damaged, but the corresponding standard component image is not found to be damaged, the defect information of the electronic component can be further extracted to determine whether the defect information meets the defect standard.

[0139] For example, if the metal plating on the upper surface of a surface-mount electronic component is missing more than 25% of the component width, or if the chipping on the top surface extends more than 0.25 mm inward from the component edge, then the electronic component is determined to be damaged.

[0140] (II) Branch 2: Testing of pin-type electronic components

[0141] 1. When a missing part is detected in a single component image of a pin-type electronic component, while the corresponding standard component image contains the electronic component, the defect of the electronic component is determined to be a missing part; 2. Obtain Mask1, an image of a single pin-type component, and Mask2, an image of a standard pin-type component, through a segmentation model. Calculate the cross-over ratio (COP) between Mask1 and Mask2. If the COP is greater than a third preset threshold, the defect of the electronic component is determined to be offset. Alternatively, calculate the difference in center point positions between Mask1 and Mask2. If the difference in center point positions is greater than a fourth preset threshold, the defect of the electronic component is determined to be offset. The formula is expressed as follows: Formula 4 Formula 5 in, Indicates intersection, union, and ratio. The cross-parallel ratio threshold representing the pin-type electronic components. This indicates the offset of the center point position. This represents the x-axis offset threshold for pin-type electronic components. This represents the offset threshold of the y-axis for pin-type electronic components.

[0142] 3. When a single component image of a pin type is found to be damaged, but no damage is detected in the corresponding standard component image, the defect information of the electronic component can be further extracted to determine whether the defect information meets the defect standard.

[0143] For example, if the internal material of a pin-type electronic component is exposed on its upper surface, causing chipping or breakage that extends more than 0.2 mm into the component's interior, then the defect in that electronic component is determined to be a breakage.

[0144] (III) Branch 3: Testing of surface mount and leaded electronic components

[0145] For each type of electronic component, the similarity between an individual component image and its corresponding standard component image is calculated. The defect category of the electronic component is determined by comparing this similarity with a set similarity threshold. The similarity calculation can be performed using one of the following methods: histogram comparison, color comparison, or gradient comparison. The similarity threshold can be set according to the requirements for over- or under-detection.

[0146] It should be noted that branch 3, as a fallback strategy for the AI ​​detection model, can be tailored according to the actual project metrics. When the project metrics are low, this branch can be omitted.

[0147] like Figure 17 The image shows examples of common electronic component defects.

[0148] It should be noted that the above-mentioned defects are only some of the defects, and the embodiments of this application do not impose any limiting requirements on the type and number of defects.

[0149] In some embodiments, for electronic components with detected defects, a visual interface can be used to compare and display the single-sided image to be inspected with a template image, and a manual re-judgment can be performed. If the re-judgment is a false detection, the re-judgment is FAIL; if the re-judgment is accurate, the re-judgment is PASS.

[0150] In some embodiments, the visual interface may support image magnification to improve the accuracy of quality inspection by quality control personnel.

[0151] like Figure 18 The image shown is a schematic diagram of the re-judgment interface for electronic component testing. The left side is the template image, and the right side is an enlarged single-sided image.

[0152] In some embodiments, after the defect detection of each electronic component in the single-sided image to be inspected is completed, the detection results of each branch can be output as a structured report (such as JSON / XML format). The report may include defect coordinates, type and overall judgment (PASS / FAIL), and the visual annotations are stored in the database to support real-time monitoring and historical traceability.

[0153] like Figure 19The diagram shown is a system architecture diagram for PCBA defect detection provided in this application embodiment. It includes three parts: initialization, image alignment, and defect detection. In the initialization part, the algorithm parameters are initialized and the corresponding template image library is loaded according to the input PCBA model. In the image alignment part, for the input image to be detected, the component corner points of the electronic components are matched with the corresponding template images in the template image library, and the image to be detected is aligned with the template images according to the matched component corner point pairs. In the defect detection part, the image to be detected is cropped using the local coordinate information of the electronic components in the template image to obtain the component image of a single electronic component, completing the preprocessing operation of the image to be detected. Then, the AI ​​model is used to perform defect detection on the component images of surface mount electronic components and the component images of pin-type electronic components. The component images can also be defect detected using computer vision algorithms. Finally, after performing post-processing operations such as manual verification and annotation on the defect detection results of each component image, the defect detection results are fused and a structured report is output.

[0154] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0155] Based on the same technical concept, this application provides an electronic device, which can be a terminal device (such as a computer or a detection device) or a server, capable of implementing the steps of the above image processing method and achieving the same technical effect.

[0156] See Figure 20 The electronic device includes a processor 2001, a memory 2002 and a communication interface 2003, which are connected via a bus 2004. Communication interface 2003 is used for sending and receiving data; The memory 2002 stores a computer program, and the processor 2001 executes the program according to the computer program. Figure 6 The steps of image processing methods.

[0157] It should be noted that, Figure 20 This is just one example; other electronic devices, not shown, may also include conventional components such as memory, power supply, buttons, and displays.

[0158] In this embodiment, the memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system and programs required for running instant messaging functions; the data storage area may store various instant messaging information and operation instruction sets. The memory may be volatile memory, such as random-access memory (RAM); it may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or it may be any other medium capable of carrying or storing a desired computer program having an instruction or data structure form and accessible by a computer, but is not limited thereto. The memory may be a combination of the above-mentioned memories. The processor may include one or more central processing units (CPUs), GPUs, or digital processing units, etc.

[0159] This application does not limit the specific connection medium between the communication interface, memory, and processor. In this application, the bus between the communication, memory, and processor is depicted with a thick line in the figure. The connection methods between other components are only for illustrative purposes and are not intended to be limiting. The bus can be divided into address bus, data bus, control bus, etc. For ease of description, only one thick line is used to describe it in the figure, but it does not mean that there is only one bus or one type of bus.

[0160] This application also provides a computer-readable storage medium for storing instructions that, when executed, can perform the steps of any of the image processing methods described in the foregoing embodiments.

[0161] This application also provides a computer program product for storing a computer program that performs the steps of any of the image processing methods described in the foregoing embodiments.

[0162] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0163] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0164] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An image processing method, characterized in that, include: Target detection is performed on a single-sided image of a printed circuit board assembly to obtain each electronic component in the single-sided image and the component category of each electronic component, wherein the component category includes pin-type and surface-mount-type; Based on the layout information of each electronic component, the electronic components are clustered to obtain multiple sets of initial component corner points for each electronic component. For each set of initial component corner points, the positions of multiple component corner points in the initial set of component corner points are fitted with straight lines to obtain the corresponding set of target component corner points. From each set of target component corner points, remove component corner points that do not meet at least one preset corner point condition; For each set of target component corner points, in the single-sided image and the template image corresponding to the single-sided image, find the first component corner point pair corresponding to the pin-type electronic component, and in the preset search range of each pair of first component corner point pairs, find the second component corner point pair corresponding to the patch-type electronic component. Based on the found first element corner point pairs and second element corner point pairs, align the single-sided image and the template image.

2. The method as described in claim 1, characterized in that, The layout information includes at least the center point coordinates, angles, and dimensions of the electronic components. The step of clustering the electronic components according to their layout information to obtain multiple initial sets of component corner points for each electronic component includes: For each electronic component, the horizontal and vertical directions of the electronic component are determined based on the coordinates and angle of its center point. Within the preset clustering ranges in the horizontal and vertical directions of the electronic components, electronic components that meet the preset size conditions are searched according to their dimensions. Electronic components that meet the preset size conditions are grouped into a set of electronic components, and the corner points of the group of electronic components are extracted to obtain an initial set of component corner points.

3. The method as described in claim 2, characterized in that, When no electronic component meeting the preset size requirements is found, the method further includes: Obtain the component category of the electronic component; When the component category is surface mount, delete the electronic component; When the component category is pin type, the component corner points of the electronic component are used as an initial set of component corner points.

4. The method as described in claim 1, characterized in that, For each initial set of component corner points, the positions of multiple component corner points in the initial set of component corner points are fitted with straight lines to obtain the corresponding target set of component corner points, including: Perform Hough line detection on the corner points of the initial component corner point set to obtain the line corresponding to the initial component corner point set; Calculate the distances from the corner points of each of the multiple components to the straight line, and determine a maximum average distance based on the multiple distances; When the maximum average distance is greater than a preset distance threshold, calculate the projection points of a batch of component corner points associated with the maximum average distance on the straight line, and replace the corresponding component corner points with the projection points to obtain the target component corner point set; When the average maximum distance is not greater than the preset distance threshold, the multiple component corner points are directly used as the target component corner point set.

5. The method as described in claim 1, characterized in that, The step of removing component corner points from each set of target component corner points that do not meet at least one preset corner point condition includes: For each corner point in each set of target component corner points, perform at least one of the following filtering operations: Calculate the area of ​​the outer rectangle of the electronic component corresponding to the corner point of the component. When the area of ​​the outer rectangle is lower than a preset area threshold, the corner point of the component is removed. Obtain the confidence level of the detection frame of the electronic component corresponding to the corner point of the component; when the confidence level is lower than a preset confidence threshold, the corner point of the component is removed. When the electronic component corresponding to the corner point of the component has a defect, the corner point of the component is removed. Calculate the average distance from the corner point of the component to its k nearest neighboring corner points. When the difference between the average distance and the global average distance is greater than a preset outlier distance, the corner point of the component is removed.

6. The method as described in claim 1, characterized in that, The step of finding the first component corner point pair corresponding to the pin-type electronic component in the single-sided image and the template image corresponding to the single-sided image for the component corner points in each set of target component corner points includes: The component corner points of the pin-type electronic components in each set of target component corner points are taken as the first component corner points, and the component corner points of the pin-type electronic components in the template image corresponding to the single-sided image are taken as the second component corner points. For each corner point of the first element, a first similarity is calculated between the electronic components of the pin class based on the layout information of the first element corner point and the layout information of each second element corner point. The second element corner point corresponding to the largest first similarity and the first element corner point are taken as a first element corner point pair of the electronic components of the pin class.

7. The method as described in claim 6, characterized in that, The step of searching for the second component corner pair corresponding to the patch-type electronic component within a preset search range for each pair of the first component corner points includes: For each pair of first component corner points, the component corner points of surface mount electronic components within a preset search range centered on the first component corner point in the first component corner point pair are taken as third component corner points, and the component corner points of surface mount electronic components within a preset search range centered on the second component corner point in the first component corner point pair are taken as fourth component corner points. Calculate a first spatial parameter between each corner point of the third element and the corner point of the first element, and calculate a second spatial parameter between each corner point of the fourth element and the corner point of the second element; For each of the third element corner points, a second similarity between the patch-type electronic components is calculated based on the layout information of the third element corner point and the layout information of each of the fourth element corner points. Combining the corresponding first spatial parameters and each of the second spatial parameters, a fourth element corner point and a third element corner point are selected as a second element corner point pair for the patch-type electronic components.

8. The method according to any one of claims 1-7, characterized in that, The step of aligning the single-sided image and the template image based on the found first element corner point pairs and second element corner point pairs includes: Based on the matching point pairs in each first element corner point pair and each second element corner point pair, a random sampling consistency algorithm is used to calculate the transformation matrix between the single-sided image and the template image. The single-sided image is transformed into the coordinate system of the template image using a transformation matrix, and pixel interpolation is performed on the default area in the transformed single-sided image.

9. The method according to any one of claims 1-7, characterized in that, After aligning the single-sided image and the template image, the method further includes: Based on the local coordinate information of each electronic component in the template image, a single component image is cropped from the single-sided image, and a standard component image is cropped from the template image; Based on the comparison between each individual component image and the corresponding standard component image, the defect detection result of the individual component image is obtained.

10. An electronic device, characterized in that, It includes a processor, a memory, and a communication interface, wherein the communication interface, the memory, and the processor are connected via a bus; The communication interface is used for sending and receiving data; The memory stores a computer program, and the processor executes the method according to any one of claims 1 to 9 based on the computer program.

Images