Printed circuit board solder joint defect detection method and system, medium

Abstract

The application relates to the technical field of printed circuit boards, in particular to a solder joint defect detection method and system for a printed circuit board and a medium. The application carries out image acquisition on different board surface areas of the printed circuit board to obtain a plurality of local board surface images; each local board surface image is cut into a plurality of detection image blocks, which are respectively input into a pre-trained solder joint defect detection model to obtain a candidate defect frame; the candidate defect frame is spliced according to the position of each detection image block in the local board surface image to obtain a local defect detection result; and then the local defect detection result is spliced to form a whole-board candidate defect frame according to the relative position relationship of each local board surface image on the printed circuit board, and a whole-board detection result is obtained through whole-board level defect confirmation. The application considers both the image detail reservation of small-size solder joint defects and the reliable output of whole-board level detection conclusions, and the detection conclusions can be reviewed and traced.

Description

Technical Field

[0001] This application relates to the field of printed circuit board technology, and in particular to a method, system, and medium for detecting solder joint defects on printed circuit boards. Background Technology

[0002] In the electronic product manufacturing process, after printed circuit boards (PCBs) undergo solder paste printing, surface mount technology (SMT), reflow soldering, through-hole soldering, and wave soldering or selective soldering processes, solder forms solder joints on the PCB. These solder joints serve both electrical connection and mechanical fixing functions, and their quality directly affects the long-term reliability of the entire device. With the continuous reduction in component pin spacing and a significant increase in the number of solder joints, defects such as bridging and insufficient solder joints appear in images as small in size, fragmented boundaries, strong reflectivity, and textures similar to normal solder joints. Manual visual inspection is insufficient to meet the production line's requirements for inspection efficiency and cycle time consistency. Post-soldering visual inspection has gradually become a crucial aspect of quality control.

[0003] In related technologies, rule-based automated optical inspection equipment requires the establishment of inspection programs for each board type, input of coordinates, component libraries, and standard image information, resulting in high debugging complexity. Furthermore, such equipment typically integrates independent conveying mechanisms and positioning fixtures, making it difficult to directly integrate into existing post-soldering stations. Deep learning-based solder joint defect recognition schemes focus on improving the feature structure of the inspection model itself. However, when the image of the entire printed circuit board is scaled to the model's input size, small defects such as solder bridging and insufficient solder occupy fewer pixels in the image, easily losing detail. Moreover, single-frame inspection results for local areas are insufficient to directly form a board-wide defect location and quality assessment, leading to insufficient repeatability and traceability of inspection conclusions.

[0004] Therefore, in the context of continuous board transfer inspection, how to balance the image details of small-sized solder joint defects with the reliable output of whole-board inspection results has become a research direction of practical significance in the industry. Summary of the Invention

[0005] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a method, system, and medium for detecting solder joint defects on printed circuit boards, which can balance the image details of small-sized solder joint defects with the reliable output of whole-board level inspection conclusions in inspection scenarios where boards are continuously transported.

[0006] A method for detecting solder joint defects on a printed circuit board according to a first aspect embodiment of this application includes: Images are acquired from different areas of the printed circuit board to obtain multiple local images of the board surface; Image segmentation is performed on each of the local plate surface images to obtain multiple detection image blocks corresponding to each of the local plate surface images; For each local board surface image, the corresponding detection image blocks are input into a pre-trained solder joint defect detection model to obtain candidate defect boxes corresponding to each detection image block; For each of the local panel images, the candidate defect boxes of each of the detected image blocks are combined according to their positions in the local panel image to obtain the local defect detection results corresponding to each local panel image. Determine the relative positional relationship of each of the local board surface images on the printed circuit board; According to the relative positional relationship, the candidate defect boxes in each of the local defect detection results are combined to form several whole plate candidate defect boxes; The entire board-level defect confirmation is performed on each of the candidate defect boxes to obtain the overall board inspection results of the printed circuit board.

[0007] A solder joint defect detection system for printed circuit boards according to a second aspect embodiment of this application includes an image acquisition device, a light source, an edge computing device, and a display device; The image acquisition device is used to acquire images of printed circuit boards; The light source is used to provide illumination for the field of view of the image acquisition device; The edge computing device is used to perform the solder joint defect detection method for printed circuit boards according to any one of the embodiments of the first aspect of this application; The display device is used to display the overall inspection results of the printed circuit board.

[0008] Thirdly, embodiments of this application provide a computer-readable storage medium storing a program that is executed by a processor to implement the solder joint defect detection method for printed circuit boards as described in any one of the embodiments of the first aspect of this application.

[0009] The method, system, and medium for detecting solder joint defects on printed circuit boards according to the embodiments of this application have at least the following beneficial effects: This application acquires images of different areas of a printed circuit board (PCB) and further segments each local image into multiple detection image blocks, which are then fed into a pre-trained solder joint defect detection model. Small-sized solder bridging and insufficient solder defects maintain a sufficient pixel ratio in the model input, and image details are not easily lost due to scaling of the entire board, ensuring the identifiability of small-sized solder joint defects. Based on this, the candidate defect boxes of each detection image block are first combined according to their positions in the local image to form local defect detection results. Then, based on the relative positional relationship of each local image on the PCB, several whole-board candidate defect boxes are combined, and the whole-board defect confirmation is used to obtain the whole-board detection results. This allows the local detection conclusions to be restored to the accurate location of the defect on the entire PCB and the whole-board quality judgment. The detection conclusions are re-inspectable and traceable. The acquisition, segmentation, detection, and combination confirmation stages work together to ensure both image details of small-sized solder joint defects and reliable output of whole-board detection conclusions in the detection scenario of continuous board transfer.

[0010] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0011] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a schematic flowchart of a method for detecting solder joint defects on a printed circuit board, provided in an embodiment of this application. Figure 2 Another schematic flowchart of a method for detecting solder joint defects on a printed circuit board provided in an embodiment of this application; Figure 3 Another schematic flowchart of a method for detecting solder joint defects on a printed circuit board provided in an embodiment of this application; Figure 4 Another schematic flowchart of a method for detecting solder joint defects on a printed circuit board provided in an embodiment of this application; Figure 5 Another schematic flowchart of a method for detecting solder joint defects on a printed circuit board provided in an embodiment of this application; Figure 6 A general flowchart of a solder joint defect detection method for printed circuit boards provided in an embodiment of this application; Figure 7 A flowchart of the board inspection and tracking method provided in the embodiments of this application; Figure 8 This is a schematic diagram of some solder joint defects such as bridging and insufficient solder collected on an actual production line according to an embodiment of this application; Figure 9This is a schematic diagram of continuous acquisition of partial panel images according to an embodiment of this application; Figure 10 This is a schematic diagram of partial panel image segmentation and detection according to an embodiment of this application; Figure 11 This is a schematic diagram of the basic network structure used in the weld joint defect detection model of this application embodiment; Figure 12 This is a schematic diagram of the overall structure of the weld joint defect detection model according to an embodiment of this application; Figure 13 This is a schematic diagram illustrating the whole-board splicing and whole-board level defect confirmation in an embodiment of this application; Figure 14 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0012] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0013] In the description of this application, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0014] In the description of this application, it should be understood that the orientation descriptions, such as up, down, left, right, front, and back, are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0015] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0016] In the description of this application, it should be noted that, unless otherwise explicitly defined, terms such as "setting," "installation," and "connection" should be interpreted broadly. Those skilled in the art can reasonably determine the specific meaning of the above terms in this application based on the specific content of the technical solution. Furthermore, the identification of specific steps in the following text does not imply a limitation on the order of steps or execution logic. The execution order and logic between each step should be understood and inferred from the content described in the embodiments.

[0017] In the electronic product manufacturing process, after printed circuit boards (PCBs) undergo solder paste printing, surface mount technology (SMT), reflow soldering, through-hole soldering, and wave soldering or selective soldering processes, solder forms solder joints on the PCB. These solder joints serve both electrical connection and mechanical fixing functions, and their quality directly affects the long-term reliability of the entire device. With the continuous reduction in component pin spacing and a significant increase in the number of solder joints, defects such as bridging and insufficient solder joints appear in images as small in size, fragmented boundaries, strong reflectivity, and textures similar to normal solder joints. Manual visual inspection is insufficient to meet the production line's requirements for inspection efficiency and cycle time consistency. Post-soldering visual inspection has gradually become a crucial aspect of quality control.

[0018] In related technologies, rule-based automated optical inspection equipment requires the establishment of inspection programs for each board type, input of coordinates, component libraries, and standard image information, resulting in high debugging complexity. Furthermore, such equipment typically integrates independent conveying mechanisms and positioning fixtures, making it difficult to directly integrate into existing post-soldering stations. Deep learning-based solder joint defect recognition schemes focus on improving the feature structure of the inspection model itself. However, when the image of the entire printed circuit board is scaled to the model's input size, small defects such as solder bridging and insufficient solder occupy fewer pixels in the image, easily losing detail. Moreover, single-frame inspection results for local areas are insufficient to directly form a board-wide defect location and quality assessment, leading to insufficient repeatability and traceability of inspection conclusions.

[0019] Therefore, in the context of continuous board transfer inspection, how to balance the image details of small-sized solder joint defects with the reliable output of whole-board inspection results has become a research direction of practical significance in the industry.

[0020] The following explanation is based on the accompanying drawings.

[0021] Reference Figure 1 The solder joint defect detection method for printed circuit boards according to the embodiments of this application may include: Step S101: Image acquisition is performed on different areas of the printed circuit board to obtain multiple local images of the board surface; Step S102: Perform image segmentation on each local plate surface image to obtain multiple detection image blocks corresponding to each local plate surface image; Step S103: For each local board surface image, input each of its corresponding detection image blocks into the pre-trained solder joint defect detection model to obtain the candidate defect box corresponding to each detection image block. Step S104: For each local panel image, according to the position of each detected image block in the local panel image, the candidate defect boxes of each detected image block are combined to obtain the local defect detection results corresponding to each local panel image. Step S105: Determine the relative positional relationship of each local board surface image on the printed circuit board; Step S106: According to the relative positional relationship, the candidate defect boxes in each local defect detection result are combined to form several whole plate candidate defect boxes; Step S107: Perform board-level defect confirmation on each candidate defect box to obtain the board-level inspection results of the printed circuit board.

[0022] In step S101 of this embodiment, images are acquired for different areas of the printed circuit board to obtain multiple local images of the board surface. It should be noted that a partial board surface image refers to an image that only covers a portion of the printed circuit board surface. Multiple partial board surface images correspond to different board surface areas, and when combined, they can cover the entire printed circuit board surface. The purpose of this application embodiment is to break down the image acquisition of the entire board into acquisitions targeting multiple partial areas. The imaging pixels of each partial board surface image are concentrated in the local board surface area, ensuring the detail resolution of solder joints in the image. The consideration for splitting the acquisition object from the entire board into multiple partial areas is that if the entire printed circuit board is photographed at once and the image is scaled to a size that the model can process, small-sized, fragmented, and highly reflective solder bridging and insufficient solder defects occupy only a few pixels in the scaled image, resulting in a significant loss of detail information. However, the processing method of acquiring partial images in different areas ensures that each board surface area retains sufficient detail in the image, providing a reliable image foundation for subsequent defect detection.

[0023] Reference Figure 2 According to some embodiments of this application, step S101 involves image acquisition of different areas of the printed circuit board to obtain multiple local images, which may include: Step S201: Continuously acquire data from the printed circuit board to obtain multiple target image frames; Step S202: The printed circuit board is positioned according to each target image frame acquired continuously to determine the board area in each target image frame; wherein, the board area refers to the image area occupied by the board surface of the printed circuit board in the target image frame. Step S203: For each target image frame, if the board area in the target image frame intersects with the acquisition trigger area, the image content corresponding to the board area is extracted from the target image frame to obtain the corresponding local board image; wherein, the acquisition trigger area refers to the image area pre-defined in the acquisition field of view for triggering image extraction.

[0024] In some embodiments, step S201 involves continuously acquiring images of the printed circuit board to obtain multiple target image frames. It should be noted that continuous acquisition refers to a processing method in which the image acquisition device outputs image frames uninterruptedly according to its own acquisition rhythm, and the target image frames are the images obtained through continuous acquisition. The purpose of this application embodiment is to continuously obtain the image within the acquisition field of view. Regardless of when the printed circuit board enters the acquisition field of view or at what speed it passes through the acquisition field of view, there are corresponding image frames available for subsequent processing. The timing of the printed circuit board reaching the acquisition position is uncertain. Intermittent acquisition may miss the image of the board passing through. Continuous acquisition covers the entire process of the board passing through the acquisition field of view with a complete frame sequence, providing a sufficient image source for subsequent positioning and cropping.

[0025] According to some embodiments of this application, step S201, which involves continuously acquiring images of a printed circuit board to obtain multiple target image frames, may include: As the printed circuit board is transported by the conveyor belt and passes through the acquisition field of view, multiple target image frames are continuously acquired. After continuously acquiring multiple target image frames from the printed circuit board in step S201, the process may further include: Cross-frame association is performed on the board regions in each target image frame to track the printed circuit board and assign board identifiers to the printed circuit board; The motion state of the printed circuit board is determined by the positional changes of the board area associated across frames in each target image frame. In response to the printed circuit board representing the motion state stopping, a data acquisition end signal carrying the board identifier is generated.

[0026] In some embodiments, as the printed circuit board is transported by a conveyor belt and passes through the acquisition field of view, multiple target image frames are continuously acquired from the printed circuit board. It should be noted that conveyor belt transport refers to a method where printed circuit boards (PCBs) are placed on a conveyor belt and continuously pass through the field of view as the conveyor belt moves. In some more specific embodiments, the image acquisition device outputs image frames in continuous acquisition mode and is positioned above, to the side, or at the end of the existing post-soldering process conveyor belt. After reflow soldering, wave soldering, or selective soldering processes, the boards continuously pass through the field of view with the conveyor belt, and multiple target image frames cover the entire process of the boards passing through the field of view. In this method, after completing soldering, the operator can directly place the PCB on the conveyor belt without precisely clamping the boards to a special fixture or pausing the conveyor belt for imaging. The inspection process can be directly integrated into the existing post-soldering station, eliminating the need for independent conveying mechanisms and positioning fixtures, and thus reducing the labor costs for board transfer and placement.

[0027] After continuously acquiring multiple target image frames from a printed circuit board, the method in this embodiment may further include: In some embodiments, board regions in each target image frame are correlated across frames to track printed circuit boards and assign board identifiers to them. It should be noted that cross-frame association refers to the process of associating board regions with similar positions and shapes as the same printed circuit board based on the bounding boxes of board regions in each consecutively acquired target image frame, thereby establishing a board tracking relationship. Board identification refers to the board number assigned to each printed circuit board entering the acquisition field of view, used to distinguish different boards. On the production line, multiple printed circuit boards typically pass through the acquisition field of view consecutively. If the board regions in each frame are not associated, the subsequent local board surface images and detection results cannot be attributed to specific boards, and the detection results for different boards are easily confused. After establishing the board tracking relationship and assigning board numbers, the board regions of each printed circuit board from entering the acquisition field of view to leaving the acquisition field of view are classified under the same board number. The corresponding local board surface images and detection results can be organized and traced through the board number.

[0028] In some embodiments, the motion state of the printed circuit board is determined based on the positional changes of the board region associated across frames in each target image frame. It should be noted that motion state refers to the state of the printed circuit board (PCB) being in motion or stationary. Board areas associated across frames belong to the same PCB, and their position changes in each target image frame reflect the board's movement along the conveyor belt. Continuous position changes indicate the board is moving, while a relatively unchanged position indicates it has stopped moving. Alternatively, the inter-frame grayscale difference between adjacent target image frames can be used to help determine if the board is moving. When the inter-frame grayscale difference is greater than a preset grayscale difference threshold, it indicates a change in the image content and that the board is moving; when the inter-frame grayscale difference is consistently less than the threshold, it indicates the board is approaching stillness. Determining the motion state provides a basis for terminating single-board acquisition. Board stopping movement means it has left the conveyor belt or the acquisition process has been interrupted, and continuing to capture images of it will no longer produce valid local board images.

[0029] In some embodiments, in response to the motion state characterizing the printed circuit board stopping, a data acquisition end signal carrying a board identifier is generated.

[0030] It should be noted that the acquisition end signal corresponds to the whole-board end signal in the detection program, which refers to the trigger signal indicating that the image acquisition of a single printed circuit board has ended. When the motion state indicates that the printed circuit board has stopped moving, continuous acquisition of that board can no longer obtain new valid images. At this time, the acquisition thread generates a whole-board end signal and transmits it to the stitching thread, notifying the subsequent processing stages that the local board surface image of that board is complete, and the organization and confirmation of the whole-board results can begin. The whole-board end signal carries a board identifier, making the signal clearly correspond to a specific printed circuit board. In scenarios where multiple boards are continuously detected, the acquisition end times of each board do not interfere with each other, and the image and results of each board can be completely closed under their respective board number names. In some more specific embodiments, the acquisition thread sends the whole-board end signal for that board after the board stops moving for a preset number of frames.

[0031] According to some embodiments of this application, step S202, which involves locating the printed circuit board based on each continuously acquired target image frame to determine the board area in each target image frame, may include: For each target image frame, candidate regions of the board surface are extracted from the target image frame according to a preset board surface color threshold; The candidate areas of the board surface are screened by contour, and the candidate areas of the board surface whose contours meet the preset geometric conditions are determined as the board areas.

[0032] It should be noted that the board surface color threshold refers to a color judgment condition pre-set based on the color of the printed circuit board substrate, and the board surface candidate region refers to the image region composed of pixels in the target image frame whose colors meet the board surface color threshold. The board surface of a printed circuit board is usually covered with solder resist ink, whose color is relatively stable and has a significant difference from the color of the conveyor belt and workbench background. The board surface can be initially separated from the target image frame using the color threshold. As one implementation method, the target image frame can be first converted from the red-green-blue color space to the hue, saturation, and brightness color space, and then the board surface candidate region can be extracted in the hue, saturation, and brightness color space based on the board surface color threshold to reduce the impact of changes in illumination intensity on color judgment. In some more specific embodiments, for the board surface covered with green solder resist ink, a hue range corresponding to green is set as the board surface color threshold. The embodiments of this application extract the board surface candidate region frame by frame based on color, which has low computational load and fast processing speed, and can meet the real-time requirements of frame-by-frame positioning in continuous acquisition scenarios.

[0033] It should be noted that contour filtering refers to detecting the outer contour of candidate areas on the board surface and determining whether the geometric shape of the outer contour meets preset geometric conditions. Preset geometric conditions refer to geometric judgment conditions pre-defined based on the shape characteristics of the printed circuit board. When extracting only based on color thresholds, other objects in the image frame with colors similar to the board surface may also be extracted as candidate areas, such as color markings or tooling components on a workbench. Printed circuit boards have regular plate-like outer contours and relatively stable area and aspect ratio ranges; contour filtering can eliminate candidate areas with similar colors but inconsistent shapes. In some more specific embodiments, preset geometric conditions include area thresholds and aspect ratio thresholds. Candidate areas with areas smaller than the area thresholds or aspect ratios deviating from the board shape are filtered out, and the remaining candidate areas are determined as board areas. Combining color extraction and contour filtering first quickly delineates the candidate range by color, then confirms its authenticity by geometric shape, ensuring that the determined board area accurately corresponds to the actual position of the printed circuit board.

[0034] In step S102 of this embodiment, image segmentation is performed on each local plate image to obtain multiple detection image blocks corresponding to each local plate image. It should be understood that a detection image block refers to an image unit obtained by segmenting a local board surface image. Each local board surface image corresponds to multiple detection image blocks, which are the basic image units subsequently fed into the detection model for processing. This embodiment further controls the image size fed into the model for processing. Even if the size of the local board surface image itself exceeds the input size of the model, the segmented detection image blocks can still be adapted to the model input. Small-sized defects maintain a sufficient pixel proportion in the model input image, and details are not diluted by secondary scaling. This processing works in conjunction with the local acquisition in step S101. Local acquisition solves the detail loss caused by whole-board imaging, while image segmentation solves the adaptation problem between the local image and the model input size. Together, they ensure that the details of solder joint defects are completely preserved before being fed into the model.

[0035] Reference Figure 3 According to some embodiments of this application, step S102 involves image segmentation for each local panel image to obtain multiple detection image blocks corresponding to each local panel image, which may include: Step S301: For each local panel image, the local panel image is segmented according to a preset image block size to obtain multiple detection image blocks corresponding to the local panel image; wherein, there is an overlapping area between adjacent detection image blocks; In step S104, the candidate defect boxes of each detected image block are stitched together according to their positions in the local panel image, which may include: Step S302: Determine the position of each candidate defect box in the local plate image according to the position of each detected image block in the local plate image; Step S303: Deduplicate and merge candidate defect boxes whose positions on the board surface are located in overlapping areas and overlap with each other to obtain the local defect detection results corresponding to each local board surface image.

[0036] In step S301 of some embodiments, for each local panel image, the local panel image is segmented according to a preset image block size to obtain multiple detection image blocks corresponding to the local panel image; wherein, there is an overlapping area between adjacent detection image blocks; It should be noted that the preset image block size refers to a pre-defined image block size that is adapted to the input of the solder joint defect detection model. Segmentation means dividing a local board image into multiple detection image blocks according to this size. The overlapping area refers to the image area jointly covered by two adjacent detection image blocks. The purpose of setting the overlapping area is to ensure that solder joints and defects located near the boundary of the detection image block can be completely presented in at least one detection image block. If boundary-aligned non-overlapping segmentation is used, solder joints that exactly cross the segmentation boundary will be segmented into two adjacent detection image blocks, and will not be complete in each detection image block. The model will find it difficult to completely identify such cut-off defects. By preserving the overlapping area between adjacent detection image blocks, defects near the boundary can be completely included in a certain detection image block. The segmentation process does not cause the defect information to be fragmented, and the detection accuracy guarantee brought by segmentation is fully utilized.

[0037] According to some embodiments of this application, step S301, for each local panel image, segments the local panel image according to a preset image block size to obtain multiple detection image blocks corresponding to the local panel image, may include: For each local panel image, the local panel image is sequentially segmented according to the preset image block size and along the preset segmentation direction to obtain each detection image block; In response to the fact that the size of the remaining region of the local panel image is less than a preset size threshold, the remaining region is incorporated into the previous detection image block; wherein, the remaining region of the local panel image refers to the region in the local panel image that has not been segmented into the detection image block after being segmented sequentially along the preset segmentation direction.

[0038] In some embodiments, for each local panel image, the local panel image is sequentially segmented along a preset segmentation direction according to a preset image block size to obtain each detection image block; It should be noted that the preset segmentation direction refers to the direction in which the segmentation operation proceeds, which can be the height or width direction of the local panel image; sequential segmentation refers to the process of segmenting the local panel image into detection image blocks one by one according to the preset image block size, starting from one end of the local panel image along the preset segmentation direction. Sequential segmentation along a fixed direction ensures that the size of each detection image block remains consistent, the adaptation relationship with the model input is stable, the segmentation rules are simple and clear, and the position of each detection image block in the local panel image is also easy to determine according to the segmentation order.

[0039] In some embodiments, in response to the size of the remaining region of a local panel image being less than a preset size threshold, the remaining region is incorporated into the previous detection image block; wherein, the remaining region of the local panel image refers to the region in the local panel image that has not been segmented into the detection image block after being segmented sequentially along a preset segmentation direction.

[0040] It should be understood that the preset size threshold refers to a pre-defined condition used to determine whether the remaining region is too narrow. The size of a local panel image is usually not an integer multiple of the preset image block size. After sequentially dividing along the preset dividing direction, a remaining region that has not been divided will remain. If the size of the remaining region is too small, and it is treated as a separate detection image block, it needs to be stretched or padded significantly before being fed into the model, resulting in image distortion and making it difficult to guarantee the reliability of the model's detection of distorted images. By incorporating the remaining region with a size smaller than the preset size threshold into the previous detection image block, the size of the previous detection image block is only slightly larger than the preset image block size. The image content remains intact, the adaptability of the model input is not affected, and the detection distortion caused by the overly narrow image block is avoided.

[0041] In step S302 of some embodiments, the position of each candidate defect box in the local plate image is determined according to the position of each detected image block in the local plate image. Specifically, the board surface position refers to the location of the candidate defect box within the local board surface image range. It is calculated based on the positions of the detected image blocks in the local board surface image and the positions of the candidate defect boxes within the detected image blocks. The positions of the candidate defect boxes output by the model are described based on the image range of each detected image block. The position descriptions of different detected image blocks are different and cannot be directly compared and organized. This embodiment of the application uniformly converts the positions of each candidate defect box to the range of the local board surface image, so that candidate defect boxes from different detected image blocks have a consistent position reference, and the stitching and subsequent processing within the entire board surface have a common position basis.

[0042] In step S303 of some embodiments, candidate defect boxes whose positions on the board surface are located in overlapping areas and overlap with each other are deduplicated and merged to obtain the local defect detection results corresponding to each local board surface image.

[0043] It should be noted that deduplication and merging refers to merging candidate defect boxes that are located in overlapping areas on the board surface, so that only one defect box is retained in the detection result for the same defect. The local defect detection result is the defect detection result within a local board surface image after deduplication and merging. The setting of overlapping areas means that the same defect may be detected by two adjacent detection image blocks. After stitching, the same defect corresponds to multiple overlapping candidate defect boxes in the local board surface image. If this is directly used for counting, it will result in duplicate counts, artificially inflating the number of defects, and the defect location will also be marked repeatedly. Deduplication and merging of overlapping candidate defect boxes ensures that each real defect appears only once in the local defect detection result, and the statistics of the number and location of defects are accurate and reliable. As one implementation method, deduplication and merging can be implemented using non-maximum suppression, that is, merging duplicate detection boxes corresponding to the same defect according to the degree of overlap between candidate defect boxes.

[0044] According to some embodiments of this application, step S303 involves deduplicating and merging candidate defect boxes whose positions on the board surface are located in overlapping areas and overlap with each other to obtain local defect detection results corresponding to each local board surface image, which may include: Deduplication and merging of candidate defect boxes located in overlapping areas on the board surface are performed to obtain the initial defect detection results; For candidate defect boxes whose board surface locations are in the overlapping area in the initial defect detection results, determine the overlapping detection image blocks; From the candidate defect boxes corresponding to the overlapping detection image blocks, select overlapping candidate boxes where the plate surface position overlaps with the plate surface position of the candidate defect box; If there are no overlapping candidate boxes in the candidate defect boxes corresponding to the overlapping detection image blocks, the candidate defect boxes are removed from the initial defect detection results to obtain the local defect detection results corresponding to the local panel image.

[0045] In some embodiments, candidate defect frames whose positions on the board surface are located in overlapping areas and overlap with each other are deduplicated and merged to obtain initial defect detection results; It should be noted that the initial defect detection result refers to the intermediate detection result after deduplication and merging, but before subsequent edge verification. Deduplication and merging first merges the overlapping candidate defect boxes in the overlapping area into a single defect box, so that only one detection record is retained for the same defect in the result, and the statistical benchmark for the number and location of defects is unified. The merged result is used as an intermediate product in the form of the initial defect detection result for further verification of the candidate defect boxes in the overlapping area.

[0046] In some embodiments, overlapping detection image blocks are determined for candidate defect boxes whose board surface positions are located in the overlapping area in the initial defect detection results; It should be noted that an overlapping detection image block refers to another detection image block that shares the same overlapping area as the detection image block used to output candidate defect boxes. The overlapping area is covered by two adjacent detection image blocks. Candidate defect boxes located within the overlapping area originate from one of the detection image blocks, while the other detection image block provides a complete image of the same area. The purpose of identifying overlapping detection image blocks is to provide a reference object for verifying candidate defect boxes. Whether a candidate defect box corresponds to a real defect can be verified by checking the detection status of the same area using overlapping detection image blocks.

[0047] In some embodiments, overlapping candidate boxes are selected from the candidate defect boxes corresponding to the overlapping detection image blocks, where the plate surface position overlaps with the plate surface position of the candidate defect box. It should be noted that overlapping candidate boxes refer to the defect boxes in the candidate defect boxes corresponding to overlapping detection image blocks, where the plate surface position overlaps with the plate surface position of the verified candidate defect box. When the real defect is located in the overlapping area, both adjacent detection image blocks will fully image it, and the defect will usually be detected, with the two detected defect boxes overlapping each other in plate surface position. Selecting overlapping candidate boxes involves searching for corresponding detections of the same defect in the detection results of overlapping detection image blocks. Finding overlapping candidate boxes indicates that the candidate defect box has been jointly detected by two adjacent detection image blocks, and its corresponding real defect has high credibility.

[0048] In some embodiments, if there is no overlapping candidate box in the candidate defect box corresponding to the overlapping detection image block, the candidate defect box is removed from the initial defect detection result to obtain the local defect detection result corresponding to the local panel image.

[0049] It should be noted that there are no overlapping candidate boxes in the candidate defect boxes corresponding to the overlapping detection image blocks. This means that the overlapping detection image blocks completely image the same area but fail to detect the corresponding defects. The verified candidate defect boxes are just isolated detections of a single detection image block near the truncation boundary, which are suspected false detections caused by the defect being cut off by the truncation boundary. Removing these candidate defect boxes from the initial defect detection results suppresses false detections caused by the truncation boundary. The remaining candidate defect boxes constitute the local defect detection results, making the number and location of defects within the local panel image range more accurate and reliable.

[0050] In step S103 of this embodiment, for each local board image, the corresponding detection image blocks are input into the pre-trained solder joint defect detection model to obtain the candidate defect box corresponding to each detection image block. It should be noted that the solder joint defect detection model refers to a neural network model pre-trained with samples, capable of identifying solder joint defects in the input image and outputting the location of the defects; the candidate defect box refers to the detection box output by the model after preliminary detection of the image block, which selects the suspected defect area. Its candidate attribute indicates that the detection box still needs to be confirmed by subsequent processing. In this application embodiment, model recognition replaces manual rule parameter tuning. Solder joints in images have strong reflectivity and large appearance fluctuations, and fixed rules are difficult to stably describe the defect morphology. However, the model trained with samples can learn the performance rules of defects from the data and has the ability to adapt to the appearance fluctuations of solder joints under different batches and different lighting conditions. The block-by-block input detection method also allows the defects in each detected image block to be independently identified by the model.

[0051] According to some embodiments of this application, the weld joint defect detection model is pre-trained in the following manner: Acquire multiple sample image blocks and their corresponding annotation information; the annotation information includes defect box annotations and weld joint structural anomaly annotations. Using each sample image patch as input and the corresponding annotation information as supervision, the main branch for defect detection and the prior auxiliary branch for weld joint structure are jointly trained. The total loss of the joint training includes the detection loss corresponding to the main branch for defect detection and the structural prior loss corresponding to the prior auxiliary branch for weld joint structure.

[0052] In some embodiments, multiple sample image blocks and corresponding annotation information for each sample image block are obtained; wherein, the annotation information includes defect box annotations and weld joint structural anomaly annotations.

[0053] It should be noted that the sample image patch refers to the image sample used to train the solder joint defect detection model. Its size can be consistent with the detection image patch obtained during the detection phase, ensuring that the model processes inputs of the same scale during both the training and detection phases. Each sample image patch corresponds to a set of annotation information, including defect bounding box annotations and solder joint structural anomaly annotations. The defect bounding box annotations mark the location and type of solder joint defects in the sample image patch, corresponding to the output format of the main defect detection branch. The solder joint structural anomaly annotations mark the locations of solder joint structural anomalies in the sample image patch at the pixel level, corresponding to the structural prior probability map output by the solder joint structural prior auxiliary branch. Since both types of annotations can be obtained simultaneously with a single sample production, the supervision signals for both branches can be constructed on the same batch of samples, and the cost of training data production does not significantly increase due to the introduction of the auxiliary branch.

[0054] In some embodiments, the defect detection main branch and the solder joint structure prior auxiliary branch are jointly trained using each sample image block as input and the corresponding annotation information of each sample image block as supervision; wherein, the total loss of the joint training includes the detection loss corresponding to the defect detection main branch and the structural prior loss corresponding to the solder joint structure prior auxiliary branch.

[0055] It needs to be clarified that joint training refers to a training method in which the same batch of sample image blocks are simultaneously input into the main branch for defect detection and the prior auxiliary branch for solder joint structure, and the outputs of the two branches are supervised by corresponding annotation information. Since the two branches share the feature extraction part in the model, joint training ensures that the shared feature extraction part is not only supervised by the defect detection task but also constrained by the solder joint structure state prediction task, prompting the extracted features to better preserve structural information such as solder joint morphology and solder distribution. Compared to training the two branches independently, joint training allows the prior auxiliary branch's understanding of the solder joint structure to feed back into the shared features, improving the main branch's feature representation of small, fragmented target defects. The feature basis upon which the two branches cooperate during the detection phase is also more consistent.

[0056] It should be noted that the total loss of joint training consists of two parts: detection loss and structural prior loss. The detection loss measures the difference between the candidate defect boxes output by the main defect detection branch and the defect box annotations. As one implementation, the detection loss can include a classification loss supervised by defect category prediction and a regression loss supervised by defect box location prediction. The structural prior loss measures the difference between the structural prior probability map output by the auxiliary branch and the weld joint structural anomaly annotations. As one implementation, the structural prior loss can use pixel-wise cross-entropy loss. During training, the model parameters can be updated globally based on the total loss until the model converges, resulting in a pre-trained weld joint defect detection model. By incorporating the two types of supervision signals into the same training process through the total loss, the box-level supervision of the main detection branch and the pixel-level supervision of the auxiliary branch each act on their respective branches, allowing the outputs of the two branches to form an effective mutual calibration relationship during the detection phase.

[0057] According to some embodiments of this application, weld joint structural anomaly annotations are generated in the following manner: Based on the marked areas of each solder joint in the sample image block, determine the pairs of adjacent solder joints; If there is a solder bridging defect annotation box between adjacent solder joint pairs, the overlapping area between the solder bridging defect annotation box and the gap area between adjacent solder joint pairs is marked as a positive sample area; If there is no solder bridging defect label box between adjacent solder joint pairs, the gap area is labeled as a negative sample area.

[0058] In some embodiments, adjacent solder joint pairs are determined based on the marked areas of each solder joint in the sample image block; It should be noted that the labeled area of ​​a solder joint refers to the region marked for each solder joint in the sample image block, and an adjacent solder joint pair refers to two solder joints that are spatially adjacent to each other in the sample image block. As one implementation method, adjacent solder joint pairs can be determined based on the spacing between the labeled areas of each solder joint, identifying two solder joints within a preset proximity range as an adjacent pair. The essence of solder bridging defects is abnormal solder bridging between adjacent solder joints. This bridging occurs at the gap between adjacent solder joints. Therefore, when constructing anomaly labels for solder joint structures, adjacent solder joint pairs are first identified, and then the labeling granularity is focused on the gap region between adjacent solder joint pairs, ensuring that the monitoring signal falls on the actual structural location where the solder bridging defect occurs, rather than being scattered across arbitrary locations in the sample image block.

[0059] In some embodiments, if there is a solder bridging defect annotation box between adjacent solder joint pairs, the overlapping area of ​​the solder bridging defect annotation box and the gap area between adjacent solder joint pairs is marked as a positive sample area. It should be noted that after identifying adjacent solder joint pairs, positive and negative sample regions can be generated based on whether a solder bridging defect annotation frame exists between the adjacent solder joint pairs. If a solder bridging defect annotation frame exists between adjacent solder joint pairs, it indicates that solder bridging has occurred between them. The overlapping area between the solder bridging defect annotation frame and the gap area between the adjacent solder joint pairs is marked as the positive sample region. The positive sample region is the pixel range actually occupied by the bridging solder in the gap area. Using the overlapping part of the solder bridging defect annotation frame and the gap area as the positive sample region, rather than directly using the entire solder bridging defect annotation frame, ensures that the positive sample region accurately covers the gap location where the bridging occurs, avoiding mislabeling the solder joint body area as a structurally abnormal area. This ensures that the monitoring signal strictly corresponds to the definition of the connection relationship probability map as "representing the probability of abnormal solder bridging between adjacent solder joints".

[0060] In some embodiments, if there is no solder bridging defect labeling box between adjacent solder joint pairs, the gap area is labeled as a negative sample area.

[0061] It should be noted that if there is no bridging defect annotation box between adjacent solder joint pairs, it indicates that the gap between the adjacent solder joint pairs is in a normal state, and the gap area is marked as a negative sample area. The negative sample area and the positive sample area together constitute a pixel-by-pixel solder joint structure anomaly annotation. The structure prior auxiliary branch uses this annotation as supervision during training, so that the output connection relationship probability map tends to output larger pixel probability values ​​at bridging positions and smaller pixel probability values ​​at normal gap positions. Since the number of gap areas in normal samples is much greater than the number of gap areas where bridging occurs, the introduction of the negative sample area can also prevent the auxiliary branch from outputting high probabilities for all gap areas, ensuring the connection relationship probability map's ability to distinguish between normal gaps and abnormal bridging.

[0062] Reference Figure 4 According to some embodiments of this application, the solder joint defect detection model includes a main defect detection branch and a priori auxiliary branch for solder joint structure; step S103, for each local board surface image, inputs its corresponding detection image blocks into the pre-trained solder joint defect detection model to obtain the candidate defect boxes corresponding to each detection image block, may include: Step S401: Input each detected image block into the solder joint defect detection model, and perform defect detection on each detected image block through the main defect detection branch to obtain the candidate defect box corresponding to each detected image block; wherein, the candidate defect box is used to indicate the solder joint defect category and detection confidence. Step S402: The solder joint structure state of each detected image block is predicted by the solder joint structure prior auxiliary branch to obtain the structure prior probability map corresponding to each detected image block; wherein, the probability value of each pixel in the structure prior probability map is used to characterize the probability that there is an abnormal solder joint structure at the corresponding pixel position. Step S403: The detection confidence of the candidate defect boxes is calibrated based on the pixel probability values ​​in the prior probability map of the structure corresponding to each candidate defect box.

[0063] In step S401 of some embodiments, each detected image block is input into the solder joint defect detection model, and defect detection is performed on each detected image block through the main defect detection branch to obtain the candidate defect box corresponding to each detected image block; wherein, the candidate defect box is used to indicate the solder joint defect category and detection confidence. It should be noted that the main branch of defect detection refers to the branch in the solder joint defect detection model that undertakes the task of defect identification. Its function is to identify solder joint defects from the detected image blocks and output candidate defect boxes. The candidate defect boxes are used to indicate the solder joint defect category and detection confidence level. The solder joint defect category indicates the type of defect selected by the candidate defect box, and the detection confidence level indicates the degree of certainty of the main branch of defect detection regarding the actual defect corresponding to the candidate defect box. In this embodiment, each detected image block provides a preliminary detection result with category and confidence level, which forms the basis for subsequent processing. It should be noted that the detection confidence level is given based on the appearance of the defect. Factors such as solder joint reflection, silkscreen texture, and local shadows may cause background areas with similar appearances to be assigned a higher detection confidence level.

[0064] According to some embodiments of this application, step S401, which performs defect detection on each detected image block through the main defect detection branch to obtain candidate defect boxes corresponding to each detected image block, may include: Multi-level feature extraction is performed on each detected image patch, and the extracted multi-level features are fused at multiple scales to obtain fused features at multiple scales; among them, the fused features at multiple scales include the high-resolution fused features with the smallest downsampling ratio. Defect detection is performed based on the fusion features at various scales to obtain candidate defect boxes corresponding to each detected image block; Predicting the solder joint structure state of each detected image block by using a priori auxiliary branch of solder joint structure can include: predicting the solder joint structure state based on high-resolution fusion features.

[0065] In some embodiments, multi-layer feature extraction is performed on each detected image block, and the extracted multi-layer features are fused at multiple scales to obtain fused features at multiple scales; wherein, the fused features at multiple scales include the high-resolution fused features with the smallest downsampling factor. It should be noted that multi-layer feature extraction refers to extracting features layer by layer from the detected image patch, resulting in multi-layer features with different levels of abstraction. Shallow features retain more image details, while deep features contain stronger semantic information. Multi-scale fusion refers to fusing features from different layers to obtain fused features at multiple scales. Among these, the fused feature with the smallest downsampling ratio has the highest resolution, i.e., the high-resolution fused feature. Tin-bonded and low-tin defects are characterized by their small size in images. After multiple downsampling steps, a large amount of detail information of small-sized defects is lost in deep features, while detection relying solely on shallow features lacks sufficient semantic information support. After multi-scale fusion, the high-resolution fused feature combines high resolution with certain semantic information, preserving the details of small-sized defects. The fused features at each scale work together to provide a feature basis for detecting defects of different sizes. As one implementation method, the weld joint defect detection model can be built on the YOLO11s network (a single-stage target detection network). The convolutional neural network backbone network performs multi-layer feature extraction on the detected image blocks, and the neck network of the feature pyramid structure performs multi-scale fusion to obtain fused features at four scales: P2, P3, P4, and P5. Among them, the downsampling step size of the P2 fusion feature is 4, and the resolution is one-quarter of the input image, which is the high-resolution fusion feature with the smallest downsampling ratio.

[0066] In some embodiments, defect detection is performed based on fused features at various scales to obtain candidate defect boxes corresponding to each detected image block; It should be noted that defect detection based on fusion features at various scales refers to identifying defects separately for each scale of fusion features and then summarizing the detection results at each scale into candidate defect boxes. Different scales of fusion features correspond to different receptive fields and resolutions, adapting to defect targets of different sizes. High-resolution fusion features are specifically designed for detecting small defects. Solder joints are densely distributed on printed circuit boards, and defects such as solder bridging and insufficient solder are small in size and diverse in shape. Features at a single scale are insufficient to detect all types of defects. By detecting defects based on multi-scale fusion features, small defects are detected using high-resolution fusion features, while relatively large defects are detected using deep fusion features. This allows the model to recognize defects of all sizes within the detected image patch. As one implementation method, the detection head of the main defect detection branch outputs candidate defect boxes based on fusion features at four scales: P2, P3, P4, and P5.

[0067] In some embodiments, predicting the solder joint structure state of each detected image block through a priori auxiliary branch of solder joint structure may include: predicting the solder joint structure state based on high-resolution fusion features.

[0068] It should be noted that solder joint structural state prediction is a pixel-by-pixel prediction process, and its prediction accuracy depends on the resolution of the input features. Solder joint structural anomalies often occupy only a small number of pixels in an image. If prediction is based on low-resolution features that have undergone multiple downsampling, the subtle manifestations of the structural anomalies have been lost during the downsampling process, and the prediction results are difficult to accurately reflect the location and range of the anomalies. High-resolution fusion features are the features with the smallest downsampling ratio and the highest resolution among the fusion features of various scales. Using high-resolution fusion features as the basis for solder joint structural state prediction preserves the subtle manifestations of structural anomalies, and the structural prior probability map output by the auxiliary branch can accurately correspond to the pixel positions in the detected image patch.

[0069] In some more specific embodiments, the solder joint structure prior auxiliary branch is connected after the P2 high-resolution feature layer, and the solder joint structure state is predicted based on the P2 fused features.

[0070] In step S402 of some embodiments, the solder joint structure state of each detected image block is predicted by the solder joint structure prior auxiliary branch to obtain the structure prior probability map corresponding to each detected image block; wherein, the probability value of each pixel in the structure prior probability map is used to characterize the probability that there is an abnormal solder joint structure at the corresponding pixel position. It should be noted that weld joint structure state prediction refers to the process of predicting whether the structural morphology of weld joints in the detected image block is normal. The formation of weld joint defects follows the inherent laws of the welding process. The defects manifest in the image not only as differences in appearance and texture but also as anomalies in the structural state of the weld joint. This correspondence between defects and structural states belongs to prior knowledge in the welding process domain. The weld joint structure prior auxiliary branch introduces this prior knowledge into the model, enabling the model to independently predict the structural state of weld joints in addition to defect identification. The model's learning of defects extends from a purely appearance-level perspective to a structural level.

[0071] It should be noted that the structural prior probability map is a two-dimensional numerical matrix output by the solder joint structure prior auxiliary branch. Its size corresponds to the spatial size of the detected image patch. The probability value of each pixel in the matrix represents the probability that a solder joint structural anomaly exists at the corresponding pixel location. It should be clarified that the structural prior probability map is not the actual captured image; the pixel probability value does not represent the image's grayscale or color information. Its value ranges from 0 to 1, with a larger value indicating a greater likelihood that the auxiliary branch considers a solder joint structural anomaly to exist at the corresponding pixel location. As one implementation method, the solder joint structure prior auxiliary branch can consist of a convolutional layer and a sigmoid activation function. The input fused features are convolved, and then the sigmoid activation function compresses the output of each pixel location to the 0-1 range, forming a pixel-wise predicted probability. Compared to giving a single judgment for the entire image, the pixel-wise probability prediction preserves the spatial distribution information of structural anomalies; the location and extent of the anomaly can be read from the structural prior probability map.

[0072] Specifically, the structural prior probability map serves to provide a structural dimension verification basis for candidate defect boxes. The main defect detection branch identifies the detected image block from the perspective of defect appearance, while the auxiliary branch predicts the same detected image block from the perspective of weld joint structural prior. The two branches process the same object but from different angles, and their outputs can be referenced against each other. At the same time, the main branch and the auxiliary branch share the model's feature extraction process. While learning the rules of defect appearance, the model is also constrained by the rules of structural state. The learned features have a stronger ability to express defects, and the stability is improved when faced with interference factors such as weld joint reflection, silkscreen texture, and local shadows.

[0073] In step S403 of some embodiments, the detection confidence of the candidate defect box is calibrated based on the pixel probability values ​​in the prior probability map of the structure corresponding to each candidate defect box.

[0074] It should be noted that calibration refers to the process of correcting the detection confidence score output by the main branch of defect detection based on the prediction results of the structural prior probability map. After the position of the candidate defect box in the detection image block is determined, the pixel probability value corresponding to it in the structural prior probability map reflects the possibility of an abnormal weld joint structure in the selected area. The detection confidence score given by the main branch of defect detection from the appearance perspective is cross-referenced with the pixel probability value given by the weld joint structure prior auxiliary branch from the structural perspective. The detection confidence score of candidate defect boxes that are detected by appearance and match the predicted structural state is supported, while the detection confidence score of candidate defect boxes that are detected by appearance but do not match the predicted structural state is lowered. False detections caused by factors such as weld joint reflection and silkscreen texture often only remain at the appearance level. These areas differ from the real defects in structural state. After confidence score calibration, false detection boxes that are only similar in appearance are suppressed, and the reliability of candidate defect boxes is improved.

[0075] According to some embodiments of this application, the solder joint structure prior auxiliary branch includes a connection relationship auxiliary branch, and the structure prior probability map includes a connection relationship probability map; wherein, the probability value of each pixel in the connection relationship probability map represents the probability that there is an abnormal solder bridging between adjacent solder joints at the corresponding pixel position; In step S403, the detection confidence of the candidate defect boxes is calibrated based on the pixel probability values ​​in the prior probability map of the structure corresponding to each candidate defect box. This may include: For candidate defect boxes with the defect category of tin bridging, the pixel probability values ​​of the candidate defect boxes in the corresponding regions of the connectivity probability map are statistically analyzed to obtain connectivity statistics. In response to a connection relationship statistic value being less than a preset calibration threshold, the detection confidence of candidate defect boxes is reduced or candidate defect boxes are removed.

[0076] It should be noted that the connection relationship auxiliary branch refers to the branch in the solder joint structure prior auxiliary branch that predicts the connection relationship of solder joints, and its output is a connection relationship probability map. Solder bridging defects refer to defects formed by excess solder bridging between adjacent solder joints or adjacent pins. Essentially, it is the presence of solder connections that should not exist between adjacent solder joints. This defect may lead to short circuits in adjacent networks and is a type of defect that requires key inspection after soldering. The probability value of each pixel in the connection relationship probability map represents the probability of abnormal solder bridging at the corresponding pixel location, that is, modeling solder bridging defects from a structural relationship perspective as an abnormal connection state between adjacent solder joints. The thin bridges formed by solder bridging are small in size in the image and similar in texture to normal solder joints. Simply identifying them from an appearance perspective can easily lead to confusion with reflective areas of solder joints. By modeling solder bridging defects as abnormal connection relationships between adjacent solder joints, the model can learn the essential behavior of solder bridging from a structural relationship perspective, and the recognition result no longer relies solely on appearance texture.

[0077] In some embodiments, for candidate defect boxes with the defect category of tin bridging, the pixel probability values ​​of the candidate defect boxes in the corresponding regions of the connectivity probability map are statistically analyzed to obtain connectivity statistics. It should be noted that the corresponding region in the connectivity probability map of the candidate defect box refers to the region obtained by mapping the position of the candidate defect box in the detection image block to the connectivity probability map; the connectivity statistics value refers to the numerical value obtained by statistically analyzing the probability values ​​of each pixel in the corresponding region, which reflects the overall degree of abnormal solder bridging in the region selected by the candidate defect box. The purpose of this application embodiment is to summarize the structural state prediction results within the selected region into a single numerical value, so that the prediction of the structural dimension can be compared with preset conditions. In some more specific embodiments, the average value of the pixel probability values ​​in the corresponding region of the candidate defect box in the connectivity probability map is calculated as the connectivity statistics value.

[0078] In some embodiments, in response to a connection statistics value being less than a preset calibration threshold, the detection confidence of candidate defect boxes is reduced or candidate defect boxes are removed.

[0079] It should be noted that the preset calibration threshold refers to a preset condition used to determine whether the connection relationship statistics are sufficient to support the detection of solder bridging. For candidate defect boxes classified as solder bridging defects, if their connection relationship statistics are less than the preset calibration threshold, it indicates that there is no obvious abnormal solder bridging at the structural level within the selected area. The detection of solder bridging in the main branch lacks structural support, and the candidate defect box is likely a false detection caused by factors such as solder joint reflection or silkscreen texture. In this case, its detection confidence should be reduced or it should be directly removed. Candidate defect boxes whose connection relationship statistics reach the preset calibration threshold have structural support for solder bridging detection and are retained. After this calibration, the detection of solder bridging defects simultaneously meets the conditions at both the appearance recognition and structural relationship levels. False detection boxes with only similar appearances are effectively suppressed, and the stability of solder bridging defect identification is improved.

[0080] According to some embodiments of this application, the solder joint structure prior auxiliary branch includes a coverage auxiliary branch, and the structure prior probability map includes an insufficient coverage probability map; wherein, the probability value of each pixel in the insufficient coverage probability map represents the probability that the solder is insufficiently covered in the solder joint area at the corresponding pixel position. In step S403, the detection confidence of the candidate defect boxes is calibrated based on the pixel probability values ​​in the prior probability map of the structure corresponding to each candidate defect box. This may include: For candidate defect boxes with the defect category of insufficient tin, the pixel probability values ​​of the candidate defect boxes in the corresponding regions of the insufficient coverage probability map are statistically analyzed to obtain the insufficient coverage statistical values. In response to a coverage insufficiency statistic being less than a preset calibration threshold, the detection confidence of candidate defect boxes is reduced or candidate defect boxes are removed.

[0081] It should be noted that the coverage auxiliary branch is another auxiliary branch parallel to the connection relationship auxiliary branch, and its output is an inadequate coverage probability map. The inadequate coverage probability map corresponds to the size of the detected image patch. The pixel probability value at each pixel position in the map is used to characterize the probability that the solder joint area at that pixel position has insufficient solder coverage. The larger the pixel probability value, the more likely that the position is in a state of insufficient solder coverage.

[0082] In some embodiments, for candidate defect boxes with a defect category of insufficient tin, the pixel probability values ​​of the candidate defect boxes in the corresponding area of ​​the insufficient coverage probability map are statistically analyzed to obtain the insufficient coverage statistical value.

[0083] It should be noted that insufficient solder refers to a solder joint with insufficient solder quantity or inadequate wetting area. This type of defect reduces the mechanical strength of the solder joint and may lead to electrical connection problems such as poor contact and cold solder joints. From an image representation perspective, the difference between insufficient solder and normal solder joints is often only reflected in subtle changes in solder coverage and surface morphology. The main detection branch is prone to misjudging solder joints with slightly incomplete morphology as insufficient solder. The coverage auxiliary branch models the structural essence of insufficient solder as insufficient solder coverage or incomplete morphology within the solder joint area and outputs this structural judgment in the form of a pixel-by-pixel probability map, providing a verification basis for candidate defect boxes of insufficient solder independent of the main detection branch.

[0084] It should be clarified that the inadequate coverage statistic is used to reflect the confidence level of the presence of insufficient solder in the area indicated by the candidate defect box from the perspective of solder coverage. The larger the inadequate coverage statistic, the more sufficient the structural basis for judging the area as having insufficient solder. As one implementation method, the average probability value of each pixel in the corresponding area of ​​the candidate defect box in the inadequate coverage probability map can be used as the inadequate coverage statistic of the candidate defect box.

[0085] In some embodiments, in response to an undercover statistic being less than a preset calibration threshold, the detection confidence of the candidate defect box is reduced or the candidate defect box is removed.

[0086] It should be noted that after obtaining the insufficient coverage statistics, these statistics can be compared with a preset calibration threshold. If the insufficient coverage statistics reach the preset calibration threshold, it indicates that the coverage auxiliary branch recognizes the insufficient solder in the candidate defect box from the perspective of solder coverage, and the detection confidence of the candidate defect box can remain unchanged. If the insufficient coverage statistics are less than the preset calibration threshold, it indicates that the area indicated by the candidate defect box lacks structural evidence of insufficient solder coverage, and the candidate defect box has a high risk of false detection. Its detection confidence can be reduced, causing it to be filtered out in subsequent screening, or the candidate defect box can be directly removed. Therefore, the false alarms of insufficient solder caused by the surface morphology and reflectivity differences of the solder joints in the main detection branch can be effectively suppressed, and the reliability of the insufficient solder defect detection results is improved.

[0087] In step S104 of this embodiment, for each local panel image, the candidate defect boxes of each detection image block are combined according to the position of each detection image block in the local panel image to obtain the local defect detection result corresponding to each local panel image. It should be noted that stitching refers to restoring the candidate defect boxes scattered in each detection image block to the range of the local board image according to the position of the detection image block in the local board image; the local defect detection result is the detection result formed by all candidate defect boxes within the range of a local board image. Segmented detection disperses the results in the image range of each detection image block. The result of a single detection image block cannot reflect the defect distribution of the local board area. Stitching according to position reorganizes the block-level results into local image-level results, restores the positional correspondence between candidate defect boxes and board areas, and provides a complete result based on local board images for subsequent organization of detection results across the entire board.

[0088] In step S105 of this embodiment, the relative positional relationship of each local board surface image on the printed circuit board is determined. It is important to clarify that the relative positional relationship refers to the relative orientation and spacing of the board area covered by each local board image on the entire printed circuit board. Each local board image is acquired separately, and each only describes a portion of the board surface; the positional relationship between them cannot be directly obtained from a single image. The embodiments of this application determine the relative positional relationship between each local board image to provide a positional basis for organizing the scattered local images into the entire board area. Subsequent steps can then use this basis to convert the detection results from each local image into a unified processing result across the entire board area.

[0089] According to some embodiments of this application, step S105, determining the relative positional relationship of each local board surface image on the printed circuit board, may include: Displacement estimation is performed on adjacent local plate images to determine the relative displacement between adjacent local plate images; Based on the relative displacement between adjacent local board images, the relative positional relationship of each local board image on the printed circuit board is determined.

[0090] In some embodiments, displacement estimation is performed on adjacent local panel images to determine the relative displacement between adjacent local panel images; It should be noted that adjacent local board images refer to two local board images whose respective covered board areas are adjacent to or partially overlap each other on the printed circuit board. Relative displacement refers to the offset between the board areas covered by two adjacent local board images in the image coordinate system, which can include displacement components along two mutually perpendicular directions. Since each local board image is acquired for a different board area of ​​the printed circuit board, each local board image itself does not carry its absolute position information on the entire printed circuit board. However, there is often overlap in content or continuity in texture between adjacent local board images, which provides a basis for estimating the relative displacement between them through image content.

[0091] It is important to clarify that displacement estimation of adjacent local board images is a crucial step in reorganizing the scattered local board images into a whole-board coordinate description. By determining the relative displacement between each pair of adjacent local board images, the local board images can be sequentially connected along their adjacency relationships, thereby deriving the relative positional relationship of each local board image on the printed circuit board. The relative positional relationship can be represented by the coordinates or arrangement order of each local board image under a unified board coordinate description. Therefore, in subsequent steps, the candidate defect boxes scattered across the local board images have a positional reference for transformation to whole-board coordinates, providing a basis for the assembly of the whole-board candidate defect boxes.

[0092] According to some embodiments of this application, displacement estimation of adjacent local panel images to determine the relative displacement between adjacent local panel images may include: Phase correlation is performed on adjacent local plate images to obtain candidate relative displacements and their confidence levels. In response to a confidence level less than a preset confidence threshold, feature point matching is performed on adjacent local plate images, and the relative displacement is determined based on the feature point matching results.

[0093] It should be noted that phase correlation operation refers to the operation of transforming two adjacent local board images to the frequency domain and determining the translation between the images based on their cross-power spectrum. In printed circuit board images, structures such as pads, traces, and silkscreens appear repeatedly in the overlapping areas of adjacent local board images. Phase correlation operation is sensitive to such overall translation relationships and has a certain robustness to changes in illumination and local reflections. It also has a relatively small computational load, making it suitable for estimating the displacement of a large number of adjacent local board images pairwise under the time constraints of online detection. In addition to the candidate relative displacement, the output of the phase correlation operation also includes the confidence level of that candidate relative displacement, which reflects the reliability of the displacement estimation result.

[0094] It is important to clarify that the confidence level provides a self-checking mechanism for the phase correlation operation results. When the overlapping areas between adjacent local board images have clear textures and sufficient content overlap, the candidate relative displacements given by the phase correlation operation usually have high confidence. However, when the overlapping area is narrow, there are large areas of blank board surfaces without components, or there is interference from reflections or foreign objects, the estimation results of the phase correlation operation may be unreliable, and the confidence level will decrease accordingly. By comparing the confidence level with a preset confidence threshold, unreliable displacement estimation results can be identified, preventing erroneous relative displacements from being propagated along the adjacency relationship and affecting the accuracy of the overall board assembly.

[0095] It should be noted that for adjacent local panel images with a confidence level lower than the preset confidence threshold, displacement estimation is re-performed using feature point matching. Feature point matching refers to extracting discriminative feature points from two adjacent local panel images and establishing a correspondence between the feature points in the two images. After the feature point correspondence is established, the relative displacement can be calculated based on the coordinate differences of each set of corresponding feature points in the two images. Feature point matching does not rely on large-scale texture overlap and can still find reliable correspondences even in scenarios where the overlapping area is sparse or disturbed, thus complementing phase correlation operations in terms of applicable conditions. This two-stage displacement estimation mechanism, with phase correlation operations as the primary method and feature point matching as a supplement, balances the efficiency and reliability of displacement estimation, providing a stable positional reference for the accurate assembly of candidate defect boxes for the entire panel.

[0096] In some embodiments, the relative positional relationship of each local board image on the printed circuit board is determined based on the relative displacement between each adjacent local board image.

[0097] It should be noted that by constructing relative positional relationships based on the relative displacement between adjacent local board images, there is no need to rely on additional mechanical positioning information or pre-defined acquisition paths. Even if the acquisition order and coverage of each local board image differ, as long as there is image content between adjacent local board images that allows for displacement estimation, the relative arrangement of each local board image on the printed circuit board can be reconstructed. This method is suitable for application scenarios in production lines where board specifications vary and the acquisition process is flexible, reducing the dependence of the inspection system on the consistency of acquisition tooling.

[0098] In step S106 of this embodiment, candidate defect frames from each local defect detection result are combined to form several whole-plate candidate defect frames according to their relative positional relationship. It should be noted that the overall candidate defect frame refers to the defect frame formed within the entire printed circuit board area after candidate defect frames are combined, and its position directly corresponds to the position of the defect on the entire printed circuit board. Based on the relative positional relationship determined in step S105, the candidate defect frames from each local defect detection result are uniformly organized into the overall board area, elevating the description of the defect from the local image level to the overall board level; the scattered results generated when the same board area is covered by multiple local board images are thus merged, and the unified confirmation of the entire board area is thus obtained.

[0099] In step S107 of this embodiment, the whole-board level defect confirmation is performed on each whole-board candidate defect box to obtain the whole-board inspection result of the printed circuit board.

[0100] It should be noted that whole-board level defect confirmation refers to the process of verifying and confirming each candidate defect box within the entire printed circuit board. The whole-board inspection result is the final inspection conclusion for the entire printed circuit board after confirmation. The candidate defect box is the initial output of the model, which may contain false detection boxes caused by reflection, image boundary factors, etc. Whole-board level defect confirmation verifies the candidate boxes at the whole-board level, improving the reliability of the final conclusion. The whole-board inspection result obtained after confirmation directly reflects the defect location and defect distribution of the entire printed circuit board, which can support subsequent re-inspection and traceability. Compared with the situation in related technologies where single-frame local inspection results are difficult to form whole-board level quality judgment, the inspection conclusion of the embodiment of this application is guaranteed in terms of completeness and reliability.

[0101] Reference Figure 5 According to some embodiments of this application, step S107 performs whole-board level defect confirmation on each whole-board candidate defect box to obtain the whole-board inspection result of the printed circuit board, which may include: Step S501: Perform whole-board level deduplication and merging on the candidate defect frames of each board that overlap in position to obtain the deduplicated candidate defect frames of the board. Step S502: For each candidate defect box after deduplication, determine the supporting defect box of the candidate defect box of the whole board; wherein, the supporting defect box refers to the candidate defect box whose position overlaps with the candidate defect box of the whole board before deduplication and merging. Step S503: Count the number of local board images that support the defect box to be assigned, and obtain the number of supported candidate defect boxes for the whole board. Step S504: In response to the number of supported items reaching the preset number of supported items threshold, the deduplicated candidate defect boxes of the entire board are retained in the overall board detection result.

[0102] In step S501, the overall candidate defect frame is formed by combining candidate defect frames scattered across various local board images. If a solder joint defect is covered by multiple local board images, multiple overlapping overall candidate defect frames may be formed in the overall board coordinate system. Overall board deduplication and merging merges these overlapping overall candidate defect frames in the overall board coordinate system, ensuring that the same actual defect corresponds to only one overall candidate defect frame after deduplication, thus preventing the same defect from being reported repeatedly in the overall board inspection results.

[0103] In step S502, the supporting defect boxes are used to record the source of candidate defect boxes collected before deduplication and merging, after deduplication of the whole board candidate defect boxes. Each candidate defect box whose position overlaps with the whole board candidate defect box before deduplication and merging is a supporting defect box of that whole board candidate defect box. By retaining the attribution information of the supporting defect boxes, the deduplication and merging process only merges the boxes without losing the detection records of defects at that location in each local board image, providing a data foundation for subsequent statistics based on the local board image dimension.

[0104] In step S503, the number of supports refers to the number of local board images to which the supporting defect box belongs, not the number of supporting defect boxes themselves. If the same solder joint defect is detected in multiple local board images, it indicates that the defect has undergone independent detection from multiple different board viewpoints, and its detection reliability is higher than that of a defect appearing only in a single local board image. Using the number of local board images as the statistical caliber can avoid the duplicate contribution of multiple candidate defect boxes in the same local board image to the statistical results, ensuring that the number of supports truly reflects the number of independent image sources that detected the defect.

[0105] In step S504, the deduplicated candidate defect boxes for the entire board are filtered by a preset support quantity threshold: candidate defect boxes whose support quantity reaches the preset support quantity threshold indicate that the corresponding defect has been detected by multiple local board surface image sources and are retained in the overall board detection result; candidate defect boxes whose support quantity does not reach the preset support quantity threshold pose a risk of false detection in a single frame and are not included in the overall board detection result. Therefore, every defect reported in the overall board detection result has been verified and confirmed across image sources, ensuring the reproducibility and traceability of the detection conclusions, and reducing the interference of false alarms on production line judgment.

[0106] According to some embodiments of this application, the method for detecting solder joint defects on printed circuit boards may further include: Based on the relative positional relationship, the images of each local board surface are stitched together to obtain the whole board stitched image of the printed circuit board; The candidate defect boxes of the entire board are overlaid on the stitched image of the entire board, and the stitched image of the entire board and the detection results of the entire board are stored.

[0107] It should be noted that a full-board image refers to an image that covers the entire surface of a printed circuit board (PCB) after being stitched together from individual partial images of the PCB according to their relative positions. Each partial image covers a different area of ​​the PCB during the acquisition phase, and a single partial image only provides a local view. The full-board image, created by stitching together the images, allows the operator to view the entire PCB from a unified perspective, providing an image basis for PCB-level defect inspection and quality traceability.

[0108] It needs further clarification that overlaying the candidate defect frames onto the overall board image refers to displaying the positions of each candidate defect frame as a selection box on the overall board image. Since the candidate defect frames are now under a unified overall board coordinate system, the actual location of each defect on the board surface is clearly visible after overlay. Operators can grasp the defect distribution of the entire printed circuit board without having to review individual board images. Re-inspection personnel can also quickly locate the defect on the board surface based on the overlay image, thus improving re-inspection efficiency.

[0109] It is important to clarify that storing the complete spliced ​​images and inspection results of each printed circuit board (PCB) is intended to ensure that the inspection process for each PCB has a complete and traceable record. When the production line subsequently disagrees with the inspection conclusions or needs to statistically analyze the quality of historical boards, the corresponding complete spliced ​​images and inspection results can be retrieved for verification, ensuring the traceability of the inspection conclusions. The stored image and result data can also serve as a source of material for model iteration and process analysis, providing data accumulation for the continuous optimization of the inspection system.

[0110] According to some embodiments of this application, the method for detecting solder joint defects on printed circuit boards may further include: The system counts the number of defects in the overall board inspection results; when the number of defects is zero, it generates an overall board judgment result indicating that the printed circuit board is qualified; when the number of defects is greater than zero, it generates an overall board judgment result indicating that the printed circuit board is unqualified.

[0111] It should be noted that the number of defects refers to the number of candidate defect frames included in the overall board inspection results. The overall board inspection results have undergone overall board-level defect confirmation, and each candidate defect frame corresponds to a confirmed solder joint defect. Therefore, the number of defects obtained from the overall board inspection results directly reflects the current solder joint quality status of the printed circuit board. Using the number of defects as the statistical basis for overall board judgment maintains a consistent correspondence between the judgment criteria and the inspection output. The judgment process does not require additional evaluation steps, making the processing flow simple and clear.

[0112] It should be noted that the overall board judgment result is used to characterize whether the printed circuit board is qualified or unqualified. When the defect count is zero, it means that no confirmed solder joint defects were found on the entire printed circuit board, generating an overall board judgment result indicating qualification; when the defect count is greater than zero, it means that there are confirmed solder joint defects on the board, generating an overall board judgment result indicating unqualified. The judgment logic uses whether the defect count is zero as the dividing line. The judgment rules are clear and the output is definite, which makes it easy for the production line to directly perform the diversion operation based on the overall board judgment result. Qualified boards flow to downstream processes, and unqualified boards are transferred to the re-inspection or rework stage.

[0113] It should be understood that the overall board judgment result is based solely on the overall board inspection result. This overall board inspection result itself undergoes multiple stages of processing, including detailed preservation of small-size defects, prior structural calibration, and overall board-level verification. The routing decisions based on the overall board judgment result inherit the reliability of each preceding stage. For the production line, the automatically generated overall board judgment result replaces the subjective judgment based on manual visual inspection of each board. The judgment standards for each board remain consistent, and the inspection cycle is unaffected by fluctuations in personnel condition.

[0114] According to some embodiments of this application, the method for detecting solder joint defects on printed circuit boards may further include: Obtain a template image corresponding to the type of the printed circuit board; wherein, the template image has at least one ignored area pre-marked. Feature point matching is performed between the template image and the printed circuit board, and each ignored region is mapped to the printed circuit board based on the feature point matching results to obtain the target ignored region; The number of defects in the overall board inspection results can be counted by excluding defects whose locations fall within the target ignore area.

[0115] It should be noted that the template image refers to a pre-established reference image corresponding to the type of the printed circuit board (PCB). The template image pre-marks at least one ignore area. Ignore areas refer to areas on the board surface that do not require solder joint defect assessment. As one implementation method, process edges, positioning holes, QR code areas, and pad areas confirmed to be unmanageable can all be marked as ignore areas in the template image. Since ignore areas are pre-marked on the template image on a board-by-board-type basis, the same markings can be reused across PCBs of the same type, eliminating the need for individual settings for each board. When switching board types, only the corresponding template image needs to be changed, thus limiting the debugging workload.

[0116] It should be noted that the ignored regions in the template image are marked below the coordinates of the template image itself. However, the actual position and angle of the acquired printed circuit board in the image may deviate from those in the template image. Directly applying the marked positions from the template image will lead to misalignment of the ignored regions. Therefore, feature point matching is performed between the template image and the printed circuit board. Based on the feature point matching results, a coordinate correspondence between the template image and the current printed circuit board is established. Then, each ignored region is mapped to the printed circuit board according to this coordinate correspondence to obtain the target ignored region. The target ignored region is the actual location of the ignored region on the current printed circuit board. Feature point matching is adaptable to differences in the pose of the board components in the image, such as translation and rotation. Even if the placement of each board in the acquisition field of view is not completely consistent, the mapped target ignored region can still accurately cover the predetermined board surface position.

[0117] It should be understood that when counting defects, defects falling into the target neglect area in the overall board inspection results are first excluded before the defect count is obtained. Although defects falling into the target neglect area may be actual detected solder joint defects, their location is not within the scope of quality control. If not excluded, such defects will be included in the defect count and may trigger a non-conformance judgment, causing qualified boards to be mistakenly judged as non-conformance. By excluding defects in the target neglect area before judgment, the overall board judgment result only reflects the defect status in the control-focused area, and the judgment caliber is consistent with the actual quality standards of the production line, thereby reducing invalid alarms and false judgments.

[0118] According to some embodiments of this application, the method for detecting solder joint defects on printed circuit boards may further include: The system stores multiple partial images of the printed circuit board, the overall board inspection results, and the corresponding inspection time to obtain historical inspection records. In response to the model update command, the selected historical detection records are exported as a sample dataset, and the weld joint defect detection model is incrementally trained based on the sample dataset to obtain the updated weld joint defect detection model.

[0119] It should be noted that historical inspection records refer to the records formed by associating and storing data generated during the inspection of each printed circuit board. These records may include multiple partial images of the circuit board, the overall inspection results, and the corresponding inspection time. The partial images preserve the original image material from the inspection process, the overall inspection results preserve the inspection conclusions, and the inspection time provides a temporal index for each record. Data generated during the inspection process accumulates naturally with the inspection cycle; the establishment of historical inspection records does not add any extra data acquisition burden, as the daily inspection process on the production line itself serves as the sample source.

[0120] It should be noted that the model update command is used to trigger the update process of the weld joint defect detection model. In response to the model update command, a portion of the historical inspection records can be selected and exported as a sample dataset. The historical inspection records simultaneously store local board images and overall board inspection results. During export, the local board images can be directly used as sample images, and the defect information at the corresponding location in the overall board inspection results can be used as the annotation basis, significantly reducing the manual input of re-annotation. The flexibility of selection allows the construction of the sample dataset to be focused on board type, time period, or defect category. For situations where the detection capability is insufficient in the initial stage of a new board type's deployment, or where the missed detection rate of a certain type of defect is high, historical inspection records for the corresponding time period and board type can be selected for export.

[0121] It should be understood that incremental training refers to continuing to train the existing solder joint defect detection model using sample datasets based on its parameters, resulting in an updated solder joint defect detection model. Compared to complete training from scratch, incremental training requires a smaller sample size and less training time, making it suitable for periodic model updates without interrupting production lines. As production line operation time accumulates, the board type, component packaging, and soldering processes of incoming materials may change, and the adaptability of the model trained in the early stages to newly emerging sample forms will gradually decrease. Through continuous accumulation of historical inspection records and repeated execution of incremental training, the solder joint defect detection model can continuously absorb real samples from the production line, ensuring that its detection capabilities evolve in sync with the actual conditions of the production line, thus guaranteeing the long-term effectiveness of the detection system.

[0122] Figure 6 This is a general flowchart of a solder joint defect detection method for printed circuit boards provided in an embodiment of this application. Figure 6 As shown, the detection process in this embodiment is organized into four layers within the solder joint defect detection system: equipment and acquisition layer, AI detection layer, whole-board post-processing layer, and result output and human-computer interaction layer. These layers are connected via queues and events, with acquisition, inference, post-processing, and result output occurring asynchronously and in parallel to match the continuous detection rhythm of the production line. The following is arranged according to… Figure 6 The process flow is explained step by step.

[0123] At the equipment and acquisition layer, after system startup, the system enters the initial detection phase, loading the necessary parameters, pre-trained solder joint defect detection model, and templates corresponding to the board type. Then, the MVS industrial camera is turned on, exposure parameters and acquisition frame rate are set, and continuous frame acquisition begins. Continuous frame acquisition, as described above, involves continuously acquiring multiple target image frames from the printed circuit board; if a blank frame occurs during frame acquisition, the system waits; if a connection error occurs, it automatically reconnects to ensure the continuity of the acquisition process.

[0124] In each target image frame obtained through continuous frame acquisition, the system performs green board frame detection to locate the PCB area and update board tracking. Green board frame detection is the determination of board area based on the board surface color threshold: the image is converted to the HSV color space, candidate areas of the board surface are extracted according to the color threshold of the printed circuit board substrate, and then the board area is obtained by filtering based on area, aspect ratio and contour conditions; board tracking is based on the cross-frame association of board areas in each target image frame acquired in continuous acquisition. The system assigns a board identifier to each printed circuit board that enters the acquisition field of view and determines whether the board is in motion by the inter-frame grayscale difference.

[0125] Then enter Figure 6The judgment step involves determining whether the board area has entered the gate line ROI and is in motion. The gate line ROI (Region of Interest) is the acquisition trigger area pre-defined in the acquisition field of view, as mentioned earlier. If the judgment result is negative, the system waits for the next frame, the interface preview continues to refresh, and no cropping is performed; if the judgment result is positive, the system crops and numbers the ROI, that is, it extracts the image content corresponding to the board area from the current target image frame to obtain a local board image, and numbers the local board image with board_id (board identifier) ​​and board_seq (board acquisition sequence number). The numbering information provides the basis for board ownership and acquisition order for subsequent whole board assembly.

[0126] In the AI ​​detection layer, the captured local board surface image is packaged into a CapturePacket and entered into the ROI queue. Figure 6 The queue decoupling, marked with dashed lines, indicates that data is transferred between the acquisition and inference stages via queues, preventing the processing speed differences between the two stages from blocking each other. The TensorRT YOLO inference service (TiledYoloInferenceService in the diagram) retrieves acquisition packets from the ROI queue and performs inference. TensorRT is NVIDIA's inference acceleration engine, YOLO is a single-stage object detection model, and the inference service is the main operating entity of the solder joint defect detection model. Tiled inference is the image segmentation detection mentioned earlier: the local board image is segmented into multiple detection image blocks, each of which is input into the model. The main defect detection branch outputs candidate defect boxes indicating the defect category and detection confidence, while the solder joint structure prior auxiliary branch outputs a structure prior probability map and calibrates the detection confidence. Then, the block detection results are merged using NMS (Non-Maximum Suppression), which determines the board position of each candidate defect box according to the position of each detected image block in the local board image. The candidate defect boxes whose board positions are in overlapping areas and overlap with each other are deduplicated and merged, and the candidate boxes for solder bridging and insufficient solder are output, thus obtaining the local defect detection results corresponding to each local board image.

[0127] In the whole board post-processing layer, each local board surface image and its local defect detection results are encapsulated into a StitchPacket and enter the stitching queue. ROI incremental stitching is performed to generate a whole board mosaic: local board surface images that arrive one by one as the board moves are incrementally stitched into the whole board image. The whole board mosaic is the whole board stitched image mentioned above. The candidate defect boxes in each local defect detection result are also stitched together under a unified whole board coordinate system to form the whole board candidate defect box. Figure 6The board end judgment step determines whether the board has left the ROI or stopped moving: if not, it returns along the dotted branch to continue accumulating and continuously receiving subsequent local board surface images; if so, it is considered that the acquisition of the board has ended, and a corresponding acquisition end signal carrying the board identifier is generated, namely BoardEndPacket in the figure.

[0128] After the board is completed, a whole-board defect confirmation is performed. The category-level NMS plus the minimum support number marked in the figure is the specific implementation of the whole-board level defect confirmation mentioned above: Under the whole-board coordinates, the candidate defect boxes with overlapping positions are deduplicated and merged at the whole-board level according to the defect category. The number of local board images to which the candidate defect boxes with overlapping positions belong before deduplication and merging are counted to obtain the support number. The whole-board candidate defect boxes with the support number reaching the minimum support number (i.e., the preset support number threshold) are retained in the whole-board detection results. Then, the ignore area judgment step is entered: If enabled, template feature matching is performed to map the ignore area and delete the specified defects. That is, the template image corresponding to the board type is obtained, the template image is matched with the current printed circuit board, the pre-marked ignore areas in the template image are mapped to the current board to obtain the target ignore area, and the defects in the whole-board detection results that fall into the target ignore area are excluded; if the ignore area is not enabled, the statistical results step is directly entered. The statistical results stage counts the number of defects in the overall board inspection results, which is further subdivided into solder bridging and insufficient solder in the graph. Based on this, a preliminary OK / NG conclusion is formed, representing the overall board judgment result indicating whether the printed circuit board is qualified or unqualified. After the statistics are completed, the system generates a BoardResultEvent (overall board result event) and transmits the result to the result output and human-computer interaction layer.

[0129] In the result output and human-computer interaction layer, the interface displays a real-time camera preview, a full-board annotation image, and a status log. The full-board annotation image is a composite image of the entire board with candidate defect boxes overlaid on it. The result judgment process is categorized based on the number of defects: if there are no defects, the judgment result is OK, indicating that the printed circuit board is qualified, the count is updated, and the inspection continues for the next board; if there are defects, the judgment result is NG, indicating that the printed circuit board is unqualified, triggering an audible and visual alarm and prompting on-site personnel to take action.

[0130] It should be noted that, Figure 6The output results and the two asynchronous saves in the human-computer interaction layer correspond to the storage of historical inspection records mentioned earlier: one path writes the inspection results to the database record (InspectionRepository in the diagram), and the other path asynchronously archives the original image of the entire board, YOLO labels, and the matching results of ignored areas into a JSON (a lightweight data exchange format) file. Asynchronous saving ensures that the storage operation does not block the main inspection process; the archived original image of the entire board and YOLO labels, i.e., image data with defect location and category annotations, can serve as the data foundation for subsequently exporting sample datasets and incrementally training the weld joint defect detection model. After receiving the stop inspection command, the system enters the shutdown phase, releases camera and thread resources, and terminates the inspection process.

[0131] Figure 7 A flowchart illustrating the board inspection and tracking method provided in this application embodiment shows... Figure 6 A detailed process for locating and triggering components in the middle-level equipment and acquisition layer. For example... Figure 7 As shown, after the detection begins, the camera continuously acquires PCB images, which is the process described above of continuously acquiring multiple target image frames for the printed circuit board.

[0132] For each target image frame, the system identifies the PCB board area: extracts the green board outline using HSV (hue, saturation, brightness) thresholds. Figure 7 The diagram below illustrates the processing effect of this step using a threshold-extracted board surface mask. The board surface area appears as a bright green in the mask. This step is the process described earlier of extracting candidate board surface regions from the target image frame based on a preset board surface color threshold, and then determining the board area through contour filtering: The substrate of the printed circuit board is usually green. After converting the target image frame to the HSV color space, the board surface color threshold is set according to the hue range of the green substrate, which can separate the candidate board surface regions from the background. Then, the contours of the candidate board surface regions are filtered according to preset geometric conditions such as area and aspect ratio. The candidate board surface regions whose contours meet the conditions are determined as the board areas.

[0133] The system then proceeds to determine whether a PCB board has been detected. If no PCB board is detected in the current target image frame, the system follows the dotted branch to the intermediate state of "not detected, continue waiting for the next frame," and returns to the frame acquisition stage, repeating the above identification for subsequent target image frames. If a PCB board is detected, the system updates the PCB position: associating the current board frame with the position of the previous frame to achieve continuous tracking. This stage is the process of cross-frame association of board areas in each target image frame, as described above. By grouping board areas with similar positions in adjacent target image frames to the same printed circuit board, the system maintains continuous tracking for each board entering the acquisition field of view and assigns a board identifier, avoiding the same board being counted repeatedly in multiple image frames.

[0134] It should be noted that the introduction of cross-frame association upgrades board localization from single-frame detection to multi-frame tracking. Single-frame detection identifies boards independently in each frame, and the results between frames do not affect each other. It is difficult for boards to maintain the same identity when they move in the field of view. Cross-frame association uses the positional relationship between frames as a link to keep the board area of ​​the same board associated in consecutive frames. Subsequent gate triggering and board end judgment are all performed based on this association relationship.

[0135] In tracking mode, the system generates a detection ROI (Region of Interest): the area to be detected is cut out according to the PCB location. Figure 7 The diagram above shows a cropping frame superimposed on a real-time image of the board surface, illustrating how the ROI detection is generated. The cropping frame is determined by the position of the board area. The next step is to determine whether the PCB has entered the center gate line. The center gate line is the acquisition trigger area pre-defined in the acquisition field of view, as mentioned earlier, and is located in the center of the acquisition field of view. If the board area has not entered the center gate line, the system enters an intermediate state of "not entering the gate line, only updating the preview and position" along the dotted line branch, only refreshing the interface preview and board position without performing acquisition, and returns to the frame acquisition stage; if the board area has entered the center gate line, the motion judgment stage begins.

[0136] It should be noted that using the center gate line as the acquisition trigger condition is to avoid the same board being acquired prematurely or repeatedly at the edge of the field of view. When the board just enters the edge of the field of view, only a part of the board surface is exposed. At this time, the captured partial board surface image is incomplete and has a large amount of overlap with the content captured later. By defining the acquisition trigger area in the middle of the acquisition field of view, and triggering the capture after the board area enters the middle with the conveyor movement, the resulting partial board surface image has more complete coverage and a more uniform acquisition density.

[0137] In the motion detection stage, the system determines whether the PCB is moving: it compares the changes in ROI between adjacent frames, that is, as mentioned earlier, it determines the motion state of the printed circuit board based on the position changes of the board area associated across frames in each target image frame. If the position of the board area changes in adjacent frames, it indicates that the board is moving. At this time, the board enters the gate line and is moving. The system acquires the ROI of that frame, that is, it extracts the image content corresponding to the board area from the current target image frame to obtain a local board image and proceeds to subsequent defect detection. If the position of the board area does not change in adjacent frames, it indicates that the board is not moving or has stopped. The system accumulates still frames. When the number of still frames reaches a threshold, it is considered that the board is finished, that is, an acquisition end signal carrying the board identifier is generated, and the acquisition and stitching process of the board is terminated. The detection results are transferred to the whole board post-processing stage.

[0138] It should be noted that by judging the motion state based on the changes in ROI between adjacent frames and determining the end of the board acquisition based on the accumulation of still frames, the start and end control of the acquisition is entirely based on the image content. This eliminates the need to install additional sensors such as encoders and photoelectric switches on the conveying mechanism, thus reducing the need for modifications to existing post-welding stations. The board acquisition is only terminated when the accumulated still frames reach a threshold, rather than upon assuming a single still frame. This avoids misjudgments caused by brief jitter or imaging fluctuations during board transport, ensuring complete board coverage before ending the acquisition.

[0139] Figure 8 This is a schematic diagram of some solder joint defects, such as bridging and insufficient solder, collected on an actual production line by the detection system of this application. Figure 8 As shown in the figure, there are six real-world images of partial board surfaces taken by an image acquisition device. Each image covers a different area of ​​the printed circuit board. The blue arrows indicate the location of solder bridging defects, and the red arrows indicate the location of insufficient solder defects.

[0140] It should be noted that, from Figure 8 As indicated by the blue arrow, solder bridging defects manifest as abnormal solder bridging between adjacent solder joints or pins: excess solder accumulates and adheres between adjacent solder joints, forming irregular solder clumps or bands with a strong reflective surface and fine, fragmented edges. Once this type of bridging occurs, it creates an electrical short circuit between adjacent pins, directly affecting the overall electrical function of the device.

[0141] It should be noted that, from Figure 8 As indicated by the red arrow, insufficient solder defects manifest as inadequate solder quantity or insufficient wetting area at the solder joint: the solder coverage at the joint is not full, the pad or lead is partially exposed, the solder joint outline is incomplete, and the difference in shape from surrounding normal solder joints is only slight. Such defects can lead to a decrease in the mechanical strength of the solder joint and may cause electrical connection problems such as poor contact and cold solder joints.

[0142] It should be noted that, Figure 8 This also intuitively reflects the detection difficulties that this application aims to solve. On the one hand, bridging and insufficient solder defects occupy a small pixel range in the image. If the image of the entire printed circuit board is scaled up to the model input size, the details of such small defects are easily lost. The processing method of this application, which segments the local board image and inputs the solder joint defect detection model in units of detection image blocks, is precisely to ensure that such small defects retain sufficient pixel details in the model input. On the other hand, bridging and insufficient solder defects have similar textures and reflective properties to normal solder joints, making it difficult to reliably distinguish them based solely on the detection frame level. This application introduces a priori auxiliary branch for solder joint structure, calibrating the detection results from two structural dimensions: whether there is abnormal solder bridging between adjacent solder joints and whether the solder coverage within the solder joint area is sufficient. This is precisely to address the challenges of bridging and insufficient solder defects. Figure 8 The design was based on the characteristic that the defect morphology shown is similar to that of a normal solder joint.

[0143] It should be noted that, Figure 8 The sample images shown are all collected by the detection system during the production line inspection process. According to the historical inspection record mechanism of this application embodiment, such defective images are continuously accumulated and archived during the inspection process. After the defect boxes and solder joint structural anomalies are marked, they can be added to the sample dataset as sample image blocks for incremental training of the solder joint defect detection model, so that the model's ability to recognize the real defect morphology on the production line can be continuously improved over time.

[0144] Figure 9 This is a schematic diagram illustrating continuous acquisition of a partial panel image according to an embodiment of this application. Figure 9 As shown, the multiple images arranged horizontally in the figure are a series of partial images of the same printed circuit board captured by the system at different acquisition times as the board moves continuously through the field of view of the image acquisition device.

[0145] from Figure 9 The changes in content across the images show that the board enters the field of view from the left, with the initial cropped images only covering the edge area. As the board continues to move, subsequent cropped images gradually cover the central area, and the positions of the circular pads, connectors, and silkscreen markings on the board shift sequentially across images. In the rightmost images, the board is nearly out of the field of view, and the cropped images again only cover the other edge. Each cropped image corresponds to a different area of ​​the board, and there is overlap in the areas covered by adjacent cropped images. The same circular pad appears repeatedly in multiple adjacent images, with its position shifting sequentially.

[0146] It should be noted that, Figure 9 The continuous acquisition method shown embodies the processing approach of this application, which replaces a single full-board image with multiple local acquisitions. Each local board image covers a local area of ​​the board surface at the native resolution at the time of acquisition. Small targets such as solder joints retain sufficient pixel details in the image, avoiding the loss of detail caused by scaling the entire board to the model input size after a single image is captured. The cropping area of ​​the local board image is determined based on the board area, the acquisition field of view, and the preset acquisition width. As one implementation method, the cropping width can be 0.2 times the width of the full image frame, and the cropping height can be determined based on the height of the board area plus the margin of the board edge.

[0147] It should be noted that the overlap of content between adjacent local board images provides a basis for whole-board stitching. The same board structure appears repeatedly in adjacent images but at different positions. Based on this, the system can estimate the relative displacement between adjacent local board images, and thus reconstruct the relative positional relationship of each local board image on the printed circuit board. The same solder joint defect may also be covered and independently detected by multiple adjacent local board images, providing a data foundation for statistically analyzing the number of defects based on the source of local board images in whole-board level defect confirmation. Acquisition triggering can be performed at fixed frame intervals or fixed time intervals. As one implementation method, the system can perform board region detection every several image frames and set a minimum time interval between two adjacent captures to achieve a balance between coverage density and inference pressure.

[0148] It should be noted that, Figure 9 The acquisition process shown is completed while the workpiece is in continuous motion, without pausing the conveyor or clamping the workpiece onto a special fixture. The detection system can be directly deployed above the existing conveyor path of the post-welding station. Throughout the entire process from when the workpiece enters the acquisition field of view to when it leaves, the capture action continues as the workpiece moves, ending when the workpiece area leaves the acquisition trigger area or the workpiece stops moving. Figure 7 The start and end controls of the board detection and tracking process shown are coordinated to form a complete single board acquisition cycle.

[0149] Figure 10 This is a schematic diagram of partial panel image segmentation and detection according to an embodiment of this application. Figure 10 As shown, Figure 10 The left side shows a complete partial image of the plate and its segmentation results. Figure 10 The middle and right sides respectively show the image tiling detection process and the mapping relationship between the detected image block coordinates and the local panel image coordinates.

[0150] like Figure 10As shown on the left, a patch coordinate system is established with the top-left corner of the local panel image as the origin. The horizontal axis (x) of the coordinate system is along the image width direction, and the vertical axis (y) is along the image height direction. The original size of the local panel image in the figure is 600 pixels by 2556 pixels. The system divides the local panel image sequentially along the vertical axis according to the preset image patch size of 600 pixels by 600 pixels, resulting in five detection image patches (labeled Tile 1 to Tile 5 in the figure). Among them, the first three detection image patches are all standard sizes of 600 pixels by 600 pixels. When segmenting to the end, the remaining area with a size smaller than the preset size threshold is merged into the previous detection image patch. In the figure, the size of Tile 4 is 600 pixels by 800 pixels, which is the detection image patch formed after the remaining area at the end is merged into the previous detection image patch. The size of Tile 5 is 600 pixels by 456 pixels. Among them, the size of Tile 5 is not smaller than the preset size threshold, so it is retained as an independent detection image patch. An overlap band is preserved between adjacent detection image blocks, namely the overlapping area between adjacent detection image blocks as mentioned above, to avoid missing edge defects.

[0151] It should be noted that the overlapping band between adjacent detection image blocks is a preventative design to address the incomplete morphology of defects at the segmentation boundary. If a solder joint defect happens to be located at the segmentation boundary, it will only retain a partial morphology in a single detection image block, reducing the model's stability in recognizing incomplete morphologies. With the overlapping band, solder joint defects located at the boundary can appear in a more complete form in two adjacent detection image blocks, ensuring that at least one detection image block provides a complete, identifiable defect morphology, thus reducing the risk of missed detections at the segmentation boundary. As one implementation method, the preset image block size can be 600 pixels by 600 pixels, and the width of the overlapping band can be 100 pixels.

[0152] like Figure 10 As shown in the middle, the image detection process is as follows: First, input the complete Patch, that is, input the complete local board surface image; Second, slide the segmentation according to the fixed window, and send each Tile to the model for detection separately, that is, each detection image block is input into the solder joint defect detection model, and the model performs inference independently on each detection image block; Third, output the detection box within the Tile, with the coordinates taking the upper left corner of the Tile as the origin, that is, the candidate defect box output by the model is initially located below the local coordinates of its respective detection image block, with the upper left corner of the detection image block as the origin of the coordinates.

[0153] like Figure 10As shown on the right, the mapping from Tile coordinates to Patch coordinates is used to restore the candidate defect boxes to the coordinates of the local board image: An offset from the top-left corner of the Tile is added to the Tile detection box, and a coordinate translation is performed. The mapping relationship is x_patch = x_tile + tile_left, y_patch = y_tile + tile_top, where x_tile and y_tile are the coordinates of the candidate defect box in the local coordinates of the detection image block, tile_left and tile_top are the coordinates of the top-left corner of the detection image block in the Patch coordinate system, and x_patch and y_patch are the coordinates of the mapped candidate defect box in the Patch coordinate system. After the coordinate translation, the candidate defect boxes output by each detection image block are unified to the coordinates of the local board image, thus determining the board position of each candidate defect box in the local board image and providing a positional reference for the stitching of candidate defect boxes.

[0154] It should be noted that, Figure 10 The bottom also indicates the post-processing logic for overlapping areas: duplicate boxes may appear within overlapping areas, and these are merged and deduplicated during the post-processing stage. Because there are overlapping areas between adjacent detection image blocks, the same solder joint defect located within an overlapping area may be detected by two adjacent detection image blocks, forming two candidate defect boxes pointing to the same defect. After unifying the candidate defect boxes to the local board surface image coordinates, the system deduplicates and merges candidate defect boxes whose board surface positions are located in overlapping areas, ensuring that only one candidate defect box is retained in the local defect detection results for the same defect, thus avoiding duplicate counting of the same defect.

[0155] Figure 11 This is a schematic diagram of the basic network structure used in the weld joint defect detection model of this application embodiment. Figure 11 As shown, the basic network is divided into a backbone (labeled as Backbone in the figure), a feature fusion part (labeled as Neck in the figure), and a detection output part (labeled as Head in the figure) in sequence according to the data flow direction: the detection image block is input into the backbone, and after feature extraction layer by layer, it is sent to the feature fusion part for multi-scale fusion. The detection output part outputs the detection result based on the multi-scale features obtained by fusion.

[0156] like Figure 11As shown on the left, the main body consists of eleven sequentially connected processing modules: the first two convolutional modules (labeled Conv in the figure), the third cross-stage feature extraction module (labeled C3K2 in the figure), the fourth convolutional module, the fifth cross-stage feature extraction module, the sixth convolutional module, the seventh cross-stage feature extraction module, the eighth convolutional module, the ninth cross-stage feature extraction module, the tenth spatial pyramid pooling module (labeled SPPF in the figure), and the eleventh cross-stage partial spatial attention module (labeled C2PSA in the figure). The detected image patches are processed layer by layer in the main body. The convolutional modules perform convolution and downsampling, gradually reducing the size of the feature map and increasing the receptive field layer by layer. The cross-stage feature extraction module splits and reassembles the features, the spatial pyramid pooling module fuses contextual information from different receptive fields, and the attention module enhances the responses in the features related to the target region. After layer-by-layer extraction in the main body, the detected image patches are converted into multi-layer features, corresponding to the multi-layer feature extraction process described earlier.

[0157] It should be noted that the multi-layer features output by the backbone differ in resolution and semantic depth: shallow features have higher resolution, preserving details such as solder joint outlines and solder surfaces, but lower semantic depth; deep features have lower resolution and higher semantic depth, but the response to small defects is easily weakened during downsampling. For defects such as bridging and insufficient solder, which are small and have fragmented boundaries, relying solely on deep features leads to insufficient detail, while relying solely on shallow features lacks semantic discriminative ability. Feature fusion is necessary to balance information from different levels.

[0158] like Figure 11 As shown in the middle section, the feature fusion part performs multi-scale fusion of the multi-layer features output from the backbone. The feature fusion part includes the 12th upsampling module (labeled Upsample in the figure), the 13th feature concatenation module (labeled Concat in the figure), the 14th cross-stage feature extraction module, the 15th upsampling module, the 16th feature concatenation module, the 17th cross-stage feature extraction module, and the 18th convolution module, the 19th feature concatenation module, the 20th cross-stage feature extraction module, the 21st convolution module, the 22nd feature concatenation module, and the 23rd cross-stage feature extraction module. Modules 12 to 17 constitute a fusion path from deep to shallow: deep features are upsampled and then concatenated with shallow features of the same resolution as the backbone along the channel dimension, and then integrated by the cross-stage feature extraction module, transferring deep semantic information layer by layer to high-resolution features. Modules 18 to 23 constitute a reverse fusion path from shallow to deep: the fused high-resolution features are downsampled by convolution and then passed back layer by layer, supplementing deep features with shallow detail information. After bidirectional fusion, the feature fusion part outputs fused features at multiple scales, which corresponds to the processing described above of multi-scale fusion of extracted multi-layer features to obtain fused features at multiple scales.

[0159] like Figure 11 As shown on the right, the detection output section includes three detection heads (labeled "Detect" in the figure), each receiving fused features at three different scales from the feature fusion section and independently outputting its prediction result. The downsampling ratios of the fused features at the three scales are different; a smaller downsampling ratio results in a higher resolution feature map and a more comprehensive response to smaller targets, while a larger downsampling ratio results in a larger receptive field and a more comprehensive representation of larger targets. Each detection head outputs a defect bounding box, defect category, and detection confidence score based on the fused features at its corresponding scale, constituting the detection result of the basic network. Figure 11 The output diagram in the upper right corner shows the effect of overlaying detection boxes onto the input image.

[0160] It should be noted that, Figure 11 The detection head of the basic network shown typically receives the outputs of three feature layers: P3, P4, and P5. For solder joint defects, the downsampling factor of the P3 layer, which has the highest resolution in the basic network, is still too large, resulting in limited response areas for small defects such as bridging and insufficient solder on its feature map. The solder joint defect detection model in this embodiment is based on this basic network, further introducing high-resolution fusion features with a smaller downsampling factor, and adding a priori auxiliary branch for solder joint structure on top of the high-resolution fusion features. The overall improved structure of the model combines… Figure 12 illustrate.

[0161] Figure 12 This is a schematic diagram of the overall structure of the weld joint defect detection model according to an embodiment of this application. Figure 12 As shown, after the detected image patch (labeled Tile in the figure) is input into the model, it is sequentially processed by the YOLO11s backbone network (labeled YOLO11sBackbone in the figure) for multi-level feature extraction, then fused at multiple scales by the feature pyramid and path aggregation network (labeled FPN / PAN Neck in the figure), and finally the detection result is output by the multi-task output head. In the figure, dashed arrows indicate the direction of supervision signal flow during the training phase, and solid arrows indicate the direction of data flow during the inference phase.

[0162] like Figure 12 As shown in the upper left, the feature fusion component, composed of FPN (Feature Pyramid Network) and PAN (Path Aggregation Network), outputs fused features at four scales: P2, P3, P4, and P5, with corresponding downsampling ratios of 1 / 4, 1 / 8, 1 / 16, and 1 / 32, respectively. P2 is the high-resolution fused feature with the smallest downsampling ratio, meaning it's the path with the smallest downsampling ratio among the multiple scale fused features mentioned earlier.

[0163] It should be noted that, compared to Figure 11 The basic network shown has three outputs: P3, P4, and P5. This embodiment adds a high-resolution feature layer, P2. P2 has a downsampling factor of 1 / 4, and its feature map resolution is twice that of P3. Small defects such as bridging and insufficient solder retain a larger response area and more complete details on the P2 feature map. With the addition of the P2 layer, the features of small defects are preserved in the fusion path and transmitted to the detection head, thus enhancing the model's ability to detect small target defects.

[0164] like Figure 12 As shown in the lower left, the multi-task output head includes three branches: the detection head performs defect detection based on the four-way fusion features of P2, P3, P4, and P5, outputting the defect bounding box, defect category, and detection confidence score, which is the main defect detection branch mentioned earlier; the connectivity prior head consists of a convolutional module and a sigmoid activation function (labeled Conv+Sigmoid in the figure), outputting a connectivity probability map based on the P2 high-resolution fusion features, which is the connectivity auxiliary branch mentioned earlier; the coverage prior head also consists of a convolutional module and a sigmoid activation function, outputting a coverage insufficiency probability map based on the P2 high-resolution fusion features, which is the coverage auxiliary branch mentioned earlier. The two prior heads are connected to the P2 feature layer by dashed arrows, indicating that the inputs of both prior heads are taken from the P2 high-resolution fusion features.

[0165] It should be noted that both prior heads are connected after the P2 high-resolution fusion feature because the determination of the solder joint structure status depends on pixel-level details: whether there is solder bridging between adjacent solder joints, and whether the solder coverage within the solder joint area is sufficient, all need to be judged based on the subtle features of the solder joint edges and solder distribution. Deep low-resolution features are difficult to retain such details. The Sigmoid activation function normalizes the output of the prior head to a value between zero and one pixel, so that the probability values ​​of each pixel in the connectivity probability map and the insufficient coverage probability map represent the probability of abnormal solder bridging and the probability of insufficient solder coverage at the corresponding pixel position, respectively.

[0166] like Figure 12As shown in the upper right, the processing on the inference output side is as follows: the detection head outputs candidate defect boxes, defect categories, and detection confidence; the connectivity probability map output by the connectivity prior head is statistically analyzed using the mean of the in-box probability map to obtain the connectivity statistics of the candidate defect boxes (labeled as connectivity confidence in the figure); the coverage insufficiency probability map output by the coverage prior head is statistically analyzed using the mean of the in-box probability map to obtain the coverage insufficiency statistics of the candidate defect boxes (labeled as coverage insufficiency confidence in the figure). Then, prior constraint confidence calibration and Tile-level NMS are performed: for candidate defect boxes with a tin-bonding defect category, the detection confidence is calibrated based on their connectivity statistics; for candidate defect boxes with a tin deficiency defect category, the detection confidence is calibrated based on their coverage insufficiency statistics; after calibration, non-maximum suppression (NMS) is performed on the candidate defect boxes within the detected image block, outputting Tile-level detection results, i.e., candidate defect boxes for tin-bonding defects and tin deficiency defects. The statistical method for the mean of the in-box probability map is the specific implementation of averaging the pixel probability values ​​of the corresponding regions in the probability map for the candidate defect boxes, as described above.

[0167] like Figure 12 As shown on the right, the supervision signal during the training phase includes three parts: the detection loss corresponding to the detection head (labeled Ldet in the figure), which uses defect box annotations to supervise the output of defect boxes, defect categories, and detection confidence; the connection relationship loss corresponding to the connection relationship prior head (labeled Lconn in the figure), which uses Focal Loss to supervise the output of the connection relationship probability map using a connection relationship supervision graph; and the coverage deficiency loss corresponding to the coverage prior head (labeled Lcov in the figure), which also uses Focal Loss to supervise the output of the coverage deficiency probability map using a coverage deficiency supervision graph. The connection relationship supervision graph and the coverage deficiency supervision graph are generated using the method described above for annotating weld joint structures. These three parts of the loss together constitute the total loss for joint training.

[0168] It should be noted that both prior heads are trained using Focal Loss. This is because of the imbalance between positive and negative samples in solder joint structural anomaly annotations: the areas with abnormal solder bridging or insufficient solder coverage in the sample image patch are far fewer than normal areas. Ordinary pixel-wise cross-entropy loss is easily dominated by the majority of negative samples, causing the prior head to tend to predict all locations as normal. Focal Loss reduces the loss weight of easily classified samples, focusing the training process on difficult-to-classify positive sample regions, thus ensuring the prior head's responsiveness to structural anomaly locations.

[0169] Figure 13 This is a schematic diagram illustrating the whole-board splicing and whole-board-level defect confirmation of an embodiment of this application. For example... Figure 13 As shown, Figure 13 The upper part shows a series of partial panel images labeled "Patch Sub-images". Figure 13 The lower left section shows the assembled result of the entire panel, labeled "Mapping Diagram". Figure 13 The bottom right section shows the deduplication and merging results labeled "NMS diagram".

[0170] like Figure 13 As shown in the upper part, the partial board surface images are arranged sequentially along the horizontal direction. They are captured by the same printed circuit board during continuous movement through the acquisition field of view. Each partial board surface image covers a different area of ​​the board, and there is content overlap between adjacent partial board surface images. The circular pad area on the board surface appears repeatedly in multiple adjacent images.

[0171] like Figure 13 As shown in the lower left, the mapping diagram is the overall board image obtained by stitching together the local board images according to their relative positions. It also shows the effect of overlaying the candidate defect boxes from the local defect detection results of each local board image onto the overall board coordinates according to their relative positions. Specifically, the process involves: estimating the displacement of adjacent local board images to determine their relative positions on the printed circuit board; incrementally stitching each local board image into the overall board image according to their relative positions to form a complete board image covering the entire board; and then stitching the candidate defect boxes from each local defect detection result together to form the overall board candidate defect boxes. The mapping diagram shows that the complete board image fully restores the overall board appearance, and each candidate defect box is distributed at its corresponding position on the board coordinates. Furthermore, in some areas, multiple candidate defect boxes overlap, meaning that the same solder joint defect is detected by multiple adjacent local board images, resulting in duplicate boxes.

[0172] like Figure 13 As shown in the lower right corner, the NMS diagram illustrates the effect of performing whole-board level defect confirmation on the whole-board candidate defect boxes in the mapping diagram: In the whole-board coordinate system, overlapping whole-board candidate defect boxes are deduplicated and merged at the whole-board level, ensuring that only one whole-board candidate defect box is retained for each defect. After a support quantity verification, whole-board candidate defect boxes with a support quantity reaching a preset support quantity threshold are retained in the whole-board detection result. Compared to the mapping diagram, overlapping duplicate boxes in the NMS diagram have been merged and cleared, resulting in clear and unique box selections for each defect, constituting the final whole-board detection result.

[0173] It should be noted that the comparison between the mapping image and the NMS image intuitively demonstrates the role of whole-board level defect confirmation. While the overlap of content between local board images ensures the integrity of board coverage, it also leads to the repeated detection of defects in overlapping areas. If the candidate defect boxes in the mapping image are directly output as the detection results, the same defect will be reported repeatedly, and the number of defects on which the whole-board determination is based will be distorted. After whole-board level deduplication and merging and support quantity verification, the whole-board detection results in the NMS image not only eliminate duplicate boxes, but also use the independent detection records of multiple local board images to verify each defect across image sources, ensuring the accuracy and repeatability of the detection conclusions.

[0174] It should be noted that, Figure 13 The spliced ​​image of the entire board shown serves as the data foundation for result output and human-computer interaction: the candidate defect boxes of the entire board are overlaid on the spliced ​​image, which is... Figure 6 The interface shown displays a full-panel annotation image, allowing operators to view the defect distribution within the entire panel. Storing the spliced ​​full-panel image along with the full-panel inspection results creates historical inspection record image material, supporting subsequent re-inspection and traceability, as well as sample dataset export.

[0175] The solder joint defect detection system for printed circuit boards according to the embodiments of this application includes an image acquisition device, a light source, an edge computing device, and a display device; Image acquisition equipment is used to acquire images of printed circuit boards; The light source is used to provide illumination for the field of view of the image acquisition device; An edge computing device is used to perform any of the solder joint defect detection methods for printed circuit boards according to any of the embodiments of this application; The display device is used to display the overall inspection results of the printed circuit board.

[0176] It should be noted that the image acquisition device is the hardware unit in the system that acquires images of the board surface. It can be mounted above the printed circuit board transport path, ensuring its field of view covers the area through which the board passes. As one implementation method, the image acquisition device can use an area-array industrial camera to continuously capture images of the field of view at a fixed frame rate, outputting a continuous sequence of image frames. After the image frame sequence output by the image acquisition device is located and triggered for cropping, it forms a partial image of the board surface, providing image input for subsequent defect detection.

[0177] On the other hand, the light source is used to illuminate the field of view of the image acquisition device. The reflective properties of the solder joint surface have a significant impact on image quality. Unstable lighting conditions can cause solder joints of the same shape to appear differently in different images, interfering with the judgment of the detection model. As one implementation method, the light source can be a strip light source or a ring light source arranged around the field of view, so that the illumination within the field of view is uniform and stable, and the outline of the solder joint and the surface morphology of the solder are clearly presented in the image, providing a stable image input for the detection model.

[0178] On the other hand, the edge computing device is the system's processing unit, deployed on the production line to execute the solder joint defect detection method for printed circuit boards according to any of the embodiments of this application. Image processing, model reasoning, board assembly, and result judgment are all performed locally on the edge computing device. Detection data does not need to be uploaded to a remote server, the detection cycle is not affected by network transmission conditions, and production line data is also retained within the production line. Compared to integrated detection equipment that combines an independent conveyor mechanism with a large host, the edge computing device is compact and can be deployed at existing post-soldering stations with image acquisition equipment and a light source, resulting in lower overall system modification costs and lower footprint requirements.

[0179] It needs to be clarified that the display device is used to display the overall inspection results of printed circuit boards. As one implementation, the display device can show a composite image of the entire board, overlaid with candidate defect frames, and simultaneously display the overall board judgment results, allowing operators to view the defect distribution and pass / fail conclusions for each printed circuit board at the workstation. The display device is connected to edge computing devices, and the display content is refreshed immediately after inspection, meeting the cycle time requirements of online inspection.

[0180] It is evident that the contents of the above embodiments of the solder joint defect detection method for printed circuit boards are all applicable to the embodiments of the solder joint defect detection system for printed circuit boards. The specific functions implemented by the embodiments of the solder joint defect detection system for printed circuit boards are the same as those of the above embodiments of the solder joint defect detection method for printed circuit boards, and the beneficial effects achieved are also the same as those achieved by the above embodiments of the solder joint defect detection method for printed circuit boards.

[0181] Reference Figure 14 , Figure 14 This illustration shows the hardware structure of an electronic device according to another embodiment. The electronic device may include: The processor 1401 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1402 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1402 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1402 and is called and executed by the processor 1401 to execute the solder joint defect detection method for printed circuit boards according to the embodiments of this application. The input / output interface 1403 is used to implement information input and output; The communication interface 1404 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1405 transmits information between various components of the device (e.g., processor 1401, memory 1402, input / output interface 1403, and communication interface 1404); The processor 1401, memory 1402, input / output interface 1403 and communication interface 1404 are connected to each other within the device via bus 1405.

[0182] This application also provides a computer program product, which includes a computer program. A processor of a computer device reads and executes the computer program, causing the computer device to perform the above-described method for detecting solder joint defects on a printed circuit board.

[0183] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0184] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0185] It should be understood that in the description of the embodiments of this application, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0186] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0187] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0188] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0189] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium may include: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media capable of storing program code.

[0190] It should also be understood that the various implementation methods provided in this application can be combined arbitrarily to achieve different technical effects.

[0191] The above is a detailed description of the embodiments of this application. However, this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for detecting solder joint defects on printed circuit boards, characterized in that, include: Images are acquired from different areas of the printed circuit board to obtain multiple local images of the board surface; Image segmentation is performed on each of the local plate surface images to obtain multiple detection image blocks corresponding to each of the local plate surface images; For each local board surface image, the corresponding detection image blocks are input into a pre-trained solder joint defect detection model to obtain candidate defect boxes corresponding to each detection image block; For each of the local panel images, the candidate defect boxes of each of the detected image blocks are combined according to their positions in the local panel image to obtain the local defect detection results corresponding to each local panel image. Determine the relative positional relationship of each of the local board surface images on the printed circuit board; According to the relative positional relationship, the candidate defect boxes in each of the local defect detection results are combined to form several whole plate candidate defect boxes; The entire board-level defect confirmation is performed on each of the candidate defect boxes to obtain the overall board inspection results of the printed circuit board.

2. The method for detecting solder joint defects on printed circuit boards according to claim 1, characterized in that, The process involves acquiring images of different areas of the printed circuit board to obtain multiple local images, including: Multiple target image frames are obtained by continuously acquiring data from the printed circuit board. The printed circuit board is positioned according to each of the continuously acquired target image frames to determine the board area in each target image frame; wherein, the board area refers to the image area occupied by the board surface of the printed circuit board in the target image frame; For each target image frame, if the board area in the target image frame intersects with the acquisition trigger area, the image content corresponding to the board area is extracted from the target image frame to obtain the corresponding local board image; wherein, the acquisition trigger area refers to the image area pre-defined in the acquisition field of view for triggering image extraction.

3. The method for detecting solder joint defects on printed circuit boards according to claim 2, characterized in that, The step of locating the printed circuit board according to each of the continuously acquired target image frames, to determine the board area in each target image frame, includes: For each target image frame, candidate regions of the board surface are extracted from the target image frame according to a preset board surface color threshold; The candidate areas of the board surface are screened by contour, and the candidate areas of the board surface whose contours meet the preset geometric conditions are determined as the board area.

4. The method for detecting solder joint defects on printed circuit boards according to claim 2, characterized in that, The continuous acquisition of images of the printed circuit board yields multiple target image frames, including: During the process of the printed circuit board being transported by the conveyor belt and passing through the acquisition field of view, the printed circuit board is continuously acquired to obtain multiple target image frames; After continuously acquiring multiple target image frames from the printed circuit board, the method further includes: Cross-frame association is performed on the board regions in each of the target image frames to track the printed circuit board and assign board identifiers to the printed circuit board; The motion state of the printed circuit board is determined based on the positional changes of the board region in each of the target image frames, which are associated across frames. In response to the motion state indicating that the printed circuit board has stopped moving, a data acquisition end signal carrying the board identifier is generated.

5. The method for detecting solder joint defects on printed circuit boards according to claim 1, characterized in that, The step of segmenting the local panel images to obtain multiple detection image blocks corresponding to each local panel image includes: For each of the local panel images, the local panel image is segmented according to a preset image block size to obtain multiple detection image blocks corresponding to the local panel image; wherein, there is an overlapping area between adjacent detection image blocks; The step of merging the candidate defect boxes of each detected image block according to their position in the local panel image includes: Based on the position of each of the detected image blocks in the local panel image, determine the panel position of each of the candidate defect boxes in the local panel image; The candidate defect boxes whose positions are located in the overlapping area and overlap with each other are deduplicated and merged to obtain the local defect detection results corresponding to each local plate image.

6. The method for detecting solder joint defects on printed circuit boards according to claim 1, characterized in that, The weld joint defect detection model includes a main defect detection branch and a priori auxiliary branch for weld joint structure. For each local board surface image, the corresponding detection image blocks are input into a pre-trained solder joint defect detection model to obtain candidate defect boxes corresponding to each detection image block, including: Each of the detected image blocks is input into the solder joint defect detection model, and defect detection is performed on each of the detected image blocks through the main defect detection branch to obtain candidate defect boxes corresponding to each of the detected image blocks; wherein, the candidate defect boxes are used to indicate the solder joint defect category and detection confidence level; The solder joint structure state is predicted for each of the detected image blocks by the prior auxiliary branch of the solder joint structure, and a prior probability map of the structure corresponding to each detected image block is obtained; wherein, the probability value of each pixel in the prior probability map of the structure is used to characterize the probability that there is an abnormal solder joint structure at the corresponding pixel position. The detection confidence of each candidate defect box is calibrated based on the pixel probability value in the prior probability map of the structure corresponding to each candidate defect box.

7. The method for detecting solder joint defects on printed circuit boards according to claim 1, characterized in that, Determining the relative positional relationship of each of the local board surface images on the printed circuit board includes: Displacement estimation is performed on adjacent local panel images to determine the relative displacement between adjacent local panel images; The relative positional relationship of each of the local board images on the printed circuit board is determined based on the relative displacement between each adjacent local board image.

8. The method for detecting solder joint defects on printed circuit boards according to claim 1, characterized in that, The step of performing board-level defect confirmation on each of the candidate defect boxes to obtain the overall board inspection result of the printed circuit board includes: The overlapping candidate defect frames of the whole board are deduplicated and merged at the whole board level to obtain the deduplicated candidate defect frames of the whole board. For each of the deduplicated candidate defect frames of the entire board, a supporting defect frame is determined; wherein, the supporting defect frame refers to the candidate defect frame whose position overlaps with the candidate defect frame of the entire board before deduplication and merging. The number of local board images to which the supported defect boxes belong is counted to obtain the number of supported candidate defect boxes for the entire board. In response to the number of supported items reaching a preset support number threshold, the deduplicated candidate defect boxes of the entire board are retained in the overall board detection result.

9. A solder joint defect detection system for printed circuit boards, characterized in that, This includes image acquisition equipment, light sources, edge computing devices, and display devices; The image acquisition device is used to acquire images of printed circuit boards; The light source is used to provide illumination for the field of view of the image acquisition device; The edge computing device is used to perform the solder joint defect detection method for printed circuit boards as described in any one of claims 1 to 8; The display device is used to display the overall inspection results of the printed circuit board.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the solder joint defect detection method for printed circuit boards as described in any one of claims 1 to 8.

Images