Production line online quality inspection method and system based on visual identification

Abstract

The invention relates to the technical field of production line quality inspection, and discloses a production line online quality inspection method and system based on visual identification, and the method comprises the steps: S1, obtaining the historical data of a production line and a standard template of a circuit board, marking key elements and common elements, building a rough detection model, and building a fine detection model; s2, collecting an image of the circuit board after surface mounting, and calling a rough detection model to detect element defects according to the standard template and the image of the circuit board after surface mounting; s3, the circuit board is fed into a reflow oven to be welded, an image of the welded circuit board is collected, and the rough detection model is called according to the image of the circuit board to detect element defects again; and S4, collecting the X-ray image of the welded circuit board, calling the precise detection model, and according to the image of the welded circuit board and the X-ray image of the welded circuit board, calling the precise detection model to carry out precise detection on the circuit board and output welding spot defects, thereby completing online quality inspection of the production line.

Description

Technical Field

[0001] This invention relates to the field of production line quality inspection technology, specifically to an online quality inspection method and system for production lines based on visual recognition. Background Technology

[0002] Circuit board assembly lines typically include solder paste printing, component placement, and reflow soldering, with online inspection stations set up after placement and reflow to control defect leakage. Online inspection usually relies on automated optical inspection, and X-ray inspection is introduced in critical packages or concealed solder joint scenarios to observe the internal morphology of the solder joints. While existing technologies can detect missing components, misalignment, polarity errors, and some soldering defects to a certain extent, several technical challenges remain in actual production line environments.

[0003] First, there are significant differences between the images of the PCB after surface mount technology (SMT) and after reflow, in terms of solder morphology, surface reflection, shadows, and contrast around components. The image distribution of the same PCB is inconsistent at different process stages, which means that detection models or rules often need to be configured separately for each process, resulting in high maintenance costs. Furthermore, threshold drift and false positives are prone to occur when changing production lines, materials, or optical conditions. Second, short-circuit defects such as solder bridging or solder bridges are essentially abnormal conductive relationships between solder joints. However, the common practice in existing technologies is to independently judge individual solder joints or individual areas, or to compare the similarity between solder joints using a uniform fixed threshold. Since different solder joints have natural differences in structural background, pad spacing, solder mask morphology, and X-ray transmission path, a uniform threshold cannot simultaneously account for both false positive and false negative rates. This is especially true in areas with high-density solder joints and fine-pitch devices, where edge defect judgment is more likely to be unstable.

[0004] Furthermore, for internal defects in solder joints, such as voids and cold solder joints, reliable identification often relies on X-ray images. However, performing high-precision inspection on all circuit boards would significantly increase computational and equipment usage, impacting production line cycle time. Conversely, sampling inspection of only a portion of circuit boards carries the risk of missed defects. Moreover, existing technologies often lack unified feature representations among different detection models, making it difficult to coordinate and optimize point defect and edge defect identification. This typically requires subsequent secondary screening using multiple thresholds or rules set manually based on experience, further increasing parameter tuning difficulty and engineering maintenance costs.

[0005] Therefore, there is an urgent need for a technical solution suitable for online quality inspection of circuit boards, which can improve the stability of the judgment of solder joint defects after reflow, especially edge defects such as solder bridging or solder bridges, under the constraints of production line cycle time, and reduce the burden of performing high-cost precision inspection on a large number of circuit boards. Summary of the Invention

[0006] In view of the above-mentioned problems, the present invention is proposed.

[0007] To solve the above technical problems, the present invention provides the following technical solution: an online quality inspection method for production lines based on visual recognition, comprising: step S1: acquiring historical data of the production line, acquiring a standard template of the circuit board, marking key components and ordinary components based on the standard template, and establishing a coarse inspection model and a fine inspection model based on the historical data of the production line; Step S2: Collect the image of the circuit board after surface mounting. Based on the standard template and the image of the circuit board after surface mounting, call the coarse inspection model to detect component defects. If a critical component defect is detected, an alarm is issued and the production line is stopped. If a common component defect is detected, it is marked and the process continues to step S3. Step S3: Send the circuit board into the reflow oven for soldering, acquire the image of the soldered circuit board, call the coarse inspection model to inspect the component defects again based on the circuit board image, if the marked ordinary component defects are detected, the circuit board is determined to be unqualified and is picked out, and step S4 is executed for the remaining circuit boards. Step S4: Acquire X-ray images of the soldered circuit board, call the precision inspection model, and perform precision inspection on the circuit board based on the soldered circuit board image and X-ray image, and output solder joint defects to complete the online quality inspection of the production line.

[0008] As a preferred embodiment of the online quality inspection method for production lines based on visual recognition described in this invention, the historical data of the production line includes standard templates of circuit boards, historical circuit board images, X-ray images of circuit boards after historical soldering, and actual defect data. The historical circuit board images include historical images of circuit boards after surface mounting and historical images of circuit boards after historical soldering. The actual defect data includes component defect data and solder joint defect data. The standard template of the circuit board is a standard reference data in image space used to describe the ideal component layout and shape of the circuit board, including the position of the component in the standard template, the outline area of ​​the component, and the mounting direction of the component. The key components include large-package integrated circuits, BGA packaged devices, QFN packaged devices, large-volume connectors and sockets, while other components on the circuit board are ordinary components.

[0009] As a preferred embodiment of the online quality inspection method for production lines based on visual recognition described in this invention, the establishment of the coarse detection model includes: for each component in the standard template, setting a template local image of the component; cropping a local image corresponding to the template local image of the component from a historical circuit board image; and normalizing and differentiating the local image and the template local image to obtain a differential image. Based on the component defect data, the component defects in the differential images are labeled. A coarse detection model is established based on the convolutional neural network. All differential images obtained from historical circuit board images and the labeled component defects are combined into a training sample set. The cross-entropy loss function is used to constrain the consistency between the output probability of the coarse detection model on the training sample set and the labeled component defects, and the coarse detection model is trained. The coarse detection model takes the difference image as input and outputs the defect score vector of each component in the difference image. The component defect score vector is then converted into the component defect probability through the Softmax function.

[0010] As a preferred embodiment of the online quality inspection method for production lines based on visual recognition described in this invention, the precision inspection model consists of a mask region convolutional neural network and an edge defect determination module.

[0011] As a preferred embodiment of the visual recognition-based online quality inspection method for production lines described in this invention, the training process of the convolutional neural network in the mask region includes: Based on historical X-ray images of circuit boards after soldering, and according to the position of the solder joints in the standard template, local X-ray images of historical solder joints are cropped from the historical X-ray images of circuit boards after soldering, and local X-ray images of historical solder joints are cropped from the historical X-ray images of circuit boards after soldering. Solder joint defects are divided into point defects and edge defects. Based on historical solder joint X-ray local images and historical solder joint local images, combined with solder joint defect data in real defect data, a mask region convolutional neural network is trained under supervision. The feature extraction network in the mask region convolutional neural network is used to obtain the precise detection features of the solder joint, and the defect judgment network is used to judge point defects based on the precise detection features of the solder joint. During the supervised training process, the model parameter update is constrained by the overall objective function, which consists of three weighted parts: point defect multi-classification loss, edge defect binary classification loss, and edge assimilation bias ranking loss. The point defect multi-class loss adopts the multi-class cross-entropy loss function; The binary classification loss for edge defects adopts the binary cross-entropy loss function. The assimilation bias ranking loss is calculated by dividing solder joint pairs into defective solder joint pairs and normal solder joint pairs according to the actual defect data, calculating the assimilation bias of each solder joint pair, randomly matching several normal solder joint pairs for each defective solder joint pair to construct a defect-normal pairing set, and calculating the ranking loss with the constraint that the assimilation bias of the defective solder joint pair is higher than that of the normal solder joint pair. When the constraint is satisfied, the assimilation bias ranking loss is zero; when the constraint is not satisfied, the assimilation bias ranking loss is included and accumulated.

[0012] As a preferred embodiment of the visual recognition-based online quality inspection method for production lines described in this invention, the edge defect determination module includes: Based on the position of the solder joints in the standard template, the solder joint pair spacing threshold of the standard template is calculated. For each solder joint, all solder joints that are less than the solder joint pair spacing threshold are included as the set of solder joint pairs for that solder joint. The solder joint and each solder joint in the set of solder joint pairs form a solder joint pair. Select R qualified circuit boards, and for each solder joint pair on the qualified R circuit boards, calculate the cosine similarity value using the fine detection features of the solder joints. Calculate the mean and standard deviation of the cosine similarity values ​​of the solder joint pairs on the R circuit boards to form a template edge reference library for that solder joint pair. Calculate the cosine similarity value of the weld point pair to be tested. Based on the cosine similarity value of the weld point pair to be tested and the template edge reference library, calculate the assimilation deviation. Based on the template edge reference library and the assimilation deviation of the weld point pair to be tested, determine the edge defects.

[0013] As a preferred embodiment of the visual recognition-based online quality inspection method for production lines described in this invention, step S2 specifically comprises: Acquire images of the circuit board after surface mounting, and align the standard template with the images of the circuit board after surface mounting; Based on the positions and dimensions of each component recorded in the standard template, a partial image of each component is extracted from the image of the board after surface mounting, and the difference is performed between the partial image and the corresponding template image to obtain the difference image of each component. The differential images of each component are input into the coarse detection model one by one to obtain the component defects of each component; When a defect in a critical component is detected, an alarm is issued and the production line is stopped to notify technicians to handle the situation. When a defect in a common component is detected, the component with the defect is marked, no alarm is issued, and the circuit board continues to be sent to the reflow oven for soldering. If no component defects are detected, the circuit board is sent to the reflow oven for soldering.

[0014] As a preferred embodiment of the visual recognition-based online quality inspection method for production lines described in this invention, step S3 specifically comprises: The circuit board is sent into a reflow oven for soldering to obtain the soldered circuit board. Acquire images of the soldered circuit board and align the standard template with the soldered circuit board image; Based on the positions and dimensions of each component recorded in the standard template, a partial image of each component is extracted from the image of the soldered circuit board, and the difference is performed between the partial image and the corresponding template image to obtain the difference image of each component after soldering. The differential images of each component after welding are input into the coarse detection model one by one to obtain the component defects of each component after welding. If the marked component is also found to have defects after soldering, the circuit board is determined to be unqualified and is rejected. Step S4 is then performed on the remaining circuit boards.

[0015] As a preferred embodiment of the visual recognition-based online quality inspection method for production lines described in this invention, step S4 specifically comprises: Acquire X-ray images and images of the circuit board after soldering. For each solder joint, extract local images and X-ray images of the solder joint based on a standard template. Local images and X-ray images of the solder joints are input into the precision detection model, and the precision detection features of each solder joint are obtained through a feature extraction network. For each pair of solder joints, a cosine similarity value is calculated based on the precise detection features of the solder joints, and an assimilation bias is calculated based on the cosine similarity value and the template edge reference library. The assimilation bias is used to determine whether the solder joint pair meets the edge defect judgment condition. When the solder joint pair meets the edge defect judgment condition, the circuit board containing the solder joint pair is output as unqualified and the fine inspection ends. When the solder joints meet the bridging criteria, the fine inspection features of each solder joint are input into the defect judgment network to determine whether there is a point defect. When there is a point defect, the circuit board where the solder joint is located is output as unqualified and the fine inspection ends. When there are no defects in the solder joint, the circuit board containing the solder joint is deemed qualified and the fine inspection is completed.

[0016] An online quality inspection system for production lines based on visual recognition, employing any of the methods described in this invention, comprising: a modeling module that acquires historical production line data, acquires a standard template for a circuit board, marks key components and ordinary components based on the standard template, and establishes a coarse inspection model and a fine inspection model based on the historical production line data; The initial inspection module acquires images of the circuit board after surface mounting. Based on the standard template and the images of the circuit board after surface mounting, it calls the coarse inspection model to detect component defects. If a critical component defect is detected, an alarm is issued and the production line is stopped. If a common component defect is detected, it is marked and the process continues to step S3. The coarse inspection module sends the circuit board into the reflow oven for soldering, acquires images of the soldered circuit board, calls the coarse inspection model based on the circuit board images to re-inspect component defects, and if a marked ordinary component defect is detected, the circuit board is determined to be unqualified and is picked out. Step S4 is executed for the remaining circuit boards. The precision inspection module acquires X-ray images of the soldered circuit board, calls the precision inspection model, and performs precision inspection on the circuit board based on the soldered circuit board image and the X-ray image, outputting solder joint defects and completing online quality inspection of the production line.

[0017] The beneficial effects of this invention are as follows: The method of this invention achieves more stable online quality inspection while ensuring online cycle time: It achieves rapid screening consistent before and after reflow through standard template alignment and coarse detection of differential images; it improves the stability of solder joint identification by judging relational edge defects of solder joint pairs through template edge reference library and assimilation deviation; it widens the gap between defective solder joint pairs and normal solder joint pairs through edge assimilation deviation sorting loss constraint feature extraction network, reducing false alarms and false negatives; and it only performs fine inspection of point defects and X-ray judgment on circuit boards that have not triggered edge defects, thereby reducing the burden of fine inspection and improving rework positioning efficiency. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is an overall flowchart of a visual recognition-based online quality inspection method for production lines, as provided in Embodiment 1 of the present invention. Detailed Implementation

[0020] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0021] This invention addresses the pain points of online quality inspection in existing circuit board production lines by proposing an improved method. Existing circuit board production lines typically have separate inspection stations after surface mount technology (SMT) and after reflow oven, but this often has two limitations: First, a one-time fine inspection is only performed after reflow oven, resulting in surface mount errors of critical components being discovered only after soldering, significantly increasing rework costs and difficulty. Second, even when inspection is deployed at both stations, independent inspection logic is often used, failing to differentiate the tolerance costs between critical and ordinary components, and failing to perform correlation analysis between the inspection results of the two stages. This leads to the repeated inspection of similar defects at different stations, consuming significant system resources, and placing an excessive burden on the fine inspection of soldering quality and internal defects, making it difficult to balance inspection coverage and accuracy while maintaining production cycle time. SMT refers to the process of a pick-and-place machine (SMT station) picking up surface mount components and placing them onto solder pads pre-printed with solder paste. A reflow oven refers to the equipment and process of heating the entire circuit board to melt solder, then cooling it to allow the solder to firmly bond the component pads to the circuit board. A pad is a small metal plate area reserved on a circuit board for soldering component leads. A solder joint is a soldered entity formed by solder in the pad area after the component is installed and soldered. Solder joints usually cover the corresponding pad and the transition area between the component lead and the pad. Their shape, volume and wetting state reflect the soldering quality at that point.

[0022] To address the aforementioned issues, this invention distinguishes between critical and ordinary components, reuses the same coarse inspection model in both the surface mount and soldering stages, and achieves a division of labor and collaboration between coarse and fine inspections. Without altering the existing hardware configuration of the circuit board production line, it can proactively intercept surface mount errors in critical components, quickly screen out structural defects in ordinary components, and concentrate system resources on fine inspection of soldering quality and internal defects. This effectively improves overall inspection efficiency while ensuring inspection accuracy.

[0023] Example 1, referring to Figure 1 As an embodiment of the present invention, a visual recognition-based online quality inspection method for production lines is provided, comprising: Step S1: Obtain historical data from the production line, obtain a standard template for the circuit board, mark key components and ordinary components based on the standard template, and establish a coarse inspection model and a fine inspection model based on the historical data from the production line.

[0024] Furthermore, historical production line data is acquired. This historical data refers to the dataset pre-collected and inspected on the target production line, used to train coarse and fine inspection models. Historical production line data includes standard circuit board templates, historical circuit board images, historical X-ray images of soldered circuit boards, and actual defect data. Historical circuit board images include images of circuit boards after surface mounting and images of circuit boards after soldering. Actual defect data includes component defect data corresponding to historical surface-mount circuit board images, component defect data and solder joint defect data corresponding to historical soldered circuit board images, and solder joint defect data corresponding to historical X-ray images of soldered circuit boards.

[0025] A circuit board image refers to a visible light image acquired by a vision inspection device to characterize the appearance of a circuit board. A vision inspection system includes an industrial camera mounted above the circuit board and an imaging light source. The industrial camera can be a linear or area array CCD / CMOS camera, used in conjunction with a ring light source, bar light source, or area array light source to illuminate the circuit board. When the circuit board passes the inspection position, a trigger signal controls the industrial camera to acquire a top-view image of the circuit board.

[0026] Obtain a standard template for the circuit board, and mark key components and ordinary components based on the standard template. The standard template is a standard reference data in image space used to describe the ideal mounting state of the circuit board, and is used to describe the component layout and solder joint layout. The standard template data includes two parts: component information and solder joint information. Component information describes the ideal mounting configuration of the circuit board at the component level, including the component's position in the standard template, the component's outline area, and the component's mounting orientation. The component's position in the standard template indicates the target placement of each component in the circuit board image, the component's outline area describes the expected area the component will occupy in the image, and the component's mounting orientation describes the correct mounting orientation for components with polarity or orientation requirements.

[0027] Solder joint information describes the ideal soldering state of the circuit board at the solder joint level, including the position of each solder joint in the standard template and the solder joint outline area. Pad position indicates the target soldering area of ​​each solder joint in the circuit board image, and solder joint outline area describes the expected coverage of the solder joint in the image after soldering.

[0028] An electrical network is defined in circuit board design as a set of conductive structures that should be electrically connected to each other. Pads and solder joints belonging to the same electrical network have the same or controlled electrical connection relationship under normal operating conditions. Different electrical networks should be electrically isolated from each other in the design and should not be directly connected.

[0029] Standard templates can be established based on circuit board design data. Specifically, based on the computer-aided design data and bill of materials data of the circuit board, the location, package type, and polarity information of each component on the circuit board are obtained, as well as the routing coordinates of each pad, the electrical network to which it belongs, and the design continuity relationship between the pads are obtained. After surface mount technology (SMT) or soldering, at least one circuit board that has passed the existing quality inspection process is selected, and the corresponding circuit board image is captured. The coordinate information in the computer-aided design data is aligned with the circuit board image, and the positions of each component and each pad are mapped onto the circuit board image. Combined with the qualified circuit board image, the outline area of ​​the components, the mounting direction of the components, and the outline area of ​​the solder joints are calibrated and corrected, thereby obtaining the ideal component layout and ideal solder joint layout of the circuit board in image space, forming a standard template. The standard template is essentially a unified set of data, indicating where each component should be located in the circuit board image, how much outline it should occupy, and what its orientation should be, as well as where each solder joint should be located in the image, how much soldering area it should occupy, which electrical network it should belong to, and which solder joints it should be connected to in the design.

[0030] In actual testing, the coarse inspection model, based on the component information in the standard template, compares and inspects the presence, placement, and installation direction of components on the circuit board image to detect component-level structural defects such as missing parts, misalignment, rotation errors, and tombstoning. The fine inspection model, based on the solder joint information in the standard template, performs detailed analysis of the solder joint area on the circuit board image and ray image after soldering, and constructs a solder joint relationship diagram by combining the electrical network identifier to which the solder joint belongs and the design continuity relationship between the solder joints, to inspect the solder joint welding quality and electrical topology consistency.

[0031] Critical components refer to components on a circuit board that, if defects occur during the surface mount or soldering process, will significantly increase the cost of rework for the entire board or severely affect the overall functionality of the board. These include large-package integrated circuits, BGA packaged devices, QFN packaged devices, large-volume connectors, and sockets. Specifically, large-package integrated circuits typically have a large package size and many pins, generally undertaking core functions such as main control, storage, and power management. These components are difficult to disassemble and rework after soldering, and failure can directly render the entire board inoperable. BGA packaged devices are connected to the circuit board through an array of solder balls, and the solder joints are completely covered by the package. Defects are mainly detected by methods such as X-rays. Once soldering defects expand or repeated repairs are needed, it can easily cause damage to the inner layer traces and pads. QFN packaged devices have pins distributed at the bottom or edge of the package in the form of no pins or short pins. The pin spacing is small and the heat dissipation pad is located at the bottom. Soldering defects are not easy to observe directly in visible light images, and repairs cause a large thermal shock to the circuit board. Large-volume connectors and sockets usually have many contacts and serve as power or signal interfaces. They are large in size and have concentrated pads. Once misalignment, incorrect holes, or poor soldering occur, the disassembly and reassembly process can easily cause the pads to fall off or mechanical damage to the board edge.

[0032] Other components on the circuit board besides the key components are ordinary components. Ordinary components typically include passive devices and small-signal devices such as chip resistors, chip capacitors, chip inductors, small-package diodes, and transistors. These components have small package sizes and are numerous, making disassembly and replacement after soldering relatively simple and having a low impact on the overall board functionality and repair costs.

[0033] It should be noted that in the circuit board production line environment targeted by this invention, critical components are typically placed using low-to-medium speed placement heads or dedicated placement heads. The placement program and fixtures meticulously verify their coordinates, polarity, and interference relationships. Therefore, the random probability of misalignment, incorrect holes, or polarity errors during the stable mass production stage is extremely low. Once these occur, they often reflect a systemic failure in the placement program or equipment. Based on this, this invention uses the pre-soldering coarse inspection results of critical components as the basis for production line safety control. Once the coarse inspection model detects a defect in a critical component during the placement stage, it issues an alarm and stops the production line, preventing systemic failures from causing multiple circuit boards to continuously enter the reflow oven and resulting in batch scrap. In contrast, defects in ordinary components are mostly random missing parts or minor misalignments. For ordinary components, pre-soldering rework and post-soldering rework are roughly equivalent in terms of single-piece operation complexity and direct cost. However, pre-soldering rework disrupts the placement production rhythm. Therefore, this invention chooses to mark defects in ordinary components only during the placement stage, allowing them to enter the reflow oven. These defects are then uniformly judged by post-soldering coarse and fine inspections and centrally reworked at the back end, which is more conducive to maintaining the overall efficiency of the production line.

[0034] Furthermore, coarse and fine inspection models are established based on historical production line data. Common defects in circuit board production mainly include: missing components (i.e., no corresponding component is mounted at a predetermined pad position); misalignment (i.e., the component is significantly deviated from the target pad area, or even crosses into adjacent pads or device areas); incorrect polarity or mounting direction (i.e., the mounting orientation of components with polarity or direction requirements is inconsistent with the standard template); tombstoning or standing (i.e., one end of a chip component is soldered while the other end is tilted up and in a basically vertical state); cold solder joint (i.e., the solder joint appears to be connected but the actual contact is insufficient or intermittent); solder bridging or solder bridges (i.e., solder joints are connected by continuous solder, forming a short circuit); voids (i.e., large air bubbles or cavities exist inside the solder joint); and abnormal solder joint appearance defects such as insufficient or excessive solder volume and solder balls. Among these, missing components, misalignment, incorrect polarity or mounting direction, and tombstoning or standing are component defects, while solder bridging or solder bridges, voids, and abnormal solder joint appearance defects are solder joint defects.

[0035] A coarse inspection model is established based on standard PCB templates, historical images of PCBs after surface mount technology (SMT) mounting, and historical images of PCBs after soldering. This model identifies component defects that become apparent immediately after SMT mounting, primarily targeting missing components, misalignment, incorrect polarity or mounting orientation, and tombstoning.

[0036] Specifically, the outline region of the component in the standard template is mapped onto the historical circuit board image, and the mapped outline region is expanded outside it by a preset margin to obtain an envelope region containing the complete outline of the target component. The pixel data corresponding to the envelope region is extracted from the historical circuit board image as a local image of the component.

[0037] Preferably, the component preset margin is calculated during the system calibration stage based on the component position tolerance (e.g., ±0.05mm) given by the placement equipment and the upper limit of the alignment error of the historical circuit board image (e.g., ±0.03mm). The sum of the two is converted into the number of pixels in the historical circuit board image at the detection resolution, and then rounded up to obtain the number of rows and columns that need to be added. When cropping a local image, the pixel range corresponding to the above-mentioned number of rows and columns is added to the top, bottom, left, and right sides of the component outline area in the standard template to form an expanded envelope area. This expanded envelope area is the local image of the component. Even if the component has the maximum allowable offset during actual placement, the actual imaging range of the component still falls completely within the local image of the component.

[0038] The circuit board image is aligned based on a standard template. According to the positions and dimensions of each component recorded in the standard template, a local image of each component is cropped from the circuit board image, and subtracted from the corresponding local image of the template to obtain the subtracted image of each component used for training. ; Where D(u,v) represents the difference image; (u,v) represents the image pixel coordinates of the element; I(u,v) represents the local image; and T(u,v) represents the local image of the template. The difference image is used to reflect the structural differences between the element region and the normal template.

[0039] The component defects in the differential images are labeled based on the component defect data. The corresponding component defect type is labeled for each differential image, including normal, missing part, misalignment, incorrect polarity or installation direction, and tombstoning, forming a training sample set.

[0040] The coarse detection model is configured as a convolutional neural network model based on difference images. The model takes the difference images of each component as input and automatically extracts structural features from the difference images through several convolutional layers, pooling layers, and fully connected layers. The output layer provides a defect scoring vector for each component, used to distinguish normal components from different types of component defects, as shown below: ; Where Z represents the component defect scoring vector; f represents the coarse detection model. The component defect scoring vector is transformed into component defect probabilities using the Softmax function: ; Where, p k Z represents the probability of component defect type k; exp represents the exponential function; k represents the k-th component defect type, for example, k=1 represents normal, k=2 represents missing part; K represents the total number of component defect types; Z k This represents the component of the component defect scoring vector on k.

[0041] In the coarse detection model training phase, based on the training sample set, the cross-entropy loss function is used to constrain the consistency between the confidence of the component defect type output by the model and the labeled component defect type. The parameters of the convolutional neural network are iteratively updated using a gradient descent optimization algorithm until the training loss converges, resulting in a coarse detection model that can adapt to both the circuit board image after patching and the circuit board image after soldering.

[0042] Furthermore, a coarse inspection model is established based on standard circuit board templates, historical images of soldered circuit boards, and X-ray images of historically soldered circuit boards. A fine inspection model is used to identify defects generated during the soldering process, such as cold solder joints, insufficient or excessive solder volume, poor wetting, solder bridging or bridging, solder balls, and voids.

[0043] Specifically, the precision inspection model consists of a mask region convolutional neural network and an edge defect determination module. Based on historical X-ray images and images of the circuit board after soldering, and according to the position of the solder joints in the standard template, local X-ray images of historical solder joints are extracted from both the historical soldering X-ray images and the historical circuit board images. The method for determining the local images of solder joints is the same as that for determining the local images of components. Based on the location of the solder joint, the model extends outwards by a preset margin for the solder joint. The preset margin for the solder joint is determined based on the positional tolerance of the pad corresponding to the solder joint and the upper limit of the alignment error of the historical circuit board images.

[0044] Using standard templates to crop local images of solder joints has the following advantages: First, standard templates provide a definite location of the solder joint on the circuit board, enabling the training data of the precision inspection model to be transformed from whole-board images into solder joint-level samples, avoiding interference from non-solder joint areas such as background lines, silkscreen characters, and component housings. Second, by using the same solder joint location to form paired local images under both visible light and X-ray imaging, the precision inspection model can simultaneously learn the surface morphology information and the internal solder morphology information of the solder joint, thereby improving the distinguishability and stability of welding quality defects.

[0045] Solder joint defects are divided into point defects and edge defects. Point defects are used to describe the abnormal soldering quality of a single solder joint, such as cold solder joint, voids, insufficient or excessive solder volume, poor wetting, solder balls, etc. Edge defects are used to describe abnormal conductivity between two solder joints, such as solder bridging or solder bridges.

[0046] Based on historical local images and X-ray images of weld joints, and combined with weld joint defect data from real defect data, a mask region convolutional neural network is trained under supervision. This mask region convolutional neural network is used to extract precise detection features characterizing the welding state of the weld joint from the local images and X-ray images, and outputs the point defect judgment result of the weld joint based on these features.

[0047] Mask Region Convolutional Neural Network (MaskR-CNN) is a target detection and instance segmentation network based on candidate regions. Its basic idea is to first extract features from the input image using a convolutional neural network, then generate candidate regions on the feature map that may contain targets, and perform alignment sampling on the candidate regions. Finally, the target category is output through the classification branch, and the pixel-level mask of the target region is output through the mask branch, thereby simultaneously achieving target localization and defect determination. In this invention, this network is used as a conventional implementation to extract features from local images and X-ray images of solder joints and output the point defect determination results of the solder joints.

[0048] Specifically, the convolutional neural network for the mask region is divided into two parts: a feature extraction network and a defect determination network. The feature extraction network encodes features from the local image and X-ray image of the solder joint, outputting the precise detection features of the solder joint. The defect determination network, based on the precise detection features of the solder joint, outputs a judgment result indicating whether the solder joint is normal or has various point defects. By using real defect data for supervised training, the model can learn visible light features such as solder surface texture, solder reflection, and contour changes, as well as X-ray features such as solder volume distribution, void shadows, internal cracks, or wetting abnormalities, thereby improving the accuracy of point defect identification.

[0049] After training, the feature extraction network in the convolutional neural network of the mask region is solidified for the subsequent precise detection feature acquisition of solder joints in the edge defect determination module. This is because edge defects (such as solder bridging or solder bridges) cannot be reliably determined by the appearance of a single solder joint; rather, it requires comparing the feature consistency or anomalous assimilation degree between adjacent solder joints. Using the precise detection features output by the trained feature extraction network can provide a unified and comparable feature representation for calculating the similarity between solder joint pairs while maintaining the ability to determine point defects. This avoids instability in similarity measurements due to differences in imaging conditions and structures across different solder joint regions. By "training the point defect model first and then solidifying the feature extraction network," the sharing of feature representations for point defect determination and edge defect determination is achieved, reducing redundant modeling and computation, and laying a consistent data foundation for the subsequent establishment of the template edge benchmark library and the calculation of assimilation bias.

[0050] To ensure that the determination of edge defects (solder bridging or solder bridges) covers the most likely solder joint pair relationships while keeping the computational scale under control, a set of candidate solder joint pairs is constructed based on the positions of solder joints in the standard template. Each solder joint in the standard template has a defined template coordinate system. For any two solder joints, the solder joint spacing function is defined as follows: ; Where d(i,j) represents the distance between solder joint i and solder joint j; This indicates the position of solder joint i on the standard template; This indicates the position of solder joint j on the standard template; This indicates the calculation of the L2 norm.

[0051] The solder joint pair spacing threshold is set based on the geometric minimum spacing of solder joints in the standard template: First, the distance between any two solder joints is calculated in the standard template, and the minimum value is taken as the minimum solder joint spacing of the template. Then, the minimum solder joint spacing distance of the template is multiplied by a fixed safety factor (preferably 2) to obtain the solder joint pair spacing threshold. The fixed safety factor is used to cover the distance deviation caused by solder morphology expansion, imaging blurring, and alignment errors after welding, so that the candidate solder joint pair set can include solder joint pairs that have a physical risk of bridging, while avoiding including solder joint pairs that are obviously far apart and unlikely to bridge in the calculation range.

[0052] For each solder joint, all solder joints whose distance to that solder joint is less than the solder joint pair spacing threshold are included as the set of solder joint pairs for that solder joint. This solder joint forms a solder joint pair with each solder joint in the set of solder joint pairs. The solder joint pairs of solder joint i are denoted as h(i), and the total number of solder joint pairs of solder joint i is H. Edge defects usually occur between spatially adjacent or closely spaced solder joints. Limiting the solder joint pairs to within the distance threshold can cover the main edge defect solder joint pairs.

[0053] R qualified circuit boards are selected. For each solder joint pair on the R qualified circuit boards, the cosine similarity value is calculated using the precise inspection features of the solder joints. For a solder joint pair ij consisting of solder joint i and solder joint j, the cosine similarity value is expressed as: ; in, This represents the cosine similarity value of the solder joints with respect to ij; This represents the precision inspection characteristics of solder joint i; This indicates the precision inspection characteristics of solder joint j.

[0054] Calculate the mean and standard deviation of the cosine similarity values ​​of solder joint pairs on R circuit boards to form a template edge reference library for each solder joint pair. The template edge reference library is obtained from qualified boards and reflects the "similarity distribution of the solder joint pair under normal welding conditions". It is essentially a statistical quantity and is used as a normalization reference for subsequent edge defect judgment. It does not participate in the model parameter update.

[0055] The cosine similarity value of the weld point pair to be tested is calculated. Based on the cosine similarity value of the weld point pair to be tested and the template edge reference library, the assimilation deviation is calculated. The edge defect is judged based on the template edge reference library and the assimilation deviation of the weld point pair to be tested, as expressed as: ; in, This indicates the assimilation bias of the solder joint to ij; This represents the mean cosine similarity value of weld points to ij pairs in the template edge reference library; This represents the standard deviation of the cosine similarity values ​​of weld points to ij in the template edge reference library.

[0056] It should be noted that assimilation bias is the degree to which the similarity of a pair of solder joints deviates from its "normal distribution center," and it is normalized using the standard deviation to make the deviations between different solder joint pairs comparable. For solder bridging or solder bridges, two solder joints that should be independent of each other often exhibit abnormal consistency in X-ray and visible light morphology when the solder is abnormally conductive, leading to an abnormally high similarity in their precision detection features, thus increasing the assimilation bias.

[0057] By establishing a template edge benchmark library with mean and standard deviation for each solder joint pair, false positives or false negatives are avoided by using a uniform similarity threshold for all solder joint pairs. Through the standardized calculation of assimilation bias, the discrimination scale for edge defect judgment remains consistent across different solder joint pairs and structural backgrounds, thereby improving the robustness and interpretability of edge defect identification such as solder bridging or solder bridges.

[0058] To ensure that the precision inspection features of weld joints simultaneously meet the requirements of "point defect determination" and "edge defect determination," a general objective function is introduced during the training phase of the precision inspection model to jointly constrain and update the model parameters of the convolutional neural network in the mask region. The general objective function is used to update the feature extraction network parameters and defect determination network parameters of the convolutional neural network in the mask region; the mean and standard deviation in the template edge benchmark library are statistically obtained fixed quantities and do not participate in parameter updates.

[0059] Specifically, the overall objective function consists of three weighted components: point defect multi-classification loss, edge defect binary classification loss, and edge assimilation bias ranking loss.

[0060] The multi-classification loss for point defects adopts the multi-classification cross-entropy loss function, which enables the defect judgment network to output the correct point defect category based on the precise detection features of the solder joint, thereby ensuring the recognition accuracy of point defects such as cold solder joints, voids, insufficient or excessive solder volume, poor wetting, and solder balls.

[0061] The edge defect binary classification loss adopts the binary cross-entropy loss function to make the solder joint fine detection features output by the feature extraction network separable at the solder joint pair level, so that defective solder joint pairs and normal solder joint pairs show significant differences in assimilation bias.

[0062] The edge assimilation bias ranking loss further constrains the relative magnitude of the assimilation biases of solder joint pairs, forcing the overall assimilation bias of defective solder joint pairs to be higher than that of normal solder joint pairs and maintaining a stable interval, thereby reducing the threshold instability problem caused by the natural similarity differences between different solder joint pairs. Solder joint pairs are divided into defective solder joint pairs and normal solder joint pairs according to the actual defect data. The assimilation bias of each solder joint pair is calculated separately. For each defective solder joint pair, several normal solder joint pairs are randomly matched to construct a defect-normal pairing set. The ranking loss is calculated with the constraint that the assimilation bias of defective solder joint pairs is higher than that of normal solder joint pairs. The edge assimilation bias ranking loss is expressed as: ; in, b represents the edge assimilation bias sorting loss; b+ represents a defective solder joint pair; b- represents a normal solder joint pair randomly matched with b+; max represents taking the maximum value; αβ represents the assimilation bias of the solder joint to αβ; m represents the sorting interval constant.

[0063] It should be noted that the value of m is determined through validation set calibration during the model building stage of the precision detection model: a portion of samples is selected from the real defect data as the validation set, and the assimilation deviation distribution of defective solder joint pairs and normal solder joint pairs is calculated respectively. The typical difference between the two distributions is statistically analyzed (for example, the difference between the smaller quantile of the assimilation deviation of defective solder joint pairs and the larger quantile of the assimilation deviation of normal solder joint pairs). This difference is used as the initial value of m. Then, m is adjusted within a small range according to the criterion that "the false alarm rate of edge defects does not exceed the preset upper limit and the false alarm rate of edge defects is minimized as much as possible", and finally solidified as the deployment parameter.

[0064] If the assimilation bias of a defective solder joint pair is not higher than that of the corresponding normal solder joint pair (or fails to meet the preset interval constraint), the difference is included in the ranking loss and accumulated; when the constraint is met, the corresponding ranking loss is zero. In this way, the overall objective function not only enables the model to determine point defects, but also drives the feature extraction network to learn feature representations that are sensitive to edge defects, so that the assimilation bias forms a stable and comparable numerical difference between defective and normal solder joint pairs, thereby improving the robustness of edge defect determination and providing an interpretable basis for threshold setting and defect localization in the subsequent detection stage.

[0065] Step S2: Collect the image of the circuit board after surface mounting. Based on the standard template and the image of the circuit board after surface mounting, call the coarse inspection model to detect component defects. If a critical component defect is detected, issue an alarm and stop the production line. If a common component defect is detected, mark it and continue to step S3.

[0066] The main difference between the image of a PCB after surface mounting and the image of a PCB after soldering lies in the appearance of the pad area: the image of a PCB after surface mounting only contains solder paste graphics and has a relatively uniform grayscale, while the image of a PCB after soldering shows solder points with metallic luster and reflective properties at the pad positions. However, the outline, position, and polarity markings of each component are consistent in spatial structure in both states.

[0067] This invention uses differential operations to map circuit board images in both the post-mount and post-soldering states into a single differential image reflecting the structural differences of component areas relative to a standard template. The standard template only describes the geometric shape and arrangement of components on the circuit board, excluding appearance features such as the grayscale and gloss of solder paste and solder joints. Thus, regardless of whether the input is a post-mount or post-soldering circuit board image, the standard template always represents the same ideal component placement state. The differential image reflects only the deviation of the current component from this ideal placement state, independent of differences in unmelted solder paste or the metallic luster of solder joints. This fundamentally ensures that the same coarse inspection model can be used in both state processes.

[0068] Specifically, the process involves acquiring images of the board after surface mounting, aligning a standard template with the image of the board after surface mounting, and extracting local images of each component from the image of the board after surface mounting according to the positions and dimensions of each component recorded in the standard template. These images are then compared with the corresponding local images of the template to obtain the difference images of each component. This allows the coarse detection model to focus only on the "structural differences between the component area and the normal design state" without directly relying on unstable factors such as brightness and reflection of the original image.

[0069] Since the image of the circuit board after surface mounting and the image of the circuit board after soldering maintain the same structural features such as component outline, position and polarity, component defects such as missing components, misalignment, incorrect polarity or mounting direction and tombstoning will all appear as stable structural differences in the differential image. Therefore, this invention uses the differential image as input to the coarse detection model, which can reliably identify these component defects that already exist before soldering in the post-surfacing stage, laying the foundation for the subsequent two coarse detections to use the same coarse detection model.

[0070] The differential images of each component are input into the coarse detection model one by one to obtain the component defects of each component. This enables the coarse detection model to achieve automatic component-level judgment on the circuit board image after surface mounting. The coarse detection model has learned the discrimination rules between "normal component differential images" and "component defect differential images" by joint training on historical circuit board images after surface mounting and soldering. Therefore, in this step, a unified component defect judgment result can be given for all components on the entire board without increasing the burden of manual re-judgment.

[0071] When a critical component defect is detected, an alarm is issued and the production line is stopped, and technical personnel are notified to handle the issue. This approach is based on the fact that the impact of critical component defects on product quality and production equipment status is far greater than that of ordinary component defects. Critical components are typically large-package integrated circuits, BGAs, QFNs, and large-volume connectors. If these components are missing, misaligned, or have incorrect polarity after placement, continuing to send them to the reflow oven will not only lead to high-risk, high-cost rework after soldering, but may even indicate systemic failures in the placement machine program, fixture positioning, or feeding system. Therefore, this invention treats such problems directly as process anomalies by issuing an alarm, stopping the production line, and notifying technical personnel once a critical component defect is identified by a coarse inspection model in the post-placement stage, prioritizing the stability of the production line and the soldering reliability of critical components.

[0072] When a common component defect is detected, the differential image of the detected defect is marked, no alarm is issued, and the circuit board continues to be sent to the reflow oven for soldering. Similarly, if no component defect is detected, the circuit board continues to be sent to the reflow oven for soldering. This is based on the process characteristic that common component defects can be centrally processed after soldering and will not cause irreversible damage when the circuit is not powered on. On the one hand, common component defects are mostly missing or partially misaligned small resistors, capacitors, inductors, etc. Even if such component defects exist in the post-mount stage, as long as the circuit is still not powered on, continuing to enter the reflow oven will not cause irreversible damage to the circuit board. On the other hand, marking the detected common component defects in the post-mount stage using differential images provides a reference for subsequent coarse inspection model checks on component defects in the post-soldering circuit board image. If the marked common component defect is still detected in the post-soldering circuit board image, the circuit board can be directly determined to be unqualified and rejected, thus achieving a two-stage collaborative judgment of "post-mount coarse inspection marking + post-soldering coarse inspection confirmation". In this way, the present invention avoids frequent production line stoppages caused by defects in common components, and ensures that defects in common components are not released after welding, thus balancing the requirements of production cycle and final quality control.

[0073] Step S3: Send the circuit board into the reflow oven for soldering, acquire the image of the soldered circuit board, call the coarse inspection model to inspect the component defects again based on the circuit board image, if the marked ordinary component defects are detected, the circuit board is determined to be unqualified and is picked out, and step S4 is performed on the remaining circuit boards.

[0074] The board with the surface mount assembly is fed into the reflow oven through a transfer mechanism. The preheating, reflow and cooling processes are completed in sequence according to the preset reflow soldering temperature profile, so that the solder paste melts and solidifies during the cooling process to form solder joints, thus obtaining the soldered board.

[0075] The process involves acquiring images of the soldered circuit board, aligning a standard template with the soldered circuit board image, extracting local images of each component from the soldered circuit board image based on the positions and dimensions of each component recorded in the standard template, and performing a difference operation between these images and the corresponding local images of the template to obtain differential images of each component after soldering. These differential images of each component after soldering are then input into a coarse inspection model to obtain the component defects of each component after soldering.

[0076] When a marked component is found to have defects after soldering, the circuit board is deemed unqualified and rejected. Step S4 is then executed for the remaining circuit boards. The coarse inspection model is called again after soldering, and the judgment is only made for components marked during the placement stage. This is based on the fact that the reflow soldering process itself has a self-correcting effect on some placement deviations, but it does not have a substantial corrective effect on serious placement errors and missing components. For positions in the circuit board image after placement that have been marked as ordinary component defects by the coarse inspection model, if there is only a slight offset or uneven solder paste distribution, the surface tension of the molten solder during reflow soldering can often provide a certain degree of traction and self-alignment of the component position. This allows the differential image of the corresponding component in the post-soldering circuit board image to recover to a normal state that conforms to the standard template. At this point, re-entering the coarse inspection model will result in a "no component defect" judgment, thus automatically filtering out these small defects that can be self-corrected by the process, avoiding the immediate rejection of the board during the placement stage, and improving the fault tolerance and utilization rate of the production line.

[0077] Conversely, for components marked during the surface mount stage that are genuinely missing, severely misaligned, or have completely incorrect polarity, even after reflow soldering, the differential image of the corresponding component in the post-soldering circuit board image will still show a structural difference that is significantly inconsistent with the standard template. Re-entering the coarse inspection model will still identify it as a component defect. In the post-soldering stage, only these "marked components that still show defects after soldering" are used as the criterion for judgment, directly identifying the circuit board as unqualified and removing it. This achieves automatic rejection of serious, non-self-correctable defects in ordinary components, intercepting such problems before they enter fine inspection and subsequent power-on testing, avoiding ineffective subsequent inspection and rework costs.

[0078] Through this two-stage coarse inspection collaborative mechanism of "marking after placement + re-inspection after soldering", the present invention leaves minor ordinary component defects that are easily corrected by reflow soldering to the process itself without interrupting the production cycle. At the same time, components that are stably identified as component defects by the coarse inspection model in both process stages are eliminated at the board level. This realizes the graded processing and screening of ordinary component defects, reduces the redundant workload in the fine inspection stage, and improves the overall quality baseline of the circuit board entering step S4 after soldering.

[0079] Step S4: Acquire X-ray images of the soldered circuit board, call the fine inspection model, and perform fine inspection on the circuit board based on the soldered circuit board image and X-ray image and output the quality inspection results.

[0080] Furthermore, X-ray images and images of the circuit board after welding were acquired, and local images and X-ray images of each solder joint were extracted based on a standard template.

[0081] Local images and X-ray images of the solder joints are input into the precision detection model, and precision detection features for each solder joint are obtained through a feature extraction network. Subsequently, a cosine similarity value is calculated for each solder joint pair based on its precision detection features, and an assimilation bias is calculated based on the cosine similarity value and a template edge reference library. The assimilation bias is used to determine whether the solder joint pair meets the edge defect judgment criteria. Let the edge defect judgment threshold be τ (τ represents the assimilation bias threshold). When... When the value is greater than or equal to τ, the edge defect judgment condition is considered to be satisfied. If the value is less than τ, the edge defect judgment condition is not met. The edge defect judgment threshold τ is determined by validation set calibration during the fine detection model establishment stage: the assimilation deviation distribution of validation samples is selected from real defect data, and the assimilation deviation value that achieves the best compromise between the two is selected as τ, based on the criterion that "the false alarm rate of edge defects does not exceed the preset upper limit and the false alarm rate of edge defects is minimized".

[0082] When a solder joint pair meets the edge defect judgment criteria, the circuit board containing the solder joint pair is deemed unqualified, and the fine inspection ends. When a solder joint does not meet the edge defect judgment condition, the fine inspection features of each solder joint are input into the defect judgment network to determine whether the solder joint has a point defect. When a point defect exists in the solder joint, the circuit board where the solder joint is located is output as unqualified and the fine inspection ends. When no point defect exists in the solder joint, the circuit board where the solder joint is located is output as qualified and the fine inspection ends.

[0083] Example 2, in an exemplary embodiment, also provides a visual recognition-based online quality inspection system for production lines, including a modeling module, which acquires historical production line data, acquires standard templates for circuit boards, marks key components and ordinary components based on the standard templates, and establishes coarse inspection models and fine inspection models based on historical production line data.

[0084] The initial inspection module acquires images of the circuit board after surface mounting. Based on the standard template and the images of the circuit board after surface mounting, it calls the coarse inspection model to detect component defects. If a critical component defect is detected, an alarm is issued and the production line is stopped. If a common component defect is detected, it is marked and the process continues to step S3.

[0085] The coarse inspection module sends the circuit board into the reflow oven for soldering, acquires images of the soldered circuit board, calls the coarse inspection model based on the circuit board images to re-inspect component defects, and if a marked ordinary component defect is detected, the circuit board is determined to be unqualified and is picked out. Step S4 is then executed for the remaining circuit boards.

[0086] The precision inspection module acquires X-ray images of the soldered circuit board, calls the precision inspection model, and performs precision inspection on the circuit board based on the soldered circuit board image and the X-ray image, outputting solder joint defects and completing online quality inspection of the production line.

[0087] If the above functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0089] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0090] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0091] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A visual recognition-based online quality inspection method for production lines, characterized in that, include: Step S1: Obtain historical data from the production line, obtain standard templates for circuit boards, mark key components and ordinary components based on the standard templates, and establish a coarse inspection model and a fine inspection model based on historical data from the production line. Step S2: Acquire an image of the board after surface mounting. Based on the standard template and the image of the board after surface mounting, call the coarse inspection model to detect component defects. If a critical component defect is detected, issue an alarm and stop the production line. If a common component defect is detected, mark it and continue to step S3. Step S3: Send the circuit board into the reflow oven for soldering, acquire the image of the soldered circuit board, call the coarse inspection model to inspect the component defects again based on the circuit board image, if the marked ordinary component defects are detected, the circuit board is determined to be unqualified and is picked out, and step S4 is executed for the remaining circuit boards. Step S4: Acquire X-ray images of the soldered circuit board, call the precision inspection model, and perform precision inspection on the circuit board based on the soldered circuit board image and X-ray image, and output solder joint defects to complete the online quality inspection of the production line.

2. The online quality inspection method for production lines based on visual recognition as described in claim 1, characterized in that, The historical data of the production line includes standard templates for circuit boards, historical circuit board images, X-ray images of circuit boards after historical soldering, and actual defect data; the historical circuit board images include historical images of circuit boards after surface mounting and historical images of circuit boards after historical soldering; the actual defect data includes component defect data and solder joint defect data; The standard template of the circuit board is a standard reference data in image space used to describe the ideal component layout and shape of the circuit board, including the position of the component in the standard template, the outline area of ​​the component, and the mounting direction of the component. The key components include large-package integrated circuits, BGA packaged devices, QFN packaged devices, large-volume connectors and sockets, while other components on the circuit board are ordinary components.

3. The online quality inspection method for production lines based on visual recognition as described in claim 2, characterized in that, The establishment of the coarse detection model includes: For each component in the standard template, a template local image of the component is set. A local image corresponding to the template local image of the component is cropped from the historical circuit board image. The local image and the template local image are normalized and differentially processed to obtain a differential image. Based on the component defect data, the component defects in the differential images are labeled. A coarse detection model is established based on the convolutional neural network. All differential images obtained from historical circuit board images and the labeled component defects are combined into a training sample set. The cross-entropy loss function is used to constrain the consistency between the output probability of the coarse detection model on the training sample set and the labeled component defects, and the coarse detection model is trained. The coarse detection model takes the difference image as input and outputs the defect score vector of each component in the difference image. The component defect score vector is then converted into the component defect probability through the Softmax function.

4. The online quality inspection method for production lines based on visual recognition as described in claim 3, characterized in that, The precision detection model consists of a mask region convolutional neural network and an edge defect determination module.

5. The online quality inspection method for production lines based on visual recognition as described in claim 4, characterized in that, The training process of the convolutional neural network in the mask region includes: Based on historical X-ray images of circuit boards after soldering, and according to the position of the solder joints in the standard template, local X-ray images of historical solder joints are cropped from the historical X-ray images of circuit boards after soldering, and local X-ray images of historical solder joints are cropped from the historical X-ray images of circuit boards after soldering. Solder joint defects are divided into point defects and edge defects. Based on historical solder joint X-ray local images and historical solder joint local images, combined with solder joint defect data in real defect data, a mask region convolutional neural network is trained under supervision. The feature extraction network in the mask region convolutional neural network is used to obtain the precise detection features of the solder joint, and the defect judgment network is used to judge point defects based on the precise detection features of the solder joint. During the supervised training process, the model parameter update is constrained by the overall objective function, which consists of three weighted parts: point defect multi-classification loss, edge defect binary classification loss, and edge assimilation bias ranking loss. The point defect multi-class loss adopts the multi-class cross-entropy loss function; The binary classification loss for edge defects adopts the binary cross-entropy loss function. The assimilation bias ranking loss is calculated by dividing solder joint pairs into defective solder joint pairs and normal solder joint pairs according to the actual defect data, calculating the assimilation bias of each solder joint pair, randomly matching several normal solder joint pairs for each defective solder joint pair to construct a defect-normal pairing set, and calculating the ranking loss with the constraint that the assimilation bias of the defective solder joint pair is higher than that of the normal solder joint pair. When the constraint is satisfied, the assimilation bias ranking loss is zero; when the constraint is not satisfied, the assimilation bias ranking loss is included and accumulated.

6. The online quality inspection method for production lines based on visual recognition as described in claim 4, characterized in that, The edge defect determination module includes: Based on the position of the solder joints in the standard template, the solder joint pair spacing threshold of the standard template is calculated. For each solder joint, all solder joints that are less than the solder joint pair spacing threshold are included as the set of solder joint pairs for that solder joint. The solder joint and each solder joint in the set of solder joint pairs form a solder joint pair. Select R qualified circuit boards, and for each solder joint pair on the qualified R circuit boards, calculate the cosine similarity value using the fine detection features of the solder joints. Calculate the mean and standard deviation of the cosine similarity values ​​of the solder joint pairs on the R circuit boards to form a template edge reference library for that solder joint pair. Calculate the cosine similarity value of the weld point pair to be tested. Based on the cosine similarity value of the weld point pair to be tested and the template edge reference library, calculate the assimilation deviation. Based on the template edge reference library and the assimilation deviation of the weld point pair to be tested, determine the edge defects.

7. The online quality inspection method for production lines based on visual recognition as described in claim 1, characterized in that, Step S2 specifically involves: Acquire images of the circuit board after surface mounting, and align the standard template with the images of the circuit board after surface mounting; Based on the positions and dimensions of each component recorded in the standard template, a partial image of each component is extracted from the image of the board after surface mounting, and the difference is performed between the partial image and the corresponding template image to obtain the difference image of each component. The differential images of each component are input into the coarse detection model one by one to obtain the component defects of each component; When a defect in a critical component is detected, an alarm is issued and the production line is stopped to notify technicians to handle the situation. When a defect in a common component is detected, the component with the defect is marked, no alarm is issued, and the circuit board continues to be sent to the reflow oven for soldering. If no component defects are detected, the circuit board is sent to the reflow oven for soldering.

8. The online quality inspection method for production lines based on visual recognition as described in claim 1, characterized in that, Step S3 specifically involves: The circuit board is sent to a reflow oven for soldering to obtain the soldered circuit board. Acquire images of the soldered circuit board and align the standard template with the soldered circuit board image; Based on the position and size of each component recorded in the standard template, a partial image of each component is extracted from the image of the soldered circuit board, and the difference is performed with the corresponding partial image of the template to obtain the difference image of each component after soldering. The differential images of each component after welding are input into the coarse detection model one by one to obtain the component defects of each component after welding. If the marked component is also found to have defects after soldering, the circuit board is determined to be unqualified and is rejected. Step S4 is then performed on the remaining circuit boards.

9. The online quality inspection method for production lines based on visual recognition as described in claim 1, characterized in that, Step S4 specifically involves: Acquire X-ray images and images of the circuit board after soldering. For each solder joint, extract local images and X-ray images of the solder joint based on a standard template. Local images and X-ray images of the solder joints are input into the precision detection model, and the precision detection features of each solder joint are obtained through a feature extraction network. For each pair of solder joints, a cosine similarity value is calculated based on the precise detection features of the solder joints, and an assimilation bias is calculated based on the cosine similarity value and the template edge reference library. The assimilation bias is used to determine whether the solder joint pair meets the edge defect judgment condition. When the solder joint pair meets the edge defect judgment condition, the circuit board containing the solder joint pair is output as unqualified and the fine inspection ends. When the solder joints meet the bridging criteria, the fine inspection features of each solder joint are input into the defect judgment network to determine whether there is a point defect. When there is a point defect, the circuit board where the solder joint is located is output as unqualified and the fine inspection ends. When there are no defects in the solder joint, the circuit board containing the solder joint is deemed qualified and the fine inspection is completed.

10. A visual recognition-based online quality inspection system for production lines, applied to the visual recognition-based online quality inspection method for production lines according to any one of claims 1 to 9, characterized in that, include: The modeling module acquires historical production line data, obtains standard templates for circuit boards, marks key components and ordinary components based on the standard templates, and establishes coarse inspection models and fine inspection models based on historical production line data. The initial inspection module acquires images of the circuit board after surface mounting. Based on the standard template and the images of the circuit board after surface mounting, it calls the coarse inspection model to detect component defects. If a critical component defect is detected, an alarm is issued and the production line is stopped. If a common component defect is detected, it is marked and the process continues to step S3. The coarse inspection module sends the circuit board into the reflow oven for soldering, acquires images of the soldered circuit board, calls the coarse inspection model based on the circuit board images to re-inspect component defects, and if a marked ordinary component defect is detected, the circuit board is determined to be unqualified and is picked out. Step S4 is executed for the remaining circuit boards. The precision inspection module acquires X-ray images of the soldered circuit board, calls the precision inspection model, and performs precision inspection on the circuit board based on the soldered circuit board image and X-ray image, outputting solder joint defects and completing online quality inspection of the production line.

Images