Method for 3D reconstruction of solder joint on PCB utilizing neural rendering

Abstract

A 3D reconstruction method for PCB solder joints based on neural rendering, including using the signed distance function (SDF) to represent the solder joint surface, combining the bidirectional reflectance distribution function (BRDF) with the calibrated incident angle to represent the light intensity of solder joint images, and then unifying them into the neural rendering framework to optimize. The method uses existing solder joint inspection equipment to collect solder joint image, and reconstructs high resolution 3D reconstruction of solder joint from a single image with low cost.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] Pursuant to 35 U.S.C. § 119 and the Paris Convention Treaty, this application claims foreign priority to Chinese Patent Application No. 202410128665.4 filed Jan. 30, 2024, the contents of which, including any intervening amendments thereto, are incorporated herein by reference. Inquiries from the public to applicants or assignees concerning this document or the related applications should be directed to: Matthias Scholl P.C., Attn.: Dr. Matthias Scholl Esq., 245 First Street, 18th Floor, Cambridge, MA 02142.BACKGROUND

[0002] The disclosure relates to the field of 3D reconstruction in computer vision and defect detection in Printed Circuit Board Surface Mounted Technology (PCB SMT), and more particularly, to a method of 3D reconstruction of a solder joint on a printed circuit board (PCB). The method is utilized for the detection of defects in the solder joint.

[0003] During the production process of a PCB, electronic components are placed on a pad with solder paste and connected to the pads through a reflow oven or high-temperature soldering iron to form solder joints. The quality of the solder joints determines the quality of signal conduction. Common defects in solder joint include missing solder, insufficient solder, and excessive solder. The current PCB production process requires the use of automated optical inspection (AOI) equipment to detect defects in solder joints. 3D solder joint AOI equipment mainly relies on high-precision laser sensors to achieve 3D imaging of solder joints, with the accuracy of the laser sensors correlating to higher costs. Conversely, 2D solder joint AOI equipment mainly uses three-color ring light sources and small-angle viewed cameras to collect images. The three-color ring light sources include red, green, and blue light sources positioned at different angles. The image of the solder joint under inspection is compared with a reference image to determine if the solder joint is acceptable. While the 2D image-based detection method is cost-effective but depends on the experience of inspectors.SUMMARY

[0004] The purpose of the disclosure is to achieve high-precision 3D reconstruction of solder joints using 2D solder joint images captured by widespread 2D AOI equipment without any increasing hardware costs. The disclosure uses the SDF to represent the solder joint surface, combines the BRDF with the calibrated incident angle to represent the light intensity of solder joint images, and then unifies them into the neural rendering framework to optimize. The proposed method uses existing widespread solder joint inspection equipment collects solder joint image, and reconstruct high resolution 3D reconstruction of solder joint from a single image with low cost.

[0005] A method for 3D reconstruction of a solder joint on a printed circuit board (PCB) utilizing neural rendering, and the method comprises:

[0006] Step 1: soldering a plurality of steel balls of different known diameters onto a blank PCB; providing an acquisition system comprising a red ring light source, a green ring light source, and a blue ring light source; adjusting an angle between each ring light source and an vertical axis to be between 10° and 70°; adjusting an intensity of each ring light source; collecting, using the acquisition system, a plurality of images of the plurality of steel balls; and calibrating an angle θ=[θr,θg,θb] between an incident light ray from each ring light source and a viewing direction v; performing multiple calibrations using the plurality of steel balls of different known diameters to determine an average angle;

[0007] Step 2: capturing, using the acquisition system, a PCB image; extracting an image patch Ip corresponding to a location of a solder joint in the PCB image; wherein a width of the image patch is assumed to be W and a height of the image patch is H, then IpϵRW×H;

[0008] Step 3: establishing a camera coordinate system with a camera center P as an origin; and defining the viewing direction {right arrow over (v)} along the negative direction of the z-axis of the camera coordinate system; sampling a light ray passing through each pixel in the image patch along the direction of the z-axis; sampling a plurality of spatial points {tilde over (x)} along the light ray passing through the camera center P; transforming the plurality of spatial points {tilde over (x)} into a normalized coordinates system; denoting the plurality of transformed spatial points as x; and encoding the coordinates of the plurality of transformed spatial points x as E(x); where, the number of the plurality of spatial points is m, the coordinates of the plurality of spatial points are denoted by {tilde over (x)}ϵRW×H×m; and E(x) is calculated as follows:E(x)=[x,sin (x), cos (x), sin (2x), cos (2x), . . . , sin (6x), cos (6x)]

[0009] Step 4: establishing a Signed Distance Field (SDF) multi-layer perceptron (MLP) network N1; inputting E(x) into the SDF MLP network to obtain a signed distance d from each of the plurality of spatial points to a surface of the solder joint; supposing F(x) represent the SDF; and defining an opacity function as follows:σ(x)=ϕ(F(E(x)))ϵ[0,1];where,ϕ⁡(F⁡(x))=s·e-s·F⁡(x)(1+e-s·F⁡(x)2)and; an analytical partial derivative where,∂F⁡(x)∂xof the SDF MLP network is calculated to obtain the normal {right arrow over (n)} of the plurality of transformed spatial points x; and s is a learnable parameter;Step 5: defining a mixed reflection model of light intensity as follows:c=Wd·Bd+Ws·Bscomputing two sets of spherical Gaussians for a specular component and a diffuse component, respectively;Bs=[eλ1(h→T⁢n→-1),eλ2(h→T⁢n→-1),… ,eλ1⁢0(h→T⁢n→-1)],Bd=[eλ1(i→T⁢n→-1),eλ2(i→T⁢n→-1),… ,eλ1⁢0(i→T⁢n→-1)],where Bs represents a spherical Gaussian basis for the specular component, Bd represents a spherical Gaussian basis for the diffuse component, λ1, λ2, . . . , λ10 represent preset parameters of the spherical Gaussians basis. {right arrow over (h)}, {right arrow over (n)}, ī represent an intermediate vector, a normal direction, and an incident light ray direction, respectively; where {right arrow over (h)}={right arrow over (v)}+{right arrow over (i)}; and the incident light ray direction is denoted as follows:i=(sin θi cos φi, sin θi sin φi, cos θi)where θi is a calibrated angle between the incident light ray and the viewing direction;φi=arctan⁢n→yn→x;{right arrow over (n)}x and {right arrow over (n)}y are components of the normal {right arrow over (n)} on the x-axis and y-axis, respectively;Step 6: constructing two Bidirectional Reflectance Distribution Function (BRDF) Multi-Layer Perceptron (MLP) networks N2 and N3 with identical network structure; inputting x into the two MLP networks N2 and N3; generating an output from each MLP network; wherein the output from BRDF MLP network N2 is a weight Wd corresponding to a 10-dimensional spherical Gaussians, and the output from BRDF MLP network N3 is a weight Ws corresponding to a 10-dimensional spherical Gaussians;c⁡(x)=Wd·Bd+Ws·Bsthe components of light intensity emitted by the red ring light source, the green ring light source, the blue ring light source at the plurality of transformed spatial points x are represented by cr, cg, cb, respectively;Step 7: rendering an image through σk, δk, ck by the rendering formula as follows:Tk=e-Σj=1k-1⁢σj⁢δj,C⁡(r)=∑k=1mTk(1-e-σk⁢δk)⁢ck.where, σk,δk, ck correspond to the variables of the k th spatial point along the light ray r through an image pixel; δk=tk+1−tk is a distance between two adjacent spatial points; tk is a depth of the k th spatial point, and C(r) represents the color of the light ray r in the image;Step 8: minimizing an objective function as shown in the following formula to train N1, N2, N3 and s; setting the number of network training rounds nt; where, after the number of training rounds is reached, the training process stops; andminN1,N2,N3,s Loss=Ip-C2+<semantics definitionURL="">❘<annotation encoding="Mathematica">"\[LeftBracketingBar]"< / annotation>< / semantics>∇F⁡(x)<semantics definitionURL="">❘<annotation encoding="Mathematica">"\[RightBracketingBar]"< / annotation>< / semantics>2-12Step 9: outputting, using the SDF MLP network, the surface with d=0 by a matching cube algorithm; where, the output surface is the surface of solder joint.The disclosed method represents the surface geometry of the solder joints using SDF. The method integrates the bidirectional reflectance distribution function (BRDF) to model light behaves when interacting with the surface of the solder joint, thereby determining the visual appearance of the solder joints. The method unifies SDF and BRDF within a neural radiation field rendering framework to enhance rendering efficiency and accuracy. Furthermore, the method utilizes the conventional solder joint inspection equipment to capture images of solder joints. From the images, the method reconstructs high-resolution 3D models of the solder joints at low cost.BRIEF DESCRIPTION OF THE DRAWINGSFIG. 1 is the framework diagram of the reconstruction method;FIG. 2 is the reconstruction flow chart;FIG. 3 is the schematic diagram of image acquisition system;FIG. 4 is the 3D reconstruction result diagram of excessive solder from three different perspectives;FIG. 5 is the 3D reconstruction result diagram of insufficient solder from three different perspectives; andFIG. 6 is the 3D reconstruction result diagram of normal solder from three different perspectives.DETAILED DESCRIPTIONThe disclosed method presents a framework illustrated in FIG. 1, with a detailed implementation process shown in FIG. 2. FIG. 3 depicts a schematic diagram of an acquisition system featuring light sources, a camera, and solder joints. The acquisition system comprises a red ring light source, a green ring light source, and a blue ring light source. The camera is disposed directly above the three ring light sources. A solder joint is positioned below the three ring light sources.Step 1: soldering a plurality of steel balls with diameters of 1 mm, 2 mm, and 4 mm on a blank PCB; adjusting an intensity of each ring light source; collecting, using the acquisition system as shown in FIG. 3, a plurality of images of the plurality of steel balls; and calibrating an angle between z-axis and an incident light ray from each ring light source, respectively, θr=28°, θg=44°, θb=60°;Step 2: capturing, using the acquisition system, a PCB image; extracting an image patch Ip corresponding to a location of a solder joint in the PCB image; where, the width of the image patch is assumed to be W and the height is H, then IpϵRW×H.

[0030] Step 3: establishing a camera coordinate system with a camera center P as an origin; and defining a viewing direction v along the negative direction of the z-axis of the camera coordinate system; sampling a light ray r passing through each pixel within the image patch along the direction of the z-axis between [−0.5,0.9]; sampling 2m (m=64) spatial points x along the light ray; diving the 2m spatial points into m uniform coarse spatial points and m fine spatial points; where, the fine spatial points is determined by leveraging the neural radiance field (NeRF) weights obtained from the coarse sampling; denoting the spatial points as {tilde over (x)}ϵRW×H×128; transforming the spatial points {tilde over (x)} from the sampling space into normalized coordinates of the camera coordinate system; denoting the transformed spatial points as x;

[0031] setting a camera intrinsic matrix,K=[f0W0fH2001];where f=max (W,H);

[0033] defining a matrix as follows:x1=[(0⁢<semantics definitionURL="">,0,1),… ,(W,0,1)<annotation encoding="Mathematica">TagBox[RowBox[List[RowBox[List[RowBox[List[",", "0", ",", "1"]], ")"]], ",", "\[Ellipsis]", " ", ",", RowBox[List["(", RowBox[List["W", ",", "0", ",", "1"]], ")"]]]], "NumberComma", Rule[SyntaxForm, "0"]]< / annotation>< / semantics>…(0,H,1),… ,(W,H,1)],x=K-1·x~replacing the third dimension of x with the sampling plane distance zϵ[−0.5,0.9].

[0035] Step 4: encoding the coordinates of the transformed spatial points x as E(x)ϵR13×3; and calculating E(x) as follows:E(x)=[x, sin (x), cos (x), sin (2x), cos (2x), . . . , sin (6x), cos (6x)]Step 5: establishing a Signed Distance Field (SDF) multi-layer perceptron (MLP) network N1; inputting E(x) into the SDF MLP network to obtain a signed distance d from each of the plurality of spatial points to a surface of the solder joint; where, the SDF MLP network N1 comprises a 9-layer perceptron configured in a fully connected mode; the 9-layer perceptron comprises an input layer, seven hidden layers, and an output layer; the input layer comprises 39 neurons; each hidden layer comprises 256 neurons; and the output layer comprises one neuron; a 1-dimensional vector represents the distance d; the weight of SDF MLP network N is initialized using a method that is commonly applied to MLPs designed for handling SDF;

[0037] supposing F(x) represent the SDF; and defining the opacity function as follows:σ(x)=ϕ(F(E(x)))ϵ[0,1]where,ϕ⁡(F⁡(x))=s·e-s·F⁡(x)(1+e-s·F⁡(x)2)and; an analytical partial derivative∂F⁡(x)∂xof the SDF MLP network is calculated to obtain the normal n of the transformed spatial points x; and s is a learnable parameter;Step 6: computing two sets of spherical Gaussians for a specular component and a diffuse component, respectively.Bs=[eλ1(h→T⁢n→-1),eλ2(h→T⁢n→-1),… ,eλ10(h→T⁢n→-1)],Bd=[eλ1(i→T⁢n→-1),eλ2(i→T⁢n→-1),… ,eλ10(i→T⁢n→-1)];where Bs represents the spherical Gaussian basis for the specular component, Bd represents a spherical Gaussian basis for diffuse component, λ1, λ2, . . . , λ10 represent preset parameters of the spherical Gaussians. {right arrow over (h)}, {right arrow over (n)}, ī represent an intermediate vector, a normal direction, and an incident light ray direction, respectively; where {right arrow over (h)}={right arrow over (v)}+{right arrow over (i)}; and the incident light ray direction is denoted as follows:{right arrow over (i)}=(sin θi cos φi, sin θi sin φi, cos θi).where θi is an calibrated angle between the incident light ray and the viewing direction;φi=arc⁢tan⁢n→yn→x;{right arrow over (n)}x and {right arrow over (n)}y are components of the normal {right arrow over (n)} on the x-axis and y-axis, respectively;Step 7: constructing two BRDF MLP networks N2 and N3 to learn the specular component and diffuse component, respectively; where, the two BRDF MLP networks N2 and N3 share an identical network structure as follows: each BRDF MLP network comprises a 4-layer perceptron configured in a fully connected mode; the 4-layer perceptron comprises an input layer, two hidden layers, and an output layer; the input layer comprises three neurons; each of the two hidden layer comprises ten neurons; and the output layer comprises ten neurons, representing a 10-dimensional spherical Gaussian weight vector; the input to the sum perceptron is x, and the output of the perceptron corresponds to the weights Ws and Wd of the 10-dimensional spherical Gaussians basis,c⁡(x)=Wd·Bd+Ws·Bsthe components of light intensity emitted by the red ring light source, the green ring light source, and the blue ring light source at the transformed spatial points x are represented by cr, cg, cb, respectively;Step 8: rendering an image through σk, δk, ck by the rendering formula as follows:Tk=e-k-1∑j=1⁢σj⁢δj,C⁡(r)=∑k=1mTk(1-e-σk⁢δk)⁢ck.where, σk, δk, ck correspond to the variables of the k th spatial point along the light ray r, δk=tk+1−tk is the distance between two adjacent spatial points, tk is the depth of the k th spatial point, and C(r) represents the color of the light ray r in the image;Step 9: minimizing an objective function as shown in the following formula to train N1, N2, N3 and s; setting the number of network training rounds nt; where, the training process stops after 20000 rounds; andminN1,N2,N3,sLoss=Ip-C2+<semantics definitionURL="">❘<annotation encoding="Mathematica">"\[LeftBracketingBar]"< / annotation>< / semantics>∇ F⁡(x)<semantics definitionURL="">❘<annotation encoding="Mathematica">"\[RightBracketingBar]"< / annotation>< / semantics>2-12Step 10: computing the distance d of the spatial points by the SDF MLP network, and outputting the surface with |d|≤ε by the matching cube algorithm; where, the output surface is the surface of the solder joint; and ε is set to a value of 0.FIG. 4 shows the reconstruction result of excessive solder from three different perspectives, with a viewpoint positioned at P=(−0.5,−0.25, 1).FIG. 5 shows the reconstruction result of insufficient solder from three different perspectives, with a viewpoint positioned at P=(0, 0, 1).FIG. 6 shows the normal solder reconstruction result from three different perspectives, with a viewpoint positioned at P=(−0.5,−0.5, 1).It will be obvious to those skilled in the art that changes and modifications may be made, and therefore, the aim in the appended claims is to cover all such changes and modifications.

Claims

1. A method for 3D reconstruction of a solder joint on a printed circuit board (PCB) utilizing neural rendering, the method comprising:1) soldering a plurality of steel balls of different known diameters onto a blank PCB; providing an acquisition system comprising a red ring light source, a green ring light source, and a blue ring light source; collecting, using the acquisition system, a plurality of images of the plurality of steel balls;2) establishing a camera coordinate system with a camera center P as an origin; and defining a viewing direction {right arrow over (v)} along the negative direction of the z-axis of the camera coordinate system;3) calibrating an angle θ=[θr, θg, θh] between an incident light ray from each ring light source and the viewing direction; and performing multiple calibrations using the plurality of steel balls of different known diameters to determine an average angle;4) capturing, using the acquisition system, a PCB image; extracting an image patch Ip corresponding to a location of a solder joint in the PCB image;wherein a width of the image patch is assumed to be W and a height of the image patch is H;5) sampling a light ray passing through each pixel in the image patch along the direction of the z-axis; sampling a plurality of spatial points {tilde over (x)} along a light ray passing through the camera center P; transforming the spatial points {tilde over (x)} into a normalized coordinates system; denoting the plurality of transformed spatial points as x; and encoding the coordinates of the plurality of transformed spatial points x as E(x); wherein the coordinates of the plurality of spatial points are denoted by {tilde over (x)}ΣRW×H×m, and the number of the plurality of spatial points is m; and E(x) is calculated as follows:E(x)=[x,sin (x),cos (x),sin (2x), cos (2x), . . . ,sin (6x), cos (6x)];6) establishing a Signed Distance Field (SDF) Multi-Layer Perceptron (MLP) network N1; inputting E(x) into the MLP network to obtain a signed distance d from each of the plurality of spatial points to a surface of the solder joint; supposing F(x) represent the SDF, and defining an opacity function as follows:σ(x)=ϕ(F(E(x)))ϵ[0,1];where,ϕ⁡(F⁡(x))=s·e-s·F⁡(x)(1+e-s·F⁡(x)2) and; an analytical partial derivative∂F⁡(x)∂x of the SDF MLP network is calculated to obtain the normal {right arrow over (n)} of the plurality of transformed spatial points x; and s is a learnable parameter;7) computing two sets of spherical Gaussians for a specular component and a diffuse component, respectively;Bs=[eλ1(h→T⁢n→-1),eλ2(h→T⁢n→-1),… ,eλ10(h→T⁢n→-1)];⁢Bd=[eλ1(i→T⁢n→-1),eλ2(i→T⁢n→-1),… ,eλ10(i→T⁢n→-1)];where Bs represents a spherical Gaussian basis for the specular component, Bd represents a spherical Gaussian basis for the diffuse component, λ1, λ2, . . . , λ10 represent preset parameters of the spherical Gaussians basis; {right arrow over (h)},{right arrow over (n)},ī represent an intermediate vector, a normal direction, and an incident light ray direction, respectively; where {right arrow over (h)}={right arrow over (v)}+{right arrow over (i)}; and the incident light ray direction is denoted as follows:{right arrow over (i)}=(sin θi cos φi, sin θi sin φi, cos θi)where θi is a calibrated angle between the incident light ray and the viewing direction;φi=arc⁢tan⁢n→yn→x,{right arrow over (n)}x and {right arrow over (n)}y are components of the normal {right arrow over (n)} on the x-axis and y-axis, respectively; a mixed reflection model of light intensity is given byc=Wd·Bd+Ws·Bs;8) constructing two Bidirectional Reflectance Distribution Function (BRDF) Multi-Layer Perceptron (MLP) networks N2 and N3 with identical network structure; inputting x into both the two MLP networks N2 and N3; generating an output from each MLP network; wherein the output from MLP network N2 is a weight Wd corresponding to a 10-dimensional spherical Gaussians, and the output from MLP network N3 is a weight Ws corresponding to a 10-dimensional spherical Gaussians; the components of intensity of light emitted by the red ring light source, the green ring light source, and the blue ring light source at the transformed sampled spatial points x are represented by cr, cg, cb;c⁡(x)=Wd·Bd+Ws·Bs;9) rendering an image C through σk, δk, ck by the rendering formula:Tk=e-k-1∑j=1⁢σj⁢δj,C⁡(r)=∑k=1mTk(1-e-σk⁢δk)⁢ck.where, σk, δk, ck correspond to the variables of the k th spatial point along a light ray r through an image pixel; δk=tk+1−tk is a distance between two adjacent spatial points; tk is a depth of the k th spatial point; and C(r) represents the color of the light ray r in the image;10) minimizing an objective function as shown in the following formula to train N1, N2, N3 and s; setting a number of network training rounds n1; wherein after the number of training rounds is reached, the training process stops; andminN1,N2,N3,sLoss=Ip-C2+<semantics definitionURL="">❘<annotation encoding="Mathematica">"\[LeftBracketingBar]"< / annotation>< / semantics>∇ F⁡(x)<semantics definitionURL="">❘<annotation encoding="Mathematica">"\[RightBracketingBar]"< / annotation>< / semantics>2-1211) computing a distance d of the plurality of spatial points by the SDF MLP network, and outputting a surface with |d|≤ε by a matching cube algorithm; wherein, the output surface is the surface of the solder joint; and ε is set to a value of 0.

2. The method of claim 1, wherein the SDF MLP network N1 comprises a 9-layer perceptron configured in a fully connected mode; the 9-layer perceptron comprises an input layer, seven hidden layers, and an output layer; the input layer comprises three neurons; each hidden layer comprises 256 neurons; and the output layer comprises one neuron.

3. The method of claim 1, wherein the BRDF MLP network N2 comprises a 4-layer perceptron configured in a fully connected mode; the 4-layer perceptron comprises an input layer, two hidden layers, and an output layer; the input layer comprises three neurons; each hidden layer comprises ten neurons; and the output layer comprises ten neurons.

Images