A method for positioning PCB solder-free optical devices
Through the fine screening method of multi-scale image processing and corner point significance score, the failure problem of traditional technology in the positioning of small and complex shapes of optical devices is solved, and high-precision device positioning and rapid assembly are achieved.
Patent Information
- Application Number
- CN202411305957.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-19
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-09-19
AI Technical Summary
In the welding-free positioning process of PCB optics, traditional edge detection and corner point extraction methods fail when facing small, complex-shaped optics and complex PCB environments, resulting in difficulty in feature extraction and affecting positioning accuracy and recognition reliability.
The fine screening method of multi-scale image processing and corner point significance score is adopted to extract key feature points of tiny non-standard optical devices, and high-precision device positioning is achieved through automated processing.
Accurate positioning and rapid assembly of micro non-standard optical devices is achieved, production efficiency and consistency are improved, and the robustness and adaptability of the system are enhanced.
Smart Images

Figure CN118823126B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of electronic component positioning, and in particular to a PCB soldering-free optical device positioning method. Background Art
[0002] There is a key technical difficulty in the solder-free positioning process of PCB optical devices. When the optical device is small in size and complex in shape, it becomes extremely difficult to accurately extract corner feature information from its image. Traditional edge detection and corner extraction often fail when facing high-reflectivity surfaces, irregular geometric shapes and tiny scales. In addition, due to the non-uniformity and surface texture of PCB substrate materials, background interference also poses challenges to feature extraction. On high-speed production lines, feature extraction needs to be completed in a very short time, which requires the algorithm to have extremely high real-time performance. However, increasing the complexity of the algorithm to enhance robustness will significantly increase the computational burden and cause processing delays. How to achieve fast, accurate and stable corner feature extraction for complex and changeable optical devices and PCB environments under limited computing resources and strict time constraints has become a technical problem that needs to be solved urgently. This problem directly affects the matching accuracy and recognition reliability of the subsequent positioning algorithm and is the key bottleneck of the entire PCB optical device solder-free positioning technology. Summary of the invention
[0003] In order to solve the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a method for positioning PCB solder-free optical devices. The method for positioning PCB solder-free optical devices can accurately extract and locate the key feature points of tiny non-standard optical devices through multi-scale image processing and fine screening of corner point significance scores, thereby achieving high-precision device positioning; the entire positioning process from image acquisition to feature extraction, screening, matching, and then to the generation of positioning information is automated, reducing manual intervention, and improving production efficiency and consistency, thereby achieving accurate positioning and rapid assembly of PCB solder-free tiny optical devices.
[0004] A PCB solder-free optical device positioning method according to the present invention comprises the following steps:
[0005] S1. Capture and downsample images of tiny non-standard optical devices to be assembled on a PCB board to obtain device image features;
[0006] S2, extracting corner candidate regions from device images at different scale levels, and preliminarily screening out initial feature sets of corners containing different materials and directions;
[0007] S3, for each corner point candidate in the initial corner point feature set, calculate the corner point significance score;
[0008] S4. Based on the corner point significance scores, secondary screening is performed on the candidate corner points, and a refined corner point feature point set of different optical device types is obtained;
[0009] S5. According to the refined corner feature point set, a stable corner feature set with uniform distribution is obtained through iterative convergence;
[0010] S6, extracting multi-scale contour edge information of the device, and calculating multi-scale curvature extreme points on the edge segments as potential corner points of complex shapes;
[0011] S7, fusing the complex shape corner points with the stable corner point feature set to construct a comprehensive corner point feature set;
[0012] S8. Construct a corner point topological structure model of the optical device based on the detected corner point comprehensive feature set, and match it with various optical device corner point template libraries to determine whether it meets the expected geometric layout;
[0013] S9. The corner point topological structure matching result is integrated with the corner point refined coordinate information to generate the positioning information of the optical device and transmit it to the PCB assembly control system.
[0014] Preferably, the step S1 specifically includes:
[0015] Collecting original images of tiny non-standard optical devices to be assembled on a PCB board, and downsampling the original images using a multi-scale image pyramid decomposition method to obtain device image features at different scale levels;
[0016] The downsampling process includes the following steps:
[0017] Preprocessing the original image to obtain a device image;
[0018] Constructing a Gaussian pyramid and a Laplacian pyramid of the device image, and obtaining image sequences and feature maps of different scales through downsampling and differential operations;
[0019] For the images at each scale layer of the pyramid, the SURF algorithm is used to extract scale-invariant features, calculate feature descriptors, and establish feature point sets;
[0020] The shape context descriptor is used to capture the global characteristics of the device and combined with the local features to form a complete device feature representation.
[0021] Preferably, the step S2 specifically includes:
[0022] According to the diversity of device materials, corner candidate regions are extracted from device images at different scale levels, and the corner candidate regions at each level are preliminarily screened to obtain an initial feature set of corner points of different materials and directions.
[0023] Preferably, the step S3 specifically includes:
[0024] For each candidate corner point in the initial feature set of the corner points, the texture features of the surrounding area are extracted, and the corner point significance score is calculated by combining the Harris corner point response function and introducing the device material recognition factor.
[0025] Preferably, the step S4 specifically includes:
[0026] Based on the corner significance score and according to the diversity of optical device types, different corner significance thresholds are set for different types of optical devices, and the corner candidate points are screened twice to obtain refined corner feature point sets for different optical device types.
[0027] Preferably, the step S5 specifically includes:
[0028] According to the refined corner feature point set of different optical device types, a non-maximum suppression algorithm is adopted, and the local neighborhood range is dynamically adjusted in combination with the corner point density. The corner point response intensity is compared in the local neighborhood, non-maximum points are suppressed, and the local optimal corner points are retained. Through iterative convergence, a stable corner point feature set with uniform distribution is obtained.
[0029] Preferably, the step S6 specifically includes:
[0030] Considering the complexity of device shape, the multi-scale contour edge information of the device is extracted, and the multi-scale curvature extreme points are calculated on the edge segments as potential corner points of complex shapes.
[0031] Preferably, the step S7 specifically includes:
[0032] The complex shape corner points are fused with the stable corner point feature set to construct a comprehensive feature set of corner points of different types, scales and directions.
[0033] Preferably, the step S8 specifically includes:
[0034] According to the detected comprehensive feature set of corner points, the geometric topological relationship between adjacent corner points of the optical device is calculated, and the corner point topological structure model of the optical device is constructed. It is matched with various optical device corner point template libraries to determine whether the corner point combination of the current optical device conforms to the expected geometric layout.
[0035] Preferably, the step S9 specifically includes:
[0036] The corner point topology structure model is matched with various optical device corner point template libraries to obtain the corner point topology structure matching result of the optical device, and the corner point topology structure matching result is integrated with the refined coordinate information of the corner point of the optical device to generate the positioning information of the optical device;
[0037] Convert the positioning information into an instruction format recognizable by a PCB assembly control system, and transmit the instruction format to the PCB assembly control system via the ModbusTCP protocol;
[0038] The instruction format includes device type, installation coordinates, posture angle, assembly force and assembly method.
[0039] The PCB solder-free optical device positioning method described in the present invention has the advantages that:
[0040] A PCB solder-free optical device positioning method of the present invention can accurately extract and locate key feature points of tiny non-standard optical devices through multi-scale image processing and fine screening of corner point significance scores, thereby realizing high-precision device positioning; combining corner point features of different materials and directions, as well as extraction of corner points of complex shapes, enables the system to cope with diverse optical devices and complex PCB board environments, thereby improving the robustness and adaptability of the system; the entire positioning process, from image acquisition to feature extraction, screening, matching, and then to the generation of positioning information, realizes automated processing, reduces manual intervention, and improves production efficiency and consistency, thereby realizing accurate positioning and rapid assembly of PCB solder-free tiny optical devices. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 The present invention is a flowchart of a method for positioning a PCB solder-free optical device. DETAILED DESCRIPTION
[0042] like Figure 1 As shown, a PCB solder-free optical device positioning method according to the present invention comprises the following steps:
[0043] S1. Capture and downsample images of tiny non-standard optical devices to be assembled on a PCB board to obtain device image features;
[0044] S2, extracting corner candidate regions from device images at different scale levels, and preliminarily screening out initial feature sets of corners containing different materials and directions;
[0045] S3, for each corner point candidate in the initial corner point feature set, calculate the corner point significance score;
[0046] S4. Based on the corner point significance scores, the candidate corner points are screened for a second time, and a refined corner point feature point set of different optical device types is obtained;
[0047] S5. According to the refined corner feature point set, a stable corner feature set with uniform distribution is obtained through iterative convergence;
[0048] S6, extracting multi-scale contour edge information of the device, and calculating multi-scale curvature extreme points on the edge segments as potential corner points of complex shapes;
[0049] S7, fusing the complex shape corner points and the stable corner point feature set to construct a comprehensive corner point feature set;
[0050] S8. Construct a corner point topological structure model of the optical device based on the detected corner point comprehensive feature set, and match it with various optical device corner point template libraries to determine whether it meets the expected geometric layout;
[0051] S9. The corner point topological structure matching result is integrated with the corner point refined coordinate information to generate the positioning information of the optical device and transmit it to the PCB assembly control system.
[0052] Furthermore, in this embodiment, step S1 specifically includes:
[0053] The original image of the tiny non-standard optical device to be assembled on the PCB board is collected, and the original image is downsampled using a multi-scale image pyramid decomposition method to obtain device image features at different scale levels; the original image is collected by a high-resolution camera;
[0054] The downsampling process includes the following steps:
[0055] Preprocessing the original image to obtain the device image;
[0056] Construct Gaussian pyramid and Laplacian pyramid of device images, and obtain image sequences and feature maps of different scales through downsampling and differential operations;
[0057] For the images at each scale layer of the pyramid, the SURF algorithm is used to extract scale-invariant features, calculate feature descriptors, and establish feature point sets;
[0058] Use shape context descriptors to capture the global features of the device and combine them with local features to form a complete device feature representation;
[0059] Specifically, a high-resolution camera is used to obtain an original image of the optical device to be assembled on the PCB board, and the obtained original image is preprocessed, including using a median filter to remove noise, applying a histogram equalization to enhance the image contrast, and obtaining a device image after processing;
[0060] Gaussian pyramids and Laplacian pyramids are constructed for the processed device images. Image sequences and feature maps of different scales are obtained through multiple downsampling and differential operations, and multi-scale edge and corner point local features are extracted.
[0061] Use Gaussian filter to downsample the image to construct Gaussian pyramid, and obtain Laplacian pyramid through difference operation between adjacent layers;
[0062] For the images of each scale layer of the pyramid, the SURF algorithm is used to extract scale-invariant features, calculate feature descriptors, establish feature point sets, and determine the corresponding relationship of devices at different scales through feature matching;
[0063] Combining the characteristics of non-standard shapes, shape context descriptors are used to capture global features and combined with local features to form a complete device feature representation;
[0064] According to the matching results, multi-scale feature information is integrated and the fine structure of the device is reconstructed using inverse wavelet transform to obtain high-precision three-dimensional point cloud data;
[0065] The reconstructed 3D point cloud data is registered with the pre-established device CAD model, and the transformation matrix is calculated using the iterative closest point algorithm to obtain the precise pose parameters of the device for subsequent PCB assembly positioning.
[0066] Here is an example:
[0067] A high-resolution camera is used to obtain the original image of the optical device to be assembled on the PCB board, with an image resolution of 4096x3072 pixels;
[0068] The original image is preprocessed by using a 5x5 window median filter to remove noise, and then histogram equalization is applied to enhance the image contrast, expanding the pixel grayscale value range from the original [0,255] to [0,4095]. A 5-layer Gaussian pyramid and Laplacian pyramid are constructed for the processed image, with each layer image size being 1 / 2 of the previous layer, and the minimum image size being 256x192 pixels;
[0069] Use 3x3 Gaussian kernel for downsampling to construct Gaussian pyramid, and make differences between adjacent layers to get Laplacian pyramid;
[0070] The SURF algorithm is applied to the images of each scale layer, the Hessian matrix threshold is set to 500, and about 1000 feature points are extracted. Each feature point is described by a 64-dimensional vector.
[0071] Combining the characteristics of non-standard shapes, using shape context descriptors, 200 sampling points are selected, divided into 5 logarithmic distances and 12 angle intervals, and a 60-dimensional global feature vector is obtained;
[0072] Feature matching was performed using FLANN (Fast Nearest Neighbor Search Library), with the distance ratio threshold set to 0.7, to obtain about 500 pairs of matching points. The fine structure of the device was reconstructed using wavelet transform, and the Daubechies4 wavelet basis function was selected to perform 3-layer decomposition and reconstruction to obtain 3D point cloud data with a resolution of 2048x1536. The reconstructed point cloud data was aligned with the pre-established CAD model, and the CAD model accuracy was 0.01mm;
[0073] The iterative closest point (ICP) algorithm was used, with the maximum number of iterations set to 50 and the convergence threshold set to 0.001 mm. The rigid body transformation matrix was calculated to obtain the 6-DOF posture parameters of the device, including 3 translations (accuracy 0.05 mm) and 3 rotation angles (accuracy 0.1 degrees). These parameters are used to guide the precise assembly and positioning of optical devices on the PCB board.
[0074] Furthermore, in this embodiment, step S2 specifically includes:
[0075] According to the diversity of device materials, corner point candidate regions are extracted from device images at different scale levels, and the corner point candidate regions at each level are preliminarily screened to obtain an initial feature set of corner points of different materials and directions.
[0076] Specifically, the device image pyramid decomposition result is obtained, and the Harris corner detection algorithm is used to process the image of each scale level in the decomposition result;
[0077] Set the threshold of the corner response function to obtain a multi-scale corner candidate set, and use the OTSU adaptive threshold segmentation algorithm to divide the image in the multi-scale corner candidate set into regions;
[0078] Determine the material type of the corner point candidate area based on the reflectivity and texture features stored in the pre-established material database;
[0079] If the material of the corner candidate area is metal, the corner response threshold is increased to a preset value;
[0080] If the material of the corner candidate area is plastic, the corner response threshold is reduced to a preset value;
[0081] The main direction of the corner candidate area is calculated using the gradient direction histogram algorithm, dividing 360 degrees into a preset number of columns;
[0082] The direction corresponding to the column where the peak value is located is selected as the main direction, and the response values of the corner points of the three adjacent scale levels are compared; the corner point with the largest response value at the middle scale is retained;
[0083] Construct an initial feature set of corner points of different materials and directions, and weight the feature points according to the material type: assign a first preset weight to the feature points of metal material, assign a second preset weight to the feature points of glass material, and assign a third preset weight to the feature points of plastic material, to obtain a weighted feature set;
[0084] Here is an example:
[0085] According to the device image pyramid decomposition results, the Harris corner detection algorithm is applied to the image at each scale level, the corner response function threshold is set to 0.01, the corner candidate area is extracted, and a multi-scale corner candidate set is obtained;
[0086] Based on the optical properties of different materials, the OTSU adaptive threshold segmentation algorithm is used to divide the images at each scale level, and the material classification of the corner point candidate areas is performed based on the reflectivity and texture features stored in the pre-established material database.
[0087] The material database contains reflectivity curves and gray-level co-occurrence matrix features of common optical device materials such as glass, plastic, and metal;
[0088] According to the material classification results, the parameters of Harris corner detection are dynamically adjusted, increasing the corner response threshold to 0.015 for metal material areas and reducing the threshold to 0.008 for plastic material areas;
[0089] The main direction of each corner candidate area is calculated using the gradient direction histogram algorithm, 360 degrees is divided into 36 columns, and the direction corresponding to the column where the peak value is located is selected as the main direction;
[0090] If there are multiple similar peaks in the gradient direction histogram, they are judged as unstable corner points and removed. By comparing the corner point response values of the three adjacent scale levels, the corner point with the largest response value at the middle scale is retained to construct an initial feature set of corner points of different materials and directions.
[0091] Weight the feature points according to the material characteristics: give the metal material feature points a weight of 1.2, the glass material 1.0, and the plastic material 0.8 to balance the contribution of different materials in the feature set;
[0092] In the image processing of optical devices on PCB boards, a 5-layer image pyramid is first constructed. The original image resolution is 4096x3072 pixels, and each layer is reduced to 1 / 2 of the previous layer. The minimum image size is 256x192 pixels;
[0093] Harris corner detection algorithm is applied to each layer of image, and the initial corner response function threshold is set to 0.01. For example, about 5000 corner candidates are detected on the 2048x1536 pixel image of the third layer. Then, the OTSU adaptive threshold segmentation algorithm is used to divide the image into regions, and the threshold usually falls between grayscale values 80-120;
[0094] The material database stores reflectivity curves and gray-level co-occurrence matrix features of common optical device materials such as glass, plastic, and metal;
[0095] The reflectivity curve records the reflectivity value in the visible light range of 400-700nm at intervals of 10nm. The grayscale co-occurrence matrix selects four directions of 0°, 45°, 90°, and 135°, with a distance of 1 pixel and quantized into 16 levels of grayscale. According to the material classification results, the Harris corner detection threshold for the metal area is adjusted to 0.015, and the plastic area is adjusted to 0.008;
[0096] The main direction of the corner point is calculated using the gradient direction histogram algorithm, 360° is divided into 36 columns, each column is 10°, and the column with the highest peak value is selected as the main direction;
[0097] If there are multiple peaks and the second highest peak exceeds 80% of the highest peak, it is determined to be an unstable corner point and removed;
[0098] In the three adjacent scale levels, the corner point response values are compared. If the middle scale response value is the largest and exceeds 1.2 times the adjacent scale, the corner point is retained;
[0099] In the final feature set, the feature points of metal, glass, and plastic materials are assigned weights of 1.2, 1.0, and 0.8, respectively;
[0100] For example, in a set of 1,000 feature points, there may be 400 metal material points, 350 glass material points, and 250 plastic material points. The weighted number of effective feature points is 1,130, which balances the contributions of different materials.
[0101] Furthermore, in this embodiment, step S3 specifically includes:
[0102] For each candidate corner point in the initial feature set of the corner point, the texture features of the surrounding area are extracted, the Harris corner point response function is combined, and the device material recognition factor is introduced to calculate the corner point significance score;
[0103] Specifically, an 11x11 pixel window centered on the coordinates of the corner candidate point is obtained, and the texture features in the window are calculated by the gray level co-occurrence matrix, where the texture features include contrast, correlation, energy and homogeneity;
[0104] Calculate the response function value of the corner candidate point according to the Harris corner detection algorithm;
[0105] If the corner candidate point is located in the metal material area, set the k value in the response function to 0.06;
[0106] If the corner candidate point is located in the glass material area, set the k value in the response function to 0.04;
[0107] If the corner candidate point is located in the plastic material area, set the k value in the response function to 0.02;
[0108] The material database stored in the kd tree structure is used to extract the identification factor corresponding to the material of the area where the corner candidate point is located;
[0109] The identification factor is obtained by using the k-nearest neighbor algorithm to identify the material. The k value of the k-nearest neighbor algorithm is set to 5;
[0110] Combining texture features, corner response function values and material recognition factors, the significance score of the corner candidate is calculated; the significance score S is determined by the formula S=0.3*(T_norm)+0.5*(H_norm)+0.2*M;
[0111] Among them, T_norm is the normalized texture feature score, H_norm is the normalized Harris corner response value, and M is the material recognition factor;
[0112] Calculate the entropy value H_entropy of the local gradient direction histogram of the corner candidate point, and correct the significance score according to the entropy value H_entropy. The corrected significance score S_final is determined by the formula S_final=S*(1+0.1*H_entropy);
[0113] Here is an example:
[0114] According to the coordinates of each candidate corner point in the initial corner point feature set, an 11x11 pixel window centered on the point is extracted from the original image. For corner points close to the edge of the image, the image boundary is expanded using the mirror filling method.
[0115] The texture features in the window are calculated by gray-level co-occurrence matrix, and four directions of 0°, 45°, 90°, and 135° are selected with a distance of 1 pixel to calculate contrast, correlation, energy, and homogeneity;
[0116] The Harris corner detection algorithm is used to calculate the corner response function value, the window size is set to 3x3 pixels, the Gaussian kernel parameter σ is 1.5, and the k value in the corner response function is dynamically adjusted according to the material. The metal material is set to 0.06, the glass material is set to 0.04, and the plastic material is set to 0.02;
[0117] Extract the identification factor corresponding to the material of the area where the corner candidate point is located from the pre-established material database. The database uses a kd tree structure to store material feature vectors. The k nearest neighbor algorithm is used for material identification, and the k value is set to 5. In the material identification results, the metal material is assigned a value of 1.2, the glass material is assigned a value of 1.0, and the plastic material is assigned a value of 0.8;
[0118] The significance score of each corner candidate is calculated by combining texture features, Harris corner response function value and material recognition factor in a weighted summation method.
[0119] The calculation formula of the significance score S is: S=0.3*(T_norm)+0.5*(H_norm)+0.2*M, where T_norm is the normalized texture feature score, H_norm is the normalized Harris corner response value, and M is the material recognition factor;
[0120] Combining the directionality and complexity of local texture, the local gradient direction histogram is introduced as an additional feature, 360° is divided into 8 direction bins, the entropy value H_entropy of the gradient direction histogram is calculated, and the significance score is corrected to: S_final=S*(1+0.1*H_entropy);
[0121] In the process of extracting corner features of optical devices on PCB boards, the original image of 4096x3072 pixels is first preprocessed. For each initial corner candidate point, an 11x11 pixel window is extracted. For points close to the edge, such as those less than 5 pixels from the left border, the extended image is filled with a mirror image.
[0122] Use the gray-level co-occurrence matrix to calculate the texture features, select four directions of 0°, 45°, 90°, and 135°, with a distance of 1 pixel, and get a 16x16 matrix;
[0123] Calculate the four eigenvalues of contrast, correlation, energy and homogeneity. For example, the contrast of a metal surface is 0.85, the correlation is 0.72, the energy is 0.06, and the homogeneity is 0.92.
[0124] Harris corner detection algorithm was applied, with a window size of 3x3 pixels, Gaussian kernel σ=1.5, and k value adjusted according to the material: 0.06 for metal, 0.04 for glass, and 0.02 for plastic;
[0125] The material database uses a kd tree to store the feature vectors of 1000 samples and uses the k nearest neighbor algorithm (k=5) for identification. For example, the probability of a point being identified as metal is 0.8, and the material factor is 1.2. In the calculation of the significance score, the normalized value of the texture feature is 0.75, and the normalized value of the Harris corner response is 0.88, and the initial score S=0.30.75+0.50.88+0.21.2=0.915 is obtained;
[0126] Combined with the local texture, the gradient direction histogram in the 11x11 window is calculated. The values of the 8 direction bins are [0.15, 0.12, 0.18, 0.10, 0.14, 0.11, 0.09, 0.11], and the entropy value H_entropy=2.97;
[0127] The final saliency score S_final = 0.915(1+0.1*2.97) = 1.187, which reflects the importance of the corner point in the complex texture area. This process is repeated for all corner point candidates in the image to form a set of feature descriptors containing position, saliency score, and material information.
[0128] Furthermore, in this embodiment, step S4 specifically includes:
[0129] Based on the corner significance score and the diversity of optical device types, different corner significance thresholds are set for different types of optical devices, and the corner candidate points are screened twice to obtain a refined corner feature point set for different optical device types.
[0130] Specifically, a pre-established optical device database and an image to be identified are obtained, wherein the input size of the image to be identified is a preset pixel value;
[0131] A preset convolutional neural network is used to identify the device type of the image to be identified, and a device type identification result is obtained;
[0132] If the device type recognition result contains overlapping devices, the region proposal network is used to generate candidate regions;
[0133] The candidate regions are screened by the non-maximum suppression algorithm to obtain the device type label and confidence score of each region;
[0134] According to the device type label, an initial corner point significance threshold corresponding to the device type label is obtained from a preset corner point significance threshold database;
[0135] For each device area in the image to be identified, the density distribution of the candidate corner points is calculated; a Gaussian kernel density estimation algorithm is used to obtain a corner density map, and the kernel bandwidth of the Gaussian kernel density estimation algorithm is set to a preset ratio of the width of the image to be identified;
[0136] According to the corner point density map and the initial corner point significance threshold, the adjusted significance threshold is calculated;
[0137] Perform secondary screening on the corner point candidates, retain the corner points whose significance scores are higher than the adjusted significance threshold, and obtain a refined corner point feature point set;
[0138] Here is an example:
[0139] Based on the pre-established optical device database, the ResNet50 convolutional neural network is used to identify the type of devices in the image. The network input size is 224x224 pixels and the cross entropy loss function is used for training.
[0140] For overlapping devices, the region proposal network is used to generate candidate regions, which are then screened using the non-maximum suppression algorithm to obtain the device type label and confidence score for each region.
[0141] For different types of optical devices, the initial corner significance threshold is set based on historical data statistical analysis, such as 0.82 for lens devices, 0.75 for prism devices, and 0.68 for optical fiber connectors;
[0142] Adjust the corner detection strategy according to the diversity of device types, and increase the sensitivity of the Harris corner detector for highly complex devices such as prisms;
[0143] For devices with smooth surfaces, such as lenses, increase the threshold of the FAST corner detector;
[0144] For each identified optical device area, the density distribution of the candidate corner points is calculated, and the Gaussian kernel density estimation algorithm is used to obtain the corner density map, and the kernel bandwidth is set to 1 / 20 of the image width;
[0145] Combined with the device type and corner point density, the corner point significance threshold is dynamically adjusted. The adjustment formula is: new threshold = initial threshold * (1 + 0.1 * normalized density value);
[0146] Based on the adjusted significance threshold, the candidate corner points are screened again, and the corner points with significance scores higher than the threshold are retained. For the corner points with scores within the range of ±0.05 of the threshold, whether to retain them is determined based on the distribution of the surrounding corner points.
[0147] Form a refined corner feature point set for each type of optical device, and record the position coordinates, significance score and device type of each feature point;
[0148] By calculating the spatial distribution entropy of the feature points of each device type, the uniformity of the screening results is verified. If the entropy value is lower than the preset threshold, the significance threshold of the corresponding device type is adjusted and the screening process is repeated until the distribution requirements are met.
[0149] In the process of extracting the corner features of optical devices, the ResNet50 convolutional neural network is first used to perform device recognition on the 4096x3072 pixel PCB image;
[0150] The network was trained on 100,000 sample images and eventually achieved an accuracy of 95% on the validation set. For the identified overlapping devices, such as two adjacent lenses, a region proposal network was used to generate 50 candidate regions. The non-maximum suppression algorithm was used to retain regions with an IoU less than 0.5, and finally three device regions were obtained with confidence levels of 0.92, 0.88, and 0.85, respectively.
[0151] Based on historical data analysis, the initial significance thresholds for lenses were set to 0.82, prisms to 0.75, and fiber connectors to 0.68;
[0152] For the identified prisms, the k value of the Harris corner detector was reduced from 0.04 to 0.03 to improve the sensitivity. For the lens, the threshold of the FAST corner detector was increased from 20 to 25, and the Gaussian kernel density estimation algorithm was used to calculate the corner density. The kernel bandwidth was set to 204.8 pixels (1 / 20 of the image width). For example, in a certain prism area, the original significance threshold was 0.75, and the calculated normalized density value was 0.6. The new threshold was adjusted to 0.75*(1+0.1*0.6)=0.795.
[0153] The 1000 candidate corner points in the prism area are screened, and the corner points with a significance score higher than 0.795 are retained. For corner points with a score in the range of 0.745-0.845, if there are more than 2 corner points with a significance score higher than the threshold in their 8-neighborhood, they are retained;
[0154] Finally, 350 feature points were obtained, and the calculated spatial distribution entropy was 3.2, which was lower than the preset threshold of 3.5. Therefore, the significance threshold was lowered by 0.01 to 0.785, and 380 feature points were re-screened. The spatial distribution entropy was increased to 3.6, which met the uniform distribution requirement.
[0155] Repeat this process for all devices, and finally form a refined corner feature point set containing position coordinates, significance scores and device type labels;
[0156] Further, the corner points falling in the key area are extracted according to the corner point significance score, and the density range of the corner point feature points and the device type in the key area is calculated. If the density of the corner point feature points in the key area is lower than the lower limit of the range, the threshold is lowered by a gradient and the corner point feature points are re-extracted; if the density is higher than the upper limit of the range, the threshold is increased by a gradient and the corner point feature points are re-extracted; if the density is within the range, the current threshold is determined as the optimal threshold of the device;
[0157] receiving a corner saliency score carrying a key region of the optical device, the corner saliency score being generated when the optical device image is processed;
[0158] According to the corner point significance score, the region growing algorithm is used to divide the key area of the optical device, and the boundary coordinates of the key area of each device are obtained; the density of the corner point feature points in the key area is calculated by the nearest neighbor distance method, and the distance from each corner point to its nearest neighbor corner point is counted, and the average value is taken as the density index;
[0159] Obtaining a density range of optical device types from a preset database, where the preset database is constructed based on statistical analysis results of historical images;
[0160] Determine whether the density index of the corner feature points is within the preset range of the device type. If it is lower than the lower limit of the preset range, lower the significance threshold; if it is higher than the upper limit of the preset range, increase the significance threshold;
[0161] Repeat the density calculation and threshold adjustment process until the optimal threshold is obtained or the maximum number of iterations is reached;
[0162] If the optimal threshold value is not obtained after the maximum number of iterations, the threshold value closest to the midpoint of the target range in the iteration process is selected as the final threshold value;
[0163] Here is an example:
[0164] According to the corner point significance score, the region growing algorithm is used to divide the key area of the optical device. The point with the highest significance score is selected as the seed point, and the growth threshold is set to 80% of the significance score to obtain the boundary coordinates of the key area of each device.
[0165] Calculate the density of corner feature points in the key area. Use the nearest neighbor distance method to count the distance from each corner point to its nearest neighbor corner point, take the average value as the density index, and divide the result by the square root of the key area for normalization.
[0166] Obtain the density range of this type of device from a preset database. The database is built based on the statistical analysis results of 10,000 historical images and contains the density distribution of various types of devices under different working conditions.
[0167] Adjust the density assessment standards according to the device type. For example, for precision optical components, the allowable density range will be reduced by 20%;
[0168] Determine whether the current density is within the preset range. If it is lower than the lower limit of the range, the significance threshold is lowered by (lower limit value - current value) / current value 0.1;
[0169] If it is higher than the upper limit of the range, the significance threshold will be increased by (current value - upper limit) / upper limit 0.1;
[0170] If it is within the range, the current threshold is recorded as the optimal threshold of the device;
[0171] For the corner feature points re-extracted after threshold adjustment, repeat the density calculation and threshold adjustment process until the optimal threshold is found or the maximum number of iterations is 10;
[0172] If the optimal threshold value is not found after the maximum number of iterations, the threshold value closest to the midpoint of the target range in the iteration process is selected as the final threshold value, and the optimal significance threshold value for each device is finally determined, and the result is updated to the preset database for reference in subsequent processing;
[0173] In the process of extracting corner features of optical devices, taking a PCB image of 4096x3072 pixels as an example, the region growing algorithm is first applied to a certain lens device;
[0174] Select the point with the highest significance score (coordinate 2048,1536, score 0.95) as the seed point, and set the growth threshold to 0.76 (80% of 0.95);
[0175] After 500 iterations, the key area with boundary coordinates from (1980,1470) to (2116,1602) is obtained, with an area of 18496 square pixels. In this area, the density of corner points is calculated using the nearest neighbor distance method. The average distance is 5.8 pixels, and the normalized density is 0.0426 (5.8 divided by the square root of 136).
[0176] According to the preset database, the density range of this type of lens under normal working conditions is 0.035-0.055. Considering that this is a precision optical component, the range is reduced by 20% to 0.038-0.052;
[0177] The current density of 0.0426 is within the range, and the current significance threshold of 0.82 is recorded as the optimal threshold. To verify the results, the thresholds are adjusted to 0.77 and 0.87 and the calculations are repeated, and the densities are 0.0362 and 0.0538, respectively, both exceeding the range. After 3 iterations, 0.82 is confirmed to be the optimal threshold;
[0178] In contrast to the processing of other devices, such as prisms, the initial threshold of 0.75 resulted in a density of 0.0612, which was outside the range of 0.045-0.060;
[0179] After 4 iterations, the thresholds were adjusted to 0.7875, 0.8156, 0.8052, and 0.8104, respectively, and the final density was 0.0573, which was within the target range. These optimal thresholds were updated to the database for reference in the subsequent processing of 2000 PCB images.
[0180] Furthermore, in this embodiment, step S5 specifically includes:
[0181] According to the refined corner feature point set of different optical device types, the non-maximum suppression algorithm is used to dynamically adjust the local neighborhood range in combination with the corner density, compare the corner response intensity in the local neighborhood, suppress the non-maximum points, retain the local optimal corner points, and obtain a stable corner feature set with uniform distribution through iterative convergence;
[0182] Specifically, a refined corner feature point set of different optical devices is obtained, and the corner point density is calculated based on the corner feature point set. The corner point density is obtained by counting the average distance of the K nearest neighbor corner points of each corner point through the K nearest neighbor algorithm.
[0183] Generate a corner point density distribution map, and use the corner point density distribution map to dynamically determine the local neighborhood range of the non-maximum suppression algorithm; judge the response strength of each corner feature point, if the response strength of the corner feature point is the local maximum, then retain the corner feature point, otherwise suppress the corner feature point;
[0184] The spatial distribution uniformity of the screened corner point set is calculated, and the spatial distribution uniformity is evaluated by the Ripley's K function to evaluate the degree of clustering of the corner point distribution;
[0185] If the spatial distribution uniformity does not meet the preset threshold, the parameters of the non-maximum suppression algorithm are adjusted to repeat the corner point screening process;
[0186] According to the feature point distribution requirements of the optical device type, the non-maximum suppression threshold of the local area is adjusted to obtain a stable corner point feature set that meets the specific requirements of various optical devices;
[0187] Here is an example:
[0188] According to the refined corner feature point set of different optical device types, the corner point density of each device is calculated, and the K nearest neighbor algorithm is used to count the average distance of the K nearest neighbor corner points of each corner point. The K value is dynamically set according to the device type, such as K=5 for lenses, K=8 for prisms, and K=6 for optical fiber connectors;
[0189] Based on the calculated corner point density distribution map, dynamically adjust the local neighborhood range of the non-maximum suppression algorithm, set a smaller neighborhood range in areas with high corner point density, such as setting a 5x5 pixel neighborhood when the density is greater than 0.8, and set a larger neighborhood range in areas with low corner point density, such as setting a 15x15 pixel neighborhood when the density is less than 0.2, to generate a neighborhood range map;
[0190] For each corner feature point, the corner response strength is compared in its corresponding local neighborhood. The Gaussian curvature is used to calculate the response strength for lens devices, and the Harris corner response function is used to calculate the response strength for prism devices. The local maximum point is retained, and the non-maximum point is suppressed to obtain a set of corner points after preliminary screening.
[0191] Calculate the uniformity of the spatial distribution of the corner point set after screening, and use Ripley's K function to evaluate the degree of clustering of the corner point distribution. In the scale range of 0 to 50 pixels, calculate the K value every 5 pixels and compare it with the theoretical K value of completely random spatial distribution. If the actual K value exceeds 1.2 times the theoretical K value, it is judged as non-uniform distribution;
[0192] If the uniformity does not meet the preset threshold, adjust the parameters of the non-maximum suppression algorithm, such as increasing or decreasing the step size of the neighborhood range, and repeat the corner point screening and uniformity evaluation process until the preset uniformity requirement is met or the number of iterations reaches 20. Finally, the screening results are verified according to the feature point distribution requirements of different optical device types. For example, lenses require that corner points are evenly distributed along the edges, and prisms require that corner points are densely distributed at the edges. By adjusting the non-maximum suppression threshold of the local area, ensure that the final corner point feature set meets the specific requirements of various optical devices;
[0193] In the 4096x3072 pixel PCB image, for the identified lens area (about 500 pixels in diameter), the K nearest neighbor algorithm is used to calculate the corner density, and K=5 is set to obtain an average nearest neighbor distance of 15 pixels. According to the density distribution, a 7x7 pixel non-maximum suppression neighborhood is set in the lens edge area (density 0.9), and a 13x13 pixel neighborhood is set in the central area (density 0.3);
[0194] For each corner point, the Gaussian curvature was calculated as the response intensity, and the curvature radius threshold was set to 10 pixels. After preliminary screening, 200 corner points were retained in the lens area;
[0195] The distribution uniformity was evaluated using Ripley's K function, with the K value calculated every 5 pixels in the range of 0-50 pixels;
[0196] The results show that at a scale of 30 pixels, the actual K value is 1.3 times the theoretical value, which exceeds the threshold of 1.2 and is considered to be non-uniformly distributed. The non-maximum suppression parameters are adjusted, the edge area neighborhood is expanded to 9x9 pixels, and the center area is reduced to 11x11 pixels. 180 corner points are obtained by re-screening.
[0197] The second Ripley's K function analysis showed that the maximum K value ratio was 1.18, which met the uniformity requirements. Combined with the lens characteristics, the edge corner point spacing was required to be no less than 10 pixels. Through local threshold adjustment, 160 evenly distributed corner points were finally obtained, including 120 on the edge and 40 in the center. The whole process was iterated 7 times and took a total of 1.2 seconds, which met the feature point distribution requirements of the lens device.
[0198] Furthermore, in this embodiment, step S6 specifically includes:
[0199] Considering the complexity of the device shape, the multi-scale contour edge information of the device is extracted, and the multi-scale curvature extreme points are calculated on the edge segments as potential corner points of complex shapes.
[0200] Specifically, the original image is obtained and multi-scale decomposition is performed using a Gaussian pyramid to obtain an image sequence of 5 scale levels;
[0201] According to the image sequence, the Canny edge detection algorithm is applied at each scale level to extract the device contour in the image sequence;
[0202] For the device contour, the Douglas-Peucker algorithm is used to perform edge segmentation processing to obtain simplified contour segments;
[0203] By using the sliding window method on the simplified contour segments, a sequence of local curvature values is calculated;
[0204] If there are extreme curvature points in the local curvature value sequence, the scale space feature point matching method is used to associate the extreme curvature points of different scales; based on the association results, a multi-scale feature descriptor is formed to determine the potential corner points of complex shapes;
[0205] Here is an example:
[0206] The original image is decomposed into five scale levels using a Gaussian pyramid. The image size of each level is half of the previous level. The minimum image size is 256x192 pixels. The image edge is filled with a mirror image with a filling width of 15 pixels.
[0207] The Canny edge detection algorithm is applied at each scale level to extract the device contour, and the σ value of the low-pass filter decreases from 1.4 at the bottom layer to 0.4 at the top layer.
[0208] The contour image of each scale level is processed by edge segmentation, and the Douglas-Peucker algorithm is used to simplify the contour. The simplification threshold is dynamically set based on the image resolution and device size. The threshold of the bottom layer is 0.1% of the image diagonal length, and the threshold of the top layer is 0.00625%;
[0209] On the simplified edge segments, the sliding window method is used to calculate the local curvature. The window size varies with the scale, with 21 pixels at the bottom layer and 3 pixels at the top layer. The window moving step is 1 pixel, and circular filling is used at the boundary.
[0210] The curvature value is calculated by the difference approximation method, and the curvature is calculated by the three-point method. The one-dimensional maximum suppression algorithm is applied to the calculated curvature value sequence to extract the curvature extreme points at each scale level. The suppression radius is set to 1 / 3 of the window size, and the threshold is set to 1.5 times the mean curvature of the scale.
[0211] The shape complexity evaluation index is introduced to calculate the fractal dimension of the contour, and the extraction density of the curvature extreme points is adjusted according to the fractal dimension;
[0212] The scale-space feature point matching method is used to associate the curvature extreme points of different scales to form a multi-scale feature descriptor, and the potential corner points of complex shapes are finally determined by comprehensively considering the information of each scale;
[0213] When processing a PCB image of 4096x3072 pixels, a 5-layer Gaussian pyramid is first constructed with a minimum image size of 256x192 pixels;
[0214] The image edge is filled with a 15-pixel-wide mirror to ensure the integrity of the edge information. The Canny algorithm is applied to each layer, with σ=1.4 for the bottom layer and σ=0.4 for the top layer to extract the contour. Taking a complex optical device as an example, 2000 edge pixels are detected at the bottom layer. The Douglas-Peucker algorithm is applied to simplify the contour. The bottom layer threshold is set to 5 pixels (0.1% of the diagonal length) and the top layer is 0.3 pixels (0.00625%). After simplification, 300 points are retained at the bottom layer.
[0215] The curvature is calculated using a sliding window, with a bottom window of 21 pixels, a step size of 1 pixel, and a boundary loop filling. The curvature is calculated using the three-point method, such as the curvature value at (100, 100) is 0.05;
[0216] One-dimensional maximum suppression extracts the curvature extreme points, with a suppression radius of 7 pixels and a threshold of 1.5 times the mean curvature of the layer, 0.03, i.e., 0.045;
[0217] The calculated contour fractal dimension is 1.3. Based on this value, the density of curvature extreme point extraction is increased by 20%. Through scale space feature matching, such as matching the bottom layer (100,100) point with the upper layer (50,50) point, a multi-scale descriptor is formed.
[0218] By integrating information from five scales, 50 potential corner points of complex shapes were finally identified on the device. Among them, the corner point (100, 100) responded at all three scales and was determined to be a stable corner point of a complex shape. The whole process took 1.5 seconds, laying the foundation for subsequent precise positioning and feature extraction.
[0219] Furthermore, in this embodiment, step S7 specifically includes:
[0220] The complex shape corners and stable corner feature sets are merged to construct a comprehensive feature set of corners of different types, scales and directions.
[0221] Specifically, the feature set of complex shape corner points and stable corner points is obtained, and the DBSCAN algorithm is used to perform spatial clustering on the feature set;
[0222] For the spatial clustering results, the center point of each cluster is calculated as the representative point;
[0223] According to the representative points, multi-scale descriptors are extracted, and the multi-scale descriptors include HOG features, SIFT features and LBP features;
[0224] Perform principal component analysis on the multi-scale descriptors to obtain the feature descriptors after dimensionality reduction;
[0225] If the feature descriptor after dimensionality reduction meets the preset conditions, a weighted voting mechanism is used to select representative corner points based on the spatial distribution of corner points, feature descriptor similarity, response strength, scale and direction information;
[0226] Obtaining the spatial uniformity index and feature diversity index of the representative corner points, and determining whether the spatial uniformity index and feature diversity index meet the preset thresholds;
[0227] If the spatial uniformity index and feature diversity index meet the preset threshold, the representative corner points are determined as the final feature set;
[0228] If the spatial uniformity index and feature diversity index do not meet the preset threshold, the parameters of the DBSCAN algorithm are adjusted and spatial clustering is re-executed;
[0229] Here is an example:
[0230] The complex shape corner points and stable corner point feature sets are spatially clustered using the DBSCAN algorithm. The clustering radius is set to 0.5% of the image diagonal length, and the minimum sample number is 0.1% of the total number of expected corner points in the image. Corner points with similar spatial positions are grouped together to obtain the corner point clustering results.
[0231] For each cluster, the cluster center is calculated as the representative point, and the multi-scale descriptor of the point is extracted, including the HOG feature of the 8x8 pixel block (9 direction bins), the SIFT feature of 4 scales (128 dimensions), and the LBP feature with a radius of 1, 2, and 3 pixels, forming a 324-dimensional feature vector, recording the scale information (Gaussian pyramid level) and direction information (main direction angle) of the point;
[0232] The principal component analysis method is used to reduce the dimensionality of the eigenvector. First, the 324x324 covariance matrix is calculated, and the eigenvalue decomposition is performed. The eigenvectors corresponding to the first 64 largest eigenvalues are selected as the projection matrix, and the original eigenvectors are projected into the 64-dimensional space to construct a new feature descriptor. At the same time, the response intensity of each corner point is calculated, and the Harris response value and curvature value are comprehensively considered, and the weighted average of the two is taken (the weights are 0.6 and 0.4 respectively);
[0233] Introduce corner type labels (complex shape or stable corners) and direction quantization (divide 360 degrees into 8 direction intervals). Based on the spatial distribution of corners (weight 0.2), feature descriptor similarity (weight 0.3), response strength (weight 0.2), scale (weight 0.15) and direction (weight 0.15) information, a weighted voting mechanism is used to select the most representative corners. Each attribute is scored according to its ranking in the cluster, and the corner with the highest total score is selected as the representative. Construct the final comprehensive feature set, each feature point contains six attributes: position, descriptor, response strength, scale, direction and type;
[0234] The spatial uniformity index and feature diversity index are used to evaluate the quality of the feature set. The spatial uniformity is measured by calculating the coefficient of variation of the nearest neighbor distance, and the feature diversity is measured by the entropy value of corner points of different types, scales, and directions. When the comprehensive score is lower than the threshold, the DBSCAN parameters are adjusted to re-cluster until the quality requirements are met.
[0235] When processing a 4096x3072 pixel PCB image, DBSCAN clustering is first performed on 1000 complex shape corners and 2000 stable corners, with the cluster radius set to 25 pixels (0.5% of the diagonal length) and the minimum number of samples set to 3 (0.1% of the total number of corners), resulting in 150 clusters;
[0236] Take one of the clusters as an example, which contains 7 corner points. The center point (1024,768) is calculated as the representative point. The multi-scale descriptor of the point is extracted. The HOG feature of the 8x8 pixel block is used to obtain an 81-dimensional vector, the SIFT feature of 4 scales is used to obtain a 128-dimensional vector, and the LBP feature of 3 radii is used to obtain a 115-dimensional vector, which are merged into a 324-dimensional feature vector. The scale information (the second layer of the pyramid) and the direction information (main direction 45 degrees) are recorded.
[0237] Through principal component analysis, the first 64 eigenvalues of the 324x324 covariance matrix account for 95% of the total. The corresponding eigenvectors are used to construct the projection matrix, and the eigenvectors are reduced to 64 dimensions. The response intensity is calculated, the Harris response value is 0.8, the curvature value is 0.6, and the weighted average is 0.72. The corner point type is complex shape, and the direction is quantized to 45 degree intervals;
[0238] In the weighted voting, the point had a spatial distribution score of 0.9, feature similarity of 0.85, response intensity of 0.72, scale of 0.8, direction of 0.75, and a total score of 3.79, ranking first in the cluster and being selected as the representative point;
[0239] The final feature set contains 150 feature points, the spatial uniformity index (nearest neighbor distance variation coefficient) is 0.3, and the feature diversity index (type, scale, directional entropy value) is 2.5, which meets the preset threshold requirements and does not require re-clustering. The whole process takes 2.3 seconds, providing a comprehensive corner feature representation for subsequent PCB image analysis;
[0240] Furthermore, different types of corner points are classified into complex shapes and stability, and the scale attribute of the corner points is analyzed. According to the change of local curvature of the edge curve where the corner points are located, small-scale and large-scale corner points are distinguished;
[0241] Directional attribute analysis is performed on the corner points. By calculating the main direction of the edge curve in the neighborhood of the corner points, a set of corner points in different directions is obtained. According to the type attribute, scale attribute and direction attribute of the corner points, a corner point feature descriptor is constructed to form a corner point feature set.
[0242] Specifically, the edge curve in the neighborhood of the corner point is obtained, and the fractal dimension of the edge curve is calculated using the box counting method;
[0243] The type of corner point is determined according to the fractal dimension. If the fractal dimension is greater than a preset threshold, the corner point is marked as a complex shape corner point.
[0244] For the edge curve where the corner point is located, the sliding window method is used to calculate the curvature value, and the scale attribute of the corner point is determined according to the window size where the maximum curvature value appears. If the maximum curvature value appears in a window smaller than the preset number of pixels, the corner point is determined to be a small-scale corner point;
[0245] In a neighborhood of preset size around the diagonal point, the Sobel operator is used to calculate the gradient magnitude and direction of each pixel;
[0246] The gradient direction histogram of edge pixels is calculated based on the gradient amplitude and direction, and the direction corresponding to the bin with the maximum response in the gradient direction histogram is selected as the main direction of the corner point;
[0247] Construct a corner feature descriptor based on the type attribute, scale attribute and direction attribute of the corner point;
[0248] Feature descriptors include type attributes, scale attributes, direction attributes and local features, among which local features are extracted and reduced in the corner neighborhood using the SIFT algorithm;
[0249] The feature descriptor is normalized by L2 norm to obtain a corner feature set including position, type, scale, direction and feature descriptor. The corner feature set is scale and rotation invariant.
[0250] Here is an example:
[0251] The shape complexity index is used to distinguish the types of corner points, and the fractal dimension of the edge curve in the neighborhood of the corner point is calculated. The box counting method is used to count the number of boxes required to cover the edge curve from 1 pixel to 10 pixels, and the fractal dimension is obtained by double logarithmic regression.
[0252] If the fractal dimension is greater than the preset threshold of 1.5, the corner point is marked as a complex shape, otherwise it is marked as a stable corner point. The scale attribute analysis is performed based on the local curvature change of the edge curve where the corner point is located, and the curvature value is calculated using the sliding window method. The window size ranges from 5 pixels to 21 pixels, and the step size is 2 pixels;
[0253] Use the three-point method to calculate the curvature. Select the first, middle and last points in the window. Use these three points to determine a circle. The curvature of the circle is the curvature value of the window.
[0254] If the maximum curvature value appears in a window smaller than 11 pixels, it is determined as a small-scale corner point, otherwise it is determined as a large-scale corner point;
[0255] Directional attribute analysis is performed on the corner points. In the 21x21 pixel neighborhood around the corner points, the Sobel operator is used to calculate the gradient magnitude and direction of each pixel. The gradient direction histogram of edge pixels is counted. 360 degrees are divided into 18 bins, each with a width of 20 degrees. The gradient magnitude is used as the weight and accumulated into the corresponding bin. The direction corresponding to the bin with the maximum response is selected as the main direction of the corner point.
[0256] According to the type attribute, scale attribute and direction attribute of the corner point, a 64-dimensional corner point feature descriptor is constructed, of which the type attribute occupies 2 dimensions (one-hot encoding), the scale attribute occupies 2 dimensions (one-hot encoding), and the direction attribute occupies 18 dimensions (gradient direction histogram). The remaining 42 dimensions use the SIFT algorithm to extract local features in the 16x16 pixel neighborhood of the corner point, and divide the area into 4x4 sub-regions. The gradient histogram of 8 directions is calculated for each sub-region to obtain 4x4x8=128-dimensional features, which are reduced to 42 dimensions through principal component analysis. For corner points located at the edge of the image, the image boundary is extended by the mirror filling method. Finally, the 64-dimensional feature descriptor is normalized by the L2 norm to improve its scale and rotation invariance, and finally a corner point feature set containing position, type, scale, direction and feature descriptors is obtained;
[0257] When processing a 4096x3072 pixel PCB image, the 1000 detected corner points are first analyzed. Taking one of the corner points (1024,768) as an example, the box counting method is used to calculate the fractal dimension in its 31x31 pixel neighborhood;
[0258] Using box sizes from 1 to 10 pixels, the number of boxes covering the edge is 310, 158, 82, 43, 23, 13, 8, 5, 3, and 2 respectively;
[0259] Through double logarithmic regression, the fractal dimension is calculated to be 1.62, which is greater than the threshold of 1.5, so it is marked as a complex shape corner. Then the scale analysis is performed, using a sliding window of 5 to 21 pixels with a step size of 2 pixels, and the curvature is calculated by the three-point method, and the curvature value sequence is obtained: 0.15, 0.12, 0.09, 0.08, 0.07, 0.06, 0.05, 0.05, 0.04;
[0260] The maximum curvature of 0.15 appears in the 5-pixel window and is determined to be a small-scale corner point. In the direction analysis, the Sobel operator is used to calculate the gradient in the 21x21 neighborhood, and the 18-bin direction histogram is obtained: [12, 18, 25, 42, 78, 120, 156, 187, 201, 198, 176, 143, 98, 65, 43, 32, 24, 19];
[0261] The main direction corresponds to the 9th bin, i.e. 160°-180°. When constructing a 64-dimensional feature descriptor, complex shapes correspond to [1,0] and small scales correspond to [1,0]. The 18-dimensional direction histogram occupies 18 dimensions after normalization.
[0262] The remaining 42 dimensions are extracted in a 16x16 neighborhood using the SIFT algorithm to obtain 128-dimensional features, which are then reduced to 42 dimensions using PCA. Finally, the 64-dimensional features are L2 normalized to obtain the final descriptor.
[0263] The whole process was repeated for 1,000 corner points, taking about 2.5 seconds, forming a complete corner point feature set including position, type, scale, direction and descriptor, laying the foundation for subsequent PCB image analysis.
[0264] Furthermore, in this embodiment, step S8 specifically includes:
[0265] According to the detected comprehensive feature set of corner points, the geometric topological relationship between adjacent corner points of the optical device is calculated, the corner point topological structure model of the optical device is constructed, and it is matched with various optical device corner point template libraries to determine whether the corner point combination of the current optical device meets the expected geometric layout;
[0266] Specifically, a comprehensive feature set of corner points is obtained, and the initial topological relationship between the corner points is constructed using the Delaunay triangulation algorithm;
[0267] According to the initial topological relationship, the Euclidean distance, angle and relative direction between each pair of adjacent corner points are calculated to obtain the feature vector of the corner point pair;
[0268] A hierarchical clustering algorithm is used to group corner points. If the clustering distance is less than the preset threshold, the corner points are classified into the same cluster to obtain multiple corner point clusters.
[0269] For each corner point cluster, the spatial distribution characteristics of the corner points in the cluster are calculated, including the corner point density, the mean and variance of the distance between corner points, and the consistency of the corner point direction, and the corner point topological structure model of the optical device is constructed;
[0270] The corner topology structure model is matched with the pre-established optical device corner template library, and the structure similarity is calculated using the graph edit distance algorithm;
[0271] If the structural similarity exceeds a preset threshold, it is determined that the corner point combination of the current optical device meets the expected geometric layout;
[0272] Conduct credibility assessment on the matching results. If the comprehensive score exceeds the preset threshold, the matching result is confirmed to be valid.
[0273] Here is an example:
[0274] According to the detected comprehensive feature set of corner points, the Delaunay triangulation algorithm is used to construct the initial topological relationship between the corner points, and the Euclidean distance, angle and relative direction between each pair of adjacent corner points are calculated to form the feature vector of the corner point pair;
[0275] For the corner points at the edge of the image, add virtual boundary points, and set the virtual point position to 20 pixels outside the edge corner point along the boundary;
[0276] Based on the feature vectors of corner point pairs, a hierarchical clustering algorithm is used to group the corner points. The single linkage method is used and the clustering threshold is set to 1.5 times the average distance of the corner point pairs. Multiple corner point clusters are obtained, each cluster representing a potential optical device.
[0277] For each corner point cluster, the spatial distribution characteristics of the corner points in the cluster are calculated, including the corner point density (the number of corner points per unit area), the mean and variance of the distance between corner points, and the consistency of the corner point direction (obtained by calculating the variance of the corner point direction vector), and the corner point topological structure model of the optical device is constructed;
[0278] The constructed corner point topological structure model is matched with the pre-established corner point template library of various optical devices, and the structural similarity is calculated using the graph edit distance algorithm. The specific steps include the calculation of node replacement cost, edge replacement cost and structural deformation cost, and the matching threshold is set to 0.8;
[0279] If the similarity exceeds the threshold, it is judged that the corner point combination of the current optical device meets the expected geometric layout. In the case of uncertain matching results, a multi-template voting mechanism is used to select the first three most similar templates for secondary matching, and the final matching result is determined based on the weighted score;
[0280] The credibility of the matching results is evaluated, taking into account the ratio of the number of corner points to the total number of template corner points (the threshold is set to 0.85) and the completeness of the spatial distribution of corner points (obtained by calculating the corner point coverage, with the threshold set to 0.9). When the comprehensive score exceeds the preset threshold of 0.88, the matching result is confirmed to be valid;
[0281] When processing a 4096x3072 pixel PCB image, 500 corner points are detected, and the Delaunay triangulation algorithm is applied to build the initial topological relationship, generating about 1400 triangles. For the 50 corner points close to the edge, virtual points are added 20 pixels outside the boundary;
[0282] Calculate the corner point pair features. For example, the distance between the corner points (1024,768) and (1056,792) is 40 pixels, the angle is 26.57 degrees, and the relative direction is 0.82.
[0283] Using single-link hierarchical clustering, we set the threshold to 60 pixels (1.5 times the average distance of 40 pixels) and obtained 20 corner point clusters. Taking one of the clusters containing 25 corner points as an example, we calculated that the corner point density is 0.005 / square pixel, the mean distance between corner points is 35 pixels, the variance is 8 pixels, and the directional consistency (directional vector variance) is 0.15;
[0284] The cluster is compared with 100 preset templates in the template library, using the graph edit distance algorithm, with a node replacement cost of 0.3, an edge replacement cost of 0.5, and a structural deformation cost of 0.2. The similarity of the best matching template is 0.85, which exceeds the threshold of 0.8;
[0285] For another cluster with a similarity of 0.75, multi-template voting was used, and the scores of the first three similar templates were 0.75, 0.72, and 0.68, respectively, and the weighted score was 0.73;
[0286] Finally, the matching credibility was evaluated. The 25 corner points of the cluster accounted for 83.3% of the 30 corner points of the template, which was lower than the 85% threshold. The spatial distribution completeness was 0.92, which exceeded the 0.9 threshold. The comprehensive score was 0.87, which was slightly lower than the 0.88 threshold. It was marked for further verification. The whole process took 1.8 seconds, providing a reliable geometric layout analysis basis for subsequent PCB device identification.
[0287] Furthermore, in this embodiment, step S9 specifically includes:
[0288] By matching the corner point topology structure model with various optical device corner point template libraries, the corner point topology structure matching result of the optical device is obtained, and the corner point topology structure matching result is integrated with the refined coordinate information of the corner point of the optical device to generate the positioning information of the optical device;
[0289] Receive the corner point topological structure matching results and corner point refined coordinate information of the optical device;
[0290] According to the matching results and coordinate information, the RANSAC algorithm is used to screen matching point pairs and eliminate outliers;
[0291] The least square method is used to transform the coordinates of the selected matching point pairs to obtain the position and direction information of the optical device in the PCB coordinate system;
[0292] Obtain PCB design drawings and device size information, and calculate the installation position of the optical device on the PCB according to the PCB design drawings and device size information;
[0293] The A* algorithm is used to optimize the installation position to obtain the optimal installation position after avoiding obstacles;
[0294] The position deviation of the optical device is estimated using the Kalman filter algorithm;
[0295] The state vector of the Kalman filter algorithm contains position, velocity, and acceleration, and the observation vector is the position measurement value;
[0296] Generate compensated positioning information based on the position deviation estimation result;
[0297] Convert the positioning information into an instruction format recognizable by the PCB assembly control system, and transmit the instruction format to the PCB assembly control system through the ModbusTCP protocol;
[0298] The instruction format includes device type, installation coordinates, posture angle, assembly force and assembly method;
[0299] Here is an example:
[0300] The corner point topology structure matching results are integrated with the refined coordinate information of the corner points. The RANSAC algorithm is used to screen the matching point pairs and remove the outliers. Then, the least squares method is used to transform the coordinates of the screened matching point pairs to calculate the precise position and direction of the optical device in the PCB coordinate system.
[0301] According to the type and size information of the optical device, combined with the PCB design drawings, the ideal installation position of the device on the PCB is calculated, and the A* algorithm is used to avoid obstacles. Considering the layout of other components on the PCB, the installation position is optimized and compared with the actual detection position to obtain the position deviation value;
[0302] The Kalman filter algorithm is used to estimate and correct the position deviation. The state vector contains position, velocity and acceleration, and the observation vector is the position measurement value. Combined with PCB manufacturing error and image distortion factors, the compensated positioning information is generated.
[0303] Select appropriate solder-free assembly methods according to device types, such as crimping or snap-on, convert positioning information and assembly methods into instruction formats recognizable by the PCB assembly control system, including device type, installation coordinates, posture angle, assembly force and assembly method parameters, and transmit them to the control system through the ModbusTCP protocol;
[0304] When processing a 4096x3072 pixel PCB image, 50 corner points of a microlens were detected;
[0305] The RANSAC algorithm was used to screen matching points, the threshold was set to 2 pixels, and 500 iterations were performed to screen out 45 valid matching points.
[0306] The coordinate transformation matrix is calculated using the least squares method, and the center position of the lens in the PCB coordinate system is (1024.5, 768.3) mm, and the rotation angle is 1.2°. Combined with the PCB design drawings, the ideal installation position is (1025.0, 768.0) mm;
[0307] The A* algorithm is used for obstacle avoidance, the grid size is set to 0.1 mm, and considering 10 surrounding components, the optimized installation position is (1024.8, 768.1) mm;
[0308] Kalman filtering was applied, with the state vector being [x, y, θ, vx, vy, ω], the observation vector being [x, y, θ], the process noise covariance Q being set to a diagonal matrix [0.01, 0.01, 0.001, 0.1, 0.1, 0.01], and the measurement noise covariance R being set to a diagonal matrix [0.1, 0.1, 0.01]. After 5 iterations, the corrected position was (1024.7, 768.2) mm, and the angle was 1.1°.
[0309] Select the crimping method for solder-free assembly, set the pressure to 5N, and write 6 holding registers using function code 16 through the ModbusTCP protocol, corresponding to x, y, θ, pressure, device type, and assembly method;
[0310] The visual sensor collects an image stream at 100fps and uses a template matching algorithm to track the lens position in real time. The deviation thresholds are set to 0.05mm and 0.5°. If the thresholds are exceeded, adjustment instructions are sent immediately. The entire process takes 0.5 seconds, achieving precise positioning and rapid assembly of tiny optical devices.
[0311] In the description of the present invention, it is necessary to understand that the directions or positional relationships indicated by directional words such as "front, back, up, down, left, right", "lateral, vertical, horizontal" and "top, bottom" are usually based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description. Unless otherwise specified, these directional words do not indicate or imply that the device or element referred to must have a specific direction or be constructed and operated in a specific direction. Therefore, they cannot be understood as limiting the scope of protection of the present invention.
[0312] For those skilled in the art, various other corresponding changes and deformations can be made according to the technical solutions and concepts described above, and all of these changes and deformations should fall within the protection scope of the claims of the present invention.
Claims
1. A PCB solder-free optical device positioning method, characterized in that: The following steps are involved: S1. Capture and downsample images of tiny non-standard optical devices to be assembled on a PCB board to obtain device image features; S2, extracting corner candidate regions from device images at different scale levels, and preliminarily screening out initial feature sets of corners containing different materials and directions; S3, for each corner point candidate in the initial corner point feature set, calculate the corner point significance score; S4, based on the corner point significance score, performing secondary screening on the candidate corner points, and obtaining a refined corner point feature point set of different optical device types; S5. According to the refined corner feature point set, a stable corner feature set with uniform distribution is obtained through iterative convergence; S6, extracting multi-scale contour edge information of the device, and calculating multi-scale curvature extreme points on the edge segments as potential corner points of complex shapes; S7, fusing the complex shape corner points with the stable corner point feature set to construct a comprehensive corner point feature set; S8. Construct a corner point topological structure model of the optical device based on the detected corner point comprehensive feature set, and match it with various optical device corner point template libraries to determine whether it meets the expected geometric layout; S9, fusing the corner point topological structure matching result with the corner point refined coordinate information to generate the positioning information of the optical device and transmit it to the PCB assembly control system; The step S3 specifically includes: For each candidate corner point in the initial feature set of the corner points, extract the texture features of the surrounding area, combine the Harris corner point response function, and introduce the device material recognition factor to calculate the corner point significance score; The step S4 specifically includes: Based on the corner significance score, according to the diversity of optical device types, different corner significance thresholds are set for different types of optical devices, and the corner candidate points are screened twice to obtain a refined corner feature point set for different optical device types; The step S5 specifically includes: According to the refined corner feature point set of different optical device types, a non-maximum suppression algorithm is adopted, and the local neighborhood range is dynamically adjusted in combination with the corner point density. The corner point response intensity is compared in the local neighborhood, non-maximum points are suppressed, and the local optimal corner points are retained. Through iterative convergence, a stable corner point feature set with uniform distribution is obtained.
2. The PCB solder-free optical device positioning method according to claim 1, characterized in that: The step S1 specifically includes: Collecting original images of tiny non-standard optical devices to be assembled on a PCB board, and downsampling the original images using a multi-scale image pyramid decomposition method to obtain device image features at different scale levels; The downsampling process includes the following steps: Preprocessing the original image to obtain a device image; Constructing a Gaussian pyramid and a Laplacian pyramid of the device image, and obtaining image sequences and feature maps of different scales through downsampling and differential operations; For the images of each scale layer of the pyramid, the SURF algorithm is used to extract scale-invariant features, calculate feature descriptors, and establish feature point sets; The shape context descriptor is used to capture the global characteristics of the device and combined with the local features to form a complete device feature representation.
3. The PCB solder-free optical device positioning method according to claim 1, characterized in that: The step S2 specifically includes: According to the diversity of device materials, corner candidate regions are extracted from device images at different scale levels, and the corner candidate regions at each level are preliminarily screened to obtain an initial feature set of corner points of different materials and directions.
4. The PCB solder-free optical device positioning method according to claim 1, characterized in that: The step S6 specifically includes: Considering the complexity of device shape, the multi-scale contour edge information of the device is extracted, and the multi-scale curvature extreme points are calculated on the edge segments as potential corner points of complex shapes.
5. The PCB solder-free optical device positioning method according to claim 1, characterized in that: The step S7 specifically includes: The complex shape corner points are fused with the stable corner point feature set to construct a comprehensive feature set of corner points of different types, scales and directions.
6. The PCB solder-free optical device positioning method according to claim 1, characterized in that: The step S8 specifically includes: According to the detected comprehensive feature set of corner points, the geometric topological relationship between adjacent corner points of the optical device is calculated, and the corner point topological structure model of the optical device is constructed. It is matched with various optical device corner point template libraries to determine whether the corner point combination of the current optical device conforms to the expected geometric layout.
7. The PCB solder-free optical device positioning method according to claim 1, characterized in that: The step S9 specifically includes: The corner point topology structure model is matched with various optical device corner point template libraries to obtain the corner point topology structure matching result of the optical device, and the corner point topology structure matching result is integrated with the refined coordinate information of the corner point of the optical device to generate the positioning information of the optical device; Convert the positioning information into an instruction format recognizable by a PCB assembly control system, and transmit the instruction format to the PCB assembly control system via the ModbusTCP protocol; The instruction format includes device type, installation coordinates, posture angle, assembly force and assembly method.
Citation Information
Patent Citations
Method and apparatus for recognizing and localizing landmarks from an image onto a map
US20110150324A1
Geosynchronization of an aerial image using localizing multiple features
WO2024042508A1
