PCBA depth completion method and system based on image guidance and storage medium

Abstract

The invention discloses a PCBA depth completion method and system based on image guidance and a storage medium. The method comprises the following steps: acquiring a two-dimensional color image and a three-dimensional depth map with holes; linearly normalizing the original depth value; the depth map and the two-dimensional color image are sent into a feature extraction network in parallel, and depth and guide feature pyramids are constructed through multiple convolution down-sampling; starting from the deepest features, performing cross-modal fusion upwards step by step; and after a residual image is output at the highest resolution layer, adding the residual image and the original depth image with the cavity element by element, and performing inverse transformation by combining an extreme value parameter during normalization, recovering a real physical depth value, and outputting a complete dense depth image. Through the end-to-end deep learning model and the two-dimensional image depth guiding mechanism, the PCBA depth completion precision is improved, and the complex boundary area completion quality is enhanced.

Description

Technical Field

[0001] This invention relates to the field of intelligent depth map completion technology in PCBA 3D inspection, and particularly to an image-guided PCBA depth completion method, system, and medium. Background Technology

[0002] In the electronics manufacturing industry, PCBA (Printed Circuit Board Assembly) quality inspection is a crucial step, as the final performance and reliability of electronic products largely depend on PCBA quality. Defective PCBAs can lead to malfunctions, performance degradation, or even complete failure in electronic products, severely impacting product market competitiveness and user experience.

[0003] In PCBA quality inspection, acquiring 3D depth information of components to detect defects such as height, thickness, and lead warping is a core technology for ensuring product reliability. 3D depth information comprehensively and accurately presents the physical morphology of components, helping to promptly identify potential defects. However, when using depth sensors to scan PCBAs and obtain depth maps, physical factors such as component gaps (e.g., components too close together obstructing the sensor's view) and strong surface reflections (e.g., BGA chips and metal pins reflecting large amounts of light) often result in localized continuous or discrete void areas (invalid data) in the depth map. These void areas affect the integrity of the depth map, preventing it from accurately reflecting the true 3D morphology of the components and interfering with subsequent depth map-based quality inspection and analysis.

[0004] To address the issue of void areas in depth maps, depth completion (hole filling) algorithms have emerged, aiming to restore complete, dense depth maps for more accurate PCBA quality inspection. Currently, mainstream PCBA 3D inspection equipment in the industry employs traditional depth map hole filling methods based on geometric priors and heuristic rules. This method typically involves a complex, multi-step post-processing workflow, first classifying components based on prior knowledge such as component package type and size, and then calling specialized algorithms to fill voids for different categories. However, due to poor physical imaging quality or high noise in the original point cloud, traditional methods often fail to achieve ideal results.

[0005] Furthermore, traditional hole-filling methods face serious technical bottlenecks when dealing with increasingly sophisticated and complex PCBA inspection needs: First, they rely on complex priors, have poor generalization ability, and the algorithms are highly dependent on manually designed heuristic rules and processing logic customized for specific components. They are prone to failure when encountering new components or unexpected surface conditions, resulting in high maintenance costs, difficulty in generalization, and inability to meet the needs of component upgrades. Second, they have low reconstruction accuracy and serious loss of detailed information. By fitting simple geometries (such as planes) to fill holes, they ignore the real geometric details of the component surface, leading to distortion of the depth map after completion, making it unsuitable for fine defect detection. Third, they fail to effectively utilize 2D image guidance information. The hole-filling process is almost entirely isolated on three-dimensional depth data, ignoring the rich guidance information in the high-resolution, high-quality two-dimensional color RGB images acquired simultaneously by AOI (Automated Optical Inspection) equipment when acquiring depth maps, resulting in a waste of key information and limiting the accuracy and effectiveness of depth completion.

[0006] Therefore, there is an urgent need for a depth completion technology that can break free from the constraints of traditional complex rules and make full use of 2D image information for high-precision guidance, so as to improve the accuracy and efficiency of PCBA quality inspection and promote the development of the electronics manufacturing industry. Summary of the Invention

[0007] In view of this, the present invention proposes an image-guided PCBA depth completion method, system and storage medium to solve the problems of existing technologies that rely on manual rules, have poor generalization and are difficult to accurately complete PCBA depth map holes.

[0008] The specific technical solution of this invention is as follows: An image-guided PCBA depth completion method includes: Acquire a two-dimensional color image of the PCBA to be processed and a three-dimensional depth map with holes; based on the minimum and maximum values ​​of the effective pixels in the current depth map, linearly normalize the original depth values ​​to improve the relative depth contrast between micro-devices. The normalized depth map and the 2D color image are used as dual inputs and fed into the feature extraction network in parallel. The depth feature pyramid and the guide image feature pyramid are constructed through multiple convolutional downsampling. Starting from the deepest feature layer, cross-modal fusion is performed step by step upwards. At each scale, the guide image features generate spatially adaptive dynamic convolution kernels to perform pixel-level filtering operations on the depth features. Image structure guidance is strengthened in edge regions and depth consistency constraints are enhanced in flat regions to achieve selective information enhancement that is content-aware. After outputting the residual map at the highest resolution layer, it is added element-wise to the original holed depth map, and an inverse transformation is performed using the extreme value parameters during normalization to recover the true physical depth value and output a complete dense depth map.

[0009] Specifically, when performing normalization, the extreme value range is calculated only based on the effective depth values ​​of the non-hole regions in the holed depth map, and a mapping function is constructed accordingly. This ensures that the normalization process is not affected by missing data, thereby maintaining the authenticity of the local depth gradient and enhancing the sensitivity to the relative height differences between small devices.

[0010] Specifically, the two feature extraction streams adopt an encoder structure that shares the number of downsampling layers and the output resolution of each layer is perfectly matched, so that the depth feature map and the guide image feature map have corresponding spatial coordinate relationships at each scale, providing a pixel-level alignment basis for cross-modal dynamic filtering.

[0011] Specifically, cross-modal fusion begins with the deepest features. In the progressive fusion process from coarse to fine, the features after high-level fusion are upsampled by interpolation and then concatenated with the deep features before fusion at lower levels. The concatenated features are then input into the dynamic filtering module of the next stage to achieve progressive modulation of local completion decisions by global context information.

[0012] Specifically, a dynamic filtering fusion module is introduced during the fusion process. The dynamic filtering fusion module uses the guiding image features at the current scale as conditional input to drive a set of lightweight convolutional layers to generate a two-dimensional convolutional kernel parameter matrix that varies with spatial location. This matrix is ​​used to perform pixel-by-pixel neighborhood weighted filtering on the same scale depth feature map to achieve local adaptive feature enhancement based on image content.

[0013] Specifically, during the training phase, the fused features at each scale are connected to an independent decoding head to generate a depth prediction map of the corresponding resolution. Each prediction map is compared with the real depth map after corresponding downsampling processing, and the L1 loss is calculated. The loss weight of the low-resolution scale is higher than that of the high-resolution scale to prioritize the correctness of the global structure.

[0014] Specifically, in the final output stage, the residual depth map is added element-wise to the original depth map with holes to obtain the complete depth map.

[0015] Specifically, the two-dimensional color images used are obtained by illuminating the PCBA surface with red, green and blue LED light sources in a synchronous or alternating state. In the feature extraction stage, the network explicitly models the reflection response differences between each color channel and extracts pseudo-geometric features related to the surface curvature to help determine the actual boundary position of the high reflectivity area.

[0016] A PCBA depth completion system based on image guidance includes: an image acquisition module for acquiring a two-dimensional color image of the PCBA to be processed and a three-dimensional depth map with holes; a preprocessing module connected to the image acquisition module, configured to calculate the minimum and maximum values ​​based only on the effective pixels in the non-hole regions of the depth map, and linearly map the original depth values ​​to the [0,1] interval to achieve adaptive normalization; a dual-stream feature extraction module for performing parallel convolution downsampling on the normalized depth map and the two-dimensional color image respectively, constructing a depth feature pyramid and a guiding image feature pyramid, with the two-stream coding structure hierarchically matched and the spatial resolution synchronized; a cross-modal dynamic fusion module for upsampling and fusing from the deepest layer features, using the guiding image features to generate spatially variable dynamic convolution kernels, performing pixel-level filtering on the depth features, strengthening structural guidance in edge regions, and enhancing consistency constraints in flat regions; and a residual compensation and inverse transformation module for receiving the residual results output from the highest resolution layer, superimposing them onto the hole regions, and combining them with normalized extreme value parameters to recover the true depth values ​​and output a complete dense depth map.

[0017] A computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, enable the processor to implement an image-guided PCBA depth completion method.

[0018] The beneficial effects of this invention are as follows: 1. A multi-scale fusion guided network architecture is adopted: 2D and 3D multi-scale features are extracted through parallel feature pyramids and iteratively fused from coarse to fine. 2D image features are used to guide the enhancement of 3D depth features, so as to achieve adaptive and highly generalizable depth completion for various PCBA components and improve completion accuracy.

[0019] 2. A dynamic filtering fusion module was designed: At the core of the network, a depth-aware guided filtering is achieved by dynamically generating content-adaptive convolution kernels. This greatly maintains the sharpness of device boundaries when filling holes, enhances the completion quality of complex boundary regions, and reduces the false alarm rate.

[0020] 3. Multi-channel composite light source color image is used: Multi-channel composite light source color image (red, green and blue LED combination) is used as guiding information to provide richer geometric clues for deep completion, more accurately guide the network to reconstruct device boundaries, and improve the success rate of completion of special surface devices.

[0021] 4. An adaptive normalization strategy was employed: based on the distribution of effective depth values ​​in the input depth map, the depth values ​​were dynamically mapped to the [0,1] normalization interval, amplifying the relative depth differences of small devices, providing a data foundation for the network to learn fine features, and improving the ability to detect small defects.

[0022] 5. Multi-scale prediction and residual output are adopted: a preliminary depth prediction map is generated at each scale to achieve multi-scale supervised learning. Finally, the residual value is added to the original depth map with holes at the highest resolution scale to obtain the final dense depth map output, which enhances the stability of the detection results and reduces the cost of manual re-inspection. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the image-guided PCBA depth completion method of the present invention. Figure 2 This is a schematic diagram of the image-guided PCBA depth completion system of the present invention. Detailed Implementation

[0025] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0026] This invention proposes an image-guided PCBA depth completion method, such as... Figure 1 As shown, the core of this method lies in constructing an end-to-end deep learning network that can automatically learn the complex mapping relationship between 2D images and 3D depth maps. Deep learning networks possess powerful non-linear fitting capabilities, automatically extracting features from massive amounts of data and learning the intrinsic connections between them, thus achieving efficient depth completion. Compared to traditional methods, this method eliminates the need for manually setting complex rules and parameters, automatically adapting to different data and scenarios, and offering greater flexibility and accuracy.

[0027] The core of the method of this invention mainly includes the following aspects: First, a multi-scale fusion-guided network architecture is proposed in this invention. This architecture employs a parallel feature pyramid structure to extract multi-scale features from both 2D color images and 3D depth maps. Image features at different scales contain different levels of information; large-scale features capture the overall structure and contours of the image, while small-scale features reflect the details and textures. This parallel extraction method effectively captures image feature information at different scales. Simultaneously, an iterative fusion mechanism from coarse to fine is introduced, using 2D image features to guide and enhance 3D depth features at each scale. At the coarse scale, the image is roughly fused and completed to obtain a preliminary depth map; then at the fine scale, the detailed information from the 2D image is further utilized to finely adjust and optimize the depth map. This step-by-step fusion approach allows the network to better combine the detailed information of the 2D image and the spatial information of the 3D depth map, thereby improving the accuracy and stability of depth completion.

[0028] Secondly, there is the dynamic filtering and fusion module. To achieve more efficient guidance, this invention features a carefully designed fusion module at the core of the network. Unlike traditional simple feature concatenation or addition methods, this module dynamically generates content-adaptive convolutional kernels for local regions in the image. Traditional feature concatenation or addition methods simply combine different features without considering the feature differences in local regions, easily leading to information loss and confusion. The fusion module designed in this invention can automatically adjust the parameters of the convolutional kernels according to the image content of local regions, achieving depth-aware guided filtering. During the filling of depth holes, this module can largely maintain the sharpness of the device boundaries, avoiding boundary blurring caused by filtering operations, thereby generating clearer and more accurate depth completion results.

[0029] Finally, there is the superior utilization of multi-channel information. Through in-depth research and practice, this invention has found that in industrial inspection scenarios, directly using multi-channel composite light source color images (a combination of red, green, and blue LEDs) as guiding information has significant advantages compared to using a single white light image. Multi-channel composite light source color images can provide richer geometric cues, including information on the reflectivity of the object's surface, color distribution, and other aspects. Different colors of light produce different effects when reflected from the object's surface. By analyzing these multi-channel images, the network can gain a more comprehensive understanding of the object's shape and structure. The network can fully utilize this rich information to more accurately guide the reconstruction of device boundaries, thereby effectively improving the accuracy of depth completion and providing more reliable data support for industrial inspection.

[0030] Figure 1This demonstrates the core process from input (2D color image + depth map with holes) processed by a deep learning model to output (complete dense depth map). The specific implementation steps of the method of this invention are as follows: Step 1: Obtain the two-dimensional color image of the PCBA to be processed and the three-dimensional depth map with holes; based on the minimum and maximum values ​​of the effective pixels in the current depth map, linearly map the original depth values ​​to the [0,1] interval to achieve adaptive normalization, so as to improve the relative depth contrast between micro-devices.

[0031] Before feeding the data into the depth completion network, this invention employs a set of key preprocessing steps to ensure the quality and applicability of the input data, laying a solid foundation for subsequent network processing. First, the system acquires the 2D color image to be processed and the corresponding 3D depth map with holes. In PCBA (Printed Circuit Board Assembly) inspection scenarios, the depth variation range of devices is typically small, and micron-level key details are crucial for accurate detection and analysis. However, these minute details are easily obscured in the raw data, affecting network performance and the final completion effect. To avoid this problem, this invention employs an adaptive normalization strategy. This strategy dynamically maps depth values ​​to a standardized interval of [0, 1] based on the distribution of effective depth values ​​in the current input depth map. Specifically, it analyzes the range of all effective depth values ​​in the depth map and then scales them to the [0, 1] interval according to a certain ratio. In this way, the relative depth differences of minute devices are amplified, enabling the network to more easily identify and learn these fine features, providing a high-quality data foundation for subsequent network processing.

[0032] Step two involves feeding the normalized depth map and the 2D color image as dual inputs into the feature extraction network in parallel. Multiple convolutional downsampling operations are used to construct a depth feature pyramid and a guiding image feature pyramid, respectively. Starting from the deepest features, cross-modal fusion is performed progressively upwards. At each scale, the guiding image features generate spatially adaptive dynamic convolution kernels, performing pixel-level filtering on the depth features. Image structure guidance is strengthened in edge regions, and depth consistency constraints are enhanced in flat regions, achieving selective information enhancement with content awareness.

[0033] The network of this invention adopts a multi-scale, iterative fusion architecture at the macroscopic level. This architecture can fully utilize information at different scales, improving the accuracy and robustness of depth completion. Its data processing flow is as follows: 1. Parallel Feature Pyramid Extraction: The network receives a preprocessed 2D color image and a 3D depth map with holes as dual inputs. To extract features from different scales, two parallel downsampling streams perform stepwise downsampling operations on the 2D color image and the 3D depth map, respectively. During downsampling, each level reduces the resolution of the image or depth map while extracting features at different levels. In this way, a depth feature pyramid and a guiding feature pyramid containing multiple scales are constructed. The depth feature pyramid mainly reflects the depth information in the depth map, while the guiding feature pyramid extracts feature information, such as edges and textures, from the 2D color image to guide depth completion.

[0034] 2. Iterative Multi-Scale Fusion: Unlike traditional single-step fusion, the feature fusion process of this invention starts from the top of the feature pyramid (the feature map with the lowest resolution) and iteratively merges downwards (towards higher resolution). At each scale, a core fusion module utilizes the 2D image features of the current scale to enhance the 3D depth features. Specifically, the fusion module analyzes information in the 2D image features, such as edges and textures, and then fuses this information with the 3D depth features, enabling the depth features to better reflect the actual structure in the image. Through this iterative fusion method, low-resolution feature information can be gradually transferred to high-resolution feature maps, improving the richness and accuracy of the features.

[0035] 3. Multi-scale Prediction and Residual Output: At each scale, the network generates a preliminary depth prediction map based on the currently fused features, achieving multi-scale supervised learning. Multi-scale supervised learning enables the network to learn effective features and mapping relationships at different scales, thereby improving the model's generalization ability. Finally, at the highest resolution scale, the network performs a residual connection between the completed depth information and the original depth map with holes. The residual connection method involves adding the completed residual value to the corresponding depth value in the original depth map to obtain the final dense depth map output. This method helps to preserve reliable depth information in the original map, while combining the completed information to improve the completeness and accuracy of the depth map.

[0036] The fusion module is crucial for implementing depth-aware guided filtering. It dynamically adjusts the feature fusion method based on the characteristics of different regions in the image, thereby improving the depth completion effect. Internally, it doesn't simply concatenate or add features; instead, it achieves deeper guidance by dynamically generating pixel-level convolutional kernels. Specifically, this module comprehensively utilizes depth features at the current scale and 2D image guidance features to dynamically generate a unique, content-adaptive convolutional kernel for each local region in the image. In regions with clear image edges, edge information is crucial for accurate depth completion. In this case, the convolutional kernel generated by the fusion module allows the guidance features to dominate, causing the network to refer more to edge information in the 2D color image when processing these regions, thus preserving boundaries and avoiding blurring or inaccuracy in the depth map at edges. In regions with flat image texture but reliable depth information, the depth information itself can already reflect the structure of objects well. In this case, the convolutional kernel generated by the fusion module allows depth features to dominate to smooth noise. In this way, it can reduce noise interference while ensuring the accuracy of depth information, improving the quality of the depth map.

[0037] Step 3: After outputting the residual map at the highest resolution layer, add it element-wise to the original depth map with holes, and perform an inverse transformation in combination with the extreme value parameters during normalization to recover the true physical depth value and output a complete dense depth map.

[0038] During the training phase, the network learns the mapping relationship from holed depth maps and 2D guide maps to complete depth maps by training on a large number of PCBA samples. PCBA samples contain various types and structures of circuit boards, possessing rich depth information and features. Thanks to the network's multi-scale prediction capability, the loss function can be calculated and supervised at multiple scales. At each scale, the difference between the predicted depth map and the true depth map is calculated, and these differences are fed back to the network as loss values. This ensures that the model is effectively trained at different granularities, enabling the network to learn features and mapping relationships at different levels from coarse to fine, improving the model's accuracy and generalization ability. During the inference phase, the network receives new input data, performs forward propagation, and directly outputs a dense, complete depth map. During forward propagation, the network performs feature extraction, fusion, and prediction operations on the input data according to the previously trained parameters and procedures. Because the data is normalized during training, the output depth map's values ​​range from [0, 1]. To obtain a true physical scale, the depth map values ​​need to be restored to their original physical scale through an inverse normalization operation. The inverse normalization operation maps the values ​​in the [0, 1] interval back to the actual depth range based on the normalization parameters used during training, thus obtaining a dense and complete depth map with true physical meaning.

[0039] The method of this invention has wide and crucial applications in many fields of electronic product manufacturing and testing, specifically in products corresponding to the following different types of equipment: (1) Online PCBA surface mount optical inspection (AOI) equipment is mainly used in the production lines of enterprises that use SMT (Surface Mount Technology) process to manufacture electronic products. It covers a wide range of products, including smartphones, tablets, and smart wearable devices in the consumer electronics field; in-vehicle entertainment systems and autonomous driving sensor modules in the automotive electronics field; and PLC controllers and industrial instruments in the industrial control electronics field. This equipment is integrated into a specific stage of the quality inspection station in the SMT production line, utilizing patented technology for online PCBA surface mount optical inspection. It can detect component height, coplanarity, warpage, and position orientation, ensuring that component installation meets design requirements, avoiding impact on circuit performance or interference. The inspection process is real-time and highly accurate, effectively preventing defective products from flowing into subsequent stages.

[0040] (2) The offline PCBA testing and analysis system is suitable for PCBA products in the R&D stage that require quality control and performance analysis, including newly developed electronic product prototypes and experimental versions of improved products. It is applied in high-end fields such as communication equipment, medical electronics, and aerospace. During the R&D stage, it can be used for design verification, first article inspection (FAI), fault diagnosis, and quality assessment. By using this patented technology to conduct offline testing and analysis of PCBA samples, R&D personnel can obtain detailed product performance data, accurately locate faults, and provide support for design improvement and product decision-making.

[0041] (3) Automated assembly and rework system, widely used in PCBA products requiring automated assembly or rework, such as circuit boards for large servers and control circuit boards for high-precision instruments. In terms of application scenarios, it provides visual guidance for PCBA automated assembly or BGA rework robots. On automated assembly lines, it guides assembly equipment to accurately pick up and place components; in the BGA rework process, it provides precise visual information for rework robots. It utilizes the high-precision three-dimensional visual guidance function in its patented technology to help identify component positions, postures, and other information, achieving precise assembly or rework, and improving production efficiency and product quality.

[0042] (4) Semiconductor packaging inspection equipment is mainly used for product inspection after various chip packaging, including integrated circuit chips in the fields of computers, communications, and consumer electronics, such as CPUs, GPUs, and communication chips. Packaging forms include BGA and QFP (Quad Flat Package). On the chip packaging production line, it uses patented technology to inspect the shape, height, coplanarity, and surface quality of the chip package. Inspecting the shape ensures compliance with design specifications, inspecting the height ensures fitting accuracy, inspecting coplanarity ensures good contact, and inspecting surface quality can detect packaging defects, ensuring that the packaging quality meets standards and improving the reliability and stability of the chip.

[0043] This invention also proposes an image-guided PCBA depth completion system, such as... Figure 2 As shown, it includes: an image acquisition module for acquiring a two-dimensional color image of the PCBA to be processed and a three-dimensional depth map with holes; a preprocessing module connected to the image acquisition module, configured to calculate the minimum and maximum values ​​based only on the effective pixels in the non-hole regions of the depth map, and linearly map the original depth values ​​to the [0,1] interval to achieve adaptive normalization; a dual-stream feature extraction module for performing parallel convolution downsampling on the normalized depth map and the two-dimensional color image respectively, constructing a depth feature pyramid and a guide image feature pyramid, with the two-stream coding structure hierarchically matched and the spatial resolution synchronized; a cross-modal dynamic fusion module for upsampling and fusing from the deepest layer features, using the guide image features to generate spatially variable dynamic convolution kernels, performing pixel-level filtering on the depth features, strengthening structural guidance in edge regions, and enhancing consistency constraints in flat regions; and a residual compensation and inverse transformation module for receiving the residual results output by the highest resolution layer, superimposing them on the hole regions, and combining them with normalized extreme value parameters to recover the true depth values ​​and output a complete dense depth map.

[0044] The present invention also proposes a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, enables the processor to implement an image-guided PCBA depth completion method.

[0045] The beneficial effects of this invention are as follows: 1. A multi-scale fusion guided network architecture is adopted: 2D and 3D multi-scale features are extracted through parallel feature pyramids and iteratively fused from coarse to fine. 2D image features are used to guide the enhancement of 3D depth features, so as to achieve adaptive and highly generalizable depth completion for various PCBA components and improve completion accuracy.

[0046] 2. A dynamic filtering fusion module was designed: At the core of the network, a depth-aware guided filtering is achieved by dynamically generating content-adaptive convolution kernels. This greatly maintains the sharpness of device boundaries when filling holes, enhances the completion quality of complex boundary regions, and reduces the false alarm rate.

[0047] 3. Multi-channel composite light source color image is used: Multi-channel composite light source color image (red, green and blue LED combination) is used as guiding information to provide richer geometric clues for deep completion, more accurately guide the network to reconstruct device boundaries, and improve the success rate of completion of special surface devices.

[0048] 4. An adaptive normalization strategy was employed: based on the distribution of effective depth values ​​in the input depth map, the depth values ​​were dynamically mapped to the [0,1] normalization interval, amplifying the relative depth differences of small devices, providing a data foundation for the network to learn fine features, and improving the ability to detect small defects.

[0049] 5. Multi-scale prediction and residual output are adopted: a preliminary depth prediction map is generated at each scale to achieve multi-scale supervised learning. Finally, the completed depth information is connected with the residual of the original depth map with holes at the highest resolution scale to obtain the final dense depth map output, which enhances the stability of the detection results and reduces the cost of manual re-inspection.

[0050] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image-guided PCBA depth completion method, characterized in that, include: Obtain a 2D color image of the PCBA to be processed and a 3D depth map with holes; Based on the minimum and maximum values ​​of the effective pixels in the current depth map, the original depth values ​​are linearly normalized to improve the relative depth contrast between tiny devices. The normalized depth map and the 2D color image are used as dual inputs and fed into the feature extraction network in parallel. The depth feature pyramid and the guide image feature pyramid are constructed through multiple convolutional downsampling. Starting from the deepest feature layer, cross-modal fusion is performed step by step upwards. At each scale, the guide image features generate spatially adaptive dynamic convolution kernels to perform pixel-level filtering operations on the depth features. Image structure guidance is strengthened in edge regions and depth consistency constraints are enhanced in flat regions to achieve selective information enhancement that is content-aware. After outputting the residual map at the highest resolution layer, it is added element-wise to the original holed depth map, and an inverse transformation is performed using the extreme value parameters during normalization to recover the true physical depth value and output a complete dense depth map.

2. The image-guided PCBA depth completion method as described in claim 1, characterized in that, When performing normalization, the extreme value range is calculated only based on the effective depth values ​​of the non-hole regions in the holed depth map, and a mapping function is constructed accordingly. This ensures that the normalization process is not affected by missing data, thereby maintaining the authenticity of the local depth gradient and enhancing the sensitivity to the relative height differences between small devices.

3. The image-guided PCBA depth completion method as described in claim 1, characterized in that, The two feature extraction streams adopt an encoder structure that shares the number of downsampling layers and the output resolution of each layer is perfectly matched, so that the depth feature map and the guide image feature map have corresponding spatial coordinate relationships at each scale, providing a pixel-level alignment basis for cross-modal dynamic filtering.

4. The image-guided PCBA depth completion method as described in claim 1, characterized in that, Cross-modal fusion begins with the deepest features. In the progressive fusion process from coarse to fine, the features after high-level fusion are upsampled by interpolation and then concatenated with the deep features before fusion at lower levels. The concatenated features are then input into the dynamic filtering module of the next stage to achieve progressive modulation of local completion decisions by global context information.

5. The image-guided PCBA depth completion method as described in claim 1, characterized in that, A dynamic filtering fusion module is introduced during the fusion process. The dynamic filtering fusion module uses the guiding image features at the current scale as conditional input to drive a set of lightweight convolutional layers to generate a two-dimensional convolutional kernel parameter matrix that varies with spatial location. This matrix is ​​used to perform pixel-by-pixel neighborhood weighted filtering on the same scale depth feature map to achieve local adaptive feature enhancement based on image content.

6. The image-guided PCBA depth completion method as described in claim 1, characterized in that, During the training phase, the fused features at each scale are connected to an independent decoding head to generate a depth prediction map of the corresponding resolution. Each prediction map is compared with the real depth map after corresponding downsampling processing, and the L1 loss is calculated. The loss weight of the low-resolution scale is higher than that of the high-resolution scale to prioritize the correctness of the global structure.

7. The image-guided PCBA depth completion method as described in claim 1, characterized in that, After outputting the residual map at the highest resolution layer, it is added element-wise to the original depth map with holes, and an inverse transformation is performed in conjunction with the extreme value parameters during normalization to recover the true physical depth value and output a complete dense depth map, thus achieving zero-disturbance update of the original effective data.

8. The image-guided PCBA depth completion method as described in claim 1, characterized in that, The two-dimensional color images used are obtained by illuminating the PCBA surface with red, green and blue LED light sources in a synchronous or alternating state. In the feature extraction stage, the network explicitly models the reflection response differences between each color channel and extracts pseudo-geometric features related to the surface curvature to help determine the actual boundary position of the high reflectivity area.

9. An image-guided PCBA depth completion system, characterized in that, include: The image acquisition module is used to acquire a two-dimensional color image of the PCBA to be processed and a three-dimensional depth map with holes; The preprocessing module, connected to the image acquisition module, is configured to calculate the minimum and maximum values ​​based only on the effective pixels in the non-hole regions of the depth map, and linearly map the original depth values ​​to the [0,1] interval to achieve adaptive normalization; The dual-stream feature extraction module performs parallel convolution downsampling on the normalized depth map and the two-dimensional color image respectively, constructing a depth feature pyramid and a guide image feature pyramid, and the two-stream coding structures are matched in hierarchy and synchronized in spatial resolution; The cross-modal dynamic fusion module upsamples and fuses features step by step from the deepest layer. It uses guided image features to generate spatially variable dynamic convolution kernels, performs pixel-level filtering on deep features, strengthens structural guidance in edge regions, and enhances consistency constraints in flat regions. The residual compensation and inverse transformation module receives the residual results output by the highest resolution layer, superimposes them onto the hole region, and combines them with the normalized extreme value parameters to recover the true depth value, outputting a complete dense depth map.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, the processor implements the image-guided PCBA depth completion method as described in any one of claims 1 to 8.

Images