A notebook mainboard visual inspection method and system

Abstract

This invention relates to the field of image detection technology, specifically to a visual inspection method and system for notebook motherboards. The method includes: inputting an original image of the notebook motherboard to be inspected into a preprocessing module to generate preliminary defect candidate regions; performing semantic parsing on the original notebook motherboard image to extract key region coordinates and non-key region texture information; extracting multiple local image patches from the original notebook motherboard image based on the key region coordinates; performing multi-scale feature extraction and high-frequency noise feature extraction on the original notebook motherboard image; and fusing the extracted features to generate a detection feature map. This invention, by analyzing the texture information of non-key regions and generating simulated defect images, enables the system to have stronger adaptability to different types and forms of defects, and can cope with changing product quality requirements.

Description

Technical Field

[0001] This invention relates to the field of image detection technology, specifically to a visual inspection method and system for notebook motherboards. Background Technology

[0002] A notebook motherboard typically refers to the main circuit board in a laptop computer. It is one of the core components of a laptop and is responsible for connecting and coordinating all other hardware components, including the central processing unit (CPU), memory (RAM), storage devices (such as solid-state drives or hard disk drives), graphics processing unit (GPU), input / output interfaces, etc.

[0003] Currently, traditional methods typically rely on manually designed features, such as edges and corners, which cannot effectively capture complex and subtle defects. This results in a high false negative rate, especially when faced with complex backgrounds or tiny defects. Furthermore, due to the lack of deep learning discrimination capabilities in traditional methods, many qualified areas may be incorrectly identified as defects, leading to frequent false positives and affecting the overall reliability of the inspection. In addition, traditional visual inspection methods often require extensive image preprocessing and feature calculation, resulting in long processing times and making them unsuitable for the needs of rapid online production inspection.

[0004] Furthermore, traditional methods often require readjusting parameters or redesigning feature extraction schemes when dealing with different types and forms of defects, lacking flexibility and making it difficult to adapt to changing product quality requirements. Moreover, many traditional methods fail to effectively utilize historical data for model optimization, resulting in a lack of continuous improvement in detection accuracy and efficiency, and a lack of feedback mechanisms to improve the detection process. Summary of the Invention

[0005] To achieve the above objectives, the present invention provides the following technical solution: a visual inspection method for notebook motherboards, comprising: The original image of the notebook motherboard is obtained. The original image of the notebook motherboard to be detected is input into the preprocessing module to generate preliminary defect candidate regions. Semantic parsing is performed on the original image of the notebook motherboard to extract the coordinates of key regions and the texture information of non-key regions. Based on the coordinates of key regions, multiple local image patches are extracted from the original image of the notebook motherboard. Multi-scale feature extraction and high-frequency noise feature extraction are performed on the original image of the notebook motherboard. The extracted features are then fused to generate a detection feature map. Surface normal map of the original motherboard image; multiple simulated defect images generated based on texture information of non-critical regions; Each local image patch is feature-encoded to obtain multiple first feature vectors, and each simulated defect image is feature-extracted to obtain multiple second feature vectors. Multiple first feature vectors are fused with multiple second feature vectors to obtain multiple third feature vectors; coordinate normalization is performed on the preliminary defect candidate region. The obtained normalized results, detection feature maps, surface normal maps, and multiple third feature vectors are input into a convolutional neural network to reconstruct the defect probability values ​​of candidate regions. Based on the reconstruction results and standard template feature vectors, the detection result map of the notebook motherboard is determined.

[0006] Preferably, the original image of the notebook motherboard to be inspected is input into the preprocessing module to generate preliminary defect candidate regions. Semantic parsing is then performed on the original notebook motherboard image to extract the coordinates of key regions and the texture information of non-key regions, including: Acquire the raw RGB image of the notebook motherboard captured by the industrial camera; convert the raw RGB image to HSV color space and perform adaptive histogram equalization on the brightness channel; Edge calculation is performed on the equalized image, and preliminary defect candidate regions are generated by combining color threshold segmentation. The original RGB image is processed to identify and mark key regions containing circuit elements and background regions, and the geometric coordinates of the key regions and the texture distribution data of the non-key regions are extracted.

[0007] Preferably, based on the coordinates of the key region, multiple local image patches are extracted from the original image of the notebook motherboard. Multi-scale feature extraction and high-frequency noise feature extraction are performed on the original image of the notebook motherboard. The extracted features are then fused to generate a detection feature map, including: Based on the geometric coordinates of key regions, multiple corresponding local image patches are cropped from the original image; The original image of the notebook motherboard is downsampled, and the basic features of the image are extracted through a residual network to obtain the basic feature map; Wavelet transform is performed on the basic feature map to extract high-frequency noise information from the image, resulting in high-frequency components. These high-frequency components are then processed through an attention mechanism module and subjected to inverse wavelet transform to obtain high-frequency noise features. Multi-scale dilated convolution is performed on the basic feature map to obtain multi-scale features; the multi-scale features and high-frequency noise features are weighted and fused to obtain an enhanced feature map; the enhanced feature map is upsampled to obtain a detection feature map.

[0008] Preferably, the surface normal map of the original image of the notebook motherboard includes: The structured light reflection features and gray-level co-occurrence matrix features of the original image of the notebook motherboard are fused to obtain hybrid texture features; The mixed texture features are weighted and calculated based on the depth information of the original image of the notebook motherboard, and the gradient direction of the mixed texture features is corrected. A predicted normal map is generated based on the corrected mixed texture features to determine the surface normal vector of each pixel in the image. Determine the principal axis direction of the original image of the notebook motherboard, and perform rotation correction on the predicted normal vector according to the principal axis direction to obtain the surface normal map.

[0009] Preferably, multiple simulated defect images are generated based on texture information of non-critical regions, including: Non-critical area texture information is classified to obtain multiple sub-texture information, and the corresponding defect morphology set is determined based on each sub-texture information. Extract a defect pattern from each defect pattern set, and combine the extracted defect patterns based on the distribution of the subtexture information corresponding to each defect pattern in the non-critical area to obtain the defect simulation parameters. Based on the coordinates of the key region, sensitive and non-sensitive regions for detection are determined, and multiple simulated defect images are generated by combining defect simulation parameters.

[0010] Preferably, feature encoding is performed on each local image patch to obtain multiple first feature vectors, and feature extraction is performed on each simulated defect image to obtain multiple second feature vectors, including: Convolution and pooling operations are performed on each local image patch to extract deep semantic information and obtain multiple first feature vectors; each simulated defect image is processed to extract initial features. The initial features are input into the feature fusion layer, which includes M feature enhancement layers and M feature compression layers connected in sequence. The M feature enhancement layers are connected to the M feature compression layers through skip connection layers. When the feature enhancement layer enhances the input feature vector, it enhances the input feature vector based on the first feature vector corresponding to the local image patch and the layer parameters corresponding to the feature enhancement layer. When the feature compression layer compresses the input feature vector, it compresses the input feature vector based on the initial features corresponding to the simulated defect image and the layer parameters corresponding to the feature enhancement layer; the output feature vector after processing by the feature fusion layer is used as multiple second feature vectors.

[0011] Preferably, multiple first feature vectors are fused with multiple second feature vectors to obtain multiple third feature vectors; coordinate normalization processing is performed on the preliminary defect candidate region, including: An attention mechanism network is used to calculate the association weights between each first feature vector and its corresponding second feature vector; the first and second feature vectors are weighted and summed based on the association weights to generate intermediate fused features. The texture description and key region coordinates related to the defect information in the simulated defect image are obtained, the defect description information is determined and encoded to obtain the fourth feature vector; the intermediate fused features and the fourth feature vector are concatenated and processed through a fully connected layer to generate multiple third feature vectors. The pixel coordinates of the initial defect candidate region are mapped, and the coordinate range of the candidate region is scaled to a preset standard range to obtain the normalized result.

[0012] Preferably, the obtained normalized result, detection feature map, surface normal map, and multiple third feature vectors are input into a convolutional neural network to reconstruct the defect probability value of the candidate region. Based on the reconstruction result and the standard template feature vector, the detection result map of the notebook motherboard is determined, including: The normalization result, detection feature map, surface normal map, and multiple third feature vectors are concatenated along the channel dimension to obtain a fused input tensor. The fused input tensor is then input into a convolutional neural network, and through multiple convolutional layers and activation function layers, the probability value of each pixel in the candidate region belonging to the defect category is reconstructed. Calculate the feature matching degree between the probability value and the preset standard template feature vector; when the feature matching degree meets the preset qualification conditions and the probability value is lower than the defect threshold, the corresponding area is determined to be a qualified area; Map the judgment results of all candidate regions back to the original image coordinate system, mark and highlight the regions judged as defects, and generate a detection result map; If the feature matching degree does not meet the qualification criteria, a second comparison is performed using the feature set of historical qualified motherboard images. If the feature similarity with the features in the historical feature set meets the criteria, the image is still determined to be qualified.

[0013] A notebook motherboard visual inspection system, applicable to the aforementioned notebook motherboard visual inspection method, includes: The image acquisition unit is used to acquire the original image of the notebook motherboard, input the original image of the notebook motherboard to be detected into the preprocessing module, generate preliminary defect candidate regions, perform semantic parsing on the original image of the notebook motherboard, and extract the coordinates of key regions and the texture information of non-key regions. The feature extraction unit is used to extract multiple local image patches from the original image of the notebook motherboard based on the coordinates of the key region, perform multi-scale feature extraction and high-frequency noise feature extraction on the original image of the notebook motherboard, fuse the extracted features, and generate a detection feature map. The defect simulation unit is used to calculate the surface normal map of the original image of the notebook motherboard; and to generate multiple simulated defect images based on the texture information of non-critical regions. The feature encoding unit is used to encode the features of each local image block to obtain multiple first feature vectors, and to extract the features of each simulated defect image to obtain multiple second feature vectors. The feature fusion unit is used to fuse multiple first feature vectors with multiple second feature vectors to obtain multiple third feature vectors; and to perform coordinate normalization processing on the preliminary defect candidate region. The detection result unit is used to input the obtained normalized result, detection feature map, surface normal map and multiple third feature vectors into the convolutional neural network to reconstruct the defect probability value of the candidate region, and determine the detection result map of the notebook motherboard based on the reconstruction result and the standard template feature vector.

[0014] Compared with the prior art, the beneficial effects of the present invention are: (1) This invention can effectively identify small and complex defects and reduce the false negative rate by performing multi-level feature extraction on the original image, such as key area coordinate extraction and multi-scale feature extraction; and by reconstructing and analyzing the detection features through a deep learning model, it can improve the ability to distinguish between qualified and defective areas, thereby reducing false detections and improving the overall reliability of detection; and by combining multiple feature extraction and fusion strategies, it can achieve fast processing while maintaining high accuracy, which is suitable for online production inspection needs and helps to improve production efficiency. (2) This invention analyzes the texture information of non-critical areas and generates simulated defect images, making the system more adaptable to different types and forms of defects and able to cope with changing product quality requirements. Furthermore, by using the feature set of historical qualified images for secondary comparison, it can not only improve the accuracy of current detection, but also provide data support for future detection, forming a closed-loop feedback mechanism to continuously optimize the detection model. Moreover, by adopting attention mechanism and feature fusion strategy, it can integrate feature information from different regions and different levels, improve the comprehensiveness and comprehensive judgment ability of the model, and make the system more intelligent. Attached Figure Description

[0015] Figure 1 This is a schematic flowchart of the overall method in one embodiment of the present invention; Figure 2 This is a schematic diagram of the overall system architecture in one embodiment of the present invention.

[0016] In the diagram: 1. Image acquisition unit; 2. Feature extraction unit; 3. Defect simulation unit; 4. Feature encoding unit; 5. Feature fusion unit; 6. Detection result unit. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1, please refer to Figure 1 This invention provides a technical solution: a visual inspection method for notebook motherboards, comprising: S1. Obtain the original image of the notebook motherboard. Input the original image of the notebook motherboard to be detected into the preprocessing module to generate preliminary defect candidate regions. Perform semantic parsing on the original image of the notebook motherboard to extract the coordinates of key regions and the texture information of non-key regions. S2. Based on the coordinates of the key regions, extract multiple local image patches from the original image of the notebook motherboard, perform multi-scale feature extraction and high-frequency noise feature extraction on the original image of the notebook motherboard, fuse the extracted features, and generate a detection feature map. S3. Compute the surface normal map of the original image of the notebook motherboard; generate multiple simulated defect images based on the texture information of non-critical regions; S4. Perform feature encoding on each local image block to obtain multiple first feature vectors, and extract features from each simulated defect image to obtain multiple second feature vectors; S5. Perform feature fusion between multiple first feature vectors and multiple second feature vectors to obtain multiple third feature vectors; perform coordinate normalization on the preliminary defect candidate region. S6. Input the obtained normalization results, detection feature map, surface normal map and multiple third feature vectors into the convolutional neural network to reconstruct the defect probability value of the candidate region, and determine the detection result map of the notebook motherboard based on the reconstruction result and the standard template feature vector.

[0019] It's important to note that, firstly, a high-quality raw image of the notebook motherboard to be inspected needs to be acquired; this image will serve as the basis for subsequent processing. The raw image is then input into the preprocessing module. At this stage, some basic image processing may be performed, such as noise reduction and contrast enhancement, to more clearly identify potential defects. Next, the system generates preliminary defect candidate regions, i.e., areas where problems may exist. The system then analyzes the image, extracting the coordinates of key areas (such as the location of important electronic components) and the texture information of non-key areas. This helps to understand which parts require focused attention. Based on the previously extracted key area coordinates, the system cuts out multiple small image patches from the raw image for detailed analysis. These patches contain key features that can help better identify defects. The system extracts features at multiple scales and high-frequency noise features from these local image patches; this means it considers details of different sizes and types, making defect detection more comprehensive. The extracted features are fused together to form a comprehensive detection feature map containing all the extracted information. Next, the system calculates the normal map of the notebook motherboard surface, which is crucial for understanding the surface's microstructure and illumination variations. Based on the texture information of non-critical areas, the system generates multiple simulated defect images; these images can be used to train the model and improve the accuracy of defect detection. For each local image patch and the simulated defect image, feature encoding is performed to obtain multiple feature vectors. These vectors are digital representations of the image content, facilitating subsequent processing. The first feature vector (from the local patch) is fused with the second feature vector (from the simulated defect) to obtain a third feature vector. Simultaneously, the coordinates of the preliminary defect candidate regions are normalized to maintain consistency across different images. Finally, the normalization result, detection feature map, normal map, and third feature vector are input into a convolutional neural network for deep learning. The network outputs the defect probability value of the candidate regions. Based on the reconstruction results and the standard template feature vector, the final detection result image of the notebook motherboard is determined, showing which regions are identified as defective. For example, consider a notebook motherboard with a faulty capacitor solder joint. First, a high-resolution image of the motherboard is captured. After preprocessing, the system identifies areas around the capacitor that may have defects. Through analysis, the system extracts local image patches of the capacitor and its surroundings, then extracts features from these areas, such as color, texture, and shape. Next, the system generates a simulated defect image of the area, such as showing a cracked solder joint. By encoding the features of these images, the system can compare the state of the capacitor in the real image with the simulated defect image, thus determining whether the solder joint of the capacitor in the original image has a problem. Finally, after processing by a convolutional neural network, the system outputs a defect probability value. If the probability value is higher than a certain threshold, the area is marked as "defective," and a corresponding detection result image is generated. In this way, the entire process can help quickly and accurately detect defects on a notebook motherboard.

[0020] In an optional embodiment, the original image of the notebook motherboard to be inspected is input to a preprocessing module to generate preliminary defect candidate regions. Semantic parsing is then performed on the original notebook motherboard image to extract key region coordinates and non-key region texture information, including: Acquire the raw RGB image of the notebook motherboard captured by the industrial camera; convert the raw RGB image to HSV color space and perform adaptive histogram equalization on the brightness channel; Edge calculation is performed on the equalized image, and preliminary defect candidate regions are generated by combining color threshold segmentation. The original RGB image is processed to identify and mark key regions containing circuit elements and background regions, and the geometric coordinates of the key regions and the texture distribution data of the non-key regions are extracted.

[0021] It should be noted that, firstly, the system uses an industrial camera to photograph the notebook motherboard, obtaining a raw RGB image containing rich color information; this image will serve as the basis for subsequent processing. Next, the RGB image is converted to the HSV (Hue, Saturation, Lightness) color space. The HSV color space can better separate color information, especially when dealing with changes in lighting conditions, making it more stable; therefore, this step is for more efficient subsequent image processing. In the HSV space, adaptive histogram equalization is performed on the lightness channel (V channel). The purpose of this step is to enhance the image contrast, making areas of different brightness more distinct, thereby improving the accuracy of subsequent defect detection. This equalization method can adaptively adjust the brightness according to the local features of the image, avoiding overall overexposure or underexposure. The equalized image undergoes edge detection; this means the system identifies areas of significant brightness variation, i.e., edges. Simultaneously, combined with color thresholding, the system segments potential defect areas based on preset color ranges (e.g., the green of a circuit board, specific colors of components, etc.). This generates preliminary candidate defect regions, which are the focus of subsequent analysis. Further processing of the original RGB image identifies and marks key areas containing circuit components (e.g., resistors, capacitors, chips) and surrounding background areas. Key areas are the primary focus because they may contain defects. Geometric coordinates (e.g., component center position, size, etc.) are extracted from the identified key areas, while texture distribution data of non-key areas are analyzed. This process helps understand the overall structure of the circuit board and background information that may affect the detection results. For example, consider a notebook motherboard with multiple electronic components. First, a high-quality RGB image is captured using an industrial camera. However, due to poor lighting conditions, some details in the image may not be clear. After converting to the HSV color space, the system performs adaptive histogram equalization on the luminance channel, improving image contrast and making subtle differences on the circuit board more apparent. Next, the system uses edge computing to identify the boundaries of circuit components. Simultaneously, based on color threshold segmentation, it identifies areas where defects may exist, such as a solder joint whose color differs from its surroundings. Subsequently, the system successfully marks the locations of resistors and capacitors, which are the critical areas. The background areas on the circuit board without components are classified as non-critical areas. By extracting the geometric coordinates of these critical areas, the system obtains the position and size information of each component. It also analyzes the texture of the background area to understand its potential impact on defect detection.

[0022] In an optional embodiment, multiple local image patches are extracted from the original image of the notebook motherboard based on the coordinates of the key region. Multi-scale feature extraction and high-frequency noise feature extraction are performed on the original image of the notebook motherboard. The extracted features are then fused to generate a detection feature map, including: Based on the geometric coordinates of key regions, multiple corresponding local image patches are cropped from the original image; The original image of the notebook motherboard is downsampled, and the basic features of the image are extracted through a residual network to obtain the basic feature map; Wavelet transform is performed on the basic feature map to extract high-frequency noise information from the image, resulting in high-frequency components. These high-frequency components are then processed through an attention mechanism module and subjected to inverse wavelet transform to obtain high-frequency noise features. Multi-scale dilated convolution is performed on the basic feature map to obtain multi-scale features; the multi-scale features and high-frequency noise features are weighted and fused to obtain an enhanced feature map; the enhanced feature map is upsampled to obtain a detection feature map.

[0023] It should be noted that, based on the geometric coordinates extracted from the key regions, the system will crop out multiple local image patches from the original image. For example, if a capacitor region on the circuit board is identified, the coordinates of this region are used to crop out a small image patch containing only that capacitor from the original image. These local image patches will serve as the basis for further analysis and processing. The original image of the notebook motherboard is downsampled, i.e., the image resolution is reduced. This step is usually to reduce computational load, making subsequent feature extraction more efficient. In this process, the size of the original image may be reduced to half or smaller, resulting in a low-resolution image. Using downsampled images, a residual network is used to extract fundamental image features. The residual network is a deep learning model that effectively captures important image information while addressing the vanishing gradient problem in deep networks. This process yields a fundamental feature map containing various features of the circuit board, such as the shape, position, and relative relationships of components. Wavelet transform is then applied to the fundamental feature map to extract high-frequency components, which often represent details and noise in the image. For example, subtle differences in solder joints on a circuit board may be revealed in the high-frequency components. The extracted high-frequency components are processed through an attention mechanism module. This process aims to highlight the details most important for defect detection while suppressing unimportant noise. In this way, the system can focus more on key defect features on the circuit board. The wavelet coefficients processed by the attention mechanism can be subjected to inverse wavelet transform to reconstruct a high-frequency noise feature map. This step is to transfer the processed high-frequency information back to the spatial domain for subsequent feature fusion. Multi-scale dilated convolution is performed on the basic feature map, which means extracting features at different receptive fields. Dilated convolution allows the network to expand the receptive field without increasing computation, helping to capture features at different scales, such as the performance of the same circuit element at different distances. The extracted multi-scale features and high-frequency noise features are weighted and fused. This step combines the information of the two types of features, so that the final feature map contains both large-scale structural information and retains detailed information, thereby improving the overall detection capability. Finally, the enhanced feature map is upsampled to restore the image to a higher resolution and form a detection feature map. This step will enable subsequent detection algorithms to work better on high-resolution images and improve detection accuracy. For example: Suppose we have acquired an original image of a notebook motherboard and identified several key components, such as resistors and capacitors. Based on the coordinates of these components, we crop out several local image patches, each containing only one component. Next, we downsample the entire original image to obtain a low-resolution image. Then, we use a residual network to extract the basic features of these images, which may yield some basic information about the shape, edges, and texture of the components. Subsequently, we perform wavelet transform to extract high-frequency components, which may capture subtle changes in solder joints. Through an attention mechanism, we emphasize the most important parts of these high-frequency features, such as abnormal solder joints. After inverse wavelet transform, we obtain a clear high-frequency noise feature map. Based on this, we perform multi-scale dilated convolution, enabling the model to simultaneously focus on the global structure and local details of the components on the circuit board. Finally, we obtain an enhanced feature map by weighted fusion of multi-scale features and high-frequency noise features. After upsampling, we obtain a detection feature map, which will be used for subsequent defect detection tasks.

[0024] In an optional embodiment, calculating the surface normal map of the original image of the notebook motherboard includes: The structured light reflection features and gray-level co-occurrence matrix features of the original image of the notebook motherboard are fused to obtain hybrid texture features; The mixed texture features are weighted and calculated based on the depth information of the original image of the notebook motherboard, and the gradient direction of the mixed texture features is corrected. A predicted normal map is generated based on the corrected mixed texture features to determine the surface normal vector of each pixel in the image. Determine the principal axis direction of the original image of the notebook motherboard, and perform rotation correction on the predicted normal vector according to the principal axis direction to obtain the surface normal map.

[0025] It's important to note that, firstly, structured light reflection features are extracted from the raw image of the notebook motherboard. These features typically originate from surface reflection information under structured light illumination, providing clues about the object's surface shape and texture. Secondly, gray-level co-occurrence matrix (GLCM) features are extracted. These are obtained by calculating the spatial relationships of pixel grayscale values ​​in the image, describing texture contrast, energy, uniformity, etc. These two types of features are then fused to generate a hybrid texture feature, simultaneously capturing both the object's shape and texture information. For example, assuming structured light technology was used when photographing the notebook motherboard, creating stripe patterns, the reflection features of these stripes help identify variations in the board's height. Simultaneously, the GLCM helps analyze the texture information of the board surface, such as the smoothness or roughness of solder joints. After fusion, a hybrid feature map integrating shape and texture information is obtained. When analyzing blended texture features, the depth information of the notebook motherboard image is used for weighted calculation. Depth information typically refers to the sharpness or blurriness of different regions in an image, which can be used to determine the distance and relative position of objects. By weighting, the gradient direction of the blended texture features can be better corrected, making the features more accurately reflect the real surface morphology. For example, if some regions in an image become blurry due to improper focus, these regions may not provide reliable normal information. Therefore, the weight of the blended texture features in these regions is reduced so that the blurriness does not affect the calculation of the normal vector. Based on the corrected blended texture features, a predicted normal map is generated. The purpose of this step is to determine the surface normal vector of each pixel in the image. The normal vector represents the orientation of the surface at a point, and it is crucial for subsequent 3D reconstruction and object surface analysis. For example, from the corrected blended texture features, the algorithm calculates the normal vector of each pixel. For instance, the top surface normal of a capacitor points upwards, while the edge normal of a circuit board may point outwards. The generated predicted normal map will be an image containing the normal information of each pixel. Determining the principal axis direction of the original image of the notebook motherboard is crucial for ensuring the consistency of the normal vector directions. The principal axis direction can be understood as the most prominent direction of an object on a two-dimensional plane, typically obtained by calculating the feature distribution in the image. Once the principal axis direction is determined, the predicted normal vectors can be rotated to ensure they align with it. For example, if analysis reveals that the principal axis direction of the circuit board is horizontal, the predicted normal vectors will be adjusted to rotate consistently in the horizontal direction, ensuring all normal vectors are compared relative to the same reference. Through these steps, the resulting surface normal map reflects the geometric features of the notebook motherboard surface and serves as a vital foundation for further inspection and analysis. This normal map helps detect defects on the circuit board, such as poor soldering or component misalignment, as these defects often exhibit abnormal normal directions or distributions in the normal map. For instance, when using the surface normal map for further analysis, an abnormal normal direction in a component may indicate a soldering problem. By observing and analyzing the normal map, potential defects can be quickly located and confirmed, improving inspection efficiency.

[0026] In an optional embodiment, multiple simulated defect images are generated based on texture information of non-critical regions, including: Non-critical area texture information is classified to obtain multiple sub-texture information, and the corresponding defect morphology set is determined based on each sub-texture information. Extract a defect pattern from each defect pattern set, and combine the extracted defect patterns based on the distribution of the subtexture information corresponding to each defect pattern in the non-critical area to obtain the defect simulation parameters. Based on the coordinates of the key region, sensitive and non-sensitive regions for detection are determined, and multiple simulated defect images are generated by combining defect simulation parameters.

[0027] It's important to note that identifying non-critical areas in the notebook motherboard image and classifying their texture information is a crucial step. The goal is to decompose the texture features of these areas into multiple sub-textures for better understanding and analysis. For example, on a notebook motherboard, non-critical areas might include areas around circuit traces or locations not involving major components. Texture analysis of these areas can categorize them into different sub-textures, such as smooth copper layers or rough solder joints. Based on each sub-texture, a corresponding set of defect morphologies is analyzed and established. Each texture type may be associated with specific defects; for instance, a rough texture on a solder joint might indicate poor soldering, while a smooth texture might indicate normal soldering. For example, the smooth copper layer sub-texture might correspond to a "missing solder joint," while the rough solder joint sub-texture might correspond to an "over-soldering" defect. These defect morphology sets aid in subsequent detection and classification. A representative defect pattern is extracted from each defect pattern set, and then these defect patterns are combined based on their distribution in non-critical areas to form defect simulation parameters. This step aims to simulate actual defect situations for more effective training and detection. For example, suppose a typical defect feature is extracted from the "missing solder joint" set, and another feature is extracted from the "over-soldered" set. By analyzing the distribution of these two defects in the image, their relative positional relationship on the circuit board can be discovered. For example, missing solder joints usually appear near capacitors, while over-soldered joints may appear near integrated circuits. Therefore, with this information, a combined model incorporating these defects can be created. Based on the coordinates of key regions, we identify which areas require special attention during the detection process (sensitive regions) and which areas can be ignored (non-sensitive regions). This aims to improve the efficiency and accuracy of detection. For example, on a notebook motherboard, the critical areas might be near the CPU and GPU, as these components are crucial to the motherboard's performance; therefore, these areas are defined as sensitive regions and require rigorous detection. Areas far from critical components may be considered non-sensitive regions, allowing for some defects. Combining the previously extracted defect simulation parameters, we generate multiple simulated defect images. These images will be used for subsequent detection algorithm training and testing to improve the system's ability to identify real defects. For example, by applying the extracted defect morphology to detect sensitive and non-sensitive regions, we can generate multiple simulated images containing different defects, such as adding an image of missing solder joints to a sensitive region and an image of over-soldering to a non-sensitive region. In this way, we can create diverse training datasets to help the model learn how to identify different types of defects.

[0028] In an optional embodiment, feature encoding is performed on each local image patch to obtain multiple first feature vectors, and feature extraction is performed on each simulated defect image to obtain multiple second feature vectors, including: Convolution and pooling operations are performed on each local image patch to extract deep semantic information and obtain multiple first feature vectors; each simulated defect image is processed to extract initial features. The initial features are input into the feature fusion layer, which includes M feature enhancement layers and M feature compression layers connected in sequence. The M feature enhancement layers are connected to the M feature compression layers through skip connection layers. When the feature enhancement layer enhances the input feature vector, it enhances the input feature vector based on the first feature vector corresponding to the local image patch and the layer parameters corresponding to the feature enhancement layer. When the feature compression layer compresses the input feature vector, it compresses the input feature vector based on the initial features corresponding to the simulated defect image and the layer parameters corresponding to the feature enhancement layer; the output feature vector after processing by the feature fusion layer is used as multiple second feature vectors.

[0029] It's important to note that when analyzing a notebook motherboard image, the entire image is segmented into multiple small local image patches. Convolution operations are then performed on these patches to extract deeper semantic information. This process typically involves multiple convolutional kernels, which capture different features such as edges, textures, and shapes. Next, pooling operations (such as max pooling or average pooling) are performed to reduce the dimensionality of the features and extract the most important information, resulting in multiple first feature vectors. For example, when processing a notebook motherboard image, it is first divided into several small patches. For a local image patch containing capacitors, convolution operations might identify the features of capacitor edges and solder joints, while pooling operations compress these features into a smaller feature vector representing the main features of that local image patch. Each simulated defect image is processed to extract initial features. These simulated defect images are created based on previously generated defect models, which help the model learn to recognize different types of defects. For example, if a simulated defect image contains features of "missing solder joints," initial features related to the missing solder joints, such as texture variations and color differences around the solder joints, can be extracted through convolution and pooling operations. The extracted initial features are then input into a feature fusion layer. This layer consists of M feature enhancement layers and M feature compression layers, connected by skip connections. This structure aims to improve the expressive power of features by alternately enhancing and compressing them. For example, assuming M=2, there are two feature enhancement layers and two feature compression layers. The first feature enhancement layer may adjust and enhance certain specific defect features, while the second layer may focus on different features. Skip connections ensure that information about the original features is preserved during feature enhancement, avoiding information loss. In the feature enhancement layer, the input feature vector is enhanced. This process is based on the first feature vector corresponding to the local image patch and the parameters of the feature enhancement layer. For example, for the feature vector of a local image patch, if the patch contains a significant missing solder joint feature, the feature enhancement layer can enhance the part related to the solder joint feature through a weighted mechanism, such as increasing the response of the area around the solder joint, thereby making the feature of the missing solder joint more prominent. In the feature compression layer, the input feature vector is compressed using the initial features corresponding to the simulated defect image and the parameters of the feature enhancement layer. The purpose of compression is to remove redundant information and retain the most effective features. For example, after feature enhancement, a certain feature vector may become very large and contain many details. The feature compression layer will remove some unnecessary details based on the importance and relevance of the features, extracting a more concise feature representation. The feature vectors processed by the feature fusion layer are output to form multiple second feature vectors. These second feature vectors will be used for subsequent classification, detection or other tasks, such as training a detection model. For example, the final output second feature vector may contain comprehensive feature information about defects such as missing solder joints and over-soldering on the notebook motherboard. This information is enhanced and compressed to better express the features, thereby improving the accuracy of subsequent detection.

[0030] In an optional embodiment, multiple first feature vectors are fused with multiple second feature vectors to obtain multiple third feature vectors; coordinate normalization processing is performed on the preliminary defect candidate region, including: An attention mechanism network is used to calculate the association weights between each first feature vector and its corresponding second feature vector; the first and second feature vectors are weighted and summed based on the association weights to generate intermediate fused features. The texture description and key region coordinates related to the defect information in the simulated defect image are obtained, the defect description information is determined and encoded to obtain the fourth feature vector; the intermediate fused features and the fourth feature vector are concatenated and processed through a fully connected layer to generate multiple third feature vectors. The pixel coordinates of the initial defect candidate region are mapped, and the coordinate range of the candidate region is scaled to a preset standard range to obtain the normalized result.

[0031] It should be noted that in this stage, an attention mechanism network is used to evaluate the correlation between each first feature vector (features from a local image patch) and its corresponding second feature vector (features processed by the feature fusion layer). By calculating the correlation weights, the model can determine which features are more important for the final defect identification. For example, suppose there is a first feature vector representing the features of a capacitor block, and the corresponding second feature vector represents the information of that block after feature fusion. The attention mechanism can calculate a correlation weight by calculating the similarity between the two. For example, the importance of the capacitor block's features for defect identification might be 0.8, while the weight of other irrelevant features might be 0.2. This means that the features of the capacitor block are more important in the final decision. By weighted summing of the first and second feature vectors and combining them with the previously calculated association weights, a new "intermediate fusion feature" is formed. This feature effectively fuses the two input features, highlighting the important parts. For example, continuing the example above, assuming the first feature vector is [0.6, 0.4] and the second feature vector is [0.5, 0.5], then after weighted summation with association weights, an intermediate fusion feature may be obtained, such as [0.64, 0.44]. This feature reflects the state of the capacitor block better than the individual features. The model needs to analyze simulated defect images to extract texture information and coordinates of key regions related to the defects. This information helps determine the specific location and nature of the defects. For example, assuming the simulated defect image contains an obvious welding defect, the model can identify the texture changes around the defect (such as uneven color, shape distortion, etc.) and record the coordinates of the defect, such as (100, 150), indicating that the defect is located in a specific region of the image. This information is encoded into a new feature representation, namely the fourth feature vector. The intermediate fusion feature is concatenated with the fourth feature vector to form a composite feature vector. Then, this composite feature is further processed through a fully connected layer to extract higher-level feature information, ultimately generating multiple third feature vectors. For example, assuming the intermediate fusion feature is [0.64, 0.44] and the fourth feature vector (containing the encoding of defect information) is [0.3, 0.7], the concatenated composite feature vector is [0.64, 0.44, 0.3, 0.7]. After processing through a fully connected layer, multiple third feature vectors may be generated, such as [0.5, 0.6, 0.4]. These features will be used for subsequent defect classification or recognition tasks. In the final step, the pixel coordinates of the initially identified defect candidate regions are mapped, scaling their coordinate range to a preset standard interval. This process ensures the consistency of the model across different image sizes or resolutions, making defect localization more reliable. For example, suppose the pixel coordinates of a preliminary defect candidate region are (120, 80), and the preset standard interval is (0, 1). Through the mapping operation, these coordinates are converted into standardized results, such as (0.12, 0.08). This means that the location of the defect is still clearly identifiable in the new coordinate system and is also suitable for subsequent processing.

[0032] In an optional embodiment, the obtained normalized result, detection feature map, surface normal map, and multiple third feature vectors are input into a convolutional neural network to reconstruct the defect probability value of the candidate region. Based on the reconstruction result and the standard template feature vector, a detection result map of the notebook motherboard is determined, including: The normalization result, detection feature map, surface normal map, and multiple third feature vectors are concatenated along the channel dimension to obtain a fused input tensor. The fused input tensor is then input into a convolutional neural network, and through multiple convolutional layers and activation function layers, the probability value of each pixel in the candidate region belonging to the defect category is reconstructed. Calculate the feature matching degree between the probability value and the preset standard template feature vector; when the feature matching degree meets the preset qualification conditions and the probability value is lower than the defect threshold, the corresponding area is determined to be a qualified area; Map the judgment results of all candidate regions back to the original image coordinate system, mark and highlight the regions judged as defects, and generate a detection result map; If the feature matching degree does not meet the qualification criteria, a second comparison is performed using the feature set of historical qualified motherboard images. If the feature similarity with the features in the historical feature set meets the criteria, the image is still determined to be qualified.

[0033] It should be noted that the normalization result, detection feature map, surface normal map, and multiple third feature vectors are concatenated along the channel dimension to form a fused input tensor. The purpose of this step is to integrate information from different sources into a unified input for subsequent processing. For example, suppose the feature size of the normalization result is (32,32,1), the size of the detection feature map is (32,32,3), the size of the surface normal map is (32,32,1), and the size of the multiple third feature vectors is (32,32,5). After concatenation along the channel dimension, the size of the resulting fused input tensor may be (32,32,10). This tensor simultaneously contains multiple information such as position, appearance, and texture, which helps the model to perform more comprehensive analysis. The fused input tensor is fed into a convolutional neural network (CNN) and processed through multiple convolutional and activation function layers. CNNs effectively extract spatial features from images and generate a probability value for each pixel belonging to a defect category. This process involves operations such as convolution, pooling, and activation, allowing the model to gradually learn the deep features of the image. For example, in the CNN processing, after several layers of convolution and activation, the model finally outputs a probability map representing the probability that each pixel belongs to a defect. For instance, some pixels in a candidate region may have a high defect probability (e.g., 0.85), while other pixels may have a low probability (e.g., 0.1), reflecting the potential presence of a defect in that region. Next, the feature matching degree between the generated probability value and the preset standard template feature vector is calculated. This process helps determine whether the currently detected feature meets the standard defect feature. For example, suppose the preset standard template feature vector is a feature description for a specific defect type (such as poor welding). When the probability map output by the model shows that the feature of the region has a high similarity to the standard template feature (for example, the score calculated by a certain similarity measurement algorithm reaches a preset threshold), it indicates that the region is likely to be qualified. When the feature matching degree meets the preset qualification condition and the probability value is lower than the defect threshold, the corresponding region is determined to be a qualified region. At this time, the model marks the region as having no defects. For example, if the probability value of a region is 0.02 (lower than the defect threshold) and the matching degree score with the standard template feature is 0.9 (higher than the qualification condition), then the region is judged to be a qualified region. The system maps the judgment results of all candidate regions back to the original image coordinate system for easy visualization and subsequent processing. Regions judged as defects are marked and highlighted to generate a detection result image. For example, in the original image, if a region with pixel coordinates (100, 150) is judged as a defective region, the system will draw a red box or mark at that location to clearly show the detected defect. If the feature matching degree does not meet the qualification criteria, a second comparison is performed using the feature set of historical qualified motherboard images. If the current feature meets the similarity criteria with some features in the historical feature set, it can still be judged as a qualified image. For example, if the feature matching degree of a certain region does not meet the standard, but by querying historical qualified motherboard images, it is found that the feature of that region meets the preset similarity criteria with the feature of historical qualified samples (e.g., the similarity score reaches 0.85), then that region can also be judged as qualified.

[0034] Example 2, please refer to Figure 2 This invention provides a technical solution: a notebook motherboard visual inspection system, applicable to the aforementioned notebook motherboard visual inspection method, comprising: Image acquisition unit 1 is used to acquire the original image of the notebook motherboard, input the original image of the notebook motherboard to be detected into the preprocessing module to generate preliminary defect candidate regions, perform semantic parsing on the original image of the notebook motherboard, and extract the coordinates of key regions and the texture information of non-key regions. Feature extraction unit 2 is used to extract multiple local image patches from the original image of the notebook motherboard based on the coordinates of the key region, perform multi-scale feature extraction and high-frequency noise feature extraction on the original image of the notebook motherboard, fuse the extracted features, and generate a detection feature map. Defect simulation unit 3 is used to calculate the surface normal map of the original image of the notebook motherboard; and to generate multiple simulated defect images based on the texture information of non-critical areas. Feature encoding unit 4 is used to encode features for each local image block to obtain multiple first feature vectors, and to extract features for each simulated defect image to obtain multiple second feature vectors; Feature fusion unit 5 is used to fuse multiple first feature vectors with multiple second feature vectors to obtain multiple third feature vectors; and to perform coordinate normalization processing on the preliminary defect candidate region. The detection result unit 6 is used to input the obtained normalized result, detection feature map, surface normal map and multiple third feature vectors into the convolutional neural network to reconstruct the defect probability value of the candidate region, and determine the detection result map of the notebook motherboard based on the reconstruction result and the standard template feature vector.

[0035] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. A visual inspection method for notebook motherboards, characterized in that, include: The original image of the notebook motherboard is obtained. The original image of the notebook motherboard to be detected is input into the preprocessing module to generate preliminary defect candidate regions. Semantic parsing is performed on the original image of the notebook motherboard to extract the coordinates of key regions and the texture information of non-key regions. Based on the coordinates of key regions, multiple local image patches are extracted from the original image of the notebook motherboard. Multi-scale feature extraction and high-frequency noise feature extraction are performed on the original image of the notebook motherboard. The extracted features are then fused to generate a detection feature map. Surface normal map of the original motherboard image; multiple simulated defect images generated based on texture information of non-critical regions; Each local image patch is feature-encoded to obtain multiple first feature vectors, and each simulated defect image is feature-extracted to obtain multiple second feature vectors. Multiple first feature vectors are fused with multiple second feature vectors to obtain multiple third feature vectors; The coordinates of the preliminary defect candidate regions are normalized. The obtained normalized results, detection feature maps, surface normal maps, and multiple third feature vectors are input into a convolutional neural network to reconstruct the defect probability values ​​of candidate regions. Based on the reconstruction results and standard template feature vectors, the detection result map of the notebook motherboard is determined.

2. The method for visual inspection of a notebook motherboard according to claim 1, characterized in that, The original image of the notebook motherboard to be inspected is input into the preprocessing module to generate preliminary defect candidate regions. Semantic parsing is performed on the original image of the notebook motherboard to extract the coordinates of key regions and the texture information of non-key regions, including: Acquire the raw RGB image of the notebook motherboard captured by the industrial camera; convert the raw RGB image to HSV color space and perform adaptive histogram equalization on the brightness channel; Edge calculation is performed on the equalized image, and preliminary defect candidate regions are generated by combining color threshold segmentation. The original RGB image is processed to identify and mark key regions containing circuit elements and background regions, and the geometric coordinates of the key regions and the texture distribution data of the non-key regions are extracted.

3. The visual inspection method for a notebook motherboard according to claim 2, characterized in that, Based on the coordinates of key regions, multiple local image patches are extracted from the original image of the notebook motherboard. Multi-scale feature extraction and high-frequency noise feature extraction are performed on the original image of the notebook motherboard. The extracted features are then fused to generate a detection feature map, including: Based on the geometric coordinates of key regions, multiple corresponding local image patches are cropped from the original image; The original image of the notebook motherboard is downsampled, and the basic features of the image are extracted through a residual network to obtain the basic feature map; Wavelet transform is performed on the basic feature map to extract high-frequency noise information from the image, resulting in high-frequency components. These high-frequency components are then processed through an attention mechanism module and subjected to inverse wavelet transform to obtain high-frequency noise features. Multi-scale dilated convolution is performed on the basic feature map to obtain multi-scale features; the multi-scale features and high-frequency noise features are weighted and fused to obtain an enhanced feature map; the enhanced feature map is upsampled to obtain a detection feature map.

4. The method for visual inspection of a notebook motherboard according to claim 3, characterized in that, The surface normal map of the original image of the computing notebook motherboard includes: The structured light reflection features and gray-level co-occurrence matrix features of the original image of the notebook motherboard are fused to obtain hybrid texture features; The mixed texture features are weighted and calculated based on the depth information of the original image of the notebook motherboard, and the gradient direction of the mixed texture features is corrected. A predicted normal map is generated based on the corrected mixed texture features to determine the surface normal vector of each pixel in the image. Determine the principal axis direction of the original image of the notebook motherboard, and perform rotation correction on the predicted normal vector according to the principal axis direction to obtain the surface normal map.

5. The visual inspection method for a notebook motherboard according to claim 4, characterized in that, Multiple simulated defect images were generated based on texture information from non-critical regions, including: Non-critical area texture information is classified to obtain multiple sub-texture information, and the corresponding defect morphology set is determined based on each sub-texture information. Extract a defect pattern from each defect pattern set, and combine the extracted defect patterns based on the distribution of the subtexture information corresponding to each defect pattern in the non-critical area to obtain the defect simulation parameters. Based on the coordinates of the key region, sensitive and non-sensitive regions for detection are determined, and multiple simulated defect images are generated by combining defect simulation parameters.

6. The method for visual inspection of a notebook motherboard according to claim 5, characterized in that, Each local image patch is feature-encoded to obtain multiple first feature vectors. Features are extracted from each simulated defect image to obtain multiple second feature vectors, including: Convolution and pooling operations are performed on each local image patch to extract deep semantic information and obtain multiple first feature vectors; each simulated defect image is processed to extract initial features. The initial features are input into the feature fusion layer, which includes M feature enhancement layers and M feature compression layers connected in sequence. The M feature enhancement layers are connected to the M feature compression layers through skip connection layers. When the feature enhancement layer enhances the input feature vector, it enhances the input feature vector based on the first feature vector corresponding to the local image patch and the layer parameters corresponding to the feature enhancement layer. When the feature compression layer compresses the input feature vector, it compresses the input feature vector based on the initial features corresponding to the simulated defect image and the layer parameters corresponding to the feature enhancement layer; the output feature vector after processing by the feature fusion layer is used as multiple second feature vectors.

7. The visual inspection method for a notebook motherboard according to claim 6, characterized in that, Multiple first feature vectors are fused with multiple second feature vectors to obtain multiple third feature vectors; The preliminary defect candidate region is subjected to coordinate normalization processing, including: An attention mechanism network is used to calculate the association weights between each first feature vector and its corresponding second feature vector; the first and second feature vectors are weighted and summed based on the association weights to generate intermediate fused features. The texture description and key region coordinates related to the defect information in the simulated defect image are obtained, the defect description information is determined and encoded to obtain the fourth feature vector; the intermediate fused features and the fourth feature vector are concatenated and processed through a fully connected layer to generate multiple third feature vectors. The pixel coordinates of the initial defect candidate region are mapped, and the coordinate range of the candidate region is scaled to a preset standard range to obtain the normalized result.

8. The method for visual inspection of a notebook motherboard according to claim 7, characterized in that, The obtained normalized results, detection feature maps, surface normal maps, and multiple third feature vectors are input into a convolutional neural network to reconstruct the defect probability values ​​of candidate regions. Based on the reconstruction results and standard template feature vectors, the detection result map of the notebook motherboard is determined, including: The normalization result, detection feature map, surface normal map, and multiple third feature vectors are concatenated along the channel dimension to obtain a fused input tensor. The fused input tensor is then input into a convolutional neural network, and through multiple convolutional layers and activation function layers, the probability value of each pixel in the candidate region belonging to the defect category is reconstructed. Calculate the feature matching degree between the probability value and the preset standard template feature vector; when the feature matching degree meets the preset qualification conditions and the probability value is lower than the defect threshold, the corresponding area is determined to be a qualified area; Map the judgment results of all candidate regions back to the original image coordinate system, mark and highlight the regions judged as defects, and generate a detection result map; If the feature matching degree does not meet the qualification criteria, a second comparison is performed using the feature set of historical qualified motherboard images. If the feature similarity with the features in the historical feature set meets the criteria, the image is still determined to be qualified.

9. A notebook motherboard visual inspection system, applicable to the notebook motherboard visual inspection method according to any one of claims 1-8, characterized in that, include: The image acquisition unit is used to acquire the original image of the notebook motherboard, input the original image of the notebook motherboard to be detected into the preprocessing module, generate preliminary defect candidate regions, perform semantic parsing on the original image of the notebook motherboard, and extract the coordinates of key regions and the texture information of non-key regions. The feature extraction unit is used to extract multiple local image patches from the original image of the notebook motherboard based on the coordinates of the key region, perform multi-scale feature extraction and high-frequency noise feature extraction on the original image of the notebook motherboard, fuse the extracted features, and generate a detection feature map. The defect simulation unit is used to calculate the surface normal map of the original image of the notebook motherboard; and to generate multiple simulated defect images based on the texture information of non-critical regions. The feature encoding unit is used to encode the features of each local image block to obtain multiple first feature vectors, and to extract the features of each simulated defect image to obtain multiple second feature vectors. The feature fusion unit is used to fuse multiple first feature vectors with multiple second feature vectors to obtain multiple third feature vectors; The coordinates of the preliminary defect candidate regions are normalized. The detection result unit is used to input the obtained normalized result, detection feature map, surface normal map and multiple third feature vectors into the convolutional neural network to reconstruct the defect probability value of the candidate region, and determine the detection result map of the notebook motherboard based on the reconstruction result and the standard template feature vector.

Images