Electronic device detection system based on computer vision

Abstract

The invention discloses an electronic device detection system based on computer vision, and relates to the technical field of electronic device automatic detection. The system comprises an image processing module, a target detection module and a defect identification module. The image processing module extracts a feature map of the electronic device image through a convolutional neural network. And the target detection module performs candidate region generation on the feature map through a region proposal network, performs binary classification analysis on each anchor frame, and outputs classification confidence. And for the anchor frames classified as targets, the positions and sizes of the anchor frames are optimized through boundary regression, redundant anchor frames are removed through non-maximum suppression, and target areas conforming to confidence are screened out. And the defect identification module is used for carrying out image block division on the target area, carrying out defect type classification in combination with a self-attention mechanism, identifying different defect modes, carrying out boundary regression optimization on the defect area, and outputting the position, the type and the confidence coefficient of the defect. Defects in the electronic device can be efficiently and accurately detected.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic device automated detection, and in particular to an electronic device detection system based on computer vision. Background Art

[0002] As electronic devices continue to develop towards high performance, miniaturization and complexity, the quality control and inspection of electronic devices are particularly important. Traditional electronic device inspection methods mostly rely on manual inspection or simple image processing technology, which is time-consuming and labor-intensive, and is easily affected by human factors, with low inspection accuracy and efficiency. In recent years, automated inspection methods based on computer vision and deep learning have gradually emerged. With their advantages in image analysis, target detection and pattern recognition, they have gradually become an important means to improve the quality and efficiency of electronic device inspection. Through convolutional neural networks (CNNs) and target detection algorithms, images of electronic devices can be automatically processed to identify potential defects, and further analyzed through precise boundary regression and defect classification, thereby achieving efficient and accurate electronic device quality inspection.

[0003] For example, the announcement CN118521591B discloses a method and system for visual detection of surface defects of optoelectronic devices, specifically a method and system for visual detection of surface defects of optoelectronic devices. The present invention first obtains the initial defect area of ​​the grayscale image of the surface of the optoelectronic device; further obtains the first defect possibility and filters out the suspected defect area; further obtains the second defect possibility and filters out the defect area to be analyzed; further obtains the grayscale change parameters of each edge pixel point of the defect area to be analyzed; further gradually adds pixels to form a corrected defect area and obtains the third defect possibility, and finally expands the defect area to be analyzed based on the third defect possibility to obtain the final defect area. The present invention analyzes the grayscale change characteristics and edge roughness of the initial defect area obtained by threshold segmentation, eliminates interference from non-defective areas, and adaptively expands the defect area to be analyzed to finally obtain an accurate defect area, thereby improving the credibility of the visual detection results.

[0004] However, existing detection methods still have some limitations. Traditional image processing methods have poor adaptability to factors such as lighting changes and background interference, and are prone to false detections and missed detections. Although deep learning methods have been applied to electronic device defect detection, they rely on a large amount of labeled data for training, and may still face problems such as high computing resource consumption and slow processing speed when faced with complex defect patterns or high-resolution images. In addition, existing methods often lack refined regression and discrimination processes when dealing with defect area boundary optimization and defect type classification, resulting in insufficient accuracy and reliability of the final detection results. Therefore, how to improve detection accuracy, reduce false detections and missed detections, and optimize detection efficiency remains an urgent problem to be solved. Summary of the invention

[0005] In view of the deficiencies of the prior art, the present invention provides an electronic device detection system based on computer vision, which solves the problems of the above-mentioned background technology.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: an electronic device detection system based on computer vision, comprising the following modules: an image processing module, a target detection module, and a defect recognition module; the image processing module is used to obtain the target electronic device image and extract the feature map of the target electronic device image through a convolutional neural network; the target detection module is used to generate candidate regions for the feature map through a region proposal network, mark the anchor boxes in the feature map, perform binary classification analysis on each anchor box, determine whether it contains the target area, and output the classification confidence of each anchor box, perform boundary regression analysis on the anchor box classified as the target, optimize the position and size of the anchor box, remove redundant anchor boxes through a non-maximum suppression algorithm, and screen the target area that meets the confidence. The defect recognition module is used to divide the target area into image blocks according to the target area output by the target detection module, feature encode each image block through a visual transformer, obtain the feature vector of each image block, capture global and local features, classify the defect type according to the output of the self-attention mechanism, identify different defect modes, perform boundary regression on the detected defect area, optimize the defect position and size, and output the position, type and confidence of the defect.

[0007] Furthermore, the specific process of extracting the feature map of the target electronic device image through the convolutional neural network is as follows: the target electronic device image is input into the convolutional neural network, the input image is convolved through the convolutional layer to extract local features, and the convolution result is nonlinearly transformed by applying the activation function; the feature map is downsampled through the pooling layer, multiple convolutional layers and pooling layers are stacked, and a multi-channel feature map is output.

[0008] Furthermore, the candidate regions of the feature map are generated through the region proposal network. The specific process of marking the anchor boxes in the feature map is as follows: the feature map extracted by the convolutional neural network is input into the region proposal network, and a sliding window operation is performed on the feature map to generate a set of fixed number of anchor boxes; multiple anchor boxes are generated at each position of the feature map, covering different scales, aspect ratios and positions, and for each anchor box, its overlap with the target area in the feature map is calculated to generate candidate regions; each anchor box is calibrated to determine the relative position of its candidate region and mark it.

[0009] Furthermore, a binary classification analysis is performed on each anchor frame to determine whether it contains the target area and output the classification confidence of each anchor frame. The specific process is as follows: for each anchor frame, features are extracted through a convolutional neural network, and the features are input into a binary classifier to determine whether the anchor frame contains the target area and output the binary classification result; at the same time, the overlap between the anchor frame and the target area is calculated, and its accuracy is evaluated by the intersection and union ratio; based on the binary classification result and the overlap, the classification confidence of each anchor frame is output to indicate whether the anchor frame accurately contains the target area.

[0010] Furthermore, the specific process of performing boundary regression analysis on the anchor frame classified as the target and optimizing the position and size of the anchor frame is as follows: for the anchor frame classified as the target, predict the boundary adjustment parameters of the anchor frame, including the position offset and the size change, through the regression network; calculate the offset of the anchor frame according to the relative position of the target area and the anchor frame, including the displacement in the horizontal and vertical directions, and the change in the aspect ratio; apply the calculated offset to the original position and size of the anchor frame, optimize the boundary of the anchor frame, make it fit the target area more accurately, and output the optimized anchor frame position and size.

[0011] Furthermore, the specific process of removing redundant anchor frames and screening target areas that meet the confidence level through the non-maximum suppression algorithm is as follows: sort all candidate anchor frames in descending order according to the classification confidence level; select the anchor frame with the highest confidence level from the sorted anchor frames as the reference frame, and calculate its overlap with other anchor frames, usually using the intersection-over-union ratio as a metric; compare the intersection-over-union ratio of each remaining anchor frame with a set threshold, and when the intersection-over-union ratio exceeds the threshold, the anchor frame is deemed redundant and is removed; repeat the above steps until all anchor frames are processed; retain anchor frames with high confidence levels and less overlap with other frames as target areas, and output the final target area after non-maximum suppression screening.

[0012] Furthermore, according to the target area output by the target detection module, the specific process of dividing the target area into image blocks is as follows: according to the coordinate information of the target area, the position of the target area in the original image is extracted, and according to the preset size or resolution requirements, the target area is divided into multiple small blocks, each small block is standardized, and the standardized image blocks are input into the visual transformer for further analysis.

[0013] Furthermore, each image block is feature encoded through a visual transformer to obtain a feature vector of each image block. The specific process of capturing global and local features is as follows: each standardized image block is input into the visual transformer, and the image block is feature extracted and encoded through the convolution layer and fully connected layer structure in the network; the local features in the image block are extracted through the convolution operation, and the local features are further compressed through the pooling layer; the local features are fused in the fully connected layer to capture the global features of the image block and generate a feature vector of the image block; the feature vector of each image block is output, which represents the high-dimensional features of the image block and contains the global and local information of the image, which is used for subsequent target recognition or classification tasks.

[0014] Furthermore, defect types are classified according to the output of the self-attention mechanism, and the specific process of identifying different defect modes is as follows: the feature vector of each image block is input into the self-attention mechanism module, and the relationship and importance between the image blocks are calculated; in the self-attention mechanism, the features of each image block are weighted by calculating the attention weight, and the defect mode classification is judged based on the weighted feature information, and the classification layer is used to output different defect types; the defect type of each target area is output, different defect modes are identified, and the defects in the target area are further processed or repaired according to the classification results.

[0015] Furthermore, the specific process of performing boundary regression on the detected defect area, optimizing the defect position and size, and outputting the defect position, type and confidence is as follows: for the detected defect area, predicting the boundary adjustment parameters of the defect area, including position offset and size change, through the regression network; calculating the horizontal and vertical displacements of the defect area and the change in the aspect ratio according to the deviation between the actual position of the defect area and the predicted boundary; applying the calculated offset to the original position and size of the defect area, adjusting the boundary of the defect area, and outputting the optimized boundary and size of the defect area; outputting the defect type according to the classification result, identifying the specific type of each defect, and indicating the detection reliability of each defect through the confidence value, and outputting the optimized position, type and confidence of each defect.

[0016] The present invention has the following beneficial effects: (1) This electronic device detection system based on computer vision realizes efficient extraction of electronic device images and accurate positioning of target areas through the collaborative work of image processing module and target detection module. The image processing module uses convolutional neural network to extract feature maps of electronic device images, improving the representation ability of image features. The target detection module generates candidate regions through region proposal network, performs binary classification analysis on each anchor frame, determines the target region and outputs the classification confidence. Through boundary regression analysis, the position and size of the anchor frame are optimized, and the non-maximum suppression algorithm is further combined to remove redundant anchor frames, and finally the target region that meets the confidence is screened out, thereby improving the detection accuracy and processing efficiency.

[0017] (2) In the defect recognition module of the electronic device inspection system based on computer vision, the visual transformer is used to divide the image blocks and encode the features, and the feature vector of each image block is obtained to effectively capture the global and local features. The self-attention mechanism is used to classify the defect types, identify different defect patterns, and further optimize the defect location and size through boundary regression. This method can accurately output the location, type and confidence of the defect, thereby realizing efficient and accurate detection of electronic devices, and improving the intelligence level and application effect of the overall detection system.

[0018] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of the electronic device detection system based on computer vision of the present invention. DETAILED DESCRIPTION

[0020] The embodiment of the present application solves the problems of low defect detection accuracy and poor detection efficiency in the process of electronic device inspection through an electronic device inspection system based on computer vision. The system uses a convolutional neural network to extract image features, generates candidate regions through a region proposal network, and performs binary classification analysis and boundary regression optimization on the anchor frame, thereby improving the positioning accuracy of the target area. Redundant anchor frames are removed through a non-maximum suppression algorithm to screen out target areas with high confidence. The defect recognition module combines a visual transformer and a self-attention mechanism to accurately classify different defect types, and performs boundary regression on defective areas to optimize defect locations and sizes. The system can achieve efficient and accurate defect detection of electronic devices to meet the needs of automated production and quality control.

[0021] The overall idea of ​​the solution in the embodiments of this application is as follows: An image of a target electronic device is acquired, and a feature map of the image of the target electronic device is extracted through a convolutional neural network.

[0022] The candidate regions of the feature map are generated through the region proposal network, the anchor boxes in the feature map are marked, and each anchor box is subjected to binary classification analysis to determine whether it contains the target area and output the classification confidence of each anchor box. The anchor boxes classified as targets are subjected to boundary regression analysis to optimize the position and size of the anchor boxes. The redundant anchor boxes are removed through the non-maximum suppression algorithm, and the target areas that meet the confidence level are screened.

[0023] According to the target area output by the target detection module, the target area is divided into image blocks, and each image block is feature encoded through the visual transformer to obtain the feature vector of each image block, capture global and local features, and classify the defect type according to the output of the self-attention mechanism to identify different defect modes. The boundary regression is performed on the detected defect area, the defect position and size are optimized, and the defect position, type and confidence are output.

[0024] See also Figure 1 , the embodiment of the present invention provides a technical solution: an electronic device detection system based on computer vision, comprising the following modules: an image processing module, a target detection module, and a defect recognition module; the image processing module is used to obtain the target electronic device image and extract the feature map of the target electronic device image through a convolutional neural network; the target detection module is used to generate candidate regions for the feature map through a region proposal network, mark the anchor boxes in the feature map, perform binary classification analysis on each anchor box, determine whether it contains the target area, and output the classification confidence of each anchor box, perform boundary regression analysis on the anchor box classified as the target, optimize the position and size of the anchor box, remove redundant anchor boxes through a non-maximum suppression algorithm, and screen the target area that meets the confidence. The defect recognition module is used to divide the target area into image blocks according to the target area output by the target detection module, perform feature encoding on each image block through a visual transformer, obtain the feature vector of each image block, capture global and local features, classify the defect type according to the output of the self-attention mechanism, identify different defect modes, perform boundary regression on the detected defect area, optimize the defect position and size, and output the position, type and confidence of the defect.

[0025] In this embodiment, the image processing module: the main task of this module is to obtain images from electronic devices and extract feature maps of the images through convolutional neural networks. CNN is a deep learning algorithm specifically used for image processing. It automatically extracts important visual features from images through operations such as convolution and pooling. This process helps to convert the original image into a feature map that is easy to analyze later. Object detection module: the task of this module is to process the extracted feature map and generate candidate regions that may contain target regions. Its key technology is the region proposal network, which is an algorithm that generates candidate regions by analyzing the feature map of the image. The region proposal network generates multiple anchor boxes (i.e., rectangular boxes of fixed size) for each position and performs binary classification analysis on these anchor boxes to determine whether each anchor box contains the target area. In the binary classification process, each anchor box is classified as "containing the target" or "not containing the target". At the same time, the module outputs the classification confidence of each anchor box, indicating whether the anchor box may contain the target. For those anchor boxes classified as targets, the module performs boundary regression analysis, that is, adjusts the position and size of the anchor box to optimize the accuracy of the anchor box. This optimization enables the anchor box to cover the target area more accurately. The module also uses the non-maximum suppression algorithm to remove redundant anchor boxes, that is, when multiple anchor boxes may cover the same target, only one optimal anchor box is retained to ensure the accuracy of the detection results. Defect recognition module: According to the target area output by the target detection module, the defect recognition module will further analyze the details in the target area. First, the module divides the target area into multiple small image blocks, and then encodes the features of each image block through the visual transformer. The visual transformer is an algorithm for processing the relationship between image blocks, which can help extract the global and local features of the image block, encode them, and convert them into feature vectors. The system uses a self-attention mechanism, which can weight the information of different parts according to the relevance of the input data, so as to focus on important areas. In this way, the system can automatically identify different types of defects and classify them. The self-attention mechanism helps to process large-scale image data and strengthen the focus on local details. For the detected defect area, the module will also perform boundary regression, that is, further adjust the position and size of the defect area. This step makes the detected defects more accurate and avoids missed detection or misjudgment. Ultimately, the defect recognition module will output the location, type, and confidence of each defect, where the location refers to the specific location of the defect in the image, the type refers to the type of defect (such as cracks, scratches, etc.), and the confidence refers to the system's confidence in the detection results.

[0026] Specifically, the specific process of extracting the feature map of the target electronic device image through the convolutional neural network is as follows: the target electronic device image is input into the convolutional neural network, the input image is convolved through the convolutional layer, local features are extracted, and the convolution result is nonlinearly transformed by applying the activation function; the feature map is downsampled through the pooling layer, multiple layers of convolutional layers and pooling layers are stacked, and a multi-channel feature map is output.

[0027] In this embodiment, input images: images of target electronic devices are input into the convolutional neural network. These images contain all the visual information of the target electronic devices, and the task of the network is to extract useful feature information from these images for subsequent processing and analysis. Convolution operation: The image is processed by the convolution layer, which applies multiple convolution kernels (filters) to perform convolution operations on the input image. The convolution kernel is a small-sized matrix (usually 3x3 or 5x5), which slides on the input image and performs weighted summation of pixels in the local area to extract local features (such as edges, corners, textures, etc.). These convolution operations help the network focus on local information at different locations in the image. Through convolution operations, CNN can extract features at different levels from the original image. In the primary layer, the convolution kernel mainly extracts simple features, such as edges and corners; in deeper layers, the convolution kernel combines these simple features to extract more complex structural features. Activation function: For each convolution result, the activation function ReLU is usually used for nonlinear transformation. The role of the activation function is to introduce nonlinear factors so that the network can learn and express complex patterns. Without the activation function, the convolution operation is only a linear transformation, which limits the expressiveness of the network. ReLU is the most commonly used activation function. It sets the part of the convolution operation result that is less than zero to zero and keeps the part that is greater than zero unchanged, thereby simplifying and accelerating the training process in terms of calculation. Pooling operation: After the convolution layer, a pooling layer is usually applied to downsample the feature map. The pooling operation helps to reduce the size of the feature map, thereby reducing the computational complexity and the burden of subsequent calculations, while retaining the most important feature information. There are usually two common ways of pooling operation: maximum pooling and average pooling. Maximum pooling: In a specific local area, the largest value is selected as the pooling result, so that the most significant features in the area can be retained. Average pooling: Calculate the average value of the pixels in the area and take the average value as the pooling result. Through the pooling layer, the size of the feature map is reduced, but the information is still retained with high quality. Pooling also has a certain degree of translation invariance, that is, it can identify objects whose relative positions in the image change. Multi-layer convolution and pooling stacking: In order to extract more complex features, multiple convolutional layers and pooling layers are usually stacked in the network. By stacking multiple layers, the network can gradually extract higher-level features (such as the shape, texture, structure, etc. of the object) from the most basic features (such as edges and corners). For example, the first few layers may focus on the edges, colors, and textures in the image, while the deeper layers can identify the specific shapes and complex structural relationships of the objects. Output multi-channel feature maps: After multiple layers of convolution and pooling, the convolutional neural network outputs a multi-channel feature map. Each channel represents the response of the image in a specific feature dimension. These feature maps contain important information about the target object in the image and are the basis for subsequent target detection and defect recognition tasks.

[0028] Specifically, the candidate regions of the feature map are generated through the region proposal network, and the specific process of marking the anchor boxes in the feature map is as follows: the feature map extracted by the convolutional neural network is input into the region proposal network, and a sliding window operation is performed on the feature map to generate a set of fixed number of anchor boxes; multiple anchor boxes are generated at each position of the feature map, covering different scales, aspect ratios and positions, and for each anchor box, its overlap with the target area in the feature map is calculated to generate candidate regions; each anchor frame is calibrated to determine the relative position of its candidate region and mark it.

[0029] In this embodiment, input feature map: the feature map previously extracted by convolutional neural network (CNN) is input into the region proposal network (RPN). Feature map is the deep image information extracted by convolutional neural network, which contains the high-order features of the target object. The task of RPN is to generate candidate regions (also called anchor boxes) that may contain the target based on these feature maps. Sliding window operation: The region proposal network performs a sliding window operation on the input feature map, which means that at each position of the feature map, RPN will slide with a fixed window size through which the image is scanned. At each window position, RPN generates multiple anchor boxes. These anchor boxes can be regarded as preliminary estimates of candidate regions, which cover different positions in the image that may contain the target. Generation of anchor boxes: At each position of the feature map, RPN generates multiple anchor boxes, which have different scales (size), aspect ratios (width-to-height ratio) and positions (relative position of the anchor box in the feature map). These different anchor boxes enable RPN to adapt to target objects of different sizes and shapes. Specifically, for each position of the feature map, RPN generates several anchor boxes, each of which represents a possible target region. Calculate overlap: For each anchor box, RPN calculates the overlap between the anchor box and the true target area (i.e., the area containing the target) in the feature map. Overlap is usually measured by intersection over union, where IoU represents the ratio of the intersection of the anchor box and the true target area to their union. The higher the IoU, the greater the overlap between the anchor box and the target area, and the more likely the candidate area is to contain the target. Generate candidate areas: Based on the overlap of each anchor box, RPN selects those anchor boxes with a high degree of overlap with the true target area as candidate areas. Usually, RPN sets a threshold, and only when the IoU of the anchor box exceeds this threshold will the anchor box be considered a valid candidate area. Position calibration: For each selected candidate area, RPN performs position calibration. Mark candidate areas: Finally, RPN marks these candidate areas based on their relative position and size, determines their exact position in the image, and prepares for subsequent classification and further processing.

[0030] Specifically, a binary classification analysis is performed on each anchor frame to determine whether it contains the target area and output the classification confidence of each anchor frame. The specific process is as follows: for each anchor frame, features are extracted through a convolutional neural network, and the features are input into a binary classifier to determine whether the anchor frame contains the target area and output the binary classification result; at the same time, the overlap between the anchor frame and the target area is calculated, and its accuracy is evaluated by the intersection and union ratio; based on the binary classification result and the overlap, the classification confidence of each anchor frame is output to indicate whether the anchor frame accurately contains the target area.

[0031] In this implementation, feature extraction: for each anchor box, features are extracted through a convolutional neural network. Assume that the extracted features are , where i represents the i-th anchor box; ; :No. The anchor boxes are feature vectors extracted from the input image. : Convolutional neural network, used to extract features from the input image. Binary classification analysis: The extracted features Input into the binary classifier and output the category label and the corresponding confidence .here is the binary classification result, Anchor box

[0032] The confidence level of whether the target area is included is usually output using the Sigmoid function: ; : Anchor box Whether to include the classification result of the target area, the value is 0 or 1. : Sigmoid activation function, which limits the output to the (0,1) interval. : The weight matrix of the classifier, used to map features to classification outputs. : The bias term of the classifier. For confidence , can be calculated by the following formula: ; : Anchor box Whether the initial classification confidence of the target area is included, the value range is (0,1). Calculate the intersection over union (loU): Calculate the anchor box and target area The intersection-over-combination ratio between them is: ;IoU : Anchor box and the actual target area The intersection ratio : Anchor box area. : The actual target area. : Intersection operation, which represents the overlapping part of two regions. U: Union operation, which represents the total area of ​​two regions. : Indicates the area of ​​the region. Calculate classification confidence: Combine the two classification results And the intersection over union (IoU) to calculate the final classification confidence .if (that is, the anchor box is classified as the target area), the IoU value is used as part of the confidence, and the final confidence Combine the output of the binary classifier to calculate: ; : The final classification confidence, indicating the final confidence that the anchor box i contains the target area. : The confidence of the binary classifier output, indicating whether the anchor box contains the target area. : The intersection-over-union ratio between the anchor box and the target area, which measures the degree of overlap.

[0033] Specifically, the specific process of performing boundary regression analysis on the anchor frame classified as the target and optimizing the position and size of the anchor frame is as follows: for the anchor frame classified as the target, predict the boundary adjustment parameters of the anchor frame, including the position offset and the size change, through the regression network; calculate the offset of the anchor frame according to the relative position of the target area and the anchor frame, including the displacement in the horizontal and vertical directions, and the change in the aspect ratio; apply the calculated offset to the original position and size of the anchor frame, optimize the boundary of the anchor frame, make it fit the target area more accurately, and output the optimized anchor frame position and size.

[0034] In this embodiment, the regression network predicts adjustment parameters: For the anchor frame classified as the target, the boundary adjustment parameters of the anchor frame are first predicted by the regression network. These parameters include: Position offset: that is, the offset between the center coordinates of the anchor frame and the center coordinates of the target area, usually calculated in the horizontal and vertical directions respectively. Size change: that is, the proportional difference between the width and height of the anchor frame and the width and height of the target area, usually described by the change of aspect ratio. Calculate the offset of the anchor frame: The offset of the anchor frame can be obtained by calculating the relative position between the target area and the anchor frame. These offsets include: Horizontal displacement: The horizontal offset between the center of the target area and the center of the anchor frame. Vertical displacement: The vertical offset between the center of the target area and the center of the anchor frame. Aspect ratio change: The change between the width and height of the anchor frame and the width and height of the target area, that is, the difference between the aspect ratio of the target area and the aspect ratio of the anchor frame. Apply the offset to optimize the anchor frame: Use the calculated offset to adjust the position and size of the anchor frame. Specifically: Position adjustment: According to the offsets in the horizontal and vertical directions, adjust the center position of the anchor frame to make it closer to the center of the target area. Size adjustment: According to the change of aspect ratio, adjust the width and height of the anchor box to make it more accurately match the size of the target area. Output optimized anchor box: Finally, after regression adjustment, output the optimized anchor box, the boundary of which will more accurately fit the position and size of the target area.

[0035] Specifically, the specific process of removing redundant anchor frames and screening target areas that meet the confidence level through the non-maximum suppression algorithm is as follows: sort all candidate anchor frames in descending order according to the classification confidence level; select the anchor frame with the highest confidence level from the sorted anchor frames as the reference frame, and calculate its overlap with other anchor frames, usually using the intersection-over-union ratio as a metric; compare the intersection-over-union ratio of each remaining anchor frame with a set threshold. When the intersection-over-union ratio exceeds the threshold, the anchor frame is deemed redundant and is removed; repeat the above steps until all anchor frames are processed; retain anchor frames with high confidence levels and less overlap with other frames as target areas, and output the final target area after non-maximum suppression screening.

[0036] In this implementation, descending sorting is performed: all candidate anchor frames are sorted in descending order according to their classification confidence, and the frame with the highest confidence is selected as the reference frame. Suppose the confidence of all anchor frames is The sorted anchor boxes are ,in . Non-maximum suppression: From the sorted anchor boxes, select the box with the highest confidence in turn and calculate its intersection-union ratio with other anchor boxes. Exceeding the set threshold When is redundant and should be removed. The specific steps are as follows: For the first anchor box , calculate its intersection with all other anchor boxes. For each anchor box ,if , then remove the anchor box This process is repeated until all anchor boxes have been processed.

[0037] Specifically, according to the target area output by the target detection module, the specific process of dividing the target area into image blocks is as follows: according to the coordinate information of the target area, the position of the target area in the original image is extracted, and according to the preset size or resolution requirements, the target area is divided into multiple small blocks, each small block is standardized, and the standardized image blocks are input into the visual transformer for further analysis.

[0038] In this embodiment, target area extraction: according to the coordinate information of the target area output by the target detection module (such as the coordinates of the upper left corner and the lower right corner), the target area is extracted from the original image. This process locates and cuts out the target area of ​​interest from the entire image. Target area division: According to the preset size or resolution requirements, the target area is divided into multiple smaller image blocks. The image block is usually divided into several small blocks evenly for more detailed analysis. These small blocks can be regarded as local areas, which are convenient for identifying finer-grained features in the target area. Standardization: After each image block is divided, it needs to be standardized. Standardization usually scales the pixel values ​​of the image block to a uniform range (such as between 0 and 1) to eliminate the differences between the image blocks due to factors such as size and brightness, and ensure the stability and accuracy of subsequent analysis. Input to the visual transformer: Each standardized image block will be input into the visual transformer for further analysis. The visual transformer is a network model based on the self-attention mechanism, which can effectively capture local and global feature information and perform fine analysis on the details in the image, such as identifying defect types and detecting target locations.

[0039] Specifically, each image block is feature encoded through a visual transformer to obtain a feature vector for each image block. The specific process of capturing global and local features is as follows: each standardized image block is input into the visual transformer, and the image block is feature extracted and encoded through the convolutional layer and fully connected layer structure in the network; the local features in the image block are extracted through the convolution operation, and the local features are further compressed through the pooling layer; the local features are fused in the fully connected layer to capture the global features of the image block and generate a feature vector for the image block; the feature vector of each image block is output, which represents the high-dimensional features of the image block and contains the global and local information of the image, which is used for subsequent target recognition or classification tasks.

[0040] In this embodiment, the standardized image block is input: First, each standardized image block is input into the visual transformer. The standardization process makes the brightness, contrast and other factors of each image block tend to be consistent, eliminating the differences between image blocks, thereby improving the stability of feature extraction. Local feature extraction (convolution operation): The initial processing of the visual transformer is performed through the convolution layer. In the convolution layer, the convolution kernel performs a convolution operation on the image block to extract local features. The convolution operation scans the image block step by step through a sliding window to extract detailed information in the image, such as local features such as edges, textures, and corners. These local features are the basic information of the image block, which helps capture the basic morphology and local details of the target area. Local feature compression (pooling layer): Next, the pooling layer further processes and compresses the local features. The pooling layer (usually maximum pooling or average pooling) reduces the size of the feature map while retaining the most important information. The pooling operation can reduce the computational complexity and help improve the robustness of the model to changes such as image deformation and scaling. Global feature fusion (fully connected layer): After being processed by the convolution and pooling layers, the local features of the image block will be sent to the fully connected layer. The fully connected layer fuses different local features and combines local information to capture the global features of the image block. The function of the fully connected layer is to synthesize the features extracted previously to form a high-dimensional feature representation that can reflect the overall information of the entire image block. Generate feature vector: After being processed by the convolution layer, pooling layer, and fully connected layer, each image block will generate a feature vector. This feature vector is a high-dimensional representation of the image block and contains global and local information of the image. The feature vector can describe various details (local features) and overall semantics (global features) in the image block, providing effective feature support for subsequent target recognition or classification tasks. Output feature vector: Finally, the feature vector of each image block will be output. This feature vector, as a representation of the image block, can be used for subsequent tasks such as target classification, defect recognition, etc. By encoding the local and global features of the image block, the system can better understand the image content, thereby improving the accuracy of detection or recognition.

[0041] Specifically, defect types are classified according to the output of the self-attention mechanism, and the specific process of identifying different defect patterns is as follows: the feature vector of each image block is input into the self-attention mechanism module, and the relationship and importance between the image blocks are calculated; in the self-attention mechanism, the features of each image block are weighted by calculating the attention weight, and the defect pattern classification is judged based on the weighted feature information, and the classification layer is used to output different defect types; the defect type of each target area is output, different defect patterns are identified, and the defects in the target area are further processed or repaired according to the classification results.

[0042] In this implementation, the feature vector is input to the self-attention mechanism; the feature vector of each image block has been encoded by the visual transformer, and then these feature vectors are input to the self-attention mechanism module. Formula: , ; : A set of feature vectors of image blocks. The feature vector of the ath image block. b: The number of image blocks. d: The dimension of the feature vector of each image block. The attention weight is obtained by calculating the relationship between each image block. The similarity is usually calculated using the dot product and normalized by the Softmax formula: ; : The query vector, key vector and value vector are respectively obtained by passing the input feature X through the weight matrix Transformed to get. : Calculate the dot product similarity between the query and the key, This is to prevent the dot product result from being too large and causing the gradient to disappear. The function converts the attention score into a probability distribution, which indicates the importance of each image block. V: value vector, indicating the weighted feature. Weighted feature processing, based on the calculated attention weight, performs weighted processing on the features of each image block. The weighted features contain the relationship and dependency information between image blocks. Formula: ; : The weighted feature contains the information about the relationship between image blocks and reflects the importance of the image blocks in the whole. Defect pattern classification uses the weighted feature vector to classify the defect type through the classification layer (such as the fully connected layer). Formula: ; : The weight matrix of the classification layer, used to map weighted features to the defect category space. The bias term of the classification layer. Softmax: The Softmax function converts the classification score into a probability distribution, which represents the predicted probability of each defect type. Output defect type By obtaining the maximum value of the classification result, the defect type formula for each target area is output: ; :: The predicted defect type category is a probability distribution The category with the largest median value. : Probability distribution of defect types output by the classification layer.

[0043] Specifically, the specific process of performing boundary regression on the detected defect area, optimizing the defect position and size, and outputting the defect position, type and confidence is as follows: for the detected defect area, predict the boundary adjustment parameters of the defect area, including position offset and size change, through the regression network; calculate the horizontal and vertical displacements of the defect area and the change in the aspect ratio according to the deviation between the actual position of the defect area and the predicted boundary; apply the calculated offset to the original position and size of the defect area, adjust the boundary of the defect area, and output the optimized boundary and size of the defect area; output the defect type according to the classification result, identify the specific type of each defect, and represent the detection reliability of each defect through the confidence value, and output the optimized position, type and confidence of each defect.

[0044] In this implementation scheme, regression predicts defect boundaries: for each detected defect area, the regression network predicts its boundary adjustment parameters, mainly including the position offset and size change of the defect area. Position offset refers to the horizontal and vertical displacement of the defect area, which indicates the difference between the current position and the predicted position of the defect area. Size change indicates the change in width and height of the defect area, that is, the difference between the actual size of the defect and the predicted size. Calculate boundary deviation: According to the deviation between the actual position of the detected defect area and the boundary predicted by regression, calculate the horizontal and vertical displacement of the defect area, as well as the change in aspect ratio. These deviations represent the difference between the current detection frame and the actual defect area, which can help adjust the size and position of the detection frame more accurately. Adjust the boundary of the defect area: Apply the calculated offset (horizontal, vertical displacement and size change) to the original position and size of the defect area. In this way, the boundary after regression is closer to the actual boundary of the defect area, thereby improving the accuracy of the detection result. Output optimized defect area: The adjusted defect area is the optimized boundary and size, which can more accurately represent the position and size of the defect. Defect type and confidence: Based on the previous classification results, the specific type of each defect is output, indicating which category the defect belongs to. The confidence value indicates the reliability of the detection result. Confidence is usually used to evaluate the detection reliability of each defect based on the classification results. A high confidence value means that the detection result is more reliable, while a low confidence value indicates that there may be a risk of false detection or missed detection. The final output is the optimized location, type and confidence of each defect, which can help further defect processing or repair work.

[0045] In summary, this application has at least the following effects: The electronic device inspection system based on computer vision can automatically extract features from electronic device images and accurately identify target areas through image processing modules and target detection modules to ensure detection accuracy and efficiency. Through binary classification analysis of anchor boxes, boundary regression and non-maximum suppression algorithm, it can not only identify target areas, but also effectively optimize the position and size of target boxes, eliminate redundant boxes and improve detection accuracy. The defect recognition module combines the feature encoding of visual transformers with the self-attention mechanism to comprehensively capture the global and local features of image blocks, effectively distinguish different types of defects, and achieve accurate defect classification. Boundary regression analysis of the detected defect area can accurately optimize the defect position and size, ensure the accurate positioning of the defect area, and enhance the reliability of defect detection. The defect type and position of each target area are output in combination with confidence, which is convenient for subsequent defect processing or repair operations, ensuring that the detection results have high accuracy and reliability. The self-attention mechanism can deeply analyze the details of the target area according to the weighted relationship between global and local features, and improve the flexibility and accuracy of defect type identification. Through the collaborative work of the three modules of image processing, target detection and defect recognition, an efficient, modular and complex scene detection system is constructed, which is easy to be applied to large-scale electronic device inspection.

[0046] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0047] The present invention is described with reference to flowcharts and / or block diagrams of systems, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0048] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0049] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0050] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0051] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An electronic device detection system based on computer vision, characterized in that: It includes the following modules: image processing module, target detection module, defect recognition module; The image processing module is used to obtain the target electronic device image and extract the feature map of the target electronic device image through a convolutional neural network; The target detection module is used to generate candidate regions for feature maps through a region proposal network, mark anchor boxes in the feature maps, perform binary classification analysis on each anchor box to determine whether it contains a target region, and output the classification confidence of each anchor box, perform boundary regression analysis on anchor boxes classified as targets, optimize the position and size of anchor boxes, remove redundant anchor boxes through a non-maximum suppression algorithm, and screen target regions that meet the confidence level; The defect recognition module is used to divide the target area into image blocks according to the target area output by the target detection module, perform feature encoding on each image block through a visual transformer, obtain a feature vector of each image block, capture global and local features, classify defect types according to the output of the self-attention mechanism, identify different defect modes, perform boundary regression on the detected defect area, optimize the defect position and size, and output the defect position, type and confidence.

2. The electronic device detection system based on computer vision according to claim 1, characterized in that: The specific process of extracting the feature map of the target electronic device image through the convolutional neural network is as follows: The target electronic device image is input into the convolutional neural network, the input image is convolved through the convolution layer to extract local features, and the activation function is applied to perform nonlinear transformation on the convolution result; The feature map is downsampled through the pooling layer, and multiple convolutional layers and pooling layers are stacked to output multi-channel feature maps.

3. The electronic device detection system based on computer vision according to claim 2, characterized in that: The specific process of generating candidate regions for feature maps through the region proposal network and marking anchor boxes in feature maps is as follows: The feature map extracted by the convolutional neural network is input into the region proposal network, and a sliding window operation is performed on the feature map to generate a set of fixed number of anchor boxes; Generate multiple anchor boxes at each position in the feature map, covering different scales, aspect ratios, and positions. For each anchor box, calculate its overlap with the target area in the feature map to generate a candidate region. Each anchor box is calibrated to determine the relative position of its candidate region and mark it.

4. The electronic device detection system based on computer vision according to claim 3, characterized in that: The specific process of performing binary classification analysis on each anchor box to determine whether it contains the target area and outputting the classification confidence of each anchor box is as follows: For each anchor box, features are extracted through a convolutional neural network and input into a binary classifier to determine whether the anchor box contains the target area and output the binary classification result; At the same time, the overlap between the anchor frame and the target area is calculated, and its accuracy is evaluated by the intersection-over-union ratio; According to the binary classification results and overlap, the classification confidence of each anchor box is output to indicate whether the anchor box accurately contains the target area.

5. The electronic device detection system based on computer vision according to claim 4, characterized in that: The specific process of performing boundary regression analysis on the anchor box classified as the target and optimizing the position and size of the anchor box is as follows: For the anchor box classified as the target, the boundary adjustment parameters of the anchor box are predicted through the regression network, including the position offset and size change; According to the relative position of the target area and the anchor frame, the offset of the anchor frame is calculated, including the horizontal and vertical displacements, as well as the change in the aspect ratio; The calculated offset is applied to the original position and size of the anchor box, the boundary of the anchor box is optimized to fit the target area more accurately, and the optimized anchor box position and size are output.

6. The electronic device detection system based on computer vision according to claim 5, characterized in that: The specific process of removing redundant anchor boxes and screening target areas that meet the confidence level through the non-maximum suppression algorithm is as follows: Sort all candidate anchor boxes in descending order according to classification confidence; From the sorted anchor boxes, select the anchor box with the highest confidence as the reference box, and calculate its overlap with other anchor boxes, usually using the intersection-over-union ratio as a metric; Compare the intersection-and-union ratio of each remaining anchor box with the set threshold. When the intersection-and-union ratio exceeds the threshold, it means that the anchor box is redundant and is removed. Repeat the above steps until all anchor boxes are processed; The anchor boxes with high confidence and less overlap with other boxes are retained as the target area, and the final target area after non-maximum suppression filtering is output.

7. The electronic device detection system based on computer vision according to claim 6, characterized in that: According to the target area output by the target detection module, the specific process of dividing the target area into image blocks is as follows: According to the coordinate information of the target area, the position of the target area in the original image is extracted, and according to the preset size or resolution requirements, the target area is divided into multiple small blocks, each small block is standardized, and the standardized image block is input into the visual transformer for further analysis.

8. The electronic device detection system based on computer vision according to claim 7, characterized in that: The specific process of encoding the features of each image block through the visual transformer, obtaining the feature vector of each image block, and capturing the global and local features is as follows: Each standardized image block is input into the visual transformer, and the image block is feature extracted and encoded through the convolutional layer and fully connected layer structure in the network; The local features in the image block are extracted through the convolution operation, and the local features are further compressed through the pooling layer; In the fully connected layer, the local features are fused to capture the global features of the image block and generate the feature vector of the image block; The feature vector of each image block is output, which represents the high-dimensional features of the image block and contains the global and local information of the image, which is used for subsequent target recognition or classification tasks.

9. The electronic device detection system based on computer vision according to claim 8, characterized in that: The specific process of classifying defect types based on the output of the self-attention mechanism and identifying different defect modes is as follows: The feature vector of each image block is input into the self-attention mechanism module to calculate the relationship and importance between image blocks; In the self-attention mechanism, the features of each image block are weighted by calculating the attention weights. Based on the weighted feature information, the defect pattern classification is judged and the classification layer is used to output different defect types. Output the defect type of each target area, identify different defect modes, and further process or repair the defects in the target area based on the classification results.

10. The electronic device detection system based on computer vision according to claim 9, characterized in that: The specific process of performing boundary regression on the detected defect area, optimizing the defect position and size, and outputting the defect position, type, and confidence is as follows: For the detected defective area, the boundary adjustment parameters of the defective area are predicted through the regression network, including the position offset and size change; According to the deviation between the actual position of the defect area and the predicted boundary, the horizontal and vertical displacements of the defect area and the change in the aspect ratio are calculated; Apply the calculated offset to the original position and size of the defect area, adjust the boundary of the defect area, and output the optimized boundary and size of the defect area; The defect type is output according to the classification result, the specific type of each defect is identified, and the detection reliability of each defect is represented by the confidence value, and the optimized position, type and confidence of each defect are output.

Images