Universal target difference detection module
By adopting a universal target difference detection neural network algorithm in industrial automation quality inspection, combined with ROI feature recognition, RPN matching and difference identification units, the problems of difficulty in defect recognition and high false alarm rate in image detection are solved, and efficient and accurate Defect detection and comprehensive determination.
Patent Information
- Application Number
- CN202411160177.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-11-22
AI Technical Summary
In the field of industrial automation quality inspection, especially in semiconductors, PCBAs, PCBs and other fields, due to inconsistent imaging conditions during image detection, defects are difficult to accurately identify, and the false alarm rate is high. The existing single image detection method cannot achieve comprehensive defect determination. , low efficiency and low accuracy.
The detection module based on the universal target difference detection neural network algorithm is adopted, which includes ROI feature recognition unit, RPN matching unit and difference identification unit. High-dimensional information is extracted through the neural network convolution layer, and the rotation translation parameters of the image are identified and matched, and similar calculations are calculated. The result of the results of the results of the shooting image and the standard image can be realized.
Effectively identifying and positioning defects in the image improves the accuracy and efficiency of detection, reduces labor costs, reduces false alarms and missed detection, and adapts to the detection needs of different scenarios.
Smart Images

Figure CN119006428A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of industrial quality inspection, in particular, to the field of industrial automation quality inspection based on image detection algorithms, and more particularly, to a universal target difference detection module based on image detection algorithms. Background Art
[0002] In the field of industrial automation quality inspection based on image detection algorithms, especially in the semiconductor, PCBA, PCB and other fields, defect identification is usually given by comparing the captured image with the standard image. Due to limited imaging conditions, there are often inconsistencies in rotation, translation, scaling, noise, etc. between the captured image and the standard image. Whether using manual detection or traditional image processing methods, when detecting defects in such images, it is often difficult to identify defects, the judgment is inaccurate, and there are too many false alarms. Specifically:
[0003] Manual inspection means that the captured image and the standard image are displayed simultaneously on the manual inspection software interface, and the defect location is judged by comprehensive analysis through an overview of all images. The disadvantage of the personnel operation scheme is that an operator needs to be assigned to the inspection station in order to respond to the defect identification needs in real time, resulting in a waste of labor costs. At the same time, due to personnel fatigue and poor positioning effect, certain errors will be caused. Contrast defects are a type of defect that relies more on the experience and judgment of the image judge. This type of defect is greatly affected by human factors during the judgment process. For the same set of images, different image judges will make comprehensive judgments based on their personal experience, and the results are often inconsistent. In the current on-site manual judgment, an experienced image judge is usually required as the leader of the image judgment team to conduct random inspections and reviews of the manual inspection results. This causes a huge waste of labor costs. For some defects that are inconsistently judged, there are corresponding yield problems such as overkill and missed inspections.
[0004] Image algorithms are used for detection, that is, algorithms are used separately for each image to calculate defect results separately. Finally, the defect results of the image are summarized, and the only defect result is obtained according to the formula or priority logic. However, the visual inspection method based on a single image has two major disadvantages. One is that for many defects, different images need to be combined for defect judgment, so that the algorithm can be designed. The algorithm based on a single image cannot achieve this function, and can only find its defect characteristics in the image, which is limited. Moreover, in the final defect synthesis link, the logic of its defect synthesis is pre-agreed and solidified in the program. It cannot be adjusted according to the actual situation, and the correctness of its preset rules and the correctness of the logical priority are questionable. In general, the single image detection method still has corresponding yield problems such as overkill and missed detection. Due to the detection algorithm for a single workstation, a single shooting scene, and a single image, during the implementation of industrial projects, the algorithm needs to be redeveloped and designed with different scenes, resulting in a great reduction in efficiency and low accuracy.
[0005] Therefore, how to process captured images and standard images with widely varying image quality and find defect locations has always been one of the important problems plaguing the image quality inspection industry. Summary of the invention
[0006] The present invention aims to overcome the above defects. In the process of research, it is found that defects that can only be judged by combining the captured image with the standard image are often more important in the process section, affecting the overall product structure parameters and basic functions. Such defects are often difficult to locate, difficult to detect, and difficult to jointly judge. Therefore, the detection of such defects directly affects the yield of the product. Therefore, the purpose of the present invention is to propose a detection module based on a universal target difference detection neural network algorithm, which can effectively and comprehensively identify defects.
[0007] The present invention provides a universal target difference detection module, which is characterized by comprising a ROI feature recognition unit, an RPN matching unit and a difference identification unit;
[0008] The ROI feature recognition unit identifies the ROI area of the captured image and the standard image, and selects the best position for output;
[0009] The RPN matching unit above searches for the best matching position through multi-channel deformation based on the ROI area of the captured image and the standard image;
[0010] The difference identification unit extracts features based on the best matching position between the captured image and the ROI area of the standard image to obtain a similarity result.
[0011] Furthermore, the present invention provides a universal target difference detection module, which is also characterized by:
[0012] The above-mentioned ROI feature recognition unit jointly calculates the captured image and the standard image, extracts the high-dimensional information of the two images through the convolution layer of the neural network, identifies the difference between the foreground and the background, and calculates the foreground area with a certain similarity. The output is the approximate ROI area of each of the two images.
[0013] Furthermore, the present invention provides a universal target difference detection module, which is also characterized by:
[0014] The ROI feature recognition unit selects random ROI frames for the two images, calculates the features of the foreground and background through the convolution kernel, and finally selects the best position for output.
[0015] Furthermore, the present invention provides a universal target difference detection module, which is also characterized by:
[0016] The above RPN matching unit obtains the rotation and translation parameters in the two ROI images, generates pictures with different scales and rotations, and sends the pictures two by two to the feature recognition network to find the best matching position.
[0017] Furthermore, the present invention provides a universal target difference detection module, which is also characterized by:
[0018] The above-mentioned difference identification unit normalizes the two completely matched areas to the same size and sends them to the convolutional neural network together. After the multi-layer deep convolutional neural network extracts features, the difference is output as a similarity vector through the fully connected layer, and the similarity result is obtained after normalization.
[0019] Furthermore, the present invention provides a universal target difference detection module, which is also characterized by:
[0020] The training process of the general object difference detection module includes:
[0021] S1. Prepare a data set, which includes a plurality of groups of captured images and standard images;
[0022] S2. Randomly divide the data set into training set, test set and validation set for training;
[0023] S3. The loss function performance in the training set is used as the training direction, the loss function performance in the test set is used as the basis for parameter adjustment, and the final effect is characterized by the loss function performance in the validation set.
[0024] Furthermore, the present invention provides a universal target difference detection module, which is also characterized by:
[0025] In step S1, the captured image and the standard image are both labeled;
[0026] The above annotations include:
[0027] The annotation box is a rectangular box with rotation and orientation, indicating the best matching position in the two images;
[0028] Similarity information, based on the similarity scores corresponding to the manifestations of the defects.
[0029] Furthermore, the present invention provides a universal target difference detection module, which is also characterized by:
[0030] The above similarity scores are expressed as 0 and 1.
[0031] Furthermore, the present invention provides a universal target difference detection module, which is also characterized by:
[0032] The above loss function includes:
[0033] ROI foreground and background segment loss, that is, the IOU of the best ROI box and the labeled box after the two images have gone through the ROI feature recognition stage;
[0034] Feature similarity loss, that is, the feature difference between the two foreground ROIs after the two images have gone through the ROI feature recognition stage;
[0035] RPN matching loss, that is, the loss of the best box and the labeled box after the two ROI images are matched by RPN;
[0036] Similarity loss, that is, for the two ROI images after RPN matching, the similarity results are obtained after feature extraction, and the difference between the similarity results and the labeled similarity scores is obtained.
[0037] The present invention also provides a carrier, characterized in that: the carrier is equipped with the universal target difference detection module and has a processor for reading external data and calling the universal target difference detection module to process data. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 , module core architecture;
[0039] Figure 2 , the framework of ROI feature recognition unit;
[0040] Figure 3 , one For both standard and photo-shoot photos;
[0041] Figure 4 , the best position of the final output;
[0042] Figure 5 ,The architecture of RPN matching unit;
[0043] Figure 6 , one For standard photos and captured photos with different zooms and rotations;
[0044] Figure 7 , Find the best matching position after RPN processing;
[0045] Figure 8 , the structure of the difference identification unit;
[0046] Fig. 9 , examples of captured images and standard images in the dataset. DETAILED DESCRIPTION
[0047] The present invention is capable of various modifications and various embodiments, and therefore specific embodiments are illustrated and described in the accompanying drawings. However, this is not intended to limit the present invention to specific implementations, but should be understood to include all modifications, equivalents, and even substitutes that fall within the concept and technical scope of the present invention.
[0048] This embodiment proposes a universal target difference detection neural network model, the input of which is a captured image and a standard image. The model can automatically calculate the focus area in the image, and perform translation and scaling on the focus area of the two images to find the best position for macro matching, and then the model can identify fine image differences and output a similarity index. Since an image segmentation model based on a deep neural network is used, in the end-to-end annotation link, it is only necessary to annotate the defect positions of the captured image and the standard image, and the model can automatically learn to obtain the features of the same position in different images. Through the convolutional neural network, the correlation between the contrasting images, the current pixel of the image and the high-dimensional features in its neighborhood is established, and finally a correct judgment is achieved.
[0049] The details are as follows:
[0050] The core architecture of the module mentioned in this embodiment is as follows Figure 1 As shown, its input is the captured image and the standard image, and its output is the matching information and the difference identification result.
[0051] The network structure consists of three main modules: ROI feature recognition, RPN matching and difference identification.
[0052] in,
[0053] ROI feature recognition link, such as Figure 2 As shown, the captured image is different from the standard image (such as Figure 3) joint calculation, through the neural network convolution layer, extract the high-dimensional information of the two images, identify the difference between the foreground and background, and calculate the foreground area with certain similarity, and the output is the approximate ROI area of the two images. Because the neural network extracts its high-dimensional macro features, the model has the function of resisting noise and color difference. The algorithm randomly selects the ROI frame of the two images, calculates the features of the foreground and background through the convolution kernel, and finally selects the best position output (such as Figure 4 ).
[0054] RPN position matching link, such as Figure 5 The algorithm shown further obtains the rotation and translation parameters in the two ROI images by generating images with different scales and rotations (such as Figure 6 ), send the pictures two by two into the feature recognition network to find the best matching position (such as Figure 7 ).
[0055] Difference identification link, such as Figure 8 The algorithm shown normalizes the two completely matching areas to the same size and sends them to the convolutional neural network together. After the multi-layer deep convolutional neural network extracts the features, the difference is output as a similarity vector through the fully connected layer, and the similarity result is obtained after normalization.
[0056] The final output is: RPN matching information, including matching position, angle, and scaling. Similarity information, a decimal between 0 and 1, the closer to 1, the more similar.
[0057] Regarding the training process of this model: First, prepare the data set, which includes the captured images and standard images (such as Fig. 9 ), both images are annotated, and the annotated box is a rectangular box with rotation and direction, indicating the best matching position in the two images. The annotation information also contains similarity information, and the similarity score is given manually according to the severity of the defect, the damage, or the area. The score is a decimal between 0 and 1, and the closer to 1, the more similar. Since the model has a certain regression computing ability, when giving the similarity annotation, only two values 0 and 1 can be given to indicate whether the defect is acceptable or whether there is a defect. The network will eventually give a non-integer value through large-scale data training.
[0058] Several sets of data are retained in the training set, 70% are randomly selected as the training set, 20% as the test set, and 10% as the validation set.
[0059] In the process of a single training, the loss function performance in the training set is used as the training direction, the loss function performance in the test set is used as the basis for parameter adjustment, and the final effect is characterized by the loss function performance in the verification set.
[0060] The loss function mainly consists of the following parts:
[0061] ROI foreground and background segment loss: the IOU of the best ROI box and the labeled box after the two images have gone through the ROI feature recognition stage;
[0062] Feature similarity loss: the feature difference between the two foreground ROIs after the two images have gone through the ROI feature recognition stage;
[0063] RPN matching loss: the loss between the best box and the labeled box after the two ROI images are matched by RPN;
[0064] Similarity loss: The difference between the similarity value and the labeled value after the similarity network is calculated.
[0065] Although the above description is centered on the embodiment, it is only an example and does not limit the present invention. It is clear to those skilled in the art that various modifications and applications not illustrated above can be made within the scope of the essential characteristics of the present embodiment. For example, each component specifically shown in the embodiment can be implemented after being modified. Moreover, various differences related to such modifications and applications should be interpreted as being included in the scope of the present invention as defined in the attached claims.
Claims
1. A universal target difference detection module, Features: It includes ROI feature recognition unit, RPN matching unit and difference identification unit; The ROI feature recognition unit recognizes the ROI areas of the captured image and the standard image, and selects the best position for output; The RPN matching unit searches for the best matching position through multi-channel deformation based on the ROI area of the captured image and the standard image; The difference identification unit extracts features based on the best matching position between the captured image and the ROI area of the standard image to obtain a similarity result.
2. A universal target difference detection module as claimed in claim 1, Features: The ROI feature recognition unit jointly calculates the captured image and the standard image, extracts high-dimensional information of the two images through the neural network convolution layer, identifies the difference between the foreground and the background, and calculates the foreground area with a certain similarity, and its output is the approximate ROI area of each of the two images.
3. A universal target difference detection module as claimed in claim 1, Features: The ROI feature recognition unit randomly selects ROI frames for the two images, calculates the features of the foreground and background through the convolution kernel, and finally selects the best position for output.
4. A universal target difference detection module as claimed in claim 1, Features: The RPN matching unit obtains rotation and translation parameters in two ROI images, generates pictures with different scales and rotations, and sends the pictures two by two to the feature recognition network to find the best matching position.
5. A universal target difference detection module as claimed in claim 1, Features: The difference identification unit normalizes the two completely matched areas to the same size and sends them to the convolutional neural network together. After the multi-layer deep convolutional neural network extracts features, the difference is output as a similarity vector through the fully connected layer, and the similarity result is obtained after normalization.
6. A universal target difference detection module as claimed in claim 1, Features: The training process of the general object difference detection module includes: S1. Prepare a data set, which includes a plurality of groups of captured images and standard images; S2. Randomly divide the data set into training set, test set and validation set for training; S3. The loss function performance in the training set is used as the training direction, the loss function performance in the test set is used as the basis for parameter adjustment, and the final effect is characterized by the loss function performance in the validation set.
7. A universal target difference detection module as claimed in claim 6, Features: In step S1, the photographed image and the standard image are both labeled; The annotations include: The annotation box is a rectangular box with rotation and orientation, indicating the best matching position in the two images; Similarity information, based on the similarity scores corresponding to the manifestations of the defects.
8. A universal target difference detection module as claimed in claim 7, Features: The similarity score is expressed as 0 and 1.
9. A universal target difference detection module as claimed in claim 6, Features: The loss function includes: ROI foreground and background segment loss, that is, the IOU of the best ROI box and the labeled box after the two images have gone through the ROI feature recognition stage; Feature similarity loss, that is, the feature difference between the two foreground ROIs after the two images have gone through the ROI feature recognition stage; RPN matching loss, that is, the loss of the best box and the labeled box after the two ROI images are matched by RPN; similarity loss, that is, the similarity result is obtained after extracting features from the two ROI images after RPN matching, and the difference between the similarity result and the labeled similarity score is obtained.
10. A carrier, Features: A processor is installed with the universal target difference detection module as claimed in any one of claims 1 to 9 and is provided to read external data and call the universal target difference detection module to process the data.
