Small sample detection method and device based on multi-image feature fusion

Abstract

The invention belongs to the technical field of image feature fusion, and particularly relates to a small sample detection method and device based on multi-image feature fusion. S10, the visible light mechanism obtains a visible light image by adjusting the height; step S20, removing fuzzy redundant features of the visible light image set through an attention system according to focal length definition, and then performing fusion to obtain a visible light image; s30, a thermal imaging mechanism obtains a thermal imaging image; s40, the ultrasonic mechanism obtains an ultrasonic image; s50, performing feature enhancement processing on the query image set, and outputting a query feature image set; step S60, performing fusion processing on the query feature image set and a support image set in a database to obtain a weighted query feature image set; and step S70, performing prediction analysis based on the weighted query feature image set, and generating a prediction result including a prediction position and a prediction type. According to the scheme, the problem that the high position or the low position of an image is blurred easily when elements have different heights is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image feature fusion, and specifically relates to a small sample detection method and device based on multi-image feature fusion. Background Art

[0002] The detection algorithm based on convolutional neural network (CNN) obtains the semantic type and location information of specific targets in the image by training a large amount of data, and is widely used in PCB surface defect detection. However, due to the limitations of the industry, there are many difficulties in building a target sample library with a large amount of data in actual scenarios. In some special scenarios, only a very small number of sample images can be obtained. In response to the above problems, small sample learning adopts the idea of ​​generalization of prior knowledge to obtain the desired results with very few training samples. Small sample object detection (FSOD) is one of the branches of small sample learning. Unlike classification, it is a more challenging task because it not only needs to distinguish the category of the target, but also needs to locate the specific location of the target.

[0003] At present, the existing announcement number is CN118587176A, which is a small sample PCB defect detection method based on meta-feature enhancement, including: inputting the PCB image to be detected into the trained defect detection model, outputting the detection frame of all detected defect targets and their predicted types; the defect detection model introduces a global feature fusion module in the support branch to improve the problem that new class and base class defects are easily confused, and introduces a self-attention module in the query branch to help solve the problem of missed detection of PCB defect targets, and finally performs dot product operation on the query enhancement feature and the support enhancement feature to generate an aggregate feature, and classifies the aggregate feature and regresses the detection frame through the detector head. The training of the defect detection model includes a meta-training stage and a meta-testing stage; the meta-training stage optimizes all parameters of the defect detection model; the meta-testing stage only optimizes the parameters of the defect detection model detector head. The method of the invention improves the detection recall rate and classification accuracy of PCB defect detection.

[0004] However, there is also a problem: the existing visible light method used by this method to obtain the visible light image in the query image is to take a photo at a fixed height and focus. This method is prone to blurring at high or low places when a component has different heights, making subsequent feature extraction difficult. Summary of the invention

[0005] This solution provides a small sample detection method based on multi-image feature fusion to solve the problem that when a component has different heights, it is easy to cause blurring of the high or low parts of the image.

[0006] This solution provides a small sample detection method based on multi-image feature fusion, including the following steps: Step S10: The visible light mechanism continuously adjusts the height through the servo motor to obtain a visible light image; Step S20: The visible light image set is subjected to an attention system to remove fuzzy redundant features according to focal length clarity, and then fused to obtain a visible light image; Step S30: The thermal imaging mechanism cooperates with the power-on test to obtain a thermal imaging image; Step S40: the ultrasonic device obtains an ultrasonic image, wherein the visible light image, the thermal imaging image and the ultrasonic image are a query image set; Step S50: performing feature enhancement processing on the query image set and outputting the query feature image set; Step S60: fusing the query feature image set with the support image set in the database to obtain a weighted query feature image set; Step S70: performing prediction analysis based on the weighted query feature image set to generate a prediction result including a prediction position and a prediction type.

[0007] The beneficial effect of this solution is that it obtains clear pictures at different heights by adjusting the height, and then removes the fuzzy redundant features according to the focal length clarity through the attention system, and then fuses them to obtain a clearer visible light image. This avoids the problem that different heights easily lead to blurry images at high or low places.

[0008] Furthermore, in step S10, when the height is adjusted by the servo motor, the height information is recorded accordingly. In step S20, the visible light image set removes fuzzy redundant features according to the focus clarity through the attention system, and then fuses to obtain the visible light image and its height information. When adjusting the height, the height information is recorded, and the height of each part can be obtained, and the height of the electronic components can be detected, which improves the troubleshooting ability.

[0009] Furthermore, the method also includes step S55: using the BI-FU multi-scale feature fusion module to first extract feature maps of multiple scales of the input support set image, and using the module to fuse the multiple feature maps of different scales to obtain a multi-scale fused feature map; then performing convolution processing on the multi-scale fused feature map to output a feature re-weighted vector; in the step S60, the query feature image set and the output feature re-weighted vector are fused to obtain a weighted query feature image set.

[0010] Furthermore, the BI-FU multi-scale feature fusion module is used to first extract feature maps of multiple scales of the input support set image, and the module is used to fuse the feature maps of multiple scales to obtain a multi-scale fused feature map, including: using VGG16 and BiFPN Layer as the multi-scale feature extraction module BI, generating multiple feature maps of multiple scales from the input support set image through VGG16, and then extracting the feature maps of multiple scales through BiFPN Layer, and using them as input of the feature fusion module FU; fusing the multiple feature maps of different scales output by the multi-scale feature extraction module BI through the feature fusion module FU to obtain the multi-scale fused feature map.

[0011] Furthermore, the feature fusion module FU fuses the feature maps of multiple different scales output by the multi-scale feature extraction module BI to obtain the multi-scale fused feature map, including: performing a convolution operation on the feature maps of the multiple different scales, compressing them to obtain a 1×1 tensor of the corresponding scale to form a feature fusion vector, and then using the feature fusion vector to re-weight the feature maps of the multiple different scales, and the re-weighted feature maps of different scales are fused into a multi-scale fused feature map through an element-by-element addition operation.

[0012] The present solution also provides a small sample detection device based on multi-image feature fusion, comprising: A camera unit, a computing unit, an attention unit and a deep learning system, wherein the camera unit is used to obtain a query image set; the computing unit is used to calculate the number of features and size data; and the attention unit is used to remove redundant parts; The camera unit comprises: Visible light mechanism: the visible light mechanism is used to obtain visible light images; Thermal imaging mechanism: the thermal imaging mechanism is used to obtain thermal imaging images; Ultrasonic mechanism: the ultrasonic mechanism is used to obtain ultrasonic images; Moving mechanism: The moving mechanism is used to change the distance between the visible light mechanism and the sample to be measured.

[0013] The principle of this scheme is as follows: the visible light mechanism obtains visible light images by continuously adjusting the height through the mobile mechanism; the visible light image set removes fuzzy redundant features according to the focus clarity through the attention system, and then fuses to obtain visible light images. The thermal imaging mechanism obtains thermal imaging images in conjunction with the power-on test; the ultrasonic mechanism obtains ultrasonic images. The visible light image, thermal imaging image and ultrasonic image are the query image set.

[0014] Then, the query image set is subjected to feature enhancement processing to output a query feature image set; the query feature image set and the support image set in the database are fused to obtain a weighted query feature image set; and prediction analysis is performed based on the weighted query feature image set to generate a prediction result including a prediction position and a prediction type.

[0015] The beneficial effect of this solution is that: this solution adjusts the height of the visible light mechanism through the moving mechanism, and then obtains visible light images of different heights with the same focal length, and then the clearest parts of each different height can be obtained according to the focus clarity. This mechanism improves the clarity of the visible light image and reduces the error.

[0016] Furthermore, it also includes: a height measurement mechanism and a marker, wherein the height measurement mechanism is used to measure and record the distance between the visible light mechanism and the sample being measured, and the marker is used to mark the height data of each position of the visible light image. The height measurement mechanism and the marker can mark the height of each part of the electronic component, so that the device can detect its height. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of a small sample detection method based on multi-image feature fusion. DETAILED DESCRIPTION

[0018] As attached Figure 1 As shown: A small sample detection method based on multi-image feature fusion, comprising the following steps: Step S10: the visible light mechanism continuously adjusts the height through the moving mechanism to obtain a visible light image, and then obtains the height information through the height measurement mechanism and the annotator; Step S20: The visible light image set is subjected to an attention system to remove fuzzy redundant features according to focal length clarity, and then fused to obtain a visible light image and its height information; Step S30: The thermal imaging mechanism cooperates with the power-on test to obtain a thermal imaging image; Step S40: the ultrasonic device obtains an ultrasonic image, wherein the visible light image, the thermal imaging image and the ultrasonic image are a query image set; Step S50: performing feature enhancement processing on the query image set and outputting the query feature image set; Step S55: using the BI-FU multi-scale feature fusion module to first extract feature maps of multiple different scales of the input support set image, and using the module to fuse the feature maps of multiple different scales to obtain a multi-scale fused feature map; then convolution processing is performed on the multi-scale fused feature map to output a feature reweighted vector; Step S60: fusing the query feature image set and the output feature reweighted vector to obtain a weighted query feature image set; Step S70: performing prediction analysis based on the weighted query feature image set to generate a prediction result including a prediction position and a prediction type.

[0019] In the support set image branch, the BI-FU multi-scale feature fusion module is used to first extract feature maps of multiple different scales of the input support set image, and the module is used to fuse the feature maps of multiple different scales to obtain a multi-scale fused feature map; then the feature re-weighting module is used to perform convolution processing on the multi-scale fused feature map, and a feature re-weighting vector is output; Then, the query set feature map and feature reweighted vector output by the two branches are fused to obtain the reweighted query set feature map.

[0020] Finally, the detection module performs prediction analysis based on the re-weighted query set feature map to generate the predicted position and predicted category results. The prediction results mainly include the predicted box (center point, size (height and width)), predicted category, category confidence, and target prediction score.

[0021] The feature fusion module FU is used to fuse the feature maps of multiple scales output by the multi-scale feature extraction module BI to obtain a multi-scale fused feature map, including: performing convolution operations on the feature maps of multiple scales, compressing them to obtain a 1×1 tensor of the corresponding scale to form a feature fusion vector, and then using the feature fusion vector to re-weight the feature maps of multiple scales. The re-weighted feature maps of different scales are fused into a multi-scale fused feature map through an element-by-element addition operation.

[0022] The present invention also provides a small sample detection device based on multi-image feature fusion, comprising: A camera unit, a computing unit, an attention unit and a deep learning system, wherein the camera unit is used to obtain a query image set; the computing unit is used to calculate the number of features and size data; and the attention unit is used to remove redundant parts; The camera unit comprises: Visible light mechanism: the visible light mechanism is used to obtain visible light images; Thermal imaging mechanism: the thermal imaging mechanism is used to obtain thermal imaging images; Ultrasonic mechanism: the ultrasonic mechanism is used to obtain ultrasonic images; Moving mechanism: The moving mechanism is used to change the distance between the visible light mechanism and the sample to be measured.

[0023] The principle of this scheme is as follows: the visible light mechanism obtains visible light images by continuously adjusting the height through the moving mechanism, and the height measurement mechanism and the annotator obtain the height information; the visible light image set removes the fuzzy redundant features according to the focal length clarity through the attention system, and then fuses them to obtain the visible light image and height information. The thermal imaging mechanism obtains thermal imaging images in conjunction with the power-on test; the ultrasonic mechanism obtains ultrasonic images, and the visible light image, thermal imaging image and ultrasonic image are the query image set.

[0024] Then, the query image set is subjected to feature enhancement processing to output a query feature image set; the query feature image set and the support image set in the database are fused to obtain a weighted query feature image set; and prediction analysis is performed based on the weighted query feature image set to generate a prediction result including a prediction position and a prediction type.

[0025] The beneficial effect of this solution is that: this solution adjusts the height of the visible light mechanism through the moving mechanism, and then obtains visible light images of different heights with the same focal length, and then the clearest parts of each different height can be obtained according to the focus clarity. This mechanism improves the clarity of the visible light image and reduces the error.

[0026] The above are only embodiments of the present invention, and the common knowledge such as the known specific structure and characteristics in the scheme is not described in detail here. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the structure of the present invention, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A small sample detection method based on multi-image feature fusion, characterized in that: The following steps are involved: Step S10: The visible light mechanism continuously adjusts the height through the servo motor to obtain a visible light image; Step S20: The visible light image set is subjected to an attention system to remove fuzzy redundant features according to focal length clarity, and then fused to obtain a visible light image; Step S30: The thermal imaging mechanism cooperates with the power-on test to obtain a thermal imaging image; Step S40: the ultrasonic device obtains an ultrasonic image, wherein the visible light image, the thermal imaging image and the ultrasonic image are a query image set; Step S50: performing feature enhancement processing on the query image set and outputting the query feature image set; Step S60: fusing the query feature image set with the support image set in the database to obtain a weighted query feature image set; Step S70: performing prediction analysis based on the weighted query feature image set to generate a prediction result including a prediction position and a prediction type.

2. The small sample detection method based on multi-image feature fusion according to claim 1 is characterized in that: In the step S10, the height information is recorded correspondingly when the height is adjusted by the servo motor. In the step S20, the visible light image set is removed from fuzzy redundant features according to focal length clarity by the attention system, and then fused to obtain the visible light image and its height information.

3. The small sample detection method based on multi-image feature fusion according to claim 1 is characterized in that: The method further includes step S55: using a BI-FU multi-scale feature fusion module to first extract feature maps of multiple scales of the input support set image, and using the module to fuse the feature maps of multiple scales to obtain a multi-scale fused feature map; then performing convolution processing on the multi-scale fused feature map to output a feature reweighted vector; In the step S60, the query feature image set and the output feature reweighted vector are fused to obtain a weighted query feature image set.

4. The small sample detection method based on multi-image feature fusion according to claim 3, characterized in that: The BI-FU multi-scale feature fusion module is used to first extract feature maps of multiple scales of the input support set image, and the module is used to fuse the feature maps of multiple scales to obtain a multi-scale fused feature map, including: using VGG16 and BiFPN Layer as the multi-scale feature extraction module BI, generating multiple feature maps of multiple scales from the input support set image through VGG16, and then extracting the feature maps of multiple scales through BiFPN Layer, and using them as input of the feature fusion module FU; fusing the feature maps of multiple scales output by the multi-scale feature extraction module BI through the feature fusion module FU to obtain the multi-scale fused feature map.

5. The small sample detection method based on multi-image feature fusion according to claim 4 is characterized in that: The feature fusion module FU is used to fuse the feature maps of multiple scales output by the multi-scale feature extraction module BI to obtain the multi-scale fused feature map, including: performing a convolution operation on the feature maps of multiple scales, compressing them to obtain a 1×1 tensor of the corresponding scale to form a feature fusion vector, and then using the feature fusion vector to re-weight the feature maps of the multiple scales. The re-weighted feature maps of different scales are fused into a multi-scale fused feature map through an element-by-element addition operation.

6. A small sample detection device based on multi-image feature fusion, characterized in that: include: A camera unit, a computing unit, an attention unit, and a deep learning system, wherein the camera unit is used to obtain a query image set; The calculation unit is used to calculate the number of features and size data; The attention unit is used to remove redundant parts; The camera unit comprises: Visible light mechanism: the visible light mechanism is used to obtain visible light images; Thermal imaging mechanism: the thermal imaging mechanism is used to obtain thermal imaging images; Ultrasonic mechanism: the ultrasonic mechanism is used to obtain ultrasonic images; Moving mechanism: The moving mechanism is used to change the distance between the visible light mechanism and the sample to be measured.

7. The small sample detection device based on multi-image feature fusion according to claim 6, characterized in that: Also includes: A height measuring mechanism and a marker, wherein the height measuring mechanism is used to measure and record the distance between the visible light mechanism and the measured sample, and the marker is used to mark the height data of each position of the visible light image.

Images