Target detection method and device based on similarity comparison
Through the object detection method based on similarity comparison, the MSNR feature extraction network generates candidate boxes and databases, which solves the information loss and long-tail problems in object detection, and improves the detection efficiency and accuracy.
Patent Information
- Application Number
- CN202211174426.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-26
- Publication Date
- 2025-08-08
- Estimated Expiration
- 2042-09-26
AI Technical Summary
The existing object detection methods have information loss or redundant boxes when processing objects of different sizes, which have high computational complexity, resulting in low detection efficiency and long tail problems, and sample imbalance leads to poor detection results.
A target detection method based on similarity comparison is adopted, and a candidate box is generated through the MSNR feature extraction network, a database of to be selected and standard feature is constructed, similarity comparison is performed, candidate box and location information with the highest confidence is obtained, error category judgment of candidate box, and missed detection rate is reduced.
It improves the detection accuracy of small and medium-sized targets, reduces the missed detection rate, solves the problem of sample imbalance, and improves the overall effect of target detection.
Smart Images
Figure CN115631352B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and in particular relates to a target detection method and device based on similarity comparison. Background Art
[0002] Object detection is a fundamental task in machine vision. Its primary purpose is to determine whether a target object exists in an image and, if so, to return the class and location of the target. The algorithm takes an image of the target to be detected as input, selects candidate boxes within the image, selects possible regions of interest, extracts features from these regions, and finally trains a classifier to calculate the features of the regions of interest to determine whether they contain the target and their category. Finally, after merging the regions, the target detection results are output.
[0003] In the prior art, a sliding window method is used to extract candidate frames for target detection. The principle of the sliding window is to use a brute force method to frame all possible candidate areas and send them to the network for recognition. When the sliding step is large, information loss will occur when the sliding step is large, and a large number of redundant frames will be generated when the step is small. In addition, since the target sizes and lengths of objects in the image vary, using a window of a uniform size for selection may result in problems such as the frame not being able to be framed or the framed areas containing many non-target areas, resulting in later classification errors. Designing windows of multiple sizes will cause a lot of calculations, resulting in a slow running speed and high complexity of the method.
[0004] Therefore, there is an urgent need to improve the defects of existing technologies and improve the efficiency of target detection. Summary of the Invention
[0005] In order to solve the above problems existing in the prior art, the present invention provides a target detection method and device based on similarity comparison.
[0006] The technical problem to be solved by the present invention is achieved through the following technical solutions:
[0007] In a first aspect, the present application provides a target detection method based on similarity comparison, comprising:
[0008] Generate candidate frames based on target image samples; the target image samples include the target to be detected;
[0009] Based on the candidate boxes, target features are extracted and a database of candidate features is constructed;
[0010] Based on existing image samples, target features are extracted and a standard feature database is constructed;
[0011] Get the candidate box with the highest confidence from the candidate feature database, compare the candidate box with the standard feature database for similarity, get the standard feature database image with the highest score, and record its category information;
[0012] Compare the similarity between the standard feature database image with the highest score and the candidate feature database image, obtain the candidate feature database image with the highest score, and record its location information;
[0013] The category information of the standard feature database image with the highest score and the position information of the candidate feature database image with the highest score are mapped to the target image sample to obtain the detection result of the target to be detected.
[0014] Optionally, the method further includes: expanding the standard feature database;
[0015] The expansion of the standard feature database includes an image retrieval module and an image storage module.
[0016] Optionally, the image retrieval module includes:
[0017] Compare the target features of the candidate frame with the features in the standard feature database one by one to obtain the score of the corresponding category of the candidate frame;
[0018] The image storage module includes:
[0019] The scores of the categories corresponding to the candidate boxes are screened, and the candidate boxes that meet the first condition are put into the standard feature database to expand the standard feature database.
[0020] Optionally, the first condition is that the score of the category corresponding to the candidate box is between 0.6 and 0.7.
[0021] Optionally, an MSNR feature extraction network is used to extract target features.
[0022] Optionally, the MSNR feature extraction network includes a first module, a second module and a third module; wherein the second module and the third module have the same structure.
[0023] Optionally, the first module includes a convolutional layer, a maximum pooling layer and a residual shrinkage module cascaded in sequence.
[0024] Optionally, the second module includes a convolutional layer, a first pooling layer, a second pooling layer, a residual shrinkage module and a feature enhancement module that are cascaded in sequence;
[0025] Among them, the feature enhancement module includes a multi-scale convolution kernel module and an attention mechanism denoising module.
[0026] In a second aspect, the present application further provides an object detection device based on similarity comparison, comprising:
[0027] A generation module is used to generate candidate boxes based on target image samples; the target image samples include the target to be detected;
[0028] The candidate feature database extraction module is used to extract target features based on candidate boxes and build a candidate feature database;
[0029] The standard feature database extraction module is used to extract target features based on existing image samples and build a standard feature database;
[0030] The first similarity comparison module is used to obtain the candidate box with the highest confidence from the candidate feature database, compare the candidate box with the standard feature database for similarity, obtain the standard feature database image with the highest score, and record its category information;
[0031] The second similarity comparison module is used to compare the image in the standard feature database with the image in the candidate feature database with the highest score, obtain the image in the candidate feature database with the highest score, and record its position information;
[0032] The result acquisition module is used to map the category information of the standard feature database image with the highest score and the position information of the candidate feature database image with the highest score to the target image sample to obtain the detection result of the target to be detected.
[0033] Beneficial effects of the present invention:
[0034] The present invention provides a target detection method and device based on similarity comparison. Based on target picture samples, multiple candidate frames are generated, and target features are extracted from the obtained candidate frames through the MSNR feature extraction network. The target features are also extracted from the existing picture samples through the MSNR feature extraction network. The image features, corresponding position information and confidence information extracted from the candidate frames are saved as a candidate feature database, and the image features and corresponding category information extracted from the existing picture samples are saved as a standard feature database. The candidate frame with the highest confidence is obtained from the candidate feature database, and the candidate frame is compared with the standard feature database for similarity to obtain the standard feature database picture with the highest score, and its category information is recorded. The standard feature database picture with the highest score is compared with the candidate feature database. The feature database is compared for similarity to obtain the candidate feature database image with the highest score, and its position information is recorded. In this embodiment, similarity comparison is used for target detection, which can solve the long-tail problem in target detection and the problem of imbalanced target detection samples in the prior art, that is, the detection effect is better for categories with more training samples, and the accuracy is higher for categories with fewer training samples; the position information and category information obtained above are mapped to the target image sample to obtain the detection result; in this way, the judgment of the wrong category of the candidate frame can be corrected, the recall rate of small sample targets can be increased, and a large number of repeated low-quality candidate target frames generated by the lowering of the confidence threshold and the increasing of the IOU threshold can be reduced, which can effectively reduce the missed detection rate of small and medium-sized targets, improve the accuracy of target positioning, and comprehensively improve the effect of target detection.
[0035] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 This is a flow chart of a target detection method based on similarity comparison provided by an embodiment of the present invention;
[0037] Figure 2 is another flow chart of a target detection method based on similarity comparison provided by an embodiment of the present invention;
[0038] Figure 3 This is a flow chart for expanding a standard feature database provided by an embodiment of the present invention;
[0039] Figure 4 Schematic diagram of the structure of the MSNR feature extraction network provided by an embodiment of the present invention;
[0040] Figure 5 This is a structural diagram of a FEM module provided by an embodiment of the present invention;
[0041] Figure 6Schematic diagram of a multi-scale convolution kernel module provided by an embodiment of the present invention;
[0042] Figure 7 This is a structural diagram of the attention mechanism noise reduction module provided by an embodiment of the present invention;
[0043] Figure 8 This is a structural diagram of a target detection device based on similarity comparison provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0044] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.
[0045] Target retrieval technologies mainly include text-based and content-based. Among them, text-based target retrieval mainly performs image retrieval through text descriptions of images, such as image name, author, year and style, which requires additional information annotation of the data, thus consuming a large amount of human and material resources; content-based image retrieval does not require additional annotation, and compares according to the characteristics of the image itself. Just by inputting the target image to be detected, similar target images of the same type can be found.
[0046] Content-Based Image Retrieval (CBIR), which began in the 1890s, is a key research area in the field of image retrieval. It primarily involves inputting relevant images for search. The system extracts features from the target image, calculates their similarity with features in a standard database, and finally outputs the image in the database that most closely resembles the target image. As demand continues to grow, search results must include not only the target image but also its coordinates within the image for precise location.
[0047] At present, the problems of content-based target detection include: low signal-to-noise ratio in images. When the target accounts for a small proportion in the image and the background is relatively cluttered, the direct use of deep learning networks to extract convolutional features for detection is not ideal and there is a high missed detection rate. It is not feasible to directly retrieve the entire image with a low signal-to-noise ratio; external environmental interference. When there are many small targets and the targets are occluded, blurred, or incomplete, the existence of these problems greatly reduces the amount of effective information extracted from the image by the feature extraction network, seriously affecting the effect of the target detection algorithm; category imbalance. When using traditional target detection methods for detection, the detection effect is better for categories with more samples during training, but the detection effect is poor for categories with fewer samples during training, that is, the long-tail problem.
[0048] In view of this, the present application provides a target detection method based on similarity comparison, designs a dynamic candidate box extraction method, and constructs an MSNR feature extraction network for similarity comparison, which can effectively reduce the missed detection rate of small and medium-sized targets, improve the accuracy of target positioning, and comprehensively improve the effect of target detection.
[0049] See Figure 1 and Figure 2 As shown, Figure 1 is a flow chart of a target detection method based on similarity comparison provided by an embodiment of the present invention. Figure 2 This is another flow chart of a target detection method based on similarity comparison provided by an embodiment of the present invention. The target detection method based on similarity comparison provided by this application includes:
[0050] S101, generating a candidate frame based on a target image sample; the target image sample includes a target to be detected;
[0051] S102: Extract target features based on the candidate boxes and build a database of candidate features.
[0052] S103: Extract target features based on existing image samples and build a standard feature database;
[0053] S104: Obtain the candidate box with the highest confidence from the candidate feature database, perform similarity comparison between the candidate box and the standard feature database, obtain the standard feature database image with the highest score, and record its category information;
[0054] S105, performing a similarity comparison between the standard feature database image with the highest score and the candidate feature database image, obtaining the candidate feature database image with the highest score, and recording its position information;
[0055] S106 : Map the category information of the standard feature database image with the highest score and the position information of the candidate feature database image with the highest score to the target image sample to obtain a detection result of the target to be detected.
[0056] For more details, please see Figure 1 and Figure 2As shown, in this embodiment, based on the target image sample, YOLOv3 or other traditional target detection methods are used, the confidence threshold is further reduced to 0.001, and the IOU threshold is increased to 0.99. On the basis of avoiding missed detection, multiple candidate frames are generated, and the target features are extracted from the obtained candidate frames through the MSNR feature extraction network, and the target features are extracted from the existing image samples through the MSNR feature extraction network; the image features, corresponding position information and confidence information extracted from the candidate frames are saved as the feature database to be selected, and the image features and corresponding category information extracted from the existing image samples are saved as the standard feature database; the candidate frame with the highest confidence is obtained from the feature database to be selected, and the candidate frame is compared with the standard feature database for similarity to obtain the standard feature database picture with the highest score, and its category information is recorded Information; perform a similarity comparison between the standard feature database image with the highest score and the candidate feature database to obtain the candidate feature database image with the highest score, and record its position information. In this embodiment, similarity comparison is used for target detection, which can solve the long-tail problem in target detection and the problem of imbalanced target detection samples in the prior art, that is, the detection effect is better for categories with more training samples, and the accuracy is higher for categories with fewer training samples; map the above-mentioned position information and category information respectively obtained to the target image sample to obtain the detection result; in this way, the judgment of the wrong category of the candidate frame can be corrected, the recall rate of small sample targets can be increased, and a large number of repeated low-quality candidate target frames generated by the lowering of the confidence threshold and the increase of the IOU threshold can be reduced, which can effectively reduce the missed detection rate of small and medium-sized targets, improve the accuracy of target positioning, and comprehensively improve the effect of target detection.
[0057] It should be noted that this embodiment improves the detection rate of the target and reduces the missed detection rate of target detection by dynamically extracting candidate boxes; optionally, the missed detection rate is reduced by 5%, and the mAP is increased by 4 percentage points compared with the traditional method, from 0.84 to 0.88, among which the small sample category is increased by 15 percentage points, from 0.62 to 0.77, effectively solving the category imbalance problem in the target process.
[0058] Optionally, when performing similarity comparison, a network such as ResNet, VGG or DenseNet can be used.
[0059] Optionally, when obtaining candidate frames, RPN or sliding window may be used.
[0060] Optionally, when screening candidate boxes, the candidate box scores may be calculated without using similarity comparison. Alternatively, a classification model may be used to classify the candidate boxes and obtain confidence scores.
[0061] See Figure 3 As shown, Figure 3 This is a flow chart of expanding a standard feature database provided by an embodiment of the present invention. In an optional embodiment of the present application, it further includes: expanding the standard feature database;
[0062] The expansion of the standard feature database includes an image retrieval module and an image storage module.
[0063] For more details, please see Figure 3 As shown, in this embodiment, the standard feature database is expanded based on the closed-loop target expansion. In this way, as the standard database is continuously expanded, the detection accuracy is improved, the data of a small number of training samples can be supplemented in time, and the small sample features can be used more efficiently.
[0064] Please continue to see Figure 3 As shown, in an optional embodiment of the present application, the image retrieval module includes:
[0065] Compare the target features of the candidate frame with the features in the standard feature database one by one to obtain the score of the corresponding category of the candidate frame;
[0066] The image storage module includes:
[0067] The scores of the categories corresponding to the candidate boxes are screened, and the candidate boxes that meet the first condition are put into the standard feature database to expand the standard feature database.
[0068] Specifically, in this embodiment, target detection results that are informative and non-redundant can be effectively added to the standard feature database, which can continuously improve the accuracy of target detection when the algorithm is running.
[0069] In an optional embodiment of the present application, the first condition is that the score of the category corresponding to the candidate box is between 0.6 and 0.7.
[0070] Specifically, in this embodiment, in the process of screening the scores of the corresponding categories of the candidate boxes, if the score is higher than 0.7 or lower than 0.6, the image features of the corresponding candidate boxes are discarded; if the score is between 0.6 and 0.7, the image features of the corresponding candidate boxes are placed in the standard feature database to expand the database. As the standard database continues to expand, the detection accuracy is improved.
[0071] See Figure 4 As shown, Figure 4 Schematic diagram of the structure of the MSNR feature extraction network provided in an embodiment of the present invention. In an optional embodiment of the present application, the MSNR feature extraction network is used to extract target features.
[0072] Specifically, the MSNR feature extraction network provided in this embodiment improves the feature extraction efficiency and reduces the time required for feature extraction. In addition, the model is lightweight and can be deployed on edge computing nodes. The addition of a denoising module in the network can effectively alleviate problems such as noise and occlusion, and reduce the missed detection rate of related targets. The effect is better than mainstream feature extraction networks such as VGG and ResNet; better results are achieved while using fewer parameters. In the drone vehicle dataset, an accuracy rate of 98.9% is achieved with 21.03M parameters.
[0073] Please continue to see Figure 4 As shown, in an optional embodiment of the present application, the MSNR feature extraction network includes a first module, a second module and a third module; wherein the second module and the third module have the same structure.
[0074] Specifically, the MSNR feature extraction network provided in this embodiment is provided with a first module, a second module and a third module in sequence.
[0075] Optionally, the first module includes a convolutional layer, a maximum pooling layer and a residual shrinkage module cascaded in sequence.
[0076] Among them, the size of the input target image sample is 224*224, the convolution layer is set with 64 convolution kernels, and the convolution kernel size is 3*3; the convolution kernel size of the maximum pooling layer is 2*2; the residual shrinkage module is improved based on the residual block, and the input of the residual shrinkage module is C*W*1, where C is the number of channels and W is the width. After passing through the two residual convolution layers, the absolute value of the result is taken, and a C*1*1 vector is obtained after passing through the global pooling layer. After passing through two fully connected layers, a 1*1*1 vector is obtained, and α is obtained through Sigmoid. The result of the global pooling layer is multiplied by α to obtain a soft threshold, and the threshold is used to threshold the original input target image sample.
[0077] Alternatively, see Figures 5 to 7 As shown, Figure 5 is a structural diagram of a FEM module provided by an embodiment of the present invention, Figure 6 : is a structural diagram of a multi-scale convolution kernel module provided by an embodiment of the present invention, Figure 7 : This is a structural diagram of the attention mechanism denoising module provided in an embodiment of the present invention, wherein the second module includes a convolutional layer, a first pooling layer, a second pooling layer, a residual shrinkage module, and a feature enhancement module that are cascaded in sequence;
[0078] Among them, the feature enhancement module includes a multi-scale convolution kernel module and an attention mechanism denoising module.
[0079] The second module is the base module (BS), which includes a convolutional layer, a first pooling layer, a second pooling layer, a residual shrinkage module and a feature enhancement module (FEM) cascaded in sequence; the convolution kernel size in the convolution layer is 3*3; the convolution kernel size of the maximum pooling layer is 2*2, and the number of channels is doubled; the FEM module passes the original features through a multiscale convolution kernel module (MSKM) and an attention mechanism noise reduction module (NRAM), and then simply superimposes the two parts of the features to obtain the final feature output, and the number of feature channels remains unchanged; it can be understood that the MSKM module passes the input through a 1*3 convolution kernel, a 3*1 convolution kernel, a 1*5 convolution kernel, a 5*1 convolution kernel, a 3*3 convolution kernel with a stride of 2, and a 3*3 convolution kernel with a stride of 3, and splices the output of each convolution kernel through a channel. The NRAM module uses a 3*3 convolution layer as input to obtain the final output result. The input is a C*W*H feature vector, which is obtained through a global pooling layer to obtain a C*1*1 feature vector. After passing through a fully connected layer, a C / 16*1*1 feature vector is obtained. After passing through a ReLU activation layer, a fully connected layer is obtained to obtain a C*1*1 feature vector. After passing through a Sigmoid layer, a C*1*1 feature vector is obtained. This feature vector is multiplied by C*W*H to obtain a new C*W*H feature vector.
[0080] Alternatively, see Figure 4 As shown, the structure of the third module is the same as that of the second module, and the feature vector processed by the third module is subjected to global average pooling to obtain the final result.
[0081] Based on the same inventive concept, see Figure 8 As shown, Figure 8 This is a structural diagram of a target detection device based on similarity comparison provided by an embodiment of the present invention. This application also provides a target detection device based on similarity comparison, which is applied to the method provided by the above embodiment of this application. Please refer to the above embodiment for the content of the method, and this application will not repeat it here. The device includes:
[0082] A generating module 201 is configured to generate a candidate frame based on a target image sample; the target image sample includes a target to be detected;
[0083] The feature database extraction module 202 is used to extract target features based on the candidate boxes and build a feature database;
[0084] The standard feature database extraction module 203 is used to extract target features based on existing image samples and build a standard feature database;
[0085] A first similarity comparison module 204 is configured to obtain a candidate frame with the highest confidence from the candidate feature database, perform a similarity comparison between the candidate frame and the standard feature database, obtain the standard feature database image with the highest score, and record its category information;
[0086] The second similarity comparison module 205 is used to perform similarity comparison between the standard feature database image with the highest score and the candidate feature database image, obtain the candidate feature database image with the highest score, and record its position information;
[0087] The result acquisition module 206 is used to map the category information of the standard feature database image with the highest score and the position information of the candidate feature database image with the highest score to the target image sample to obtain the detection result of the target to be detected.
[0088] This application provides a target detection method based on similarity comparison, designs a dynamic candidate box extraction method, and constructs an MSNR feature extraction network for similarity comparison, which can effectively reduce the missed detection rate of small and medium-sized targets, improve the accuracy of target positioning, and comprehensively improve the effect of target detection.
[0089] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.
Claims
1. A target detection method based on similarity comparison, characterized in that: include: Generate candidate boxes based on target image samples; The target image sample includes a target to be detected; Based on the candidate frame, target features are extracted to build a database of candidate features; Based on existing image samples, target features are extracted and a standard feature database is constructed; wherein, an MSNR feature extraction network is used for target feature extraction, and the MSNR feature extraction network includes a first module, a second module, and a third module; the second module and the third module have the same structure; the first module includes a convolutional layer, a maximum pooling layer, and a residual shrinkage module cascaded in sequence; the second module includes a convolutional layer, a first pooling layer, a second pooling layer, a residual shrinkage module, and a feature enhancement module cascaded in sequence; The feature enhancement module includes a multi-scale convolution kernel module and an attention mechanism noise reduction module; Obtain the candidate box with the highest confidence from the candidate feature database, perform similarity comparison between the candidate box and the standard feature database, obtain the standard feature database image with the highest score, and record its category information; Compare the image in the standard feature database with the highest score with the image in the candidate feature database for similarity, obtain the image in the candidate feature database with the highest score, and record its position information; The category information of the standard feature database image with the highest score and the position information of the selected feature database image with the highest score are mapped to the target image sample to obtain a detection result of the target to be detected.
2. The target detection method based on similarity comparison according to claim 1, characterized in that: Also includes: Expanding the standard feature database; The expansion of the standard feature database includes an image retrieval module and an image storage module.
3. The target detection method based on similarity comparison according to claim 2, characterized in that: The image retrieval module includes: Comparing the target features of the candidate frame with the features in the standard feature database one by one to obtain the score of the corresponding category of the candidate frame; The image storage module includes: The scores of the categories corresponding to the candidate boxes are screened, and the candidate boxes that meet the first condition are put into the standard feature database to expand the standard feature database.
4. The target detection method based on similarity comparison according to claim 3, characterized in that: The first condition is that the score of the category corresponding to the candidate box is between 0.6 and 0.
7.
5. A target detection device based on similarity comparison, characterized in that: include: The generation module is used to generate candidate boxes based on target image samples; The target image sample includes a target to be detected; A candidate feature database extraction module is used to extract target features based on the candidate box and construct a candidate feature database; wherein, the target feature extraction is performed using an MSNR feature extraction network, and the MSNR feature extraction network includes a first module, a second module, and a third module; the second module and the third module have the same structure; the first module includes a convolution layer, a maximum pooling layer, and a residual shrinkage module cascaded in sequence; the second module includes a convolution layer, a first pooling layer, a second pooling layer, a residual shrinkage module, and a feature enhancement module cascaded in sequence; the feature enhancement module includes a multi-scale convolution kernel module and an attention mechanism denoising module; The standard feature database extraction module is used to extract target features based on existing image samples and build a standard feature database; A first similarity comparison module is configured to obtain a candidate frame with the highest confidence from the candidate feature database, perform a similarity comparison between the candidate frame and the standard feature database, obtain the standard feature database image with the highest score, and record its category information; A second similarity comparison module is used to compare the image in the standard feature database with the image in the candidate feature database to obtain the image in the candidate feature database with the highest score and record its position information; The result acquisition module is used to map the category information of the standard feature database image with the highest score and the position information of the candidate feature database image with the highest score to the target image sample to obtain the detection result of the target to be detected.