A small sample target detection method and system based on multi-modal fusion and a medium

Abstract

The application discloses a kind of based on multimodal fusion small sample target detection method, system and medium, the method includes: obtaining coaxial visible light image and infrared image;Visible light image and infrared image are input into based on double-branch feature extraction network, feature extraction is carried out, and visible light modal feature map and infrared light feature map are obtained;Visible light modal feature map and infrared light feature map are fused, and the multimodal fusion feature map after fusion is obtained;Multimodal fusion feature map is input into improved RPN network, multimodal fusion feature map is input into ROI network, and each model training is carried out, and basic small sample target detection model is obtained;Basic small sample target detection model is adjusted, and the final small sample target detection model related to target task is obtained;And using final small sample target detection model, target task detection is carried out to the small sample to be measured.The application is good to the detection effect of small sample target detection, and detection performance is high.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a small sample target detection method and system based on multi-modal fusion and a medium. BACKGROUND

[0002] At present, artificial intelligence with deep learning as the core technology is developing rapidly, and deep learning algorithm is widely used in image classification, image target detection and other fields. However, the implementation of the above technology depends on the support of large data samples, and in some practical application scenarios, such as disaster prevention and reduction, large area monitoring, national defense and other fields, it is very expensive and difficult to collect a large number of labeled data. Therefore, how to realize the detection of targets by using a small amount of labeled data, that is, the detection of targets in the case of small samples, has become a hot and difficult point in the field of machine learning.

[0003] At present, most small sample target detection methods are still based on the idea of combining traditional target detection methods with small sample learning, and the algorithm idea and structure are improved to optimize. However, due to the limited features extracted by these methods because of the small number of samples, the algorithm effect cannot reach the level of practical application, and how to learn as much as possible under the condition of a small amount of samples is the key problem of research.

[0004] However, the existing small sample target detection method has the problems of poor detection effect, low detection performance, and great influence of samples. SUMMARY

[0005] The technical problem to be solved by the present application is that the existing small sample target detection method has the problems of poor detection effect, low detection performance, and great influence of samples. The purpose of the present application is to provide a small sample target detection method and system based on multi-modal fusion, which can realize detection and recognition under extreme conditions such as poor lighting and partial occlusion of targets. According to the characteristics of infrared images and visible light images, a double-branch feature extraction network composed of a residual network and an improved residual network is designed, and the improved residual feature extraction network introduces variable convolution to better extract image features. A method of Soft-NMS co-action is also designed to reduce the missed detection problem caused by multiple target overlaps. The present application uses multi-modal fusion features to improve the target recognition ability of the model, and then fine-tunes the model with a small amount of small sample class samples to realize target detection under the condition of small samples.

[0006] The present application is realized by the following technical scheme:

[0007] In a first aspect, the present application provides a small sample target detection method based on multi-modal fusion, which comprises:

[0008] acquiring a visible light image and an infrared image coaxially;

[0009] input the visible light image and the infrared image into a double-branch feature extraction network to perform feature extraction, to obtain a visible light modal feature map and an infrared light feature map;

[0010] fuse the visible light modal feature map and the infrared light feature map to obtain a fused multi-modal fusion feature map;

[0011] input the multi-modal fusion feature map into an improved RPN network, input the multi-modal fusion feature map into an ROI network, and perform model training to obtain a basic small sample target detection model;

[0012] adjust the basic small sample target detection model to obtain a final small sample target detection model related to a target task, and use the final small sample target detection model to perform target task detection on a to-be-detected small sample.

[0013] The application first performs coaxial visible light image and infrared image acquisition, inputs the visible light image and the infrared image into a double-branch feature extraction network designed by the application to perform feature extraction, the double-branch feature extraction network includes an original residual network and an improved residual network, inputs the extracted multi-modal fusion feature map into an RPN, inputs the multi-modal fusion feature map into an ROI network, and performs model pre-training; then fine-tunes the pre-trained model under the condition of a small sample for a target task to obtain a target detection network related to the target task. The application greatly reduces the dependence on the number of training samples by applying the multi-modal information fusion idea to a two-stage target detection network, reduces the influence of extreme environments such as poor lighting and partial occlusion of the target, improves the learning ability for the task, strengthens the robustness of the algorithm, and solves the problem of poor small sample detection effect at the present stage.

[0014] Further, the double-branch feature extraction network includes an original residual feature extraction network and a residual network feature extraction network based on variable convolution;

[0015] The original residual feature extraction network is used to perform feature extraction on the infrared image to obtain an infrared light feature map;

[0016] The residual network feature extraction network based on variable convolution is used to perform feature extraction on the visible light image to obtain a visible light modal feature map.

[0017] Further, the sampling of the variable convolution in the residual network feature extraction network based on variable convolution is represented as:

[0018]

[0019] wherein y(p0) is the feature map output at each p0 position, w(p n) is a sampling weight, x is an input feature map, p0 is each position on an output feature map y, p n Each position in R is listed, R is a receptive field, and Δp n is an offset.

[0020] Further, the multi-modal fusion feature map is input into the improved RPN network, the multi-modal fusion feature map is input into the ROI network, and each model is trained to obtain a basic small sample target detection model, which includes:

[0021] The multi-modal fusion feature map F is input into the improved RPN network to generate an anchor box Anchor box; the anchor box Anchor box is cropped and filtered to output a Bbox bounding box and a class score, that is, the best target Rois are obtained;

[0022] The multi-modal fusion feature map F is input into the ROI network, and the target Rois output by the improved RPN network are input into the ROI network; after ROI pooling, the output of the full connection layer is obtained, the classification and regression are completed, and a trained basic small sample target detection model is obtained.

[0023] Further, the improved RPN network uses linear Soft-NMS to remove redundant detection boxes to obtain the best target Rois; the expression of linear Soft-NMS is:

[0024]

[0025] Where s i is a score, N t is a threshold, and iou(M, b i ) is an overlap rate.

[0026] Further, the basic small sample target detection model is trained, specifically:

[0027] According to the basic small sample target detection model, all layer parameters except the last layer are frozen, and the last layer parameters are initialized;

[0028] The last layer parameters are adjusted by using a small sample fine-tuning dataset, and a final small sample target detection model is obtained by combining a cosine similarity classifier.

[0029] Further, the small sample fine-tuning dataset is obtained by taking sample numbers K=1, 2, 3, 5, and 10 on base classes and new classes respectively.

[0030] In a second aspect, the application further provides a small sample target detection system based on multi-modal fusion, which is used to implement the small sample target detection method based on multi-modal fusion; the system includes:

[0031] a data acquisition module configured to acquire coaxial visible light images and infrared images;

[0032] a dual-branch feature extraction network module configured to input the visible light images and the infrared images into the dual-branch feature extraction network for feature extraction to obtain visible light modal feature maps and infrared light feature maps;

[0033] a multi-modal feature fusion module configured to fuse the visible light modal feature maps and the infrared light feature maps to obtain fused multi-modal fusion feature maps;

[0034] a classification and regression module configured to input the multi-modal fusion feature maps into an improved RPN network, input the multi-modal fusion feature maps into an ROI network, and perform model training to obtain a basic small sample target detection model;

[0035] an adjustment module configured to adjust the basic small sample target detection model to obtain a final small sample target detection model related to a target task, and use the final small sample target detection model to perform target task detection on a to-be-detected small sample.

[0036] In a third aspect, the present application further provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the multi-modal fusion-based small sample target detection method when executing the computer program.

[0037] In a fourth aspect, the present application further provides a computer readable storage medium storing a computer program, wherein the computer program is executable on a processor to implement the multi-modal fusion-based small sample target detection method.

[0038] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0039] 1. The present application first acquires coaxial visible light images and infrared images, inputs the visible light images and the infrared images into the dual-branch feature extraction network designed in the present application for feature extraction, the dual-branch feature extraction network comprising an original residual network and an improved residual network, inputs the extracted multi-modal feature maps into an RPN, inputs the multi-modal fusion feature maps into an ROI network, and performs model pre-training; then, the pre-trained model is fine-tuned under the condition of small samples for a target task to obtain a target detection network related to the target task. The present application greatly reduces the dependence on the number of training samples by applying the multi-modal information fusion idea to the two-stage target detection network, reduces the influence of extreme environments such as poor lighting and partial occlusion of the target, improves the learning ability for the task, strengthens the robustness of the algorithm, and solves the problem of poor detection effect of the small sample detection at the present stage.

[0040] 2、The double-branch feature extraction network designed in the application adopts different feature extraction models for different characteristics of images, wherein the improved feature extraction model adopts the idea of variable convolution to increase the spatial sampling position and improve the processing capability for geometric transformation, so that the target movement, size scaling, rotation and the like can be better coped with, and the information of the visible light image can be efficiently learned under the condition of a small amount of samples. Meanwhile, the fusion module of the application is a feature-level fusion, and compared with the common image-level fusion, the modal features are more complete, and the detection and recognition effect is improved. In addition, the Soft-NMS is used to replace the traditional NMS for improvement, which can effectively solve the problems of complex background, target occlusion, overlap and the like, reduce the missed detection rate of the algorithm, and improve the detection effect. BRIEF DESCRIPTION OF DRAWINGS

[0041] The accompanying drawings, which are included to provide a further understanding of the embodiments of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings:

[0042] Figure 1 A flow chart of a small sample target detection method based on multi-modal fusion of the application;

[0043] Figure 2 A detailed flow chart of a small sample target detection method based on multi-modal fusion of the application;

[0044] Figure 3 A residual block diagram of a double-branch feature extraction network based on the application;

[0045] Figure 4 A structure block diagram of a small sample target detection system based on multi-modal fusion of the application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical scheme and advantages of the application clearer, more specific embodiments and the accompanying drawings are used to further explain the application, and the illustrative embodiments and the description thereof are only used to explain the application, and do not limit the application.

[0047] The existing small sample target detection method has the problems of poor detection effect, low detection performance and great influence of samples. And under the condition of a small sample, it is difficult to detect and recognize only one modality information of an object, and multi-modal information can complement each other in features because it describes the target from different angles. Based on this, the application considers introducing multi-modal learning to improve the algorithm performance on the basis of a small sample, and using the visible light modality and infrared modality information of the image to improve the detection and recognition performance.

[0048] Currently, common visible light and infrared multi-mode fusion methods are based on image-level fusion (in the data processing stage, the infrared image and the visible light image are first fused into a single image as input). This results in the loss of some information from each modality. Based on this, a dual-branch multi-modal fusion small sample target detection network is designed to address the characteristics of different sensors. The dual-branch network is characterized by feature extraction networks designed separately for different models, which can better extract the subtle features of different models. Furthermore, a modal feature fusion network is designed to perform feature-level fusion, increasing the number of effective features and effectively improving detection and recognition performance.

[0049] This invention first acquires coaxial visible light and infrared images. These images are then input into a two-branch feature extraction network (RPN) designed in this invention for feature extraction. The RPN comprises an original residual network and an improved residual network. The extracted multimodal feature maps are input into the RPN, and the fused multimodal feature maps are input into the ROI network for model pre-training. Then, the pre-trained model is fine-tuned for the target task under small sample conditions to obtain a target detection network relevant to the target task. This invention significantly reduces the dependence on the number of training samples by applying the concept of multimodal information fusion to the two-stage target detection network, mitigating the impact of extreme environments such as poor lighting and partial target occlusion, improving the learning ability for the task, and enhancing the robustness of the algorithm, thus solving the problem of poor detection performance with small sample sizes in current methods.

[0050] Example 1

[0051] like Figure 1 As shown, this invention provides a small-sample target detection method based on multimodal fusion, the method comprising:

[0052] Acquire coaxial visible light and infrared images;

[0053] Visible light images and infrared images are input into a dual-branch feature extraction network for feature extraction, resulting in visible light modal feature maps and infrared light feature maps.

[0054] The visible light modal feature map and the infrared light feature map are fused to obtain the fused multimodal feature map;

[0055] The multimodal feature maps are input into the improved RPN network, the multimodal fused feature maps are input into the ROI network, and each model is trained to obtain a basic few-shot object detection model.

[0056] The basic small sample target detection model is adjusted to obtain the final small sample target detection model related to the target task; and the final small sample target detection model is used to detect the target task in the small sample to be tested.

[0057] likeFigure 2 The specific implementation is as follows:

[0058] Step 1, collect coaxial visible light images S l and infrared images Q l Data sets, l = 1, 2,..., m, m + 1,..., l, l is the total number of categories, and training data sets and test data sets are made. First, the collected data is divided into base class data sets X base and new class data sets X new , wherein the base class data set is composed of k classes of visible light image and infrared image data, and the new class data set is composed of the remaining classes of visible light image and infrared image data:

[0059] X base = {S i , Q i}, i = 1, 2,..., k

[0060] X new = {S j , Q j}, j = m + 1, m + 2,..., l

[0061] The training data set X train is composed of base class data, and the test data set X test is composed of part of the base class and the new class;

[0062] X train = X base

[0063] X test = X base ∪X new

[0064] In the specific implementation, step 1 is divided into non-intersecting base class and new class in the ratio of 4:1, and the data in the base class and the new class is visible light image and infrared image with labels. In the experiment, K images are extracted from each category respectively; then the base class is randomly extracted as the training set and the verification set in the ratio of 7:3;

[0065] Specifically, the visible light images and infrared images used in the experiment are one-to-one corresponding, wherein the base class contains a large number of samples, and the new class only has a small number of samples. K = 1, 2, 3, 5, 10.

[0066] Step 2, according to the characteristics of visible light images and infrared images, a double-branch feature extraction network is constructed; the double-branch feature extraction network includes a residual feature extraction network (ResNet101) and a residual network feature extraction network based on variable convolution (improved ResNet101); as Figure 3As shown, (a) is a processing flowchart of an original residual block in an original residual feature extraction network, and (b) is a processing flowchart of a variable convolution residual block in a variable convolution-based residual network feature extraction network.

[0067] The original sampling of the original residual feature extraction network can be expressed as:

[0068]

[0069] The sampling of the variable convolution in the variable convolution-based residual network feature extraction network is expressed as:

[0070]

[0071] where y(p0) is the feature map output at each p0 position, w(p n ) is the sampling weight, x is the input feature map, p0 is each position on the output feature map y, p n lists each position in R, R is the receptive field, and Δp n is the offset.

[0072] Step 3: input the visible light image and the infrared image into the dual-branch feature extraction network for feature extraction; specifically, the original residual feature extraction network is used to extract features from the infrared image to obtain an infrared light feature map f h .

[0073] Based on the fact that the visible light image contains features such as shape and color, the variable convolution-based residual network feature extraction network is used to extract features from the visible light image to obtain a visible light modal feature map f k .

[0074] Step 4: improve the RPN network by replacing the traditional NMS with a linear Soft-NMS to improve the detection performance of the algorithm in complex scenes, and the input of the ROI layer is the multi-modal feature obtained by fusing the visible light modal feature and the infrared modal feature, which can improve the classification accuracy of the target; the expression of the linear Soft-NMS is:

[0075]

[0076] where s i is the score, N t is the threshold, and iou(M, b i ) is the overlap rate.

[0077] Step 5: fuse the visible light modal feature map f k and the infrared light feature map f h to obtain a fused multi-modal fusion feature map F; the feature fusion formula is:

[0078]

[0079] wherein F is the fused multi-modal fusion feature map, f k is the visible light modal feature map, f h is the infrared modal feature map, is point-by-point addition.

[0080] Step 6, input the multi-modal fusion feature map F obtained in step 5 into the improved RPN network, first perform sliding window processing on the picture, and then obtain a series of detection boxes B and their corresponding scores S. Unlike NMS which suppresses all overlapping prediction boxes, linear Soft-NMS attenuates the detection score of a detection box B1 which has a high overlap with a prediction box M. By setting a confidence threshold, the score less than the threshold is removed, and the score greater than the threshold is retained to determine the position of the foreground proposal box.

[0081] Step 7, input the multi-modal fusion feature map F and the information output by the improved RPN network in step 6 into the ROI Pooling layer. Using multi-modal feature maps instead of single-modal feature maps in traditional algorithms can provide more information for classification and recognition under small sample conditions, and increase accuracy. Specifically:

[0082] input the visible light modal feature map f k into the improved RPN network to generate anchor boxes; perform cropping and filtering on the anchor boxes to output Bbox bounding boxes and class scores, i.e. obtain the best target Rois;

[0083] input the multi-modal fusion feature map F into the ROI network, and pass the target Rois output by the improved RPN network into the ROI network; after ROI pooling, enter the full connection layer to output, complete classification and regression.

[0084] Step 8, perform model training iteration to obtain a pre-trained model with excellent performance on the base class (i.e. a basic small sample target detection model);

[0085] In the iteration training process of the above basic small sample target detection model, the total network loss is composed of three parts, represented as:

[0086] L total = L rpn + L cls + L loc

[0087] wherein L total is the total network loss, L rpn is the RPN network loss, L cls is the network classification loss, and L locThe network boundary regression loss.

[0088] Step 9, sample the number K=1, 2, 3, 5, 10 of base classes and new classes respectively to obtain small sample fine-tuning data sets, and the remaining data sets are used for verification;

[0089] Step 10, introduce a cosine similarity classifier in the fine-tuning stage, and the evaluation function of the cosine similarity is:

[0090]

[0091] Wherein, S i,j is the similarity score between the input x of the i-th target proposal and the weight vector of the j-th class, F(x) is the input feature, and omega j is the weight vector of the j-th target, and alpha is a scale factor.

[0092] Step 11, according to the basic small sample target detection model obtained in step 8, freeze all layer parameters except the last layer, and initialize the last layer parameters; adjust the last layer parameters with the small sample fine-tuning data set to obtain a detector with excellent detection performance for base classes and new classes (i.e. the final small sample target detection model).

[0093] Compared with the prior art, the present application has the following beneficial effects:

[0094] The double-branch feature extraction network designed in the present application adopts different feature extraction models for different characteristics of images, wherein the improved feature extraction model adopts the idea of variable convolution to increase the spatial sampling position and improve the processing capability for geometric transformation, so that the information of visible light images can be efficiently learned under the condition of a small number of samples in the case of target movement, size scaling and rotation.

[0095] The Soft-NMS adopted in the present application replaces the traditional NMS to improve it, which can effectively solve the problems of complex background, target occlusion and overlap, reduce the missed detection rate of the algorithm, and improve the detection effect.

[0096] The method of fusing visible light modal features and infrared modal features adopted in the present application is used to improve the performance of the small sample algorithm, mainly to improve its classification performance. Many scholars' researches show that under the condition of small sample, the positioning of the target detection network to the target has little influence, but it is difficult to learn useful features of the target under such a small number of samples. The multi-modal feature fusion idea of the present application combines the features of different modalities, enhances the learning ability and representativeness of the target features, and reduces the influence of samples on the algorithm. Compared with the traditional small sample detection algorithm, it has better robustness and detection effect.

[0097] Example 2

[0098] AsFigure 4 The difference between the embodiment and embodiment 1 is that the embodiment provides a small sample target detection system based on multi-modal fusion, which is used to implement the small sample target detection method based on multi-modal fusion in embodiment 1; the system comprises:

[0099] A data acquisition module is configured to acquire the coaxial visible light image and infrared image.

[0100] A dual-branch feature extraction network module is configured to input the visible light image and the infrared image into the dual-branch feature extraction network for feature extraction to obtain a visible light modal feature map and an infrared light feature map.

[0101] A multi-modal feature fusion module is configured to perform feature fusion on the visible light modal feature map and the infrared light feature map to obtain a fused multi-modal fusion feature map.

[0102] A classification and regression module is configured to input the multi-modal fusion feature map into the improved RPN network, input the multi-modal fusion feature map into the ROI network, and perform model training to obtain a basic small sample target detection model.

[0103] An adjustment module is configured to adjust the basic small sample target detection model to obtain a final small sample target detection model related to a target task, and use the final small sample target detection model to perform target task detection on a to-be-detected small sample.

[0104] The execution process of each module is performed according to the process steps of the small sample target detection method based on multi-modal fusion in embodiment 1, and will not be described again in this embodiment.

[0105] Meanwhile, the application further provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the small sample target detection method based on multi-modal fusion when executing the computer program.

[0106] Meanwhile, the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the small sample target detection method based on multi-modal fusion.

[0107] Those skilled in the art will appreciate that embodiments of the application can be readily used as software, hardware, or a combination of software and hardware. In one

[0108] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or a combination of flowcharts and / or blocks in the flowcharts can be implemented by computer program instructions. Figure 1 means for carrying out functions specified in the flowchart block or blocks.

[0109] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or a combination of flowcharts and / or blocks in the flowcharts can be implemented by computer program instructions. Figure 1 means for carrying out functions specified in the flowchart block or blocks.

[0110] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or a combination of flowcharts and / or blocks in the flowcharts can be implemented by computer program instructions. Figure 1 means for carrying out functions specified in the flowchart block or blocks.

[0111] The above description is only specific embodiments of the present application, and is not intended to limit the protection scope of the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall fall within the protection scope of the present application.

Claims

1. A small sample target detection method based on multi-modal fusion, characterized in that, The method comprises: acquiring coaxial visible light images and infrared images; inputting the visible light images and infrared images into a double-branch feature extraction network for feature extraction to obtain visible light modal feature maps and infrared light feature maps; performing feature fusion on the visible light modal feature maps and infrared light feature maps to obtain multi-modal fusion feature maps; inputting the multi-modal fusion feature maps into an improved RPN network, inputting the multi-modal fusion feature maps into an ROI network, and performing model training to obtain a basic small sample target detection model; adjusting the basic small sample target detection model to obtain a final small sample target detection model related to a target task, and using the final small sample target detection model to perform target task detection on a to-be-detected small sample; the double-branch feature extraction network comprises an original residual feature extraction network and a variable convolution-based residual network feature extraction network; the original residual feature extraction network is used to perform feature extraction on the infrared images to obtain infrared light feature maps; the variable convolution-based residual network feature extraction network is used to perform feature extraction on the visible light images to obtain visible light modal feature maps; sampling of the variable convolution in the variable convolution-based residual network feature extraction network is represented as: ； wherein, is each feature map output for each is a sampling weight, is an input feature map, is each position on the output feature map y, enumerates each position in R, is a receptive field, is an offset; the inputting of the multi-modal fusion feature maps into the improved RPN network, the inputting of the multi-modal fusion feature maps into the ROI network, and the performing of model training to obtain the basic small sample target detection model comprise: inputting the multi-modal fusion feature maps into the improved RPN network to generate anchor boxes Anchor box; performing cropping and filtering on the anchor boxes Anchor box to output Bbox bounding boxes and class scores, i.e., obtaining best target Rois; inputting the multi-modal fusion feature maps into the ROI network and inputting the target Rois output by the improved RPN network into the ROI network; after ROI pooling, entering a fully connected layer to output, completing classification and regression, and obtaining a trained basic small sample target detection model; the improved RPN network uses linear Soft-NMS to remove redundant detection boxes to obtain best target Rois; an expression of the linear Soft-NMS is: ； wherein, is a score, is a threshold, is an overlap rate.

2. The small sample target detection method based on multi-modal fusion according to claim 1, characterized in that, the adjusting of the basic small sample target detection model is specifically: freezing all layer parameters except the last layer according to the basic small sample target detection model, and initializing the last layer parameters; adjusting the last layer parameters using a small sample fine-tuning dataset, and combining a cosine similarity classifier to obtain a final small sample target detection model.

3. The small sample target detection method based on multi-modal fusion according to claim 2, characterized in that, The small sample fine-tuning dataset is a small sample fine-tuning dataset obtained by taking sample numbers K=1, 2, 3, 5, and 10 on base classes and new classes.

4. A small sample target detection system based on multi-modal fusion, characterized in that, The system uses a multi-modal fusion-based small sample target detection method according to any one of claims 1 to 3; and the system comprises: a data acquisition module configured to acquire coaxial visible light images and infrared images; The dual-branch feature extraction network module is configured to input the visible light image and the infrared image into the dual-branch feature extraction network, perform feature extraction, and obtain a visible light modality feature map and an infrared light feature map; The multi-modal feature fusion module is configured to perform feature fusion on the visible light modality feature map and the infrared light feature map, and obtain a fused multi-modal fusion feature map; The classification and regression module is configured to input the multi-modal fusion feature map into an improved RPN network, input the multi-modal fusion feature map into an ROI network, and perform model training, to obtain a basic small sample target detection model; The adjustment module is configured to adjust the basic small sample target detection model, to obtain a final small sample target detection model related to a target task, and perform target task detection on a to-be-detected small sample by using the final small sample target detection model.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The computer program is executed by the processor to implement the small sample target detection method based on multi-modal fusion according to any one of claims 1 to 3.

6. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 5. The computer program is executed by the processor to implement the small sample target detection method based on multi-modal fusion according to any one of claims 1 to 3.

Images