Small-sample target detection method based on multi-view feature fusion and meta-feature encoding

Abstract

The present application proposes a small sample target detection method based on multi-view feature fusion and meta-feature encoding, including: multi-view embedding and fusion of the image data to be detected to obtain fused image data; using a feature extraction network to extract image features from the fused image data, and combining the region proposal network to determine the candidate regions of the image data, and performing RoI pooling on the candidate regions to obtain RoI features; inputting the RoI features into the fully connected network and the meta-feature encoder respectively to obtain classification features, regression features and meta-features, and fusing the meta-features and classification features to obtain the final classification features; determining the location and classification of the target in the image data according to the regression features and the final classification features through the classifier and regressor. The present invention using the above scheme can ensure the generalization of new classes when performing small sample target detection, and realize the effective retention of base class knowledge, alleviating the problem of catastrophic knowledge forgetting.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a small sample target detection method and device based on multi-view feature fusion and meta-feature encoding. Background Art

[0002] In recent years, the development of deep convolutional neural networks has greatly promoted the progress of related algorithms in the field of computer vision. As one of the research branches in the field of computer vision, object detection mainly classifies and locates objects in a given image. General object detection methods are mainly divided into two categories: two-stage detection networks and single-stage detection networks. However, no matter which type, it is necessary to pre-train on a large number of annotated data sets first, resulting in a single coverage scene. When migrating to scenes with scarce annotated data, it is easy to cause a decrease in detection accuracy. On the one hand, data acquisition and annotation require a lot of manpower and time; on the other hand, for fields such as medicine and military, it is difficult to obtain sufficient samples due to privacy and security issues.

[0003] Based on this situation, few-shot learning has gradually attracted the attention of academia and industry. Few-shot learning aims to mine the potential features of samples through a small amount of labeled data, and then fit a robust feature space through a small amount of data to complete related visual perception tasks. Few-shot object detection, as one of the important branches in the field of few-shot learning, aims to complete the classification and positioning of corresponding objects through a small amount of new class labeled data.

[0004] At present, there are two evaluation methods for small sample target detection methods: FSOD and G-FSOD. FSOD only focuses on the detection performance of new classes, aiming to build a detection model for detecting new small targets, while G-FSOD focuses on the performance of both base classes and new classes, aiming to build a general target detection model. The current mainstream method for small sample target detection belongs to FSOD evaluation, which ignores the detection performance of base classes. Under the G-FSOD evaluation method, existing small sample target detection methods based on transfer learning often suffer from catastrophic forgetting problems. Summary of the invention

[0005] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0006] To this end, the first purpose of this application is to propose a small sample target detection method based on multi-view feature fusion and meta-feature encoding, which solves the technical problem that existing methods cannot achieve the optimal balance between base class knowledge and new class knowledge in the model training and fine-tuning stages. It can effectively retain the base class knowledge while ensuring the generalization of new classes, thereby alleviating the problem of catastrophic knowledge forgetting.

[0007] The second objective of this application is to propose a small sample target detection model based on multi-view feature fusion and meta-feature encoding.

[0008] The third objective of this application is to propose a small sample model training method.

[0009] The fourth objective of the present application is to provide a computer device.

[0010] To achieve the above-mentioned purpose, the first aspect of the present application proposes a small sample target detection method based on multi-view feature fusion and meta-feature encoding, including: performing multi-view embedding fusion on the image data to be detected to obtain fused image data, wherein the fused image data is fused data of image data of different viewpoints corresponding to the image data to be detected; using a feature extraction network to extract image features from the fused image data, and using a region proposal network to determine the candidate region of the image data based on the extracted feature map, and performing RoI pooling on the candidate region to obtain RoI features; inputting the RoI features into a fully connected network and a meta-feature encoder respectively, obtaining the classification features and regression features output by the fully connected network and the meta-features output by the meta-feature encoder, and fusing the meta-features with the classification features to obtain the final classification features; determining the positioning and classification of the target in the image data according to the regression features and the final classification features through a classifier and a regressor.

[0011] The small sample target detection method based on multi-view feature fusion and meta-feature encoding in the embodiment of the present application obtains highly generalized representations of base classes and new classes through a multi-view feature fusion method, and then retains the base class meta-features during base class training by using a meta-feature encoder, thereby effectively retaining the base class knowledge when fine-tuning the new class, thereby reducing the difficulty of learning the new class while retaining the detection accuracy of the base class, thereby improving the accuracy of the model under the G-FSOD evaluation method.

[0012] Optionally, in one embodiment of the present application, multi-view embedding and fusion are performed on the image data to obtain fused image data, including:

[0013] Perform data augmentation on the image data to obtain image data from different perspectives;

[0014] Perform inter-view feature fusion on image data from different viewpoints to obtain preliminary fused image data;

[0015] All the preliminarily fused image data are stacked in the batch dimension so that the image data of each batch remains independent, thereby obtaining the fused image data.

[0016] Optionally, in one embodiment of the present application, the process of data augmentation is expressed as:

[0017]

[0018] Among them, m and n represent the type sequence and amplitude sequence of data augmentation, num_aug represents the data augmentation type, max_amp represents the maximum augmentation amplitude, RandomInts(x,y) represents randomly generating an integer that is greater than or equal to x and less than or equal to y, and RandomFloats(x,y) represents randomly generating a floating-point number that is greater than or equal to x and less than or equal to y;

[0019] The process of feature fusion between views is expressed as:

[0020]

[0021] Among them, λ in represents the weight of each view when the features between views are fused, λ in is a learnable parameter;

[0022] The process of stacking over the batch dimension is expressed as:

[0023]

[0024] in, Represents the stacked data.

[0025] Optionally, in one embodiment of the present application, determining a candidate region of the image data based on the extracted feature map using a region proposal network includes:

[0026] Input the extracted feature map into the region proposal network to obtain candidate region proposals;

[0027] The candidate regions are determined based on the extracted feature maps and the candidate region proposals.

[0028] Optionally, in one embodiment of the present application, the RoI feature is input into a meta-feature encoder, and the meta-feature is output, including:

[0029] The RoI features are respectively passed through multiple parallel convolutions to form multiple RoI features, and the width and height of the multiple RoI features are flattened to obtain multiple sets of feature vectors;

[0030] Input multiple groups of feature vectors into the multi-head self-attention mechanism to remove redundancy and obtain the features after removing redundancy;

[0031] The de-redundant features and the learnable features are used through a cross-attention mechanism to determine the meta-feature retrieval space and the meta-features contained in the retrieval space, where the learnable features are obtained by random initialization.

[0032] Optionally, in one embodiment of the present application, multiple groups of feature vectors are input into a multi-head self-attention mechanism for redundancy removal, including:

[0033] Multiple sets of feature vectors are input into the multi-head self-attention network, the attention matrix is ​​output, and the attention matrix is ​​input into the feedforward neural network to output the de-redundant features. The feedforward neural network consists of a linear layer, a ReLU activation function layer, and a linear layer. The de-redundant process is expressed as:

[0034]

[0035] Among them, FFN is a feedforward neural network, d is the feature dimension, multiple groups of feature vectors include Q, K, V, Q, K, V are transformer feature vector inputs, Q, K, V represent Query, Key, and Value respectively, and matmul() represents matrix multiplication.

[0036] Optionally, in one embodiment of the present application, the meta-features and the classification features are fused to obtain the final classification features, including:

[0037] The meta-features are weighted by the defined weights to obtain the semantic representation vector of the RoI;

[0038] The semantic representation vector of RoI is superimposed with the classification feature to obtain the final classification feature.

[0039] To achieve the above-mentioned purpose, the second embodiment of the present invention proposes a small sample target detection model based on multi-view feature fusion and meta-feature encoding, including a multi-view embedding fusion module, a feature extraction network, a region proposal network, an R-CNN detection network, a meta-feature encoder, and a classification module. The R-CNN detection network includes a RoI pooling module, wherein:

[0040] A multi-view embedding and melting module is used to perform multi-view embedding and fusion on the image data to be detected to obtain fused image data, wherein the fused image data is fused data of image data of different viewpoints corresponding to the image data to be detected;

[0041] A feature extraction network is used to extract image features from the fused image data to obtain a feature map;

[0042] A region proposal network is used to determine candidate regions of image data based on the extracted feature maps;

[0043] R-CNN detection network is used to perform RoI pooling on the candidate regions through RoI pooling to obtain RoI features, and to determine classification features and regression features through RoI features;

[0044] A meta-feature encoder, used to determine meta-features based on RoI features;

[0045] The classification module is used to fuse the meta-features and classification features to obtain the final classification features, and determine the location and classification of the target in the image data based on the regression features and the final classification features through the classifier and regressor.

[0046] To achieve the above purpose, the third aspect of the present invention proposes a small sample model training method, which is applied to the above image target detection model. The method includes:

[0047] Acquire training data, wherein the training data includes base class data and new class data, and the amount of the new class data is less than the amount of the base class data;

[0048] Use base class data to pre-train the image target detection model for target detection tasks. During pre-training, use the final classification features and regression features for positioning and classification. Calculate the loss based on the prediction results and labels, and perform gradient optimization based on the loss to update the model parameters.

[0049] The image object detection model is fine-tuned using new class data, and the parameters of the meta-feature encoder are frozen during fine-tuning.

[0050] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a computer device, including: a processor; a memory for storing executable instructions of the processor, when the processor executes the executable instructions on the memory, the above-mentioned small sample target detection method based on multi-view feature fusion and meta-feature encoding is implemented.

[0051] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0053] Figure 1 A flowchart of a small sample target detection method based on multi-view feature fusion and meta-feature encoding provided in Example 1 of the present application;

[0054] Figure 2 This is an overall architecture diagram of a multi-view feature embedding and fusion algorithm based on data augmentation according to an embodiment of the present application;

[0055] Figure 3 This is a structural diagram of a meta-feature encoder according to an embodiment of the present application;

[0056] Figure 4 This is a flowchart of the execution of the meta-feature encoder of an embodiment of the present application;

[0057] Figure 5 A schematic diagram of the structure of a small sample target detection model based on multi-view feature fusion and meta-feature encoding provided in an embodiment of the present application;

[0058] Figure 6 A framework diagram of a small sample target detection model based on multi-view feature fusion and meta-feature encoding according to an embodiment of the present application;

[0059] Figure 7 A flowchart of a small sample model training method provided in an embodiment of the present application;

[0060] Figure 8 This is a flowchart of pre-training base class data in an embodiment of the present application;

[0061] Fig. 9 This is an execution flow chart of fine-tuning new type data according to an embodiment of the present application. DETAILED DESCRIPTION

[0062] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0063] In response to the problems of the prior art, the present application proposes a small sample target detection method based on multi-view feature fusion and meta-feature encoding. The method belongs to small sample target detection. The method is based on the transfer learning paradigm. Specifically, the small sample target detection based on transfer learning uses the existing training model to perform target detection on a small number of data sets. Its main advantage is that it can achieve better performance on a limited data set and can quickly adapt to new tasks and fields. This method usually uses a pre-trained model to train the model on a large-scale data set to learn a general feature representation. Then, these general features are applied to a new small sample data set for fine-tuning to adapt to specific tasks and fields. This method can effectively reduce the risk of overfitting of the model on a small sample data set and improve the generalization ability of the model.

[0064] This application first proposes a multi-view feature embedding and fusion algorithm based on data augmentation, and uses multi-view generation technology to design a highly generalized multi-view feature embedding and fusion method to improve the model representation ability and enhance the model generalization; secondly, a meta-feature encoder based on the attention mechanism is proposed. The meta-feature retrieval space is obtained through the attention mechanism and feature embedding module, and the meta-feature retrieval space is weighted to the classification head of the model, thereby enhancing the performance of the classification head.

[0065] The following describes a small sample target detection method and device based on multi-view feature fusion and meta-feature coding according to an embodiment of the present application with reference to the accompanying drawings.

[0066] Figure 1 A flowchart of a small sample target detection method based on multi-view feature fusion and meta-feature encoding provided in Example 1 of the present application.

[0067] like Figure 1 As shown, the small sample target detection method based on multi-view feature fusion and meta-feature encoding includes the following steps:

[0068] Step 101, performing multi-view embedding fusion on the image data to be detected to obtain fused image data, wherein the fused image data is fused data of image data of different viewpoints corresponding to the image data to be detected;

[0069] Step 102, using a feature extraction network to extract image features from the fused image data, and using a region proposal network to determine candidate regions of the image data based on the extracted feature map, and performing RoI pooling on the candidate regions to obtain RoI features;

[0070] Step 103, inputting the RoI features into the fully connected network and the meta-feature encoder respectively, obtaining the classification features and regression features output by the fully connected network and the meta-features output by the meta-feature encoder, and fusing the meta-features with the classification features to obtain the final classification features;

[0071] Step 104: Determine the location and classification of the target in the image data based on the regression features and the final classification features through the classifier and the regressor.

[0072] The small sample target detection method based on multi-view feature fusion and meta-feature encoding in the embodiment of the present application obtains highly generalized representations of base classes and new classes through a multi-view feature fusion method, and then retains the base class meta-features during base class training by using a meta-feature encoder, thereby effectively retaining the base class knowledge when fine-tuning the new class, thereby reducing the difficulty of learning the new class while retaining the detection accuracy of the base class, thereby improving the accuracy of the model under the G-FSOD evaluation method.

[0073] Optionally, in one embodiment of the present application, multi-view embedding and fusion is achieved through a multi-view feature embedding and fusion algorithm based on data augmentation. Specifically, through the generation and fusion of multi-view data, the model needs to be optimized under the supervision of multi-view data each time the gradient is updated, thereby improving the stability of the model gradient update. Figure 2 As shown in the figure, the overall architecture of the algorithm consists of three parts: multi-view sample generation (MG) of images, feature fusion (MF) between views, and multi-view data embedding (MFE).

[0074] During the training process, any sample x i Multi-view data will be generated through MF. The MF module contains a data augmentation pool. For any data augmentation, a set of sequences m and n will be randomly generated, which represent the type sequence and amplitude sequence of data augmentation. For any sample x i The process of generating data of a perspective is shown in formula (1), where num_aug represents the type of data augmentation and max_amp represents the maximum augmentation amplitude.

[0075]

[0076] Formula (1) can generate a series of perspective samples [x i,v1 ,x i,v2 ,...,x i,vn ], these samples will be fused in MG, and each combination can be expressed as:

[0077]

[0078] Through the operation of formula (1) and formula (2), the samples are transformed into samples of multiple perspectives. These samples will eventually be stacked up in the batch dimension and remain independent in the subsequent network processing.

[0079]

[0080] Specifically, this means superimposing the fused image data in the batch dimension, and the batch value will be expanded; for example, the original input to the backbone network during training is (n, c, h, w), where n is the batch. Assuming that the number of multi-viewpoints for each image is 4, the input after multi-viewpoint feature fusion is (4n, c, h, w). When stacking, different types of images will appear in the same batch, but these images will contain multiple views.

[0081] Through the generation, embedding and fusion of the above steps, multi-perspective data features can be obtained. On the one hand, the data features can be made more robust, highly generalized representations can be obtained, and the model classification performance can be enhanced; on the other hand, the stability of the model gradient update can be improved through the supervision and optimization of multi-perspective data.

[0082] Optionally, in one embodiment of the present application, determining a candidate region of the image data based on the extracted feature map using a region proposal network includes:

[0083] Input the extracted feature map into the region proposal network to obtain candidate region proposals;

[0084] The candidate regions are determined based on the extracted feature maps and the candidate region proposals.

[0085] Optionally, in one embodiment of the present application, in human vision, when distinguishing two different objects, they will naturally distinguish them by the detailed features of the objects, for example, dogs and cats have different ears, eyes, body shapes, fur colors, etc. In this embodiment, such a series of features is called meta-features, that is, the semantic representation of an object can be composed of a series of meta-features, and each meta-feature has a different contribution to the semantic representation of the object, that is, a different weight. Assume that there is a series of meta-features (s1, s2, s3, s4, ..., s n ), whose weight vector is (w1,w2,w3,w4,…w n ), then a category can be characterized as:

[0086] Rep=w 1 *s 1 +w 2 *s 2 +w 3 *s 3 +w 4 *s 4 +......+w n *s n (4)

[0087] In order to obtain meta-feature knowledge, this embodiment uses the attention mechanism to learn meta-features and designs a meta-feature encoder, whose structure is as follows: Figure 3 As shown in the figure. In the small sample target detection method based on transfer learning, the base class has a rich sample set, and the model can fully learn on the base class. In the R-CNN detection head, the foreground RoI (Region of Interest) contains the semantic information of the target and has the conditions for extracting meta-features. Therefore, the meta-feature encoder of this embodiment mainly encodes the RoI features. The meta-feature encoding process proposed in this embodiment is as follows Figure 4 As shown, the specific execution process is as follows:

[0088] (1) The region proposal network outputs candidate region proposals;

[0089] (2) The feature extractor obtains a series of RoI features based on the candidate region proposals. Specifically, the RoI features output by the network are 4x4 features with 2048 channels.

[0090] (3) The RoI features with 2048 channels are output according to the original network structure to obtain bounding box regression features and bounding box classification features;

[0091] (4) Copy the RoI feature and use three parallel 1x1 convolutions to form three RoI features. Flatten the width and height of each feature to form a feature vector of (N, 2048, H*W), i.e., Q, K, and V.

[0092] (5) The three groups of feature vectors Q, K, and V are sent to the multi-head self-attention mechanism for redundancy removal. First, the three groups of feature vectors Q, K, and V are sent to the multi-head self-attention network to obtain the attention matrix, and then the attention matrix is ​​sent to the FFN (feedforward neural network), whose network structure consists of a linear layer-ReLU activation function layer-linear layer, and outputs the features after redundancy removal. The process can be expressed as:

[0093]

[0094] Where d represents the feature dimension;

[0095] (6) The de-redundant features and learnable features are combined to form a meta-feature retrieval space through a cross-attention mechanism;

[0096] Specifically, the learnable feature is a learnable vector obtained by random initialization. The de-redundant features and the learnable features can obtain the meta-features of the learnable feature dimension (fixed dimension) through the cross-attention mechanism;

[0097] (7) In the bounding box classification module, the meta-features in the meta-feature retrieval space are weighted by defining a set of weights to form a semantic representation vector of the current RoI;

[0098] (8) Superimpose the bounding box classification features and the RoI semantic representation vector and use them as the final classification features to improve classification robustness;

[0099] (9) The bounding box classifier and bounding box regressor locate and classify the bounding box based on the regression features and the final classification features.

[0100] The meta-feature encoder removes redundancy and superimposes attention on the RoI through the attention mechanism and additional feature embedding vectors, extracts and separates the high-response features in the region, and independently generates meta-features, thereby forming a series of meta-feature representation spaces. These meta-features and meta-feature representation spaces contain the semantic feature information of the RoI, which helps to enhance the classification features of the instance, especially under small sample conditions, and can be combined with the classification features of the original network to enhance the classification effect of the network.

[0101] In order to implement the above embodiments, the present application also proposes a small sample target detection model based on multi-view feature fusion and meta-feature encoding.

[0102] Figure 5 A schematic diagram of the structure of a small sample target detection model based on multi-view feature fusion and meta-feature encoding provided in an embodiment of the present application.

[0103] like Figure 5 As shown, the small sample target detection model based on multi-view feature fusion and meta-feature encoding includes a multi-view embedding fusion module, a feature extraction network, a region proposal network, an R-CNN detection network, a meta-feature encoder, and a classification module. The R-CNN detection network includes a RoI pooling module, wherein:

[0104] The multi-view embedding and melting module is used to perform multi-view embedding and fusion on the image data to be detected to obtain fused image data, wherein the fused image data is fused data of image data of different viewpoints corresponding to the image data to be detected;

[0105] The feature extraction network is used to extract image features from the fused image data to obtain a feature map;

[0106] The region proposal network is used to determine candidate regions of the image data based on the extracted feature map;

[0107] The R-CNN detection network is used to perform RoI pooling on the candidate region through the RoI pooling to obtain RoI features, and determine classification features and regression features through the RoI features;

[0108] The meta-feature encoder is used to determine a meta-feature based on the RoI feature;

[0109] The classification module is used to fuse the meta-features and the classification features to obtain final classification features, and determine the location and classification of the target in the image data according to the regression features and the final classification features through a classifier and a regressor.

[0110] Specifically, Figure 6: is a framework diagram of a small sample target detection model based on multi-view feature fusion and meta-feature encoding in this embodiment, such as Figure 6 As shown in the figure, the framework mainly includes: multi-view embedding fusion, feature extraction backbone network, region proposal network RPN, R-CNN detection network and meta-feature encoder. The overall structure of the network refers to the two-stage target detection network Faster-RCNN and the small sample target detection algorithm DeFRCN based on transfer learning, among which multi-view embedding fusion and meta-feature encoder are new modules proposed in this embodiment. First, multi-view embedding fusion is performed on both the base class and the new class data to obtain highly generalized feature representations of the base class and new class samples, thereby increasing the escape efficiency of the model during the optimization process. Then the model extracts features from the fused data through the backbone network, and the region proposal network outputs the foreground and background proposal regions, which are sent to the R-CNN detection network, and the foreground and background feature representations of uniform size are obtained through RoI pooling, i.e., RoI (Region of Interest), which is further regressed and classified. In particular, the meta-feature encoder forms a meta-feature retrieval space by further mining the RoI, and weights it to the classification head.

[0111] It should be noted that the aforementioned explanation of the embodiment of the small sample target detection method based on multi-view feature fusion and meta-feature coding is also applicable to the small sample target detection device based on multi-view feature fusion and meta-feature coding of this embodiment, and will not be repeated here.

[0112] In order to implement the above embodiment, the present application also proposes a small sample model training method, which is applied to the above image target detection model.

[0113] Figure 7 A flowchart of a small sample model training method provided in an embodiment of the present application.

[0114] like Figure 7 As shown, the small sample target detection method based on multi-view feature fusion and meta-feature encoding includes the following steps:

[0115] Step 701, obtaining training data, wherein the training data includes base class data and new class data, and the amount of new class data is less than that of base class data;

[0116] Step 702, using the base class data to pre-train the image target detection model for the target detection task, using the final classification features and regression features for positioning and classification during pre-training, calculating the loss based on the prediction results and labels, and performing gradient optimization based on the loss to update the model parameters;

[0117] Step 703: fine-tune the image object detection model using the new class data, and keep the parameters of the meta-feature encoder in a frozen state during the fine-tuning training.

[0118] Specifically, the training process is divided into two stages: the first stage is to pre-train the target detection task on a large amount of base class data to learn common features; the second stage is to fine-tune on a small amount of new class data. In the current small sample target detection paradigm, these small amounts of data are often limited to 1, 2, 3, 5, 10, and 30 instances. If the fine-tuning data and the test set only contain new classes, it is the traditional FSOD evaluation method; if the fine-tuning data and the test set contain base classes and new classes, it is the GFSOD evaluation method. This application focuses more on the GFSOD evaluation method.

[0119] Specifically, Figure 8 This is the execution flow chart for pre-training the base class data, such as Figure 8 As shown, including:

[0120] (1) Preprocess the base class images. First, standardize the images and scale the pixel data to a mean of 0 and a variance of 1. Then, randomly select the data augmentation method and augmentation amplitude in the random augmentation pool of the MG module to generate multiple sets of augmented data, which are then stacked and fused.

[0121] (2) Input the preprocessed base class data into the backbone network to obtain the feature map of the base class image data;

[0122] (3) The feature map is fed into the region proposal network to generate foreground and background region proposals, and a series of regions of interest are obtained by combining the original feature map;

[0123] (4) Perform bilinear interpolation pooling on the region of interest to obtain the RoI features of the base class image data with fixed channels and fixed sizes;

[0124] (5) The RoI features of the base class image data are fed into the original network structure to obtain classification features and regression features, and the RoI features are copied and fed into the meta-feature encoder to obtain the meta-features of the base class data and the semantic representation vector of the base class image RoI;

[0125] (6) Fusion of classification features and the semantic representation vector of RoI as the final classification features;

[0126] (7) Use the final classification features and regression features to locate and classify the bounding box, calculate the loss between the prediction results and the labels, perform gradient optimization based on the loss, and update the network parameters.

[0127] Specifically, Fig. 9 The execution flow chart for fine-tuning new types of data is as follows: Fig. 9 As shown in Figure 2, the fine-tuning process is similar to the base class pre-training, with the main differences being:

[0128] (1) At the training data level, under the G-FSOD evaluation method, the training data for the new class fine-tuning process contains a small amount of base class data and a small amount of new class data, and the number of instances of these data is limited to 1, 2, 3, 5, 10, and 30;

[0129] (2) During the model fine-tuning training process, the parameters of the meta-feature encoder are frozen, that is, the features of the meta-feature encoder are not updated. These parameters retain the meta-features of the base class data and effectively retain the base class knowledge. In the forward process, these base class meta-features are fused with the classification features and weighted to the classification head to obtain semantically enhanced classification features. The bounding box classifier and bounding box regressor of the model will classify and locate based on the enhanced classification features and regression features. The network is updated based on the loss of a small amount of sample data to obtain a detector that can have relatively good detection results for both base classes and new classes under the G-FSOD evaluation condition.

[0130] The model training method of this embodiment retains the meta-features of the base class containing semantic features and the meta-feature retrieval space in the first-stage pre-training process of the model through the meta-knowledge encoder; in the second-stage model fine-tuning process, the parameters of the meta-feature encoder are frozen to retain the effective base class knowledge, reduce the training difficulty in the fine-tuning stage, improve the model's overall detection performance for the base class and the new class, and improve the performance under the G-FSOD evaluation conditions.

[0131] In order to implement the above embodiments, the present invention further proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method described in the above embodiments is implemented.

[0132] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0133] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0134] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0135] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0136] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0137] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0138] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0139] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A small sample target detection method based on multi-view feature fusion and meta-feature encoding, characterized in that: The following steps are involved: Perform multi-view embedding fusion on the image data to be detected to obtain fused image data, wherein the fused image data is fused data of image data of different viewpoints corresponding to the image data to be detected; Using a feature extraction network to extract image features from the fused image data, using a region proposal network to determine candidate regions of the image data based on the extracted feature map, and performing RoI pooling on the candidate regions to obtain RoI features; Inputting the RoI features into a fully connected network and a meta-feature encoder respectively, obtaining classification features and regression features output by the fully connected network and meta-features output by the meta-feature encoder, and fusing the meta-features with the classification features to obtain final classification features; Determine the location and classification of the target in the image data through the classifier and regressor according to the regression features and the final classification features; The method of performing multi-view embedding fusion on the image data to be detected to obtain fused image data includes: Performing data augmentation on the image data to be detected to obtain image data of different viewing angles; Performing inter-view feature fusion on the image data of different view angles to obtain preliminary fused image data; Stacking all the preliminarily fused image data in the batch dimension so that the image data of each batch remains independent, thereby obtaining the fused image data; The process of data augmentation is expressed as: Among them, m and n represent the type sequence and amplitude sequence of data augmentation, num_aug represents the data augmentation type, max_amp represents the maximum augmentation amplitude, RandomInts(x,y) represents randomly generating an integer that is greater than or equal to x and less than or equal to y, and RandomFloats(x,y) represents randomly generating a floating-point number that is greater than or equal to x and less than or equal to y; The process of feature fusion between views is expressed as: Among them, λ in represents the weight of each view when the features between views are fused, λ in is a learnable parameter; The process of stacking on the batch dimension is expressed as: in, Represents the stacked data; The RoI feature is input into a meta-feature encoder, and the meta-feature is output, including: The RoI features are respectively subjected to multiple parallel convolutions to form multiple RoI features, and the multiple RoI features are flattened in width and height to obtain multiple groups of feature vectors; Inputting the multiple groups of feature vectors into a multi-head self-attention mechanism to remove redundancy, thereby obtaining features after removing redundancy; The de-redundant features and the learnable features are used to determine the meta-feature retrieval space and the meta-features contained in the retrieval space through a cross-attention mechanism, wherein the learnable features are obtained by random initialization.

2. The method according to claim 1, characterized in that The method of using a region proposal network to determine a candidate region of the image data based on the extracted feature map includes: Inputting the extracted feature map into the region proposal network to obtain candidate region proposals; Determine a candidate region based on the extracted feature map and the candidate region proposal.

3. The method according to claim 1, characterized in that The inputting the multiple groups of feature vectors into the multi-head self-attention mechanism to remove redundancy includes: The multiple groups of feature vectors are input into a multi-head self-attention network, an attention matrix is ​​output, and the attention matrix is ​​input into a feedforward neural network, and the features after de-redundancy are output, wherein the feedforward neural network is composed of a linear layer, a ReLU activation function layer, and a linear layer, and the de-redundancy process is expressed as: Wherein, FFN is a feedforward neural network, d is a feature dimension, the multiple groups of feature vectors include Q, K, V, Q, K, V are transformer feature vector inputs, Q, K, V represent Query, Key, Value respectively, and matmul() represents matrix multiplication.

4. The method according to claim 1, characterized in that The fusing of the meta-features and the classification features to obtain final classification features includes: The meta-features are weighted by defined weights to obtain a semantic representation vector of the RoI; The semantic representation vector of the RoI is superimposed with the classification feature to obtain a final classification feature.

5. A method for constructing a small sample target detection model based on multi-view feature fusion and meta-feature encoding, characterized in that: The small sample target detection model is used to implement the small sample target detection method based on multi-view feature fusion and meta-feature encoding as claimed in claim 1, and the small sample target detection model includes a multi-view embedding fusion module, a feature extraction network, a region proposal network, an R-CNN detection network, a meta-feature encoder, and a classification module. The R-CNN detection network includes an RoI pooling module, wherein: The multi-view embedding and melting module is used to perform multi-view embedding and fusion on the image data to be detected to obtain fused image data, wherein the fused image data is fused data of image data of different viewpoints corresponding to the image data to be detected; The feature extraction network is used to extract image features from the fused image data to obtain a feature map; The region proposal network is used to determine candidate regions of the image data based on the extracted feature map; The R-CNN detection network is used to perform RoI pooling on the candidate region through the RoI pooling to obtain RoI features, and determine classification features and regression features through the RoI features; The meta-feature encoder is used to determine a meta-feature based on the RoI feature; The classification module is used to fuse the meta-features and the classification features to obtain final classification features, and determine the location and classification of the target in the image data according to the regression features and the final classification features through a classifier and a regressor.

6. A small sample model training method, characterized in that: The method is applied to the small sample target detection model as claimed in claim 5, and the method comprises the following steps: Acquire training data, wherein the training data includes base class data and new class data, and the amount of the new class data is less than the amount of the base class data; Pre-training the small sample target detection model for the target detection task using the base class data, performing positioning and classification using the final classification features and regression features during pre-training, calculating the loss based on the prediction results and labels, performing gradient optimization based on the loss, and updating the model parameters; The small sample target detection model is fine-tuned using the new class data, and the parameters of the meta-feature encoder are frozen during the fine-tuning training.

7. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

Images