A target detection method, device, and storage medium based on few-shot learning

Abstract

This application discloses a target detection method, apparatus, and storage medium based on few-shot learning. It relates to the field of image processing technology. Specifically, the method includes: an electronic device using a neural network to extract features from a support image set and a target image, obtaining multiple first feature information of the support image set and second feature information of the target image. Then, the electronic device extracts features from the second feature information based on the multiple first feature information to obtain a target feature map. The electronic device generates third feature information based on the feature vectors of each of the multiple first feature information, where the feature vectors indicate the correlation between the multiple sample images. Finally, the electronic device determines the target in the target image based on the target feature map and the third feature information. This method addresses the problem of low recognition accuracy in existing few-shot learning target detection algorithms.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a target detection method, apparatus and storage medium based on few-shot learning. Background Technology

[0002] Deep learning-based image object detection algorithms typically use large datasets as training sets to ensure the accuracy of the neural network model in identifying targets in random scenes. However, using large datasets for training results in lengthy training times and low model training efficiency. Therefore, this paper proposes an object detection algorithm based on few-shot learning, which uses only a small number of sample datasets to complete model training, thereby improving training efficiency.

[0003] Currently, target detection algorithms based on few-shot learning suffer from severe overfitting when identifying targets, resulting in low accuracy in target detection. Summary of the Invention

[0004] This application provides a target detection method, apparatus, and storage medium based on few-shot learning, which addresses the problem of low recognition accuracy in existing few-shot learning target detection algorithms.

[0005] Firstly, this application provides a target detection method based on few-shot learning. This method can be applied to electronic devices with processing capabilities. Specifically, the method includes: the electronic device using a pre-set neural network to extract features from a support image set and a target image, obtaining multiple first feature information of the support image set and second feature information of the target image. The target image includes a target to be detected, and the support image set includes multiple sample images. The type of the target in each sample image is the same as the type of the target to be detected. One sample image corresponds to one piece of first feature information, and different sample images correspond to different pieces of first feature information. Then, the electronic device extracts features from the second feature information based on the multiple first feature information to obtain a target feature map. The electronic device generates third feature information based on the feature vectors of each of the multiple first feature information, where the feature vectors indicate the correlation between the multiple sample images. Finally, the electronic device determines the target to be detected in the target image based on the target feature map and the third feature information.

[0006] In the embodiments of this application, the feature vectors of multiple first feature information generate third feature information that include the correlation between multiple sample images. That is, when extracting features, the correlation between multiple sample images is considered, thereby utilizing more information between sample images for target recognition, which can improve the accuracy of target recognition.

[0007] Optionally, the first feature information includes feature information in three dimensions. Generating third feature information based on the feature vectors of each of the multiple first feature information pieces involves: performing dimensionality reduction on the multiple first feature information pieces to obtain multiple first feature vectors, with each piece of first feature information corresponding one-to-one with a feature vector; then encoding the multiple first feature vectors to obtain multiple second feature vectors; then generating a third feature vector based on the multiple second feature vectors, where the third feature vector includes feature information obtained by weighted summation of the multiple second feature vectors; and finally, performing dimensionality increase on the third feature vector to obtain the third feature information.

[0008] The first feature information in this embodiment includes multi-dimensional feature information. The electronic device performs dimensionality reduction processing on the multiple first feature information to more accurately extract the correlation between them. The resulting multiple second feature vectors are then weighted and summed to obtain a third feature vector, allowing the correlations among the multiple first feature information to be combined, resulting in more accurate correlation information. Then, the third feature vector obtained based on the correlations is subjected to dimensionality increase processing, enabling it to be used for subsequent target detection. When the multiple second feature vectors are weighted and summed to obtain the third feature vector, a learnable weight coefficient is set for each second feature vector. During model training, this weight coefficient is continuously adjusted based on each training result, ensuring that the final trained model has high accuracy in actual target detection.

[0009] Optionally, encoding multiple first feature vectors to obtain multiple second feature vectors includes: determining the association information between the multiple first feature vectors based on the encoding, and determining multiple second feature vectors based on the association information and the multiple first feature vectors.

[0010] Optionally, feature extraction is performed on the second feature information based on multiple first feature information to obtain a feature map to be detected. This includes: using multiple first feature information as convolution kernels to extract features from the second feature information to obtain multiple fourth feature information, where each of the fourth feature information corresponds one-to-one with the first feature information. Then, the multiple fourth feature information is input into a preset region generation network (RPN) to obtain an initial feature map to be detected. Feature extraction is then performed on the initial feature map to be detected to generate a new feature map to be detected.

[0011] Optionally, the feature map to be detected and the third feature information include channel-dimensional information. Determining the target of the image to be detected based on the feature map to be detected and the third feature map includes: performing pooling processing on the feature map to be detected and the third feature information separately; concatenating the pooled feature map to be detected and the third feature information along the channel dimension; performing a fully connected operation on the concatenated feature map to obtain a first classification result and a first regression result; extracting features from the feature map to be detected using the third feature information as the convolution kernel; performing a fully connected operation on the extracted feature results to obtain a second classification result and a second regression result; concatenating the feature map to be detected and the third feature information along the channel dimension; extracting features from the concatenated feature map to obtain a third classification result and a third regression result; and determining the target of the image to be detected based on the first classification result, the second classification result, the third classification result, the first regression result, the second regression result, and the third regression result.

[0012] In this embodiment, the electronic device processes the feature map to be detected and the third feature map through three different branches, which can more fully identify the feature map to be detected and the third feature map. Finally, the classification result and regression result of the target to be detected are determined through the classification result and regression result of the three different branches, making the identification of the target to be detected more accurate.

[0013] Optionally, the target in the image to be detected is determined based on the first classification result, the second classification result, the third classification result, the first regression result, the second regression result, and the third regression result. This includes: obtaining the classification result of the target by weighted summation of the first classification result, the second classification result, and the third classification result; and obtaining the regression result of the target by weighted summation of the first regression result, the second regression result, and the third regression result.

[0014] In this embodiment, the electronic device sets a weight coefficient for each branch, and can continuously adjust the weight of each branch according to the training results during model training, so that the trained model has higher accuracy when it is actually used to detect targets.

[0015] Optionally, generating a detection bounding map based on an initial detection bounding map includes: extracting features from the initial detection bounding map using residual blocks, performing an interest region alignment operation on the feature extraction results, and obtaining the detection bounding map.

[0016] Secondly, this application provides a target detection device. The device includes an extraction unit, an encoding unit, and a recognition unit. The extraction unit uses a preset neural network to extract features from a support image set and a target image, obtaining multiple first feature information of the support image set and second feature information of the target image. The target image includes a target to be detected, and the support image set includes multiple sample images. The type of the target in each sample image is the same as the type of the target to be detected. One sample image corresponds to one piece of first feature information, and different sample images correspond to different pieces of first feature information. The extraction unit is further used to extract features from the second feature information based on the multiple first feature information to obtain a target feature map. The encoding unit generates third feature information based on the feature vectors of the multiple first feature information, where the feature vectors indicate the correlation between the multiple sample images. The recognition unit determines the target to be detected in the target image based on the target feature map and the third feature information.

[0017] Optionally, the encoding unit is specifically used for: performing dimensionality reduction processing on multiple first feature information to obtain multiple first feature vectors, with each first feature information corresponding one-to-one with a first feature vector; then encoding the multiple first feature vectors to obtain multiple second feature vectors; then generating a third feature vector based on the multiple second feature vectors, the third feature vector including feature information obtained by weighted summation of the multiple second feature vectors; and finally performing dimensionality increase processing on the third feature vector to obtain third feature information.

[0018] Optionally, the encoding unit is specifically used to: determine the association information between multiple first feature vectors based on the encoding, and determine multiple second feature vectors based on the association information and the multiple first feature vectors.

[0019] Optionally, the extraction unit is specifically used to: extract features from the second feature information using multiple first feature information as convolution kernels to obtain multiple fourth feature information, wherein each of the multiple fourth feature information corresponds one-to-one with the multiple first feature information. Then, the multiple fourth feature information is input into a preset region generation network (RPN) to obtain an initial feature map to be detected. Feature extraction is performed on the initial feature map to be detected to generate a new feature map to be detected.

[0020] Optionally, the recognition unit is specifically used for: performing pooling processing on the feature map to be detected and the third feature information respectively; concatenating the pooled feature map to be detected and the third feature information along the channel dimension; performing a fully connected operation on the concatenated feature map to obtain a first classification result and a first regression result; extracting features from the feature map to be detected using the third feature information as the convolution kernel; performing a fully connected operation on the extracted feature results to obtain a second classification result and a second regression result; concatenating the feature map to be detected and the third feature information along the channel dimension; extracting features from the concatenated feature map to obtain a third classification result and a third regression result; and determining the target in the image to be detected based on the first classification result, the second classification result, the third classification result, the first regression result, the second regression result, and the third regression result.

[0021] Optionally, the identification unit is specifically used to: obtain the classification result of the target to be detected by weighted summation of the first classification result, the second classification result, and the third classification result; and obtain the regression result of the target to be detected by weighted summation of the first regression result, the second regression result, and the third regression result.

[0022] Optionally, the extraction unit is specifically used to: extract features from the initial block image to be detected through residual blocks, perform interest region alignment operation on the feature extraction results, and obtain the block image to be detected.

[0023] Thirdly, embodiments of this application provide an electronic device including at least one processor and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method described in any one of the first aspects.

[0024] Fourthly, embodiments of this application provide a computer storage medium storing computer program instructions for performing the method described in any one of the first aspects.

[0025] Fifthly, embodiments of this application also provide a computer program product, which includes computer program code that, when run on a computer, causes the computer to perform the steps described in the first aspect and the various optional steps.

[0026] In a sixth aspect, embodiments of this application also provide a chip, the chip including a processor and an interface, the interface being used to communicate with the processor and receive information from other devices; the processor being used to execute the methods described in the first aspect and any possible implementation thereof. Attached Figure Description

[0027] Figure 1A schematic diagram illustrating a feature extraction process provided in an embodiment of this application;

[0028] Figure 2 A schematic diagram of feature information provided in an embodiment of this application;

[0029] Figure 3 A flowchart illustrating the target detection method based on few-shot learning provided in this application embodiment;

[0030] Figure 4 A schematic diagram illustrating another feature extraction process provided in an embodiment of this application;

[0031] Figure 5 A schematic diagram illustrating another feature extraction process provided in an embodiment of this application;

[0032] Figure 6 This is a schematic diagram of the target detection device provided in the embodiments of this application;

[0033] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0034] This application relates to object detection. It uses an object detection model to extract features from the image to be detected, analyzes the extracted features, and obtains the classification probability distribution of each feature in the image to be detected in order to determine the final object detection result.

[0035] To obtain relatively accurate object detection results, large datasets are typically used as training sets, resulting in long training times and low model training efficiency. When there is limited data corresponding to target types not present in the training set, the accuracy of the model based on this training set in identifying the target is also low. Therefore, this paper proposes training neural network models based on few-shot learning. Few-shot learning does not require a large dataset as a training set, thus eliminating the need for extensive data collection and storage, saving training time. Few-shot learning refers to using a small set of support images as the training set, making it applicable to scenarios with limited sample sizes and resulting in higher model training efficiency. However, current few-shot learning-based object detection algorithms suffer from severe overfitting, leading to low object detection accuracy.

[0036] In view of this, embodiments of this application provide a target detection method based on few-shot learning. This method identifies the target to be detected by supporting the correlation information between multiple sample images in the image set, which can improve the accuracy of target detection.

[0037] The technical solutions provided in the embodiments of this application are described below. For ease of understanding, several neural network models involved in the embodiments of this application are first introduced.

[0038] 1) A feedforward neural network (FFN) is a simple artificial neural network. A feedforward neural network consists of at least two neural network layers, each containing several neurons. In a feedforward neural network, information is transmitted in only one direction.

[0039] 2) A Convolutional Neural Network (CNN) is a feedforward neural network with a convolutional structure. A CNN includes convolutional layers, pooling layers, and fully connected layers. The image to be detected undergoes a series of processing steps through multiple convolutional layers, pooling layers, and fully connected layers, ultimately outputting the recognition result of the target object. A convolutional layer typically includes several feature planes, each composed of rectangularly arranged neural units. Neural units within a feature plane use the same convolutional kernel to process the input image. The kernel contains weights shared by all neural units, meaning that neural units within a feature plane share weights for feature extraction. This shared weights make feature extraction independent of feature location. Convolutional kernels can be initialized as matrices of random size, and during the training process, they can learn appropriate weights. Furthermore, the direct benefit of shared weights is reduced connections between layers in the CNN, thus lowering the risk of overfitting.

[0040] 3) Region proposal networks (RPNs) generate multiple candidate region proposals from the input image to specify the location of the target to be detected. These proposals are used to locate the target, facilitating subsequent target recognition within the candidate region. After the input image is processed by the RPN to generate multiple candidate region proposals, the proposals may differ in size. Therefore, a Region of Interest (ROI) Alignment operation can be used to correct the proposals, ensuring that the final generated proposals have the same size, which is determined by preset parameters of the RPN.

[0041] First, a few-shot learning-based object detection model is constructed. In this embodiment, the training dataset (denoted as the Base dataset) includes multiple labeled data sets (referred to as labeled data). The labeled data can be data from public datasets or historical datasets. To ensure the object detection model can identify more types of objects, the labeled data can include as many labels as possible; for example, the labeled data may include data from the Support Set and the Query Set. It should be understood that the Query Set refers to the dataset uploaded to the server, which can be obtained by querying the server. The Support Set includes multiple types of data (e.g., N classes), and each class includes multiple images (e.g., K images).

[0042] Taking CNN as an example, when building a target detection model based on few-shot learning, a Support Set and a Query Set can be used as inputs to the CNN model for feature extraction. Taking a Support Set containing K images of the target to be detected as an example, feature extraction from the Support Set yields K features of dimension C*H*W, i.e., the feature map of the Support Set (denoted as Fs). Similarly, feature extraction from the Query Set yields its feature map (denoted as Fq). Subsequently, a cross-correlation operation is performed on Fs and Fq to obtain the cross-correlation result Fc. Specifically, Fs is used as the convolution kernel to convolve Fq, resulting in Fc. Then, Fc is input into the RPN network to generate candidate bounding boxes (proposals).

[0043] To better utilize the relationships between data and obtain a more accurate object detection model, this embodiment of the application can perform intra-class relation extraction and feature weighting on the Support Set. Following the example above, Fs undergoes dimensional transformation and reshape operations to obtain K features of dimension d. These K d-dimensional features are combined into a feature sequence S (i.e., S has a feature dimension of K*d), and S is input into a Transformer encoder for encoding. The resulting feature (denoted as S_T, with a dimension of K*d) can characterize the similarity relationship between these K d-dimensional features. Since the K Support images may have different impacts on detection, S_T also characterizes the influence relationship between these K d-dimensional features. When establishing a small-sample-based object detection model in this embodiment of the application, calculating S_T based on Fs allows the established object detection model to utilize the information from the input K Support images to the greatest extent, thereby improving the accuracy of object detection.

[0044] Furthermore, embodiments of this application can also perform feature fusion on the K support image information to extract as much feature information as possible from the K support images, thereby improving the detection capability of general targets. The result of the weighted feature fusion (dimension 1*d) is reshaped to obtain feature 1×C. s ′×H s ′×W s ′, denoted as S_F.

[0045] The above describes how to construct an object detection model based on few-shot learning. After obtaining the object detection model, it can be tested. This embodiment uses a small number of labeled detection datasets (denoted as Noveldataset) to test the model. The Novel dataset contains a small number of labeled detection datasets, and the target types it contains are those not present in the training set. In other words, the Base dataset contains the detection dataset, and the dataset is empty. The Novel dataset also includes a Support Set and a Query Set. This method allows for good target recognition even with a small number of labeled data. After testing the object detection model, target detection, or target recognition, is performed based on this model.

[0046] To facilitate understanding, we will first introduce the relevant concepts of the target detection process.

[0047] Please see Figure 1 This illustrates the process of feature extraction from an image to be detected using a convolutional neural network with convolutional kernels. The image to be detected includes a target object with a shape similar to "X", and the image size is C*H*W. A convolutional kernel of size C′*H′*W′ is used to extract the "\" feature from the image. The feature information obtained through one feature extraction is denoted as C″*H″*W″. This feature information includes the information of the "\" feature in the target object.

[0048] Feature information of size C″*H″*W″, such as Figure 2As shown, this feature information includes numerical information in three dimensions: channel dimension C, high dimension H, and width dimension W. Channel dimension C represents the number of layers in the image to be detected. When the image to be detected has information in three channels (R, G, and B), the number of layers (channel dimension C) is 3. In the high dimension H and the width dimension W, the information of the image to be detected is presented numerically, representing the grayscale values ​​of the image. It can be understood that the image recognition process involves calculating these numerical values ​​to obtain a probability distribution, from which the final classification result can be derived. In convolutional neural networks, a single feature extraction is insufficient to identify the target; multiple rounds of feature extraction are often required to ensure a more accurate final classification by the convolutional neural network.

[0049] This application proposes a target detection method based on few-shot learning, building upon traditional convolutional neural networks.

[0050] Please see Figure 3 This is a flowchart illustrating the target detection method based on few-shot learning provided in this application embodiment. This method can be executed by an electronic device with processing capabilities, such as a personal computer. This application embodiment does not limit the type of neural network used, such as FNN, CNN, etc. Figure 3 The process shown uses CNN as an example.

[0051] S301. The electronic device uses a convolutional neural network (CNN) to extract features from the supporting image set and the image to be detected, thereby obtaining multiple first feature information of the supporting image set and second feature information of the image to be detected.

[0052] The image to be detected includes the target to be detected. Feature extraction from the image to be detected means extracting the features of the target to be detected. The support image set is a set of images including multiple sample images, where the type of the target included in these sample images is the same as the type of the target to be detected. For example, if the target to be detected is a type of tree, and the image to be detected is a forest, then the multiple sample images in the support image set are multiple sample images of that type of tree. Feature extraction from the support image set means extracting the features of the target to be detected from the multiple sample images. This application does not limit the number of sample images included in the support image set; the number of sample images in the support image set can be one or more. Few-shot learning uses the support image set to identify the target to be detected. Compared to traditional convolutional neural network models, this method can maintain the accuracy of target recognition even with a relatively small training set.

[0053] For example, suppose the supporting image set includes K sample images, such as Figure 4As shown, the electronic device uses a weight-sharing CNN to extract features from the supporting image set and the image to be detected, respectively. For ease of description, in this embodiment, the feature information obtained after feature extraction from a sample image is referred to as the first feature information (denoted as Fs), and its dimension is C. s '*H s '*W s 'Then the K sample images correspond to K first feature information Fs. The feature information obtained by the electronic device after feature extraction of the image to be detected is called second feature information, which can be denoted as Fq, and its dimension is C. q '*H q '*W q '.

[0054] S302. The electronic device extracts features from the second feature information based on multiple first feature information to obtain the feature map to be detected.

[0055] See Figure 5 The electronic device extracts features from the second feature information based on K first feature information, and further extracts feature information about the target to be detected from the second feature information. The electronic device can perform a convolution operation on the second feature information of the image to be detected using the K first feature information as convolution kernels, generating K fourth feature information, each of which corresponds one-to-one with a first feature information. The process of performing a convolution operation on the second feature information of the image to be detected using the K first feature information as convolution kernels can be understood as cross-correlating the K first feature information with the second feature information. Since the feature extraction of the second feature information of the image to be detected is based on the first feature information of multiple sample images, and the type of the sample images is the same as the type of the target to be detected in the image to be detected, the generated fourth feature information further highlights the features of the target to be detected.

[0056] The electronic device inputs K fourth feature information into a preset Region Generation Network (RPN) to obtain initial detection bounding boxes. It should be noted that the number and size of the initial detection bounding boxes generated by the RPN are determined by the RPN's preset parameters; that is, the number of initial detection bounding boxes can be one or more, and the size of each initial detection bounding box is also different. The N initial detection bounding boxes generated by the RPN also include feature information of the target object, but feature extraction from these N initial detection bounding boxes is insufficient. Directly performing pooling and fully connected operations on these N initial detection bounding boxes may yield inaccurate recognition results. Therefore, further feature extraction is required from these N initial detection bounding boxes.

[0057] Specifically, the electronic device performs further feature extraction on N initial detection bounding boxes through residual blocks. The sizes of the N detection bounding boxes after feature extraction may be inconsistent. Therefore, after feature extraction, a Region of Interest Align (ROI Align) operation can be performed on the N detection bounding boxes to obtain N detection bounding boxes of the same size, denoted as Q_F.

[0058] S303, the electronic device generates third feature information based on the feature vectors of each of the multiple first feature information, and the feature vectors are used to indicate the correlation between the multiple sample images.

[0059] Since the types of multiple sample images are the same as the type of the target to be detected in the image to be detected, the K first feature information extracted by the pre-defined convolutional neural network (CNN) have a certain correlation. Applying this information representing the correlation to the recognition of the target to be detected will significantly improve the accuracy of the target recognition.

[0060] The first feature information includes a size of C. s '*H s '*W s The electronic device performs dimensionality reduction on the K first feature information to obtain K first feature vectors that correspond one-to-one with the K first feature information. For example, the first feature vector obtained after dimensionality reduction can be represented as [a1, a2, a3... an], that is, the numerical information of the three dimensions in the first feature information is sequentially filled into a two-dimensional sequence to achieve dimensionality reduction of the first feature information, and K first feature vectors [a1, a2, a3... an] are obtained in the same way.

[0061] Since the K first feature vectors are all obtained from K first feature information, they also include the feature information of the K sample images. There is a correlation between the K first feature vectors; the information in the K first feature vectors that represents the correlation between the same features of the sample images is called correlation information. Electronic devices encode the K first feature vectors [a1, a2, a3…an] to extract the correlation information between them.

[0062] Electronic devices can use a transformer encoder to encode K first feature vectors [a1, a2, a3, ..., an]. Of course, there can be various encoding methods, as long as the associated information can be extracted. After the electronic device inputs the K first feature vectors [a1, a2, a3, ..., an] into the transformer encoder, it first generates a feature matrix. Each row of the feature matrix S corresponds to a first eigenvector. Then, the electronic device uses the preset position encoding vector PE and the learnable matrix W... q Wk W v The associated information G is calculated according to the following formula (1):

[0063] Q = (S + PE)W q

[0064] K = (S + PE)W k

[0065] V = (S + PE)W v

[0066]

[0067] The specific form of the position encoding vector PE is as follows: PE (pos,2i) =sin(pos / 10000) 2i / d ), PE (pos,2i+1) =cos(pos / 10000) 2i / d The learnable matrices W_q, W_k, and W_v can be adjusted based on the extraction of association information to find the set of learnable matrices W_q, W_k, and W_v that best extracts association information during model training. The dimensions of matrices Q, K, and V are consistent with the dimensions of the feature matrix S. The softmax function can convert the input into a probability distribution. Therefore, multiplying the Q matrix by the transpose of the K matrix can calculate the feature similarity. Then, the result of the dot product is input into the softmax function, and the obtained association information G can represent the association relationship between the K first eigenvectors in the feature matrix S.

[0068] Then, the electronic device multiplies the associated information G with the matrix V generated from the K first feature vectors to obtain matrix Z, which is equivalent to assigning the extracted associated information G to each first feature vector. Then, matrix Z is input into a preset feedforward neural network FFN to obtain feature matrix S_T. S_T satisfies equation (2):

[0069] Z = GV

[0070] S_T=FFN(Z) (2)

[0071] The resulting feature matrix S_T has the same form as the feature matrix S. Therefore, the feature matrix S_T can represent K second eigenvectors, and each of the K second eigenvectors contains the response values ​​of the other second eigenvectors.

[0072] After obtaining K second feature vectors, the electronic device assigns a learning weight coefficient ζ to each second feature vector according to K preset learning weight coefficients ζ1, ζ2, ζ3...ζk, and performs a weighted summation to obtain a third feature vector. The third feature vector can be represented as [b1, b2, b3...bn]. Then, the electronic device performs dimensionality upscaling based on the third feature information to obtain third feature information, which can be denoted as S_F, with a size of C. s *H s *W s .

[0073] S304. The electronic device determines the target to be detected in the image based on the feature map to be detected and the third feature information.

[0074] A feature map to be detected has a dimension of C. Q *H Q *W Q Then the dimension of the N feature maps Q_F to be detected is N*C. Q *H Q *W Q The size of the third feature information S_F is C. s *H s *W s Since the size of the feature map to be detected is determined by the preset parameters of the Region Generation Network (RPN), the parameters can be set so that the feature maps Q_F and S_F to be detected have the same size in the channel dimension C, height dimension H, and width dimension W.

[0075] In this embodiment, the extracted features Q_F and S_F are input into the first, second, and third branches respectively to obtain the target detection results. These three branches can be a global branch, a local branch, and a deep semantic branch. The three different branches are used for feature extraction at different levels, and each branch outputs classification and regression results. Combining the classification and regression results of the three branches determines the final classification and regression results of the target to be detected, making the final result more accurate. There is no specific order in which the three branches are extracted, and they can be combined in any way, as long as the classification and regression effects of these three branches are achieved.

[0076] The electronic device obtains the first classification result and the first regression result through the first branch. The electronic device performs pooling processing on the N feature maps Q_F to be detected and the third feature information S_F respectively, obtaining the N pooled feature maps Q_F to be detected, whose dimension information is N*C. Q *1*1, obtain the third feature information S_F after pooling, whose dimension information is C. s*1*1. Then, the electronic device concatenates the pooled N feature maps Q_F to be detected with the pooled third feature information S_F along the channel dimension C. The dimension information of the concatenated feature map is N*(C). Q +C s Then, a fully connected operation is performed on the concatenated feature map to obtain the first classification result and the first regression result. The first classification result has N*2 dimensionality information, and the first regression result has N*4 dimensionality information.

[0077] The electronic device obtains the second classification and second regression results through the second branch. The electronic device performs convolution operations on N feature maps Q_F to be detected using the third feature information S_F as the convolution kernel, making the size of the feature map Q_F in the channel dimension C, height dimension H, and width dimension W all equal to 1, thus obtaining a feature map with dimensionality N*1*1*1. This feature map is then fully connected to obtain the second classification and second regression results. Similarly, the second classification result has dimensionality N*2, and the second regression result has dimensionality N*4.

[0078] The electronic device obtains the third classification and third regression results through the third branch. The electronic device concatenates the N feature maps Q_F to be detected and the third feature information S_F along the channel dimension C to obtain the concatenated feature map, whose dimension information is N*(C). Q +C s )*(H Q )*(W Q ) or N*(C Q +C s )*(H s )*(W s The concatenated feature map is then processed using a pre-defined convolutional neural network to extract features, resulting in a feature map with dimensionality N*(C). Q +C s Then perform a full connection operation (N*1) to obtain the third classification result and the third regression result. The third classification result has N*2 dimensionality information, and the third regression result has N*4 dimensionality information.

[0079] The electronic device performs a weighted summation of the first, second, and third classification results based on the set classification weighting coefficients α1, α2, and α3 to obtain the final classification result of the target to be detected. Similarly, the electronic device performs a weighted summation of the first, second, and third regression results based on the set regression weighting coefficients β1, β2, and β3 to obtain the final regression result of the target to be detected.

[0080] In this embodiment, different classification and regression results are weighted by setting classification weighting coefficients and regression weighting coefficients. The classification weighting coefficients and regression weighting coefficients can be continuously adjusted during model training to determine the optimal classification weighting coefficients and regression weighting coefficients based on the training data, so as to obtain more accurate classification results.

[0081] The embodiments of this application can also include the loss (cls) of the classification result of the aforementioned target detection method. loss ) and the loss of the regression results (reg) loss The learnable weight coefficients α and β are configured separately, and then the weighted sum is obtained to obtain the final loss of the model in recognizing the target. loss total loss Satisfy the following formula:

[0082] total loss =α*cls loss +β*reg loss

[0083] Among them, cls loss Using binary classification cross-entropy loss, while reg loss Using smooth L1 loss, the weight coefficients α and β can be learned to indicate the proportion of the classification result loss to the regression result loss. The values ​​of α and β can be adjusted according to the actual situation during model training. The final loss, total, is obtained. loss Then, backpropagation is performed to update the model parameters, and finally, the accuracy of the model in identifying the target to be detected is improved through continuous training.

[0084] Please see Figure 6 Based on the same inventive concept, this application also provides a target detection device 600. The target detection device 600 includes an extraction unit 601, an encoding unit 602, and a recognition unit 603. The extraction unit 601 is used to extract features from a support image set and a target image using a preset neural network, obtaining multiple first feature information of the support image set and second feature information of the target image. The target image includes a target to be detected, and the support image set includes multiple sample images. The type of the target in each sample image is the same as the type of the target to be detected. One sample image corresponds to one first feature information, and different sample images correspond to different first feature information. The extraction unit 601 is also used to extract features from the second feature information based on the multiple first feature information to obtain a target feature map. The encoding unit 602 is used to generate third feature information based on the feature vectors of the multiple first feature information, where the feature vectors indicate the correlation between the multiple sample images. The recognition unit 603 is used to determine the target to be detected in the target image based on the target feature map and the third feature information.

[0085] Optionally, the encoding unit 602 is specifically used for: performing dimensionality reduction processing on multiple first feature information to obtain multiple first feature vectors, wherein each of the multiple first feature information corresponds one-to-one with the multiple first feature vectors; then, encoding the multiple first feature vectors to obtain multiple second feature vectors; then, generating a third feature vector based on the multiple second feature vectors, wherein the third feature vector includes feature information obtained by weighted summation of the multiple second feature vectors; and finally, performing dimensionality increase processing on the third feature vector to obtain third feature information.

[0086] Optionally, the encoding unit 602 is specifically used to: determine the association information between multiple first feature vectors according to the encoding, and determine multiple second feature vectors according to the association information and the multiple first feature vectors.

[0087] Optionally, the extraction unit 601 is specifically used to: extract features from the second feature information using multiple first feature information as convolution kernels to obtain multiple fourth feature information, wherein the multiple fourth feature information corresponds one-to-one with the multiple first feature information. Then, the multiple fourth feature information is input into a preset region generation network (RPN) to obtain an initial feature map to be detected. Feature extraction is performed on the initial feature map to be detected to generate a new feature map to be detected.

[0088] Optionally, the recognition unit 603 is specifically used for: performing pooling processing on the feature map to be detected and the third feature information respectively; concatenating the pooled feature map to be detected and the third feature information in the channel dimension; performing a fully connected operation on the concatenated feature map to obtain a first classification result and a first regression result; extracting features from the feature map to be detected using the third feature information as the convolution kernel; performing a fully connected operation on the feature extraction result to obtain a second classification result and a second regression result; concatenating the feature map to be detected and the third feature information in the channel dimension; extracting features from the concatenated feature map to obtain a third classification result and a third regression result; and determining the target of the image to be detected based on the first classification result, the second classification result, the third classification result, the first regression result, the second regression result, and the third regression result.

[0089] Optionally, the identification unit 603 is specifically used to: obtain the classification result of the target to be detected by weighted summation of the first classification result, the second classification result, and the third classification result; and obtain the regression result of the target to be detected by weighted summation of the first regression result, the second regression result, and the third regression result.

[0090] Optionally, the extraction unit 601 is specifically used to: extract features from the initial block image to be detected through residual blocks, perform interest region alignment operation on the feature extraction results, and obtain the block image to be detected.

[0091] Please see Figure 7Based on the same inventive concept, this application provides an electronic device 700, which includes at least one processor 701, at least one memory 702, and computer program instructions stored in the memory. When the computer program instructions are executed by the processor, the aforementioned gate motor driving method is implemented.

[0092] Optionally, the processor 701 may be a central processing unit, an application-specific integrated circuit (ASIC), one or more integrated circuits for controlling program execution, a hardware circuit developed using a field-programmable gate array (FPGA), or a baseband processor.

[0093] Optionally, the read-write lock operating device further includes a memory 702 connected to at least one processor 701. The memory 702 may include read-only memory (ROM), random access memory (RAM), and disk storage. The memory 702 is used to store data required for the processor 701 to run. There may be one or more memories 702. The memory 702 is used in… Figure 7 It is shown together, but it should be noted that the memory 702 is not a mandatory functional module, therefore in Figure 7 It is shown in dashed lines.

[0094] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the aforementioned gate motor driving method.

[0095] In specific implementation, computer-readable storage media include: Universal Serial Bus flash drive (USB), portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other storage media that can store program code.

[0096] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0097] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0098] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0099] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0100] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: Universal Serial Bus flash disks, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0101] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A target detection method based on few-shot learning, characterized in that, include: A neural network is used to extract features from a support image set and a target image to obtain multiple first feature information of the support image set and second feature information of the target image. The target image includes a target to be detected, and the support image set includes multiple sample images. The type of the target in each sample image is the same as the type of the target to be detected. One sample image corresponds to one first feature information, and the first feature information corresponding to different sample images is different. Based on the multiple first feature information, feature extraction is performed on the second feature information to obtain the feature map to be detected; Third feature information is generated based on the feature vectors of each of the plurality of first feature information, wherein the feature vectors are used to indicate the correlation between the plurality of sample images; The target to be detected in the image to be detected is determined based on the feature map to be detected and the third feature information. Wherein, the feature map to be detected and the third feature information include channel dimension information, and determining the target to be detected in the image to be detected based on the feature map to be detected and the third feature information includes: Pooling is performed on the feature map to be detected and the third feature information respectively. The pooled feature map to be detected and the third feature information are then concatenated along the channel dimension. A fully connected operation is then performed on the concatenated feature map to obtain the first classification result and the first regression result. The third feature information is used as the convolution kernel to extract features from the feature map to be detected. The result of feature extraction is then subjected to a fully connected operation to obtain the second classification result and the second regression result. The feature map to be detected and the third feature information are concatenated along the channel dimension, and feature extraction is performed on the concatenated feature map to obtain the third classification result and the third regression result; The target to be detected in the image to be detected is determined based on the first classification result, the second classification result, the third classification result, the first regression result, the second regression result, and the third regression result.

2. The method as described in claim 1, characterized in that, The first feature information includes feature information in three dimensions. A third feature information is generated based on the feature vectors of each of the multiple first feature information items, including: The plurality of first feature information is subjected to dimensionality reduction processing to obtain a plurality of first feature vectors, wherein the plurality of first feature information and the plurality of first feature vectors correspond one-to-one; Encode the plurality of first feature vectors to obtain a plurality of second feature vectors; A third feature vector is generated based on the plurality of second feature vectors, wherein the third feature vector includes feature information obtained by weighted summation of the plurality of second feature vectors; The third feature vector is subjected to dimensionality increase processing to obtain the third feature information.

3. The method as described in claim 2, characterized in that, Encoding the plurality of first feature vectors yields a plurality of second feature vectors, including: The association information among the plurality of first feature vectors is determined based on the encoding; The plurality of second feature vectors are determined based on the association information and the plurality of first feature vectors.

4. The method as described in claim 1, characterized in that, Based on the multiple first feature information, feature extraction is performed on the second feature information to obtain a feature map to be detected, including: The second feature information is extracted using the multiple first feature information as convolution kernels to obtain multiple fourth feature information, and the multiple fourth feature information corresponds one-to-one with the multiple first feature information. The multiple fourth feature information is input into a preset region generation network (RPN) to obtain an initial feature map to be detected. Feature extraction is performed on the initial feature map to be detected to generate the new feature map.

5. The method as described in claim 1, characterized in that, Determining the target to be detected in the image based on the first classification result, the second classification result, the third classification result, the first regression result, the second regression result, and the third regression result includes: The classification result of the target to be detected is obtained by weighted summation of the first classification result, the second classification result, and the third classification result; The regression result of the target to be detected is obtained by weighted summation of the first regression result, the second regression result, and the third regression result.

6. The method as described in claim 4, characterized in that, Generating the detection block diagram based on the initial detection block diagram includes: The initial block diagram to be detected is subjected to feature extraction through residual blocks, and the result of feature extraction is subjected to region of interest alignment to obtain the block diagram to be detected.

7. A target detection device, characterized in that, include: The extraction unit is used to extract features from a support image set and a target image using a neural network to obtain multiple first feature information of the support image set and second feature information of the target image. The target image includes a target to be detected, the support image set includes multiple sample images, the type of the target in each sample image is the same as the type of the target to be detected, one sample image corresponds to one first feature information, and the first feature information corresponding to different sample images is different. The extraction unit is further configured to extract features from the second feature information based on the plurality of first feature information to obtain a feature map to be detected; An encoding unit is configured to generate third feature information based on the feature vectors of each of the plurality of first feature information, wherein the feature vectors are used to indicate the correlation between the plurality of sample images; The recognition unit is configured to determine the target to be detected in the image to be detected based on the feature map to be detected and the third feature information; wherein the feature map to be detected and the third feature information include information of the channel dimension; When the recognition unit determines the target to be detected in the image to be detected based on the feature map to be detected and the third feature information, it specifically performs the following steps: pooling processing on the feature map to be detected and the third feature information respectively; concatenating the pooled feature map to be detected and the third feature information in the channel dimension; performing a fully connected operation on the concatenated feature map to obtain a first classification result and a first regression result; extracting features from the feature map to be detected using the third feature information as a convolution kernel; performing a fully connected operation on the extracted feature results to obtain a second classification result and a second regression result; concatenating the feature map to be detected and the third feature information in the channel dimension; extracting features from the concatenated feature map to obtain a third classification result and a third regression result; and determining the target to be detected in the image to be detected based on the first classification result, the second classification result, the third classification result, the first regression result, the second regression result, and the third regression result.

8. An electronic device, characterized in that, include: At least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by a processor, the method described in any one of claims 1-6 is implemented.

Images