Small sample image target detection method, model training method and device

Abstract

The invention discloses a small sample image target detection method and a model training method and device. The training method comprises the following steps: acquiring image data; inputting the image data into a small sample image target detection model to perform feature extraction and target detection on the image data to obtain prediction probability distribution of the image data and corresponding small sample target detection loss; performing style migration on the image data, and performing feature extraction and target detection on the image data after the style migration to obtain prediction probability distribution of the image data after the style migration and corresponding small sample target detection loss; and determining the total loss of training according to the prediction probability distribution of the image data before and after the style migration and the corresponding small sample target detection loss, and optimizing the parameters of the small sample image target detection model according to the total loss.

Description

Small sample image target detection method, model training method and device Technical Field

[0001] The embodiments of the present disclosure relate to, but are not limited to, the field of machine learning technology, and in particular to a small sample image target detection method, a model training method, and a device. Background Art

[0002] Object detection is a core task in computer vision, aiming to locate and identify objects in images. With the rapid development of machine learning, object detection has achieved tremendous success in recent years and has been successfully applied to numerous fields, including autonomous driving, land exploration, and smart healthcare.

[0003] However, current deep learning-based object detection methods often rely on large amounts of labeled data to train the model, requiring significant human resources and time. In real-world applications, it is difficult to obtain and collect large quantities of relevant images for many specific tasks. In this case, traditional methods are prone to overfitting, resulting in a large number of missed detections. To alleviate this problem, small-sample object detection methods have been gradually proposed to reduce the data dependence of the model. These methods can detect new categories of objects even with limited sample sizes. Despite this, most existing small-sample object detection methods assume that the target domain data, which contains a small number of samples, and the source domain data have the same statistical distribution, that is, they have similar scenes. However, the reality is that samples in the target domain are not only difficult to obtain but also have different domain distributions. The degradation of fine-grained features of samples caused by scene switching and significant morphological changes has brought huge challenges to the field of computer vision.

[0004] Summary of the Invention

[0005] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0006] The disclosed embodiments also provide a training method for a small-sample image target detection model, comprising: obtaining image data; inputting the image data into the small-sample image target detection model to perform feature extraction and target detection on the image data to obtain a predicted probability distribution of the image data and a corresponding small-sample target detection loss; performing style transfer on the image data, and performing feature extraction and target detection on the image data after style transfer to obtain a predicted probability distribution of the image data after style transfer and a corresponding small-sample target detection loss; determining the total loss of training based on the predicted probability distribution of the image data before and after style transfer and the corresponding small-sample target detection loss, and optimizing the parameters of the small-sample image target detection model based on the total loss.

[0007] An embodiment of the present disclosure also provides a training device for a small sample image target detection model, comprising a memory; and a processor connected to the memory, wherein the memory is used to store instructions, and the processor is configured to execute the steps of the training method for the small sample image target detection model described in any embodiment of the present disclosure based on the instructions stored in the memory.

[0008] An embodiment of the present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it implements the training method of the small sample image target detection model described in any embodiment of the present disclosure.

[0009] An embodiment of the present disclosure also provides a training device for a small sample image target detection model, comprising an image acquisition module, a first detection module, a second detection module and a self-supervision module; wherein: the image acquisition module is configured to acquire image data; the first detection module is configured to input the image data into the small sample image target detection model to perform feature extraction and target detection on the image data, and obtain a predicted probability distribution of the image data and a corresponding small sample target detection loss; the second detection module is configured to perform style transfer on the image data, and perform feature extraction and target detection on the image data after style transfer, and obtain a predicted probability distribution of the image data after style transfer and a corresponding small sample target detection loss; the self-supervision module is configured to determine the total loss of training based on the predicted probability distribution of the image data before and after style transfer and the corresponding small sample target detection loss, and optimize the parameters of the small sample image target detection model based on the total loss.

[0010] An embodiment of the present disclosure also provides a small sample image target detection method, including: obtaining a trained small sample image target detection model, wherein the small sample image target detection model is trained using the training method of the small sample image target detection model described in any embodiment of the present disclosure; inputting the image to be identified into the trained small sample image target detection model to obtain an image recognition result.

[0011] An embodiment of the present disclosure also provides a small sample image target detection device, comprising a memory; and a processor connected to the memory, wherein the memory is used to store instructions, and the processor is configured to execute the steps of the small sample image target detection method as described in any embodiment of the present disclosure based on the instructions stored in the memory.

[0012] An embodiment of the present disclosure also provides a small sample image target detection device, comprising a storage module and an input module; wherein: the storage module is configured to store a trained small sample image target detection model, and the small sample image target detection model is trained using the training method of the small sample image target detection model as described in any embodiment of the present disclosure; the input module is configured to input the image to be recognized into the trained small sample image target detection model to obtain an image recognition result.

[0013] An embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the small sample image target detection method as described in any embodiment of the present disclosure.

[0014] Other aspects will become apparent upon reading and understanding the drawings and detailed description.

[0015] Summary of the Figures

[0016] The accompanying drawings are intended to provide a further understanding of the technical solutions of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solutions of the present disclosure and do not constitute a limitation of the technical solutions of the present disclosure. The shapes and sizes of the components in the drawings do not reflect the actual scale and are intended only to illustrate the contents of the present disclosure.

[0017] FIG1 is a flow chart of a method for training a small sample image target detection model according to an exemplary embodiment of the present disclosure;

[0018] FIG2 is a schematic diagram of the structure of a small sample image target detection model provided by an exemplary embodiment of the present disclosure;

[0019] FIG3 is a schematic structural diagram of a feature extractor provided by an exemplary embodiment of the present disclosure;

[0020] FIG4 is a schematic diagram of the structure of a feature extraction block provided by an exemplary embodiment of the present disclosure;

[0021] FIG5A is a schematic structural diagram of a training model for a small sample image target detection model provided by an exemplary embodiment of the present disclosure;

[0022] FIG5B is a schematic structural diagram of a style transfer module provided by an exemplary embodiment of the present disclosure;

[0023] FIG6 is a schematic structural diagram of a training device for a small sample image target detection model provided by an exemplary embodiment of the present disclosure;

[0024] FIG7A is a schematic diagram of a flow chart of a method for detecting an object in a small sample image according to an exemplary embodiment of the present disclosure;

[0025] FIG7B is a schematic structural diagram of a small sample image target detection device provided by an exemplary embodiment of the present disclosure;

[0026] FIG8 is a schematic structural diagram of another training device for a small sample image target detection model provided by an exemplary embodiment of the present disclosure;

[0027] FIG9 is a schematic structural diagram of another small sample image target detection device provided by an exemplary embodiment of the present disclosure.

[0028] Details

[0029] To make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other in any manner.

[0030] Unless otherwise defined, the technical or scientific terms used in the embodiments of the present disclosure should have the ordinary meaning understood by people with ordinary skills in the field to which the present disclosure belongs. The words "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. The words "include" or "comprising" and similar words mean that the elements or objects preceding the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0031] As shown in FIG1 , an embodiment of the present disclosure provides a training method for a small sample image target detection model, including:

[0032] Step 101: Acquire image data;

[0033] Step 102: Input the image data into a small sample image target detection model to perform feature extraction and target detection on the image data, thereby obtaining a predicted probability distribution of the image data and a corresponding small sample target detection loss.

[0034] Step 103: Perform style transfer on the image data, and perform feature extraction and object detection on the style-transferred image data to obtain a predicted probability distribution of the style-transferred image data and a corresponding small-sample object detection loss;

[0035] Step 104: Determine the total loss of training based on the predicted probability distribution of the image data before and after style transfer and the corresponding small sample object detection loss, and optimize the parameters of the small sample image object detection model based on the total loss.

[0036] In the embodiment of the present disclosure, during the training process of the small sample image target detection model, multiple rounds of training processes need to be iterated. The multiple rounds of training processes are achieved by performing style transfer on one or more channels of image data and performing consistency constraints on the target detection results of the image data before style transfer and the target detection results of the image data after style transfer.

[0037] In the embodiments of the present disclosure, style transfer refers to a technique of applying the style of other images to the current image while preserving the content of the current image as much as possible. In this case, the current image before style transfer can be called the source image, the other image can be called the style image, and the current image after style transfer can be called the generated image.

[0038] Since in cross-domain situations, the differences between source domain data and target domain data are mainly reflected in different image styles, the embodiments of the present disclosure transfer the style of the style image to the source image, so that the model can be exposed to more images with diverse styles during the training phase, thereby enhancing the robustness of the model to images of different styles, improving the migration capability from the source domain to the target domain, and ultimately improving the recognition capability in the target domain during the testing phase.

[0039] In the embodiment of the present disclosure, the source domain refers to the source of the data and model we have. During the training phase, we use the data from the source domain to train the model so that it has a certain generalization ability. The target domain is where we want to apply the model. In practical applications, we often need to migrate the trained model to the target domain and implement tasks such as prediction or classification. The target domain may have certain differences from the source domain, such as changes in data distribution, missing features, etc. These differences will have a negative impact on the performance of the model, resulting in the performance in the target domain not being as good as in the source domain. The multi-round training process of the embodiment of the present disclosure performs style transfer on one or more images of the source domain, and implements consistency constraints on the target detection results of the image before style transfer and the target detection results of the image after style transfer.

[0040] In some exemplary embodiments, as shown in FIG2 , the small sample image target detection model may include: a feature extractor and a small sample target detector, wherein the feature extractor is configured to perform feature extraction on the input image and input the feature extraction result into the small sample target detector; and the small sample target detector is configured to perform target detection on the feature extraction result to obtain a target detection result.

[0041] In some exemplary embodiments, the feature extractor may be any image feature extraction network.

[0042] For example, the feature extractor can be a Vision Transformer (ViT) model. ViT is a novel neural network architecture that treats images as sequential data, captures spatial and temporal information in images through a self-attention mechanism, and captures positional information in images through position encoding. It has achieved excellent results in the field of computer vision. However, the disclosed embodiments are not limited to this, and the structure of the feature extractor can be configured as needed.

[0043] In some exemplary embodiments, as shown in FIG3 , the feature extractor may include an embedding layer, a feature encoding layer, and a fully connected layer connected in sequence, wherein the embedding layer is configured to convert an input image into a vector sequence; the feature encoding layer is configured to extract feature vectors from the vector sequence; and the fully connected layer is configured to integrate the feature vectors extracted by the feature encoding layer.

[0044] In some exemplary embodiments, the feature encoding layer includes a plurality of sequentially connected feature extraction blocks, and each feature extraction block has the same structure.

[0045] In some exemplary embodiments, the structure of each feature extraction block can be an encoder structure based on a transformer (a deep learning model based on a self-attention mechanism). Exemplarily, as shown in FIG4 , the structure of each feature extraction block can include: a normalization (Norm) layer, a multi-head attention (Multi-Head Attention) layer, and a multilayer perceptron (MLP) layer, wherein the normalization layer is configured to normalize the input vector; the multi-head attention layer is configured to divide the normalized input vector into multiple parts and perform independent attention calculations on each part; and the multilayer perceptron layer is configured to map multiple input data sets to a single output data set.

[0046] In some exemplary embodiments, the small sample target detector can be any small sample target detection network based on distance comparison. Exemplarily, the small sample target detector can be a graph neural network (GNN), however, the embodiments of the present disclosure are not limited to this. For input data, the small sample target detector can predict the corresponding class probability distribution P and the corresponding loss function L.

[0047] In some exemplary embodiments, performing style transfer on image data includes:

[0048] Randomly sample M channels of image data from the source domain data, where M is a natural number greater than or equal to 2, and each channel of image data includes b frames of image data, where b is a natural number greater than or equal to 1;

[0049] The following operation is performed on at least one of the M paths of image data: one or more images from the other paths of image data or pre-specified image data are selected as style images, and the style of the selected style image is transferred to one or more image data in the current path of image data to obtain one or more generated images, where the style of the generated image is determined by the low-frequency component of the style image after wavelet transformation.

[0050] In some exemplary embodiments, M=2, and the image data in the first path of image data and the image data in the second path of image data are used as source images and style images for each other (i.e., each path of image data is a style image of an image in the other path of image data except for its own path of image data), thereby obtaining multiple generated images. Specifically, the image in the first path of image data is used as the source image, and the image in the second path of image data is used as the style image, and the style of the image in the second path of image data is transferred to the image in the first path of image data, thereby obtaining one or more generated images; at the same time, the image in the second path of image data is used as the source image, and the image in the first path of image data is used as the style image, and the style of the image in the first path of image data is transferred to the image in the second path of image data, thereby obtaining one or more generated images.

[0051] In the embodiment of the present disclosure, the two paths of image data refer to two batches of data sampled from the same source domain training set using different random number seeds. The sampling sets are the same, and the sampling process is completely random.

[0052] In some exemplary embodiments, each channel of image data follows an N-way-K-shot model, where N represents the number of categories, K represents the number of labeled samples in each category as a support set, and each category also includes a portion of labeled samples as a validation set. The model is trained using the data in the support set and evaluated using the data in the validation set.

[0053] In large-scale supervised learning methods, the training and test sets are mixed and then randomly split proportionally, ensuring that the data distribution of the training and test sets is consistent. For classification problems, for example, the categories in the training and test sets are identical after the split, differing only in the number of samples. However, in the small-sample domain, the training and test sets are not interleaved, meaning that categories in the test set do not appear in the training set. This is because small-sample object detection models require the ability to learn to learn. When presented with an unknown category, the model can summarize the characteristics of that category with only a few samples. Therefore, categories in the test set cannot appear in the training set.

[0054] The small-shot object detection problem typically divides the target dataset into categorically disjoint base classes and novel classes. The base class data has ample labeled data and original images, while the novel classes have only a small number of available samples. This problem can be viewed as learning generalized knowledge from the base class data to identify novel class objects. This problem is also known as the N-way-K-shot problem, where each class is defined to contain K labeled instances. The value of N varies depending on the task, and K is often chosen to be 1, 2, 3, 5, 10, or 30.

[0055] In order to simulate the small sample size in the test phase, the training set and the validation set can also be sampled in the same N-way-K-shot format. In each round of training, N categories are sampled from the training set, and K+Q samples are sampled for each category. These K+Q samples are then divided into: Support Set and Query Set, where the Support Set contains N categories, K samples for each category, and the Query Set contains N categories, Q samples for each category. Both the Support Set and the Query Set come from the training set, and both contain the same categories, but the number of samples in each category can be different. The Support Set is used to train and learn the model, while the Query Set is used to evaluate and verify the model.

[0056] In some exemplary embodiments, N is 5 and K is 1. However, the embodiments of the present disclosure are not limited thereto, and the values ​​of N and K can be set as needed.

[0057] In some exemplary embodiments, the training method of the small sample image object detection model includes: iterating multiple rounds of training until the model converges, each round of training including:

[0058] Randomly sampling one or more channels of image data in the source domain, each channel of image data includes one or more images;

[0059] Perform style transfer on one or more image data;

[0060] Check whether the model meets the convergence condition, where the convergence condition includes: the object detection results of the image before style transfer and the object detection results of the image after style transfer meet the consistency constraint condition;

[0061] When the model does not meet the convergence conditions, the model parameters are optimized.

[0062] The disclosed embodiments perform style transfer on one or more channels of image data, allowing the model to be exposed to more images with diverse styles during the training phase, thereby enhancing the model's robustness to images of different styles, improving the ability to transfer from the source domain to the target domain, and ultimately improving the recognition ability in the target domain during the testing phase.

[0063] In some exemplary embodiments, the training method of the small sample image object detection model includes:

[0064] A style transfer module is provided after at least one feature extraction block to obtain a training model of a small sample image object detection model, wherein the style transfer module is used to transfer the style of another channel of image data to the current channel of image data;

[0065] Iterate multiple rounds of training until the model converges. Each round of training includes:

[0066] Randomly sample two channels of image data from the source domain data;

[0067] The two image data are input into the small sample image target detection model respectively to obtain the predicted probability distribution of each image data and the corresponding small sample target detection loss;

[0068] The two image data are input into the training model of the small sample image object detection model respectively to obtain the predicted probability distribution after style transfer of each image data and the corresponding small sample object detection loss;

[0069] Calculate the corresponding consistency loss based on the predicted probability distribution of each image data channel and the predicted probability distribution of each image data channel after style transfer;

[0070] The overall target loss is calculated based on the consistency loss corresponding to each image data, the small sample target detection loss, and the small sample target detection loss after style transfer. When the overall target loss does not meet the model convergence conditions, the model parameters are optimized.

[0071] In the embodiment of the present disclosure, during each round of training, two paths of image data may be randomly sampled from the source domain data, or more than two paths of image data (such as three paths, four paths, etc.) may be randomly sampled from the source domain data. The embodiment of the present disclosure does not impose any restrictions on this.

[0072] In the embodiment of the present disclosure, during each round of training, multiple channels of randomly sampled image data can be input into the small sample image target detection model and its training model respectively, or only one or part of the channels of image data can be input into the small sample image target detection model and its training model respectively. The embodiment of the present disclosure does not impose any restrictions on this.

[0073] In some exemplary embodiments, when K is greater than 1, that is, the number of annotated samples included in each category as a support set is greater than 1, the first image in the first channel of image data and the first image in the second channel of image data can be made to serve as each other's source image and style image, the second image in the first channel of image data and the second image in the second channel of image data can be made to serve as each other's source image and style image, and so on. The Kth image in the first channel of image data and the Kth image in the second channel of image data can be made to serve as each other's source image and style image. However, the presently disclosed embodiments are not limited to this.

[0074] In other exemplary embodiments, the first image in the first path of image data and all images in the second path of image data may be respectively organized into K groups, with the two images in each group being each other's source image and style image; the second image in the first path of image data and all images in the second path of image data may be respectively organized into K groups, with the two images in each group being each other's source image and style image... The Kth image in the first path of image data and all images in the second path of image data may be respectively organized into K groups, with the two images in each group being each other's source image and style image.

[0075] In some exemplary embodiments, the plurality of feature extraction blocks includes X feature extraction blocks connected in sequence, where X is a natural number greater than or equal to 2, and at least one style transfer module is disposed after one or more feature extraction blocks from the first feature extraction block to the X-1th feature extraction block, wherein the first feature extraction block to the X-1th feature extraction block are disposed according to the data flow direction. Because the features extracted by the last feature extraction block are too semantically related, style transfer based on such features generally does not bring benefits. Therefore, the present disclosure does not provide a style transfer module after the last feature extraction block.

[0076] In some exemplary embodiments, the style transfer modules include X-1, the i-th style transfer module is arranged between the i-th feature extraction block and the i+1-th feature extraction block, and i is a natural number between 1 and X-1.

[0077] For example, as shown in FIG5A , where X=4, there are four feature extraction blocks: feature extraction block 1, feature extraction block 2, feature extraction block 3, and feature extraction block 4. There are three style transfer modules: style transfer module 1, style transfer module 2, and style transfer module 3. Style transfer module 1 is located between feature extraction block 1 and feature extraction block 2, style transfer module 2 is located between feature extraction block 2 and feature extraction block 3, and style transfer module 3 is located between feature extraction block 3 and feature extraction block 4. The parameters of the feature extractors and small-sample object detectors in both paths are identical. In other examples, the number of feature extraction blocks and style transfer modules can be set as needed.

[0078] In some exemplary embodiments, as shown in FIG5B , the style transfer module includes a forward wavelet transform module, a style exchange module, and an inverse wavelet transform module;

[0079] A forward wavelet transform module is configured to split the input feature vector into high-frequency components and low-frequency components;

[0080] a style exchange module configured to replace the low-frequency component of the data of the current channel with the low-frequency component of the data of the other channel;

[0081] The inverse wavelet transform module is configured to recombine the high frequency component and the replaced low frequency component into a feature vector.

[0082] Since the style information has been extracted in the low-frequency components, the style exchange module of the embodiment of the present disclosure only acts on the low-frequency components of the image. The style transfer module provided by the embodiment of the present disclosure first uses the forward wavelet transform module to decompose the high- and low-frequency components of the overall visual features extracted by the feature extraction block; then uses the style exchange module to exchange the style information of the various low-frequency components of the two-way data to obtain the low-frequency components after style transfer; finally, uses the inverse wavelet transform module to reconstruct the high-frequency components and the low-frequency components after style transfer, and the reconstructed features are used as input for subsequent feature extraction blocks. This process is repeated in sequence after feature extraction block 1, feature extraction block 2, and feature extraction block 3 until the data passes through feature extraction block 4 and the last fully connected layer to obtain the final feature representation of style transfer.

[0083] Since the feature vectors extracted by the feature extraction blocks at the same position in the multi-channel data have the same size, the i-th style exchange module in each channel replaces the low-frequency components split out by the i-th forward wavelet transform module in the channel with the low-frequency components split out by the i-th forward wavelet transform module in other channels. For example, taking the first path of data as an example, the first style exchange module in the first path (located in the style transfer module 1) replaces the low-frequency component split out by the first forward wavelet transform module in the first path with the low-frequency component split out by the first forward wavelet transform module in other paths (such as the second path) (located in the style transfer module 1), the second style exchange module (located in the style transfer module 2) replaces the low-frequency component split out by the second forward wavelet transform module in the first path with the low-frequency component split out by the second forward wavelet transform module in other paths (such as the second path) (located in the style transfer module 2)... The X-1th style exchange module (located in the style transfer module X-1) replaces the low-frequency component split out by the X-1th forward wavelet transform module in the first path with the low-frequency component split out by the X-1th forward wavelet transform module in other paths (such as the second path) (located in the style transfer module X-1).

[0084] In some exemplary embodiments, the style exchange module replaces the low-frequency components of the current data stream with the low-frequency components of the other data stream using the following calculation formula:

[0085] Among them, x a Represents the low-frequency component of the first channel of image data, x b Represents the low-frequency component of the second channel image data, u(x a ) represents x a The data mean, σ(x a ) represents x a The data variance, u(x b ) represents x b The data mean, σ(x b ) represents x b The data variance, Represents the low-frequency component of the first channel of image data after replacement, Represents the low-frequency component of the second channel of image data after replacement.

[0086] Forward Wavelet Transform (DWT) and Inverse Discrete Wavelet Transform (IDWT) are two classic image processing methods. DWT can decompose the input image into low-frequency components and high-frequency components, wherein the low-frequency components can extract visual information such as style and shape in the image, while the high-frequency components can retain detail information such as texture in the image. The disclosed embodiment uses DWT to easily decompose the high- and low-frequency components of the image and extract the low-frequency information from the overall visual features. IDWT can reversely reconstruct the original image through the low-frequency components and high-frequency components decomposed by DWT. In particular, the two processes of DWT and IDWT are completely reversible. The process itself will not lose any image information, and the shapes of the input and output remain consistent, so that it can be plug-and-play inserted into the feature extractor for use.

[0087] The style exchange module can calculate x by the following formula a and x b Their respective means and variances: u(x a ),σ(x a )、u(x b ) and σ(x b ), where u represents the data mean of x and σ represents the data variance of x.

[0088] Wherein, subscript b is the number of images in one channel of image data; subscript c is the image feature dimension (the data mean u and data variance σ in formulas (1) and (2) are related to b and c), H and W are the length and width of the image, respectively, and ε is a very small value used to prevent the sign from being zero. For example, ε = 0.0001, however, this disclosure does not limit this.

[0089] In the embodiment of the present disclosure, when performing style transfer, the style and content information of each image data are first disassembled: given data x, its statistics u and σ are calculated. The combination of u and σ represents the style information of the data, and (xu) / σ represents the content information after removing the style information. Then, the styles of the two image data are exchanged: given x to be exchanged, a and x b ,On the basis of extracting their respective styles and contents, the above formulas (1) and (2) are used to realize the style exchange of the two-way image data.

[0090] In some exemplary embodiments, calculating the overall object loss based on the corresponding consistency loss, the few-shot object detection loss, and the few-shot object detection loss after style transfer includes:

[0091] Calculate the weighted sum of the corresponding consistency loss, small sample target detection loss, and small sample target detection loss after style transfer, and use the calculated weighted sum as the overall target loss.

[0092] In some exemplary embodiments, calculating the corresponding consistency loss based on the predicted probability distribution of the image data and the predicted probability distribution after style transfer of the image data includes:

[0093] The predicted probability distribution P0 of the image data and the predicted probability distribution P after style transfer of the image data are calculated according to the following formula SA The mean P of the probability distribution between avg :

[0094] Calculate the first distance KL(P0,P avg ), and the second distance KL(P SA ,P avg );

[0095] The first distance KL (P0, P avg ) and the second distance KL(P SA ,P avg ) as the corresponding consistency loss L SSL :

[0096] The disclosed embodiment uses a self-supervisor to compare the predicted probability distribution P0 without style transfer and the predicted probability distribution P after style transfer. SA Consistency constraints are imposed (i.e., by constraining the two predicted probability distributions P0 and P SA is similar) to eliminate the negative effects such as semantic shift that may be caused by style transfer.

[0097] In some exemplary embodiments, the distance between two predicted probability distributions is measured using the KL (Kullback-Leibler Divergence) discrete distance. The KL distance, also known as relative entropy, is used to measure the difference between two probability distributions in the same event space. Its physical meaning is: in the same event space, if the event space of probability distribution P(x) is encoded using probability distribution Q(x), how many bits are added to the average encoding length of each basic event. We use D(P||Q) to represent the KL distance, and its calculation formula is as follows:

[0098] When two probability distributions are exactly the same, that is, P(x) = Q(x), their relative entropy is 0.

[0099] In the embodiment of the present disclosure, the self-supervisor first obtains two predicted probability distributions P0 and P SA The average value P avg (i.e. the mean of the probability distribution), and then calculate P0, P SA and the probability distribution mean P avg The KL distance between them is taken as the average of the two KL distances as the final consistency loss L SSL The specific calculation method is shown in formulas (6) to (7):

[0100] In the cross-domain case, there are visual differences between the source domain and target domain data. The training method of the small sample image target detection model provided by the embodiment of the present disclosure improves the robustness of the model to images of different styles by enhancing the style diversity of the source domain data, thereby enhancing the model's ability to generate generalization from the source domain to the target domain.

[0101] In some exemplary embodiments, the training method of the small sample image object detection model includes the following steps:

[0102] (1) Randomly sample two data x in the source domain data a and x b As network input (not limited to two channels, in other examples, three or more channels of data can also be sampled); the two channels of data are two batches of data sampled from the same source domain training set with different random number seeds. The sampling set is the same, and the sampling process does not have any additional requirements on the image category and is completely random.

[0103] (2) The two data x a and x b They are fed into the feature extractor and small sample target detector respectively to obtain the predicted probability distribution of their respective test sets and And the corresponding small sample target detection loss function as well as

[0104] (3) For input data x a and x b In the process of extracting visual features with the feature extractor, image style transfer is performed separately, and the features after style transfer are input into the small sample object detector again to obtain the predicted probability distribution of each validation set after style transfer. and And the corresponding small sample target detection loss function as well as

[0105] (4) x a Two prediction results and x b Two prediction results Both are input into the supervisor to obtain their own consistency loss as well as

[0106] (5) Using small sample target detection loss And two consistency losses The weighted sum of the two together constitutes the overall target loss of this training process, and the model parameters are optimized according to the value of the overall target loss.

[0107] (6) Repeat steps (1) to (5) multiple times until the model converges and the model training process is completed.

[0108] The training method of the small sample image target detection model provided by the embodiment of the present disclosure uses the small sample target detection loss generated by the small sample target detector and the consistency loss generated by the self-supervisor as supervision information to optimize the network parameters during the model training stage. The four modules used in the training stage of the embodiment of the present disclosure, namely the forward wavelet transform module, the style transfer module, the inverse wavelet transform module and the self-supervisor, do not have any network parameters that need to be learned, and are only used to improve the generalization ability of the model by transferring style during the training stage. Therefore, the small sample image target detection model trained by the training method of the embodiment of the present disclosure only needs to randomly sample data in the target domain during the testing stage, and send it to the trained feature extractor and small sample target detector in sequence to obtain the prediction results.

[0109] The training method of the small sample image target detection model of the embodiment of the present disclosure improves the robustness of the model to different styles by enhancing the style diversity of the source domain image, thereby improving the recognition performance of the model in the unknown target domain and solving the problem of visual differences in domains in cross-domain small sample learning. The present disclosure uses wavelet transform for cross-domain small sample image recognition and performs feature enhancement operations on low-frequency components. In addition, the four modules used in the training phase of the present disclosure, namely the forward wavelet transform module, style exchange module, inverse wavelet transform module and self-supervisor, are all plug-and-play and can be easily used to enhance other cross-domain small sample recognition methods. The training method of the small sample image target detection model of the embodiment of the present disclosure can be used for tasks with single source domain training and without any target domain data, and does not require adjustment during the testing phase. Experimental results demonstrate the effectiveness of the method of the present disclosure.

[0110] As shown in FIG6 , the embodiment of the present disclosure further provides a training device for a small sample image target detection model, including an image acquisition module 601 , a first detection module 602 , a second detection module 603 , and a self-supervision module 604 ; wherein:

[0111] An image acquisition module 601 is configured to acquire image data;

[0112] The first detection module 602 is configured to input the image data into a small sample image object detection model to perform feature extraction and object detection on the image data, thereby obtaining a predicted probability distribution of the image data and a corresponding small sample object detection loss;

[0113] The second detection module 603 is configured to perform style transfer on the image data, and perform feature extraction and object detection on the style-transferred image data to obtain a predicted probability distribution of the style-transferred image data and a corresponding small-sample object detection loss;

[0114] The self-supervision module 604 is configured to determine the total loss of training based on the predicted probability distribution of the image data before and after style transfer and the corresponding small sample object detection loss, and optimize the parameters of the small sample image object detection model based on the total loss.

[0115] In some exemplary embodiments, the training device for the small sample image target detection model also includes a storage module, which is configured to store the small sample image target detection model and the training model of the small sample image target detection model, wherein the small sample image target detection model is used to perform feature extraction and target detection on the input image data; the training model of the small sample image target detection model is used to perform style transfer on the input image data, and perform feature extraction and target detection on the image data after style transfer.

[0116] In some exemplary embodiments, a small sample image object detection model includes: a feature extractor and a small sample object detector;

[0117] The feature extractor is configured to extract features from the input image and input the feature extraction results into the small sample object detector; the feature extractor includes an embedding layer, a feature encoding layer and a fully connected layer connected in sequence, the embedding layer is configured to convert the input image into a vector sequence; the feature encoding layer is configured to extract a feature vector from the vector sequence; the fully connected layer is configured to integrate the feature vectors extracted by the feature encoding layer;

[0118] The small sample target detector is configured to perform target detection on the feature extraction result to obtain a target detection result.

[0119] In some exemplary embodiments, the feature extractor includes a feature encoding layer, which includes a plurality of feature extraction blocks; a style transfer module is provided after at least one feature extraction block to obtain a training model for a small sample image object detection model, and the style transfer module is used to transfer the style of another image to the current image;

[0120] The image data is input into the training model of the small sample image object detection model to perform style transfer on the image data, and feature extraction and object detection are performed on the style-transferred image data.

[0121] In some exemplary embodiments, the style transfer module includes a forward wavelet transform module, a style exchange module, and an inverse wavelet transform module;

[0122] A forward wavelet transform module is configured to split the input feature vector into high-frequency components and low-frequency components;

[0123] a style exchange module configured to replace the low-frequency component of the data of the current channel with the low-frequency component of the data of the other channel;

[0124] The inverse wavelet transform module is configured to recombine the high frequency component and the replaced low frequency component into a feature vector.

[0125] In some exemplary embodiments, the style exchange module replaces the low-frequency components of the current data stream with the low-frequency components of the other data stream using the following calculation formula:

[0126] Among them, x a Represents the low-frequency component of the first channel of data, x b Represents the low-frequency component of the second channel data, u(x a ) represents x a The data mean, σ(x a ) represents x a The data variance, u(x b ) represents x b The data mean, σ(x b ) represents x b The data variance, Represents the low-frequency component of the first channel of data after replacement, Represents the low-frequency component of the second channel of data after replacement.

[0127] In some exemplary embodiments, the feature extraction blocks include X, where X is a natural number greater than or equal to 2, and the first feature extraction block to the Xth feature extraction block are arranged according to the data flow direction; the style transfer module includes at least one, and each style transfer module is arranged between two feature extraction blocks.

[0128] In some exemplary embodiments, the style transfer modules include X-1, the i-th style transfer module is arranged between the i-th feature extraction block and the i+1-th feature extraction block, and i is a natural number between 1 and X-1.

[0129] In some exemplary embodiments, the feature extractor is a ViT neural network, and the feature extraction block is a Transformer-based encoder structure.

[0130] In some exemplary embodiments, the small sample target detector is a small sample target detector based on distance comparison.

[0131] In some exemplary embodiments, the self-supervisor determines the total loss of training based on the predicted probability distribution of the image data before and after style transfer and the corresponding few-shot object detection loss, including:

[0132] Calculate the corresponding consistency loss based on the predicted probability distribution of the image data and the predicted probability distribution after the style transfer of the image data. The predicted probability distribution is the probability that the image data includes the small sample detection target.

[0133] The overall object loss is calculated based on the corresponding consistency loss, few-shot object detection loss, and few-shot object detection loss after style transfer.

[0134] In some exemplary embodiments, the self-supervisor calculates the overall target loss based on the corresponding consistency loss, the few-shot target detection loss, and the few-shot target detection loss after style transfer, including:

[0135] Calculate the weighted sum of the few-shot object detection loss, the few-shot object detection loss after style transfer, and the consistency loss, and use the calculated weighted sum as the overall object loss.

[0136] In some exemplary embodiments, the self-supervisor calculates a corresponding consistency loss based on the predicted probability distribution of the image data and the predicted probability distribution after style transfer of the image data, including:

[0137] The predicted probability distribution P0 of the image data and the predicted probability distribution P after style transfer of the image data are calculated according to the following formula SA The mean P of the probability distribution between avg :

[0138] Calculate the first distance KL(P0,P avg ), and the second distance KL(P SA ,P avg );

[0139] The first distance KL (P0, P avg ) and the second distance KL(P SA ,P avg ) as the corresponding consistency loss L SSL :

[0140] As shown in FIG7A , the embodiment of the present disclosure further provides a method for detecting an object in a small sample image, including:

[0141] Step 701: Obtain a trained small sample image target detection model, where the small sample image target detection model is trained using the training method for the small sample image target detection model described in any embodiment of the present disclosure;

[0142] Step 702: Input the image to be recognized into the trained small sample image target detection model to obtain an image recognition result.

[0143] The small sample image object detection method of the embodiment of the present disclosure enables the model to be exposed to more images with diverse styles during the training phase, thereby enhancing the model's robustness to images of different styles, improving the ability to migrate from the source domain to the target domain, and ultimately improving the recognition ability in the target domain during the testing phase.

[0144] In some exemplary embodiments, as shown in FIG2 , the small sample image target detection model may include: a feature extractor and a small sample target detector, wherein the feature extractor is configured to perform feature extraction on the input image and input the feature extraction result into the small sample target detector; and the small sample target detector is configured to perform target detection on the feature extraction result to obtain a target detection result.

[0145] In some exemplary embodiments, as shown in FIG5A , the training model of the small-sample image target detection model may include: a feature extractor, a style transfer module, a small-sample target detector, and a self-supervisor, wherein the feature extractor is configured to perform feature extraction on the input image and input the feature extraction result into the small-sample target detector; the style transfer module is configured to perform style transfer on one or more images in the source domain during the feature extraction process performed by the feature extractor; the small-sample target detector is configured to perform target detection on the feature extraction result to obtain a target detection result; and the self-supervisor is configured to detect whether the target detection result of the image before style transfer and the target detection result of the image after style transfer meet consistency constraints.

[0146] In some exemplary embodiments, the feature extractor may be any image feature extraction network. For example, the feature extractor may be a ViT model.

[0147] In some exemplary embodiments, as shown in FIG3 and FIG5A , the feature extractor may include an embedding layer, a feature encoding layer, and a fully connected layer connected in sequence, wherein the feature encoding layer includes multiple feature extraction blocks connected in sequence, each having the same structure. The style transfer module is provided after at least one feature extraction block.

[0148] In some exemplary embodiments, the number of feature extraction blocks is X, where X is a natural number greater than or equal to 2, and at least one style transfer module is provided after one or more feature extraction blocks from the first feature extraction block to the X-1th feature extraction block, wherein the first feature extraction block to the X-1th feature extraction block are provided according to the data flow direction. Because the features extracted by the last feature extraction block are too semantically related, style transfer based on such features generally does not bring benefits. Therefore, the present disclosure does not provide a style transfer module after the last feature extraction block.

[0149] In some exemplary embodiments, the style transfer modules include X-1, the i-th style transfer module is arranged after the i-th feature extraction block, and i is a natural number between 1 and X-1.

[0150] For example, as shown in FIG5A , where X=4, there are four feature extraction blocks: feature extraction block 1, feature extraction block 2, feature extraction block 3, and feature extraction block 4. There are three style transfer modules: style transfer module 1, style transfer module 2, and style transfer module 3. Style transfer module 1 is located between feature extraction block 1 and feature extraction block 2, style transfer module 2 is located between feature extraction block 2 and feature extraction block 3, and style transfer module 3 is located between feature extraction block 3 and feature extraction block 4. The parameters of the feature extractors and small-sample object detectors in both paths are identical. In other examples, the number of feature extraction blocks and style transfer modules can be set as needed.

[0151] In some exemplary embodiments, as shown in FIG5B , the style transfer module includes a forward wavelet transform module, a style exchange module, and an inverse wavelet transform module;

[0152] A forward wavelet transform module is configured to split the input feature vector into high-frequency components and low-frequency components;

[0153] a style exchange module configured to replace the low-frequency component of the data of the current channel with the low-frequency component of the data of the other channel;

[0154] The inverse wavelet transform module is configured to recombine the high frequency component and the replaced low frequency component into a feature vector.

[0155] In this disclosed embodiment, the overall visual features extracted by the feature extraction block are first decomposed into high- and low-frequency components using a forward wavelet transform module. The style information of the various low-frequency components of the two data streams is then exchanged using a style exchange module to obtain the style-transferred low-frequency components. Finally, an inverse wavelet transform module is used to reconstruct the high-frequency components and the style-transferred low-frequency components. The reconstructed features serve as input for subsequent feature extraction blocks. This process is repeated after feature extraction blocks 1, 2, and 3, until the data passes through feature extraction block 4 and the last fully connected layer, resulting in the final style-transferred feature representation.

[0156] Since the visual information has been extracted in the low-frequency components, the style exchange module of the embodiment of the present disclosure only operates on the low-frequency components of the image.

[0157] In some exemplary embodiments, the style exchange module replaces the low-frequency components of the current data stream with the low-frequency components of the other data stream using the following calculation formula:

[0158] Among them, x a Represents the low-frequency component of the first channel of data, x b Represents the low-frequency component of the second channel data, u(x a ) represents x a The data mean, σ(x a ) represents x a The data variance, u(x b ) represents x b The data mean, σ(x b ) represents x b The data variance, Represents the low-frequency component of the first channel of data after replacement, Represents the low-frequency component of the second channel of data after replacement.

[0159] Forward wavelet transform and inverse wavelet transform, as two classic image processing methods, DWT can decompose the input image into low-frequency components and high-frequency components, among which the low-frequency components will extract visual information such as style and shape in the image, while the high-frequency components will retain detail information such as texture in the image. The disclosed embodiment uses DWT to easily decompose the high- and low-frequency components of the image and extract the low-frequency information from the overall visual features. The IDWT can reversely reconstruct the original image through the low-frequency components and high-frequency components decomposed by DWT. In particular, the two processes of DWT and IDWT are completely reversible. The process itself will not lose any image information, and the shapes of the input and output remain consistent, so that it can be plug-and-play inserted into the feature extractor for use.

[0160] The style exchange module can calculate x by the following formula a and x b Their respective means and variances: u(x a ),σ(x a )、u(x b ) and σ(x b ), where u represents the data mean of x and σ represents the data variance of x.

[0161] Wherein, the subscript b is the number of images in one channel of data; c is the image feature dimension, H and W are the length and width of the image respectively, and ε is a very small value used to prevent the sign from being 0. For example, ε=0.0001, however, this disclosure does not limit this.

[0162] In the embodiment of the present disclosure, when performing style transfer, the style and content information of each data are first disassembled: given data x, its statistics u and σ are calculated. The combination of u and σ represents the style information of the data, and (xu) / σ represents the content information after removing the style information. Then, the styles of the two data are exchanged: given x to be exchanged, a and x b ,On the basis of extracting their respective styles and contents, the above formulas (1) and (2) are used to realize the style exchange of the two data.

[0163] In some exemplary embodiments, the small sample target detector can be any small sample target detection network based on distance comparison. For example, the small sample target detector can be a graph neural network, however, the embodiments of the present disclosure are not limited to this. For input data, the small sample target detector can predict the corresponding class probability distribution P and the corresponding loss function L.

[0164] In some exemplary embodiments, performing style transfer on one or more channels of data includes:

[0165] Select one or more images from other data paths as style images, and transfer the style of the selected style images to one or more images in this path to obtain one or more generated images.

[0166] In some exemplary embodiments, during each round of training, two data paths are randomly sampled in the source domain, and the images in the two data paths are used as source images and style images of each other (each data path in the two data paths is a style image of an image in the other data path except its own data path), and multiple generated images are obtained, that is, the image in the first data path is used as the source image, the image in the second data path is used as the style image, and the style of the image in the second data path is transferred to the image in the first data path, to obtain one or more generated images; at the same time, the image in the second data path is used as the source image, the image in the first data path is used as the style image, and the style of the image in the first data path is transferred to the image in the second data path, to obtain one or more generated images.

[0167] In the embodiment of the present disclosure, during each round of training, two paths of data may be randomly sampled from the source domain data, or more than two paths of data (such as three paths, four paths, etc.) may be randomly sampled from the source domain data. The embodiment of the present disclosure does not impose any restrictions on this.

[0168] In some exemplary embodiments, the small sample image object detection method includes the following steps:

[0169] (1) Randomly sample two data x in the source domain data a and x b As network input (not limited to two channels, in other examples, three or more channels of data can also be sampled); the two channels of data are two batches of data sampled from the same source domain training set with different random number seeds. The sampling set is the same, and the sampling process does not have any additional requirements on the image category and is completely random.

[0170] (2) The two data x a and x b They are fed into the feature extractor and small sample target detector respectively to obtain the predicted probability distribution of their respective test sets and And the corresponding small sample target detection loss function as well as

[0171] (3) For input data x a and x b In the process of extracting visual features with the feature extractor, image style transfer is performed separately, and the features after style transfer are input into the small sample object detector again to obtain the predicted probability distribution of each validation set after style transfer. and And the corresponding small sample target detection loss function as well as

[0172] (4) xa Two prediction results and x b Two prediction results Both are input into the self-supervisor to obtain their own consistency loss function as well as

[0173] (5) Using small sample target detection loss And two consistency losses The weighted sum of constitutes the overall target loss function of this training process, and the model parameters are optimized according to the value of the overall target loss function.

[0174] (6) Repeat steps (1) to (5) multiple times until the model converges and the model training process is completed.

[0175] (7) Randomly sample a path of data in the target domain and send it to the trained feature extractor and small sample target detector in turn to obtain the predicted probability distribution, and take the category with the best probability distribution as the result of this prediction.

[0176] (8) Repeat step (7) multiple times and take the average prediction accuracy as the final model performance.

[0177] The small sample image target detection method provided by the embodiment of the present disclosure uses the small sample target detection loss generated by the small sample target detector and the consistency loss generated by the self-supervisor as supervision information to optimize the network parameters during the model training stage. The four modules used in the training stage of the embodiment of the present disclosure, namely the forward wavelet transform module, the style transfer module, the inverse wavelet transform module and the self-supervisor, do not have any network parameters that need to be learned, and are only used to improve the generalization ability of the model by transferring style during the training stage. Therefore, the small sample image target detection model generated by the embodiment of the present disclosure only needs to randomly sample data in the target domain during the testing stage, and send it to the trained feature extractor and small sample target detector in sequence to obtain the prediction results.

[0178] The small-sample image object detection method of the disclosed embodiments enhances the style diversity of source domain images to improve the model's robustness to different styles. This in turn improves the model's recognition performance in unknown target domains and addresses the visual differences between domains in cross-domain small-sample learning. This disclosure utilizes wavelet transforms for cross-domain small-sample image recognition and performs style transfer on low-frequency components.

[0179] As shown in FIG7B , the embodiment of the present disclosure further provides a small sample image target detection device, including a storage module 703 and an input module 704 ; wherein:

[0180] A storage module 703 is configured to store a trained small sample image target detection model, where the small sample image target detection model is trained using the training method for the small sample image target detection model described in any embodiment of the present disclosure;

[0181] The input module 704 is configured to input the image to be recognized into the trained small sample image target detection model to obtain an image recognition result.

[0182] In the disclosed embodiment, the training process of the small sample image target detection model can refer to the above description and will not be repeated here.

[0183] An embodiment of the present disclosure also provides a training device for a small sample image target detection model, comprising a memory; and a processor connected to the memory, the memory being used to store instructions, the processor being configured to execute the steps of the training method for a small sample image target detection model as described in any embodiment of the present disclosure based on the instructions stored in the memory.

[0184] As shown in FIG8 , in one example, a training apparatus for a small sample image object detection model may include: a first processor 810, a first memory 820, a first bus system 830, and a first transceiver 840, wherein the first processor 810, the first memory 820, and the first transceiver 840 are connected via the first bus system 830, the first memory 820 is used to store instructions, and the first processor 810 is used to execute the instructions stored in the first memory 820 to control the first transceiver 840 to transmit and receive signals. Specifically, the first processor 810 iterates multiple rounds of training, which are achieved by performing style transfer on one or more images and applying consistency constraints to the object detection results of the image before style transfer and the object detection results of the image after style transfer.

[0185] It should be understood that the first processor 810 may be a central processing unit (CPU), or may be another general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0186] The first memory 820 may include a read-only memory and a random access memory, and provides instructions and data to the first processor 810. A portion of the first memory 820 may also include a non-volatile random access memory. For example, the first memory 820 may also store device type information.

[0187] In addition to the data bus, the first bus system 830 may also include a power bus, a control bus, a status signal bus, etc. However, for the sake of clarity, various buses are labeled as the first bus system 830 in FIG.

[0188] During implementation, the processing performed by the processing device can be completed by the hardware integrated logic circuit in the first processor 810 or by instructions in the form of software. That is, the method steps of the embodiment of the present disclosure can be embodied as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the first memory 820, and the first processor 810 reads the information in the first memory 820 and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0189] The present disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for training a small-sample image object detection model as described in any of the embodiments of the present disclosure. The method for driving the training of a small-sample image object detection model by executing executable instructions is substantially the same as the method for training a small-sample image object detection model provided in the aforementioned embodiments of the present disclosure and is not further described here.

[0190] In some possible embodiments, various aspects of the training method for a small sample image target detection model provided by the present disclosure may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the training method for a small sample image target detection model according to various exemplary embodiments of the present disclosure described above in this specification. For example, the computer device may execute the training method for a small sample image target detection model recorded in the embodiments of the present disclosure.

[0191] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0192] An embodiment of the present disclosure also provides a small sample image target detection device, comprising a memory; and a processor connected to the memory, wherein the memory is used to store instructions, and the processor is configured to execute the steps of the small sample image target detection method as described in any embodiment of the present disclosure based on the instructions stored in the memory.

[0193] As shown in FIG9 , in one example, a small sample image object detection apparatus may include: a second processor 910, a second memory 920, a second bus system 930, and a second transceiver 940. The second processor 910, the second memory 920, and the second transceiver 940 are connected via the second bus system 930. The second memory 920 is used to store instructions, and the second processor 910 is used to execute the instructions stored in the second memory 920 to control the second transceiver 940 to transmit and receive signals. Specifically, under the control of the first processor 810, the first transceiver 840 may obtain a trained small sample image object detection model and an image to be recognized. The second processor 910 inputs the image to be recognized into the trained small sample image object detection model to obtain an image recognition result. The training process of the small sample image object detection model is achieved by performing style transfer on one or more images and constraining the consistency of the object detection results of the image before style transfer and the object detection results of the image after style transfer.

[0194] It should be understood that the second processor 910 may be a central processing unit (CPU), or may be another general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0195] The second memory 920 may include a read-only memory and a random access memory, and provide instructions and data to the second processor 910. A portion of the second memory 920 may also include a non-volatile random access memory. For example, the second memory 920 may also store device type information.

[0196] In addition to the data bus, the second bus system 930 may also include a power bus, a control bus, a status signal bus, etc. However, for the sake of clarity, various buses are labeled as the second bus system 930 in FIG.

[0197] During the implementation process, the processing performed by the processing device can be completed by the hardware integrated logic circuit in the second processor 910 or by instructions in the form of software. That is, the method steps of the embodiment of the present disclosure can be embodied as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the second memory 920, and the second processor 910 reads the information in the second memory 920 and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0198] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method for detecting an object in a small sample image as described in any embodiment of the present disclosure is implemented.

[0199] In some possible embodiments, various aspects of the small sample image target detection method provided in the present application can also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the small sample image target detection method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device can execute the small sample image target detection method recorded in the embodiments of the present application.

[0200] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0201] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable, and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media generally embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0202] It should be noted that the above-described embodiments or implementations are merely illustrative and not restrictive. Therefore, the present disclosure is not limited to what is specifically shown and described herein. Various modifications, substitutions, or omissions may be made to the forms and details of the implementations without departing from the scope of the present disclosure.

Claims

1. A training method for a small sample image object detection model, comprising: Get image data; Inputting the image data into a small sample image target detection model to perform feature extraction and target detection on the image data, thereby obtaining a predicted probability distribution of the image data and a corresponding small sample target detection loss; Performing style transfer on the image data, and performing feature extraction and object detection on the style-transferred image data to obtain a predicted probability distribution of the style-transferred image data and a corresponding small-sample object detection loss; According to the predicted probability distribution of the image data before and after style transfer and the corresponding small sample target detection loss, the total loss of training is determined, and the parameters of the small sample image target detection model are optimized according to the total loss.

2. The training method according to claim 1, wherein: The performing style transfer on the image data includes: Randomly sampling M channels of image data, where M is a natural number greater than or equal to 2, and each channel of image data includes one or more images; The following operation is performed on at least one of the M paths of image data: one or more image data from other paths of image data or pre-specified image data is selected as a style image, and the style of the style image is transferred to one or more image data in the current path of image data to obtain one or more generated images, where the style of the generated image is determined by the low-frequency component of the style image after wavelet transformation.

3. The training method according to claim 1, wherein: Determining the total loss of training based on the predicted probability distribution of the image data before and after style transfer and the corresponding small sample object detection loss includes: Calculating a corresponding consistency loss based on a predicted probability distribution of the image data and a predicted probability distribution of the image data after style transfer, wherein the predicted probability distribution is a probability that the image data includes a small sample detection target; The overall object loss is calculated based on the corresponding consistency loss, the small-shot object detection loss, and the small-shot object detection loss after style transfer.

4. The training method according to claim 3, wherein: The calculating the overall target loss according to the corresponding consistency loss, the small sample target detection loss, and the small sample target detection loss after style transfer includes: A weighted sum of the small-sample target detection loss, the small-sample target detection loss after style transfer, and the consistency loss is calculated, and the calculated weighted sum is used as the overall target loss.

5. The training method according to claim 3, wherein: The calculating corresponding consistency loss according to the predicted probability distribution of the image data and the predicted probability distribution after the style transfer of the image data includes: The predicted probability distribution P0 of the image data and the predicted probability distribution P after the style transfer of the image data are calculated according to the following formula SA The mean P of the probability distribution between avg : Calculate the first distance KL(P0, P avg ), and the difference between the predicted probability distribution after the image data style migration and the calculated probability distribution mean The second distance KL(P SA ,P avg ); The first distance KL (P0, P avg ) and the second distance KL(P SA ,P avg ) as the corresponding consistency loss L SSL :

6. The training method according to claim 1, wherein: The small sample image target detection model includes: a feature extractor and a small sample target detector; The feature extractor is configured to extract features from an input image and input the feature extraction results into a small sample object detector; the feature extractor comprises an embedding layer, a feature encoding layer, and a fully connected layer connected in sequence, the embedding layer is configured to convert the input image into a vector sequence; the feature encoding layer is configured to extract a feature vector from the vector sequence; the fully connected layer is configured to integrate the feature vectors extracted by the feature encoding layer; The small sample target detector is configured to perform target detection on the feature extraction result to obtain a target detection result.

7. The training method according to claim 1, wherein: The feature extractor includes a feature encoding layer, which includes multiple feature extraction blocks; a style transfer module is set after at least one of the feature extraction blocks to obtain a training model of the small sample image object detection model, and the style transfer module is used to transfer the style of another image to the current image; The image data is input into a training model of the small sample image target detection model to perform style transfer on the image data, and feature extraction and target detection are performed on the image data after style transfer.

8. The training method according to claim 7, wherein: The style transfer module includes a forward wavelet transform module, a style exchange module and an inverse wavelet transform module; The forward wavelet transform module is configured to split the input feature vector into a high-frequency component and a low-frequency component; The style exchange module is configured to replace the low-frequency component of the current data with the low-frequency component of the other data; The inverse wavelet transform module is configured to recombine the high-frequency component and the replaced low-frequency component into a feature vector.

9. The training method according to claim 8, wherein: The style exchange module replaces the low-frequency component of the current data with the low-frequency component of the other data using the following calculation formula: Among them, x a Represents the low-frequency component of the first channel of data, x b Represents the low-frequency component of the second channel data, u(x a ) represents x a The data mean, σ(x a ) represents x a The data variance, u(x b ) represents x b The data mean, σ(x b ) represents x b The data variance, Represents the low-frequency component of the first channel of data after replacement, Represents the low-frequency component of the second channel of data after replacement.

10. The training method according to claim 7, wherein: The feature extraction blocks include X, where X is a natural number greater than or equal to 2, and the first feature extraction block to the Xth feature extraction block are arranged according to the data flow direction; the style transfer module includes at least one, and each style transfer module is arranged between two feature extraction blocks.

11. The training method according to claim 10, wherein: The style transfer modules include X-1, the i-th style transfer module is arranged between the i-th feature extraction block and the i+1-th feature extraction block, and i is a natural number between 1 and X-1.

12. The training method according to claim 7, wherein: The feature extractor is a ViT neural network, and the feature extraction block is a Transformer-based encoder structure.

13. The training method according to claim 7, wherein: The small sample target detector is a small sample target detector based on distance comparison.

14. The training method according to claim 1, wherein: The image data includes images of N categories, each category includes K+Q images, wherein the K images are divided into a support set, the Q images are divided into a validation set, and N, K and Q are all natural numbers greater than or equal to 1.

15. A training device for a small sample image target detection model, comprising a memory; and a processor connected to the memory, the memory being used to store instructions, the processor being configured to execute the steps of the training method for a small sample image target detection model as described in any one of claims 1 to 14 based on the instructions stored in the memory.

16. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method for a small sample image target detection model according to any one of claims 1 to 14.

17. A method for detecting an object in a small sample image, comprising: Obtaining a trained small sample image target detection model, wherein the small sample image target detection model is trained using the training method for a small sample image target detection model according to any one of claims 1 to 14; The image to be recognized is input into the trained small sample image target detection model to obtain an image recognition result.

18. A small sample image target detection device, comprising a memory; and a processor connected to the memory, the memory being used to store instructions, the processor being configured to execute the steps of the small sample image target detection method as described in claim 17 based on the instructions stored in the memory.

19. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method for detecting an object in a small sample image according to claim 17 is implemented.

20. A small sample image target detection device, comprising a storage module and an input module; wherein: The storage module is configured to store a trained small sample image target detection model, wherein the small sample image target detection model is trained using the training method for a small sample image target detection model according to any one of claims 1 to 14; The input module is configured to input the image to be recognized into the trained small sample image target detection model to obtain an image recognition result.

21. A training device for a small sample image target detection model, comprising an image acquisition module, a first detection module, a second detection module, and a self-supervision module; wherein: The image acquisition module is configured to acquire image data; The first detection module is configured to input the image data into a small sample image object detection model to perform feature extraction and object detection on the image data, thereby obtaining a predicted probability distribution of the image data and a corresponding small sample object detection loss; The second detection module is configured to perform style transfer on the image data, and perform feature extraction and object detection on the style-transferred image data to obtain a predicted probability distribution of the style-transferred image data and a corresponding small-sample object detection loss; The self-supervision module is configured to determine the total loss of training based on the predicted probability distribution of the image data before and after style transfer and the corresponding small sample target detection loss, and optimize the parameters of the small sample image target detection model based on the total loss.

Images