A cross-domain small sample target detection method based on scale matching and semantic enhancement

Abstract

The application provides a cross-domain small sample target detection method based on scale matching and semantic enhancement. For an input image to be detected, first, differential data enhancement is performed according to the input difference of a teacher network and a student network. Then, cross-domain scale shapes are optimally matched, so that the problem of scale shape affecting detection precision in different domains is solved from the perspective of cross domains. Then, data samples are interpolated or extrapolated by using a semantic space, so as to generate new data samples, thereby expanding the data, and solving the problem of data scarcity in cross-domain small sample target detection. Finally, knowledge distillation is performed, the teacher model is taken as the last training model, and the output result of the teacher model is the final detection result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection and computer vision, and particularly relates to a cross-domain small sample target detection method based on scale matching and semantic enhancement. BACKGROUND

[0002] In the field of target detection, small sample setting is an important and challenging topic. Traditional small sample target detection paradigm often involves two stages: first, pre-training on the source dataset composed of base classes, and then migrating the pre-trained detector to the target dataset, which contains a small number of instances belonging to completely new classes that do not overlap with the base classes, but the data domains are similar. However, this setting has significant limitations in practical applications, especially in specific fields such as medicine and aviation. In these fields, due to the difficulty and sensitivity of data collection, it is difficult or almost impossible to obtain a sufficient number of target class samples, not to mention meeting the assumption of similar domains. With the rapid development of computer vision technology, target detection, as one of the key tasks, has made significant progress. However, in practical applications, cross-domain small sample target detection remains a challenging problem. This is mainly due to the large differences in scale, shape, texture, etc. of targets in different domains or scenarios, and the increased difficulty of model generalization due to small sample data. Traditional target detection methods mainly rely on a large amount of labeled data for training, but in the cross-domain small sample scenario, such data is often difficult to obtain. Therefore, how to effectively utilize limited labeled data and overcome cross-domain differences has become a problem to be solved in the field of target detection. SUMMARY

[0003] Therefore, the purpose of the present application is to provide a cross-domain small sample target detection method based on scale matching and semantic enhancement, which uses a strong-weak enhancement architecture based on knowledge distillation to improve anchor box matching in cross-domain scenarios and uses semantic space to interpolate or extrapolate data samples, implicitly expanding the data and increasing the diversity of samples.

[0004] To achieve the above purpose, the present application adopts the following technical solution: a cross-domain small sample target detection method based on scale matching and semantic enhancement, comprising the following steps:

[0005] Step S1: training features by inputting data into a ResNet50 backbone network through a cross-domain small sample target detection general training set and using strong and weak data augmentation methods on the data respectively;

[0006] Step S2: using the trained feature extraction network as a feature extraction module and introducing a cross-domain scale shape optimization matching module to solve the problem of scale shape affecting detection accuracy in different domains;

[0007] Step S3: Design a dynamic semantic covariance enhancement module to use the semantic space to interpolate or extrapolate data samples to generate new data samples, thereby expanding the data;

[0008] Step S4: Perform knowledge distillation. The teacher model and the student model are trained in a collaborative manner. The teacher model uses distillation loss to guide the student model to promote its performance. The student model continuously optimizes the parameter weights through the exponential moving average (EMA) strategy, gradually approaching the teacher model. Finally, the features of the teacher model are used for cross-domain small-sample target detection.

[0009] In a preferred embodiment, step S1 specifically includes the following steps:

[0010] Step S11: Obtain a publicly available small sample segmentation training set from the Internet and obtain relevant annotations for the training data; the model is trained on a base class dataset with a large number of annotated instances for each class, and then the model is transferred and fine-tuned to the target dataset where each class has only a few samples; the base class dataset and the new class dataset come from different fields, with the MSCOCO2017 dataset set as the base class dataset and the ArTaxOr, UODD, and DIOR datasets set as new class datasets, whose fields are biology, underwater, and aviation, respectively;

[0011] Step S12: After the data is loaded, different data augmentations are performed on the teacher network and the student network. The teacher network uses a weak augmentation method, i.e., random flipping, while the student network implements augmentation measures, including random color jittering, grayscale, Gaussian blurring, and random shearing, to simulate the complex changes in real scenes and force the student network to learn more robust feature representations.

[0012] Step S13: Initialize the network weights and parameters of the pre-trained ResNet50 feature extraction network for the image to be detected in the cross-domain small sample target detection dataset.

[0013] In a preferred embodiment, step S2 specifically includes the following steps:

[0014] Step S21: Define A to represent the detection box, B to represent the ground truth box, Area(·) to represent the area of ​​the corresponding region, and A∩B to represent the intersection area of ​​the detection box and the ground truth box. First, obtain the width w and height h of each detection box, as well as the width w′ and height h′ of the ground truth box, and calculate their areas Area(A) and Area(B). The specific calculation is as follows:

[0015] Area(A)=w×h, Area(B)=w′×h′

[0016] Step S22: Calculate the width difference s of the detection frame and the real frame center point c w and the height difference s c h, and the distance sigma between them; the calculation formula is as follows:

[0017]

[0018] Where x' min , x' max are the minimum and maximum horizontal coordinates of the real frame respectively, x min , x max are the minimum and maximum horizontal coordinates of the detection frame respectively; similarly, y' min , y' max are the minimum and maximum vertical coordinates of the real frame respectively, y min , y max are the minimum and maximum vertical coordinates of the detection frame respectively;

[0019] Step S23: According to the angle threshold value threshold and the calculation formula of the center point angle α, the angle loss angle_cost is obtained; the calculation formula of the center point angle is as follows, and the angle loss calculation formula is as follows:

[0020]

[0021] Where σ represents the scale of the anchor frame;

[0022] Step S24: Calculate the convex hull width cw and the convex hull height ch of the detection frame and the real frame, and the center point distance sigma, and then calculate the distance loss distance_cost and the shape loss shape_cost; wherein the convex hull width cw refers to the maximum distance of the detection frame and the real frame in the horizontal direction, that is, the maximum overlap length of the detection frame and the real frame in the horizontal direction; the convex hull height ch refers to the maximum distance of the detection frame and the real frame in the vertical direction, that is, the maximum overlap height of the detection frame and the real frame in the vertical direction; the specific calculation formula is as follows:

[0023] CW = max(w, w'), ch = max(h, h')

[0024]

[0025] distance_cost = 2-exp(γ·rho x )-exp(γ·rho y )

[0026]

[0027] where w is the width of the bounding box, h is the height of the bounding box, w' is the width of the real bounding box, and h' is the height of the real bounding box; in the calculation of the shape and distance losses, a rho value is introduced to measure the shape difference between the bounding box and the real bounding box; rho x refers to the difference in width between the bounding box and the real bounding box; rho y refers to the difference in height between the bounding box and the real bounding box; the calculation of the rho value involves a center point difference s c w and s c h are the ratios of the convex hull width cw and the convex hull height ch to the bounding box width w and the bounding box height h; ∈ is a very small positive number to prevent the calculation of rho from dividing by zero when the convex hull width cw and the convex hull height ch are zero; γ represents a weight parameter to adjust the relative importance of the shape and distance losses; the greater γ is, the greater the influence of the shape loss and the distance loss on the dssom value is; the smaller γ is, the relatively smaller the influence of the shape loss and the distance loss is;

[0028] Step S25: Calculate the area irtter of the intersection region according to the calculation formula of the intersection region. area The specific calculation is as follows:

[0029] irtter area = (x max_inter -x min_inter ) × (y max_inter -y min_inter )

[0030] where x max_inter and x min_inter are the maximum and minimum horizontal coordinates of the intersection region; y max_inter and y min_inter are the maximum and minimum vertical coordinates of the intersection region.

[0031] Step S26: Finally, divide the area of the intersection region by the area of the union region, and subtract half of the distance loss and the shape loss to obtain the final IoU loss value L dssom The specific calculation formula is as follows:

[0032]

[0033] In a preferred embodiment, step S3 specifically includes the following steps:

[0034] Step S31: Design an estimation module for the covariance matrix to dynamically estimate the covariance matrix of different categories of features according to the input feature data; this module maintains the covariance matrix, mean matrix, and sample number information of each category during the training process, and updates these statistical information according to new samples;

[0035] First, reshape the features into the shape of N×1×A to match the dimensions required by subsequent operations, where N and A are the number of samples and the number of features, respectively. Then, calculate the one-hot encoding of the label to determine the category membership of each sample. Next, calculate the mean vector ave of the feature data under each category. c,a , as the representative vector of the category feature; ave c，a The specific calculation formula is as follows:

[0036]

[0037] Among them, c represents the category index, a represents the dimension of the feature vector, features i , a is the reshaped feature value, i represents the sample index, onehot c,i is the one-hot category vector of sample i, Amount c , a represents the number of features of category c;

[0038] Step S32: Based on the above covariance matrix generation, data expansion of implicit semantics is performed; first, the feature weight matrix M_ij is obtained from the network weights, which maps the features to the category space; the shape of this matrix is ​​N×C×A, where: N is the number of samples, C is the number of categories, and A is the number of features; the formula of the feature weight matrix M_ij is as follows:

[0039] M_ij=weight_m×ones(N,C,A)

[0040] Among them, weight_m is the weight parameter of the network, and ones(N, C, A) is a full-1 tensor with shape N×C×A;

[0041] Step S33: Obtain the target feature weight matrix M from the feature weight matrix according to the true label labels _ kj, used to calculate the pseudo label of the sample; the shape of the target feature weight matrix is ​​the same as the feature weight matrix; the formula of the target feature weight matrix M_kj is as follows:

[0042] M_kj=gather(M_ij, 1, labels)

[0043] Where gather(·) indicates indexing according to a given dimension;

[0044] Step S34: Calculate the pseudo label isda_aug_y of the sample based on the feature weight matrix, the target feature weight matrix, and the covariance matrix; the pseudo label will be used to calculate the loss value. The formula of the pseudo label isda_aug_y is as follows:

[0045] sigma2=ratio×(weight_m-M_kj) 2 ×CoVariance c,a

[0046] isda_aug_y=y+0.5×sigma2

[0047] Among them, ratio is the ratio parameter of loss, CoVariance c , a is the covariance matrix obtained from the covariance estimator, y is the original output of the model, sigma2 represents the adjustment value of the pseudo label of the sample, which adjusts the original output to a form more suitable for the target domain;

[0048] Step S35: Finally, the cross entropy loss function is used to calculate the loss value between the pseudo label and the true label; this loss value will be used as the training target of the model, and the loss value will be minimized by optimizing the network parameters; the loss value L aug The formula is as follows:

[0049] L aug =CE(isda_aug_y, labels)

[0050] Among them, CE represents the cross entropy loss function, and labels is the true label of the sample.

[0051] In a preferred embodiment, step S4 specifically includes the following steps:

[0052] Step S41: First, based on the classic framework of cross-domain object detection, it is divided into two phases, namely the training phase and the testing phase. In the training phase, the base class data is used to train the detector to improve detection performance. Subsequently, before fine-tuning the new class, the initialized detector parameters are copied to the student model. Before the formal distillation, the student model needs to go through a "running-in" phase. In this phase, the model is fine-tuned for a specific number of new class data using the standard detection supervision loss to adapt to the needs of cross-domain small-sample object detection. After the running-in is completed, the weights of the student model will be synchronized to the teacher model to prepare for subsequent distillation training. During distillation, the teacher model and the student model are trained in a collaborative manner.

[0053] Step S42: Use the knowledge distillation architecture to take the strongly enhanced image as the input of the student model and the weakly enhanced image as the input of the teacher model to provide reliable pseudo labels; in the supervision branch, calculate the supervised detection loss of the student model: the classification loss L cls and the localization loss L loc ; In new class samples In the case of supervised detection loss L S The calculation formula is as follows:

[0054]

[0055] wherein, represents an image value, represents a label real value, s refers to a new class sample, i refers to the i-th of the new class sample, L cls represents a classification loss, L loc represents a positioning loss;

[0056] Step S43: the teacher and the student share the same architecture, and are initialized with the same weights after the aging step; the image is independently processed by the student and the teacher; the teacher is used to generate thousands of frame candidates for the weak enhanced version, and redundancy is eliminated through non-maximum suppression NMS; the student model uses the strong enhanced version for prediction, and obtains the distillation loss L D by calculating the detection loss between the student prediction and the pseudo label; D The specific calculation formula of L

[0057]

[0058] wherein, is the correct result in the distillation case, is the generated prediction result in the distillation case;

[0059] Step S44: exponential moving average EMA is used to separate the teacher and the student; the teacher is used to infer and evaluate the test image; it is worth noting that in the inference stage, no data enhancement operation will be performed on the input image;

[0060] So far, before knowledge distillation, the loss function calculation formula is as follows:

[0061] L=L S +L dssom +L aug

[0062] wherein, L represents the value of the loss function, L S represents the value of the supervised detection loss, L dssom represents the loss value of the cross-domain scale shape optimization matching module generated in step S2, L aug represents the loss value of the dynamic semantic covariance enhancement module generated in step S3;

[0063] After the knowledge distillation calculation, the new loss function calculation formula is as follows:

[0064] L=L S +L D +L dssom +Laug

[0065] wherein L D represents distillation loss;

[0066] Step S45: test the trained teacher model on the validation set to obtain the final detection accuracy, and obtain the training model adapted to the new class data set.

[0067] Compared with the prior art, the present application has the following beneficial effects:

[0068] 1. Unlike small sample target detection, considering the disjointedness of base classes and new classes, small sample learning is introduced on the basis of cross-domain target detection, which can better fit the actual application scenario.

[0069] 2. The teacher and student networks in knowledge distillation are used, and the two develop cooperatively to achieve excellent detection performance in different fields and with fewer samples, greatly improving the generalization ability of the model.

[0070] 3. The scale and shape differences between the detection frame and the real frame are considered, and the scale and shape matching module is designed to solve the problem of scale and shape affecting detection accuracy in different fields from the perspective of cross-domain.

[0071] 4. The problem of data scarcity in cross-domain small sample target detection is further solved, and an implicit semantic data augmentation method called dynamic semantic covariance enhancement module is proposed, which uses semantic space to interpolate or extrapolate data samples to generate new data samples, thereby expanding the data. BRIEF DESCRIPTION OF DRAWINGS

[0072] Figure 1 The method implementation flowchart of the preferred embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0073] The present application will be further described below in conjunction with the drawings and examples.

[0074] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0075] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form, and it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0076] like Figure 1 As shown, the trial example of the present invention provides a cross-domain small sample target detection method based on scale matching and semantic enhancement, comprising the following steps:

[0077] Step S1: Using a cross-domain small sample target detection universal training set, and applying strong and weak data augmentation methods to the data, the data is fed into the ResNet50 backbone network for feature training. This specifically includes the following steps:

[0078] Step S11: Obtain a publicly available small-sample segmentation training set from the internet and obtain the relevant annotations for the training data. The model is trained on a base dataset with a large number of annotated examples per class, and then the model is transferred and fine-tuned to the target dataset with only a small number of examples per class. The base and new class datasets come from different domains: the MSCOCO2017 dataset is used as the base dataset, while the ArTaxOr, UODD, and DIOR datasets are used as new class datasets, covering the biological, underwater, and aviation fields, respectively.

[0079] Step S12: Due to the knowledge distillation architecture, different data augmentations are performed on the teacher and student networks after data loading. The teacher network only uses a weak augmentation method, namely random flipping, to maintain its stability relative to the original data. The student network, on the other hand, implements more intensive augmentation measures, including random color jittering, grayscale conversion, Gaussian blurring, and random cropping, to simulate the complex changes in real scenes and force the student network to learn more robust feature representations.

[0080] Step S13: Initialize the network weights and parameters of the pre-trained ResNet50 feature extraction network for the image to be detected in the cross-domain small sample target detection dataset.

[0081] Step S2: Use the trained feature extraction network as the feature extraction module and introduce a cross-domain scale shape optimization matching module to solve the problem that scale shapes in different domains affect detection accuracy. Specifically, it includes the following steps:

[0082] Step S21: Define A as the detection box, B as the ground truth box, Area(·) as the area of ​​the corresponding region, and A∩B as the intersection of the detection box and the ground truth box. First, obtain the width w and height h of each detection box, as well as the width w′ and height h′ of the ground truth box, and calculate their areas Area(A) and Area(B). The specific calculation is as follows:

[0083] Area(A)=w×h, Area(B)=w′×h′

[0084] Step S22: Calculate the width difference s between the center point of the detection frame and the real frame c w and height difference s c h, and the distance sigma between them. The calculation formula is as follows:

[0085]

[0086] Step S23: Calculate the angle loss angle_cost according to the angle threshold threshold and the calculation formula of the center point angle. The calculation formula of the center point angle is as follows:

[0087]

[0088] Step S24: Calculate the convex hull width cw and convex hull height ch of the detection box and the ground truth box, as well as the center point distance sigma, and then calculate the distance loss distance_cost and shape loss shape_cost. The convex hull width cw refers to the maximum horizontal distance between the detection box and the ground truth box, that is, the maximum horizontal overlap length of the detection box and the ground truth box; the convex hull height ch refers to the maximum vertical distance between the detection box and the ground truth box, that is, the maximum vertical overlap height of the detection box and the ground truth box. The specific calculation formula is as follows:

[0089] CW=max(w,w′), ch=max(h,h′)

[0090]

[0091] distance_cost = 2-exp(γ·rho x )-exp(γ·rho y )

[0092]

[0093] Among them, the rho value is introduced in the calculation of shape and distance loss to measure the shape difference between the detection box and the real box. xRefers to the difference in width between the detection box and the real box; rho y Refers to the difference in height between the detection box and the real box; the calculation of the rho value involves the center point difference s c w and s c The ratio of h to the convex hull width cw and convex hull height ch. ∈ is a very small positive number used to prevent the calculation of rho from being divided by zero when the convex hull width cw and convex hull height ch are zero. γ represents a weight parameter used to adjust the relative importance of shape and distance losses. The larger γ is, the greater the impact of shape loss and distance loss on the dssom value; the smaller γ is, the smaller the impact of shape loss and distance loss is.

[0094] Step S25: Calculate the area of ​​the intersection region according to the calculation formula of the intersection region. area , the specific calculation is as follows:

[0095] inter_area=(x max_inter -x min_inter )×(y max_inter -y min_inter )

[0096] Step S26: Finally, divide the area of ​​the intersection region by the area of ​​the union region, and then subtract half of the distance loss and shape loss to obtain the final IoU loss value L dssom , the specific calculation formula is as follows:

[0097]

[0098] By comprehensively considering the differences in scale and shape, the matching degree between the detection frame and the true frame can be evaluated more accurately. Taking into account the differences in scale and shape between the detection frame and the true frame, it has better adaptability to the scale and shape changes of the target. In cross-domain scenarios, since data from different fields may have different scale and shape characteristics, L dssom It can more accurately measure the degree of match between the detection box and the true box.

[0099] Step S3: Design a dynamic semantic covariance enhancement module to use semantic space to interpolate or extrapolate data samples to generate new data samples, thereby expanding the data. Specifically, it includes the following steps:

[0100] Step S31: In order to enhance the features generated by the model, it is necessary to calculate the data augmentation value according to the covariance matrix, the model weight and the sample label. An estimation module of the covariance matrix is designed to dynamically estimate the covariance matrix of different categories of features according to the input feature data. This module maintains the covariance matrix, mean matrix and sample number information of each category during the training process, and updates these statistical information according to the new samples;

[0101] Firstly, the features are reshaped into the shape of N x 1 x A to match the dimensions required by the subsequent operations, and N and A are the number of samples and the number of features, respectively. Then, the one-hot encoding of the label is calculated to determine the class membership of each sample. Next, the mean vector ave c,a of the feature data under each category is calculated as the representative vector of the features of this category. The specific calculation formula of ave c，a is as follows:

[0102]

[0103] Where c represents the category index, a represents the dimension of the feature vector, features i，a is the reshaped feature value, i represents the sample index, onehot c，i is the one-hot category vector of sample i, and Amount c，a represents the number of features of category c;

[0104] Step S32: On the basis of the above covariance matrix generation, implicit semantic data augmentation is performed. Firstly, this section obtains the feature weight matrix M_ij from the weights of the network, which maps the features to the category space. The shape of this matrix is N x C x A, where N is the number of samples, C is the number of categories, and A is the number of features. The formula of the feature weight matrix M_ij is as follows:

[0105] M_ij = weight_m x ones(N, C, A)

[0106] Where weight_m is the weight parameter of the network, and ones(N, C, A) is a full 1 tensor with shape N x C x A;

[0107] Step S33: According to the real label labels, the target feature weight matrix M_kj is obtained from the feature weight matrix, which is used to calculate the pseudo label of the sample. The shape of the target feature weight matrix is the same as that of the feature weight matrix. The formula of the target feature weight matrix M_kj is as follows:

[0108] M_kj = gather(M_ij, 1, labels)

[0109] Where gather(·) indicates indexing according to a given dimension;

[0110] Step S34: Calculate the pseudo label isda_aug_y of the sample based on the feature weight matrix, the target feature weight matrix, and the covariance matrix. The pseudo label will be used to calculate the loss value. The formula of the pseudo label isda_aug_y is as follows:

[0111] sigma2=ratio×(weight_m-M_kj) 2 ×CoVariance c,a

[0112] isda_aug_y=y+0.5×sigma2

[0113] Among them, ratio is the ratio parameter of loss, CoVariance c，a is the covariance matrix obtained from the covariance estimator, y is the original output of the model, and sigma2 represents the adjustment value of the pseudo label of the sample, which adjusts the original output to a form more suitable for the target domain;

[0114] Step S35: Finally, the cross entropy loss function is used to calculate the loss value between the pseudo label and the true label. This loss value will be used as the training target of the model and minimized by optimizing the network parameters. aug The formula is as follows:

[0115] L aug =CE(isda_aug_y, labels)

[0116] Among them, CE represents the cross entropy loss function, and labels is the true label of the sample.

[0117] Step S4: Perform knowledge distillation. The teacher model and student model are trained in a collaborative manner. The teacher model uses distillation loss to guide the student model, promoting its performance improvement. The student model continuously optimizes parameter weights using an exponential moving average (EMA) strategy, gradually moving closer to the teacher model. Finally, the teacher model's features are used for cross-domain small-sample object detection. Specifically, the following steps are included:

[0118] Step S41: First, based on the classic framework of cross-domain target detection, it is divided into two major stages, namely the training stage and the testing stage. In the training stage, the base class data is used to train the detector to improve the detection performance. Subsequently, before fine-tuning the new class, the initialized detector parameters are copied to the student model. The student model needs to go through a "running-in" stage before formal distillation. In this stage, the model uses standard detection supervision loss to fine-tune a specific number of new class data to meet the needs of cross-domain small sample target detection. After the running-in is completed, the weights of the student model will be synchronized to the teacher model to prepare for subsequent distillation training. During distillation, the teacher model and the student model are trained in a collaborative manner;

[0119] Step S42: The knowledge distillation architecture is used to take the strongly enhanced image as the input of the student model and the weakly enhanced image as the input of the teacher model to provide reliable pseudo labels. The purpose of this is to increase the diversity of the student model, thereby improving the performance and generalization ability of the model. In the supervision branch, the supervised detection loss (classification loss L cls and the localization loss L loc ). In the new class sample In the case of supervised detection loss L S The calculation formula is as follows:

[0120]

[0121] in, It represents the image value. It represents the true value of the label, s refers to the new class sample, i refers to the i-th new class sample, L cls Represents the classification loss, L loc It represents the positioning loss;

[0122] Step S43: The teacher and student share the same architecture and are initialized with the same weights after the burn-in step. The image is processed independently by the student and the teacher. The teacher is used to generate thousands of box candidates for the weakly enhanced version and remove redundancy through non-maximum suppression (NMS). Although most redundant boxes are removed, there may still be some non-foreground candidate boxes. Therefore, only candidates with foreground scores above the threshold are retained as pseudo boxes. The student model uses the strongly enhanced version to make predictions and obtains the distillation loss L by calculating the detection loss between the student predictions and the pseudo labels. D . L D The specific calculation formula is as follows:

[0123]

[0124] in, For the correct result in the case of distillation, Predictions generated for the distillation case;

[0125] Step S44: To alleviate the overfitting problem, an exponential moving average (EMA) is used to separate the teacher and the student. The teacher is used to perform inference and evaluation on the test images. It is worth noting that no data augmentation operations are performed on the input images during the inference phase.

[0126] At this point, before knowledge distillation, the loss function calculation formula is as follows:

[0127] L=L S +L dssom +L aug

[0128] Among them, L represents the value of the loss function, L S It represents the supervised detection loss value, L dssom It represents the loss value of the cross-domain scale shape optimization matching module generated in step S2, L aug It represents the loss value of the dynamic semantic covariance enhancement module generated in step S3;

[0129] After the knowledge distillation calculation, the new loss function calculation formula is as follows:

[0130] L=L S +L D +L dssom +L aug

[0131] Among them, L D It represents the distillation loss;

[0132] Step S45: Test the trained teacher model on the validation set to obtain the final detection accuracy and obtain a training model that is suitable for the new class data set.

[0133] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0134] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0135] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0136] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0137] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

[0138] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive various other forms of cross-domain small sample target detection methods based on scale matching and semantic enhancement under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of this invention should be covered by this patent.

Claims

1. A cross-domain small sample target detection method based on scale matching and semantic enhancement, characterized by: The following steps are involved: Step S1: Use a cross-domain small sample target detection universal training set, and use strong and weak data enhancement methods on the data respectively, and send the data into the ResNet50 backbone network to train features; Step S2: Using the trained feature extraction network as the feature extraction module, introducing a cross-domain scale shape optimization matching module to solve the problem that the scale shape in different domains affects the detection accuracy; Step S3: Design a dynamic semantic covariance enhancement module to use the semantic space to interpolate or extrapolate data samples to generate new data samples, thereby expanding the data; Step S4: Perform knowledge distillation, where the teacher model and the student model are trained in a collaborative manner. The teacher model uses distillation loss to guide the student model and improve its performance. The student model continuously optimizes parameter weights through the exponential moving average (EMA) strategy, gradually approaching the teacher model; and finally uses the characteristics of the teacher model for cross-domain small sample target detection; Step S2 specifically includes the following steps: Step S21: Define A to represent the detection box, B to represent the ground truth box, Area(·) to represent the area of ​​the corresponding region, and A∩B to represent the intersection area of ​​the detection box and the ground truth box. First, obtain the width w and height h of each detection box, as well as the width w′ and height h′ of the ground truth box, and calculate their areas Area(A) and Area(B). The specific calculation is as follows: Area(A)=w×h, Area(B)=w′×h′ Step S22: Calculate the width difference s between the center point of the detection frame and the real frame c w and height difference s c h, and the distance sigma between them; the calculation formula is as follows: Where x′ min , x′ max are the minimum and maximum horizontal coordinates of the real box, respectively, min 、x max are the minimum and maximum horizontal coordinates of the detection box respectively; similarly, y′ min , y′ max are the minimum and maximum ordinates of the real box, y min 、y max are the minimum and maximum vertical coordinates of the detection box respectively; Step S23: Calculate the angle loss angle_cost according to the calculation formula of the angle threshold threshold and the center point angle α; the calculation formula of the center point angle is as follows, and the specific calculation formula of the angle loss is as follows: Among them, σ represents the scale of the anchor box; Step S24: Calculate the convex hull width cw and convex hull height ch of the detection box and the true box, as well as the center point distance sigma, and then calculate the distance loss distance_cost and shape loss shape_cost; the convex hull width cw refers to the maximum horizontal distance between the detection box and the true box, that is, the maximum overlapping length of the detection box and the true box in the horizontal direction; the convex hull height ch refers to the maximum vertical distance between the detection box and the true box, that is, the maximum overlapping height of the detection box and the true box in the vertical direction; the specific calculation formula is as follows: cw=max(w,w′),ch=max(h,h′) distance_cost=2-exp(γ·rho x )-exp(γ·rho y ) Among them, w is the width of the detection box, h is the height of the detection box, w′ is the width of the real box, and h′ is the height of the real box. In the calculation of shape and distance loss, the rho value is introduced to measure the shape difference between the detection box and the real box; rho x Refers to the difference in width between the detection box and the real box; rho y Refers to the difference in height between the detection box and the real box; the calculation of the rho value involves the center point difference s c w and s c The ratio of h to the convex hull width cw and the convex hull height ch; ∈ is a very small positive number used to prevent the calculation of rho from being divided by zero when the convex hull width cw and the convex hull height ch are zero; γ represents a weight parameter used to adjust the relative importance of shape and distance losses; the larger γ is, the greater the influence of shape loss and distance loss on the dssom value; the smaller γ is, the smaller the influence of shape loss and distance loss is; Step S25: Calculate the area of ​​the intersection region according to the calculation formula of the intersection region. area , the specific calculation is as follows: inter area =(x max_inter -x min_inter )×(y max_inter -y min_inter ) Among them, x max_inter and x min_inter is the maximum and minimum horizontal coordinates of the intersection area; y max_inter and y min_inter are the maximum and minimum ordinates of the intersection area; Step S26: Finally, divide the area of ​​the intersection region by the area of ​​the union region, and then subtract half of the distance loss and shape loss to obtain the final IoU loss value L dssom , the specific calculation formula is as follows:

2. The cross-domain small sample target detection method based on scale matching and semantic enhancement according to claim 1 is characterized in that: Step S1 specifically includes the following steps: Step S11: Obtain a publicly available small sample segmentation training set from the Internet and obtain relevant annotations for the training data; the model is trained on a base class dataset with a large number of annotated instances for each class, and then the model is transferred and fine-tuned to the target dataset where each class has only a few samples; the base class dataset and the new class dataset come from different fields, with the MSCOCO2017 dataset set as the base class dataset and the ArTaxOr, UODD, and DIOR datasets set as new class datasets, whose fields are biology, underwater, and aviation, respectively; Step S12: After the data is loaded, different data augmentations are performed on the teacher network and the student network. The teacher network uses a weak augmentation method, i.e., random flipping, while the student network implements augmentation measures, including random color jittering, grayscale, Gaussian blurring, and random shearing, to simulate the complex changes in real scenes and force the student network to learn more robust feature representations. Step S13: Initialize the network weights and parameters of the pre-trained ResNet50 feature extraction network for the image to be detected in the cross-domain small sample target detection dataset.

3. The cross-domain small sample target detection method based on scale matching and semantic enhancement according to claim 1 is characterized in that: Step S3 specifically includes the following steps: Step S31: Design a covariance matrix estimation module to dynamically estimate the covariance matrix of different category features based on the input feature data; this module maintains the covariance matrix, mean matrix and sample size information of each category during the training process, and updates these statistical information based on new samples; First, reshape the features into the shape of N×1×A to match the dimensions required by subsequent operations, where N and A are the number of samples and the number of features, respectively. Then, calculate the one-hot encoding of the label to determine the category membership of each sample. Next, calculate the mean vector ave of the feature data under each category. c，a , as the representative vector of the category feature; ave c，a The specific calculation formula is as follows: Among them, c represents the category index, a represents the dimension of the feature vector, features i，a is the reshaped feature value, i represents the sample index, onehot c，i is the one-hot category vector of sample i, Amount c，a The number of features representing category c; Step S32: Based on the above covariance matrix generation, data expansion of implicit semantics is performed; first, the feature weight matrix M_ij is obtained from the network weights, which maps the features to the category space; the shape of this matrix is ​​N×C×A, where: N is the number of samples, C is the number of categories, and A is the number of features; the formula of the feature weight matrix M_ij is as follows: M_ij=weight_m×ones(N,C,A) Among them, weight_m is the weight parameter of the network, and ones(N, C, A) is a full-1 tensor with shape N×C×A; Step S33: Obtain the target feature weight matrix M_kj from the feature weight matrix according to the true label labels, which is used to calculate the pseudo label of the sample; the shape of the target feature weight matrix is ​​the same as the feature weight matrix; the formula of the target feature weight matrix M_kj is as follows: M_kj=gather(M_ij, 1, labels) Where gather(·) indicates indexing according to a given dimension; Step S34: Calculate the pseudo label isda_aug_y of the sample based on the feature weight matrix, the target feature weight matrix, and the covariance matrix; the pseudo label will be used to calculate the loss value. The formula of the pseudo label isda_aug_y is as follows: sigma2=ratio×(weight_m-M_kj) 2 ×CoVariance c，a isda_aug_y=y+0.5×sigma2 Among them, ratio is the ratio parameter of loss, CoVariance c，a is the covariance matrix obtained from the covariance estimator, y is the original output of the model, and sigma2 represents the adjustment value of the pseudo label of the sample, which adjusts the original output to a form more suitable for the target domain; Step S35: Finally, the cross entropy loss function is used to calculate the loss value between the pseudo label and the true label; this loss value will be used as the training target of the model and minimized by optimizing the network parameters; the loss value L aug The formula is as follows: L aug =CE(fish_aug_y, labels) Among them, CE represents the cross entropy loss function, and labels is the true label of the sample.

4. The cross-domain small sample target detection method based on scale matching and semantic enhancement according to claim 1 is characterized in that: Step S4 specifically includes the following steps: Step S41: First, based on the classic framework of cross-domain object detection, it is divided into two phases, namely the training phase and the testing phase. In the training phase, the base class data is used to train the detector to improve detection performance. Subsequently, before fine-tuning the new class, the initialized detector parameters are copied to the student model. Before the formal distillation, the student model needs to go through a "running-in" phase. During this phase, the model is fine-tuned for a specific number of new class data using the standard detection supervision loss to adapt to the needs of cross-domain small-sample object detection. After the running-in is completed, the weights of the student model will be synchronized to the teacher model to prepare for subsequent distillation training. During distillation, the teacher model and the student model are trained in a collaborative manner. Step S42: Use the knowledge distillation architecture to take the strongly enhanced image as the input of the student model and the weakly enhanced image as the input of the teacher model to provide reliable pseudo labels; in the supervision branch, calculate the supervised detection loss of the student model: the classification loss L cls and the localization loss L loc ; In new class samples In the case of supervised detection loss L S The calculation formula is as follows: in, It represents the image value. It represents the true value of the label, s refers to the new class sample, i refers to the i-th new class sample, L cls Represents the classification loss, L loc It represents the positioning loss; Step S43: The teacher and student share the same architecture and are initialized with the same weights after the burn-in step; the image is processed independently by the student and teacher; the teacher is used to generate thousands of box proposals for the weakly enhanced version and remove redundancy through non-maximum suppression (NMS); the student model uses the strongly enhanced version for prediction and obtains the distillation loss L by calculating the detection loss between the student prediction and the pseudo-label D ;L D The specific calculation formula is as follows: in, For the correct result in the case of distillation, Predictions generated for the distillation case; Step S44: An exponential moving average (EMA) is used to separate the teacher and the student. The teacher is used to perform reasoning and evaluation on the test image. It is worth noting that no data augmentation is performed on the input image during the reasoning phase. At this point, before knowledge distillation, the loss function calculation formula is as follows: L=L S +L dssom +L aug Among them, L represents the value of the loss function, L S It represents the supervised detection loss value, L dssom It represents the loss value of the cross-domain scale shape optimization matching module generated in step S2, L aug It represents the loss value of the dynamic semantic covariance enhancement module generated in step S3; After the knowledge distillation calculation, the new loss function calculation formula is as follows: L=L S +L D +L dssom +L aug Among them, L D It represents the distillation loss; Step S45: Test the trained teacher model on the validation set to obtain the final detection accuracy and obtain a training model that is suitable for the new class data set.

Images