Cross-domain small sample target detection method based on scale matching and semantic enhancement

By introducing scale matching and semantic enhancement methods in object detection, combining knowledge distillation and dynamic semantic covariance enhancement, data scarcity and model generalization difficulty in cross-domain small sample object detection are solved, and detection performance and adaptation are improved sex.

CN118941851AActive Publication Date: 2024-11-12FUZHOU UNIV

Patent Information

Application Number
CN202410976440.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-20
Publication Date
2024-11-12
Estimated Expiration
2044-07-20

AI Technical Summary

Technical Problem

In the field of target detection, especially in specific fields such as medical care and aviation, cross-domain small sample target detection has problems of data scarcity and difficulty in generalizing the model. It is difficult for traditional methods to effectively utilize limited labeled data and overcome cross-domain differences.

Method used

Using a cross-domain small sample object detection method based on scale matching and semantic enhancement, the cross-domain scale shape optimization matching module and dynamic semantic covariance enhancement module are introduced through the strength enhancement architecture of knowledge distillation, and the cross-domain scale shape optimization matching module and dynamic semantic covariance enhancement module are introduced to expand implicit semantic data and improve anchors. Box matching and detection accuracy.

Benefits of technology

In different fields and with few samples, the generalization ability of the model is significantly improved, the impact of scale and shape differences is overcome, and a data augmentation method with implicit semantics is provided, which enhances detection performance and adapts to practical application scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941851A_ABST
    Figure CN118941851A_ABST
Patent Text Reader

Abstract

The invention provides a cross-domain small sample target detection method based on scale matching and semantic enhancement, and the method comprises the steps: carrying out the differential data enhancement of an input to-be-detected image according to the input difference of a teacher network and a student network; and then optimization matching is carried out on the cross-domain scale shape, and the problem that the detection precision is influenced by the scale shape under the condition of different domains is solved from the cross-domain perspective. And then interpolation or extrapolation is carried out on the data sample by using the semantic space to generate a new data sample, so that the data is expanded, and the problem of data scarcity existing in cross-domain small sample target detection is solved. And finally, knowledge distillation is carried out, the teacher model is used as a final training model, and an output result is a final detection result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of target detection and computer vision technology, and in particular to a cross-domain small sample target detection method based on scale matching and semantic enhancement. Background Art

[0002] In the field of object detection, the small sample setting is an important and challenging topic. The traditional small sample object detection paradigm often involves two stages: first, pre-training on a source dataset consisting of base classes, and then transferring the pre-trained detector to a target dataset, which contains a small number of instances belonging to new categories that do not overlap with the base classes but have similar data domains. However, this setting has significant limitations in practical applications, especially in specific fields such as medicine and aviation. In these fields, due to the difficulty and sensitivity of data collection, it is difficult or almost impossible to obtain a sufficient number of target category samples, let alone satisfy the assumption of domain similarity. With the rapid development of computer vision technology, object detection, as one of the key tasks, has made significant progress. However, in practical applications, cross-domain small sample object detection is still a challenging problem. This is mainly due to the large differences in scale, shape, texture, etc. between objects in different fields or scenarios, and the small sample data increases the generalization difficulty of the model. Traditional object detection methods mainly rely on a large amount of labeled data for training, but such data is often difficult to obtain in cross-domain small sample scenarios. Therefore, how to effectively utilize limited annotated data and overcome cross-domain differences has become a pressing problem in the field of object detection. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a cross-domain small sample target detection method based on scale matching and semantic enhancement, which improves the anchor box matching in cross-domain situations and uses the semantic space to interpolate or extrapolate data samples, thereby performing implicit semantic data expansion and increasing sample diversity.

[0004] To achieve the above object, the present invention adopts the following technical solution: a cross-domain small sample target detection method based on scale matching and semantic enhancement, comprising the following steps:

[0005] Step S1: Use a cross-domain small sample target detection universal training set, and use strong and weak data enhancement methods on the data respectively, and send the data to the ResNet50 backbone network for feature training;

[0006] Step S2: Using the trained feature extraction network as the feature extraction module, introducing a cross-domain scale shape optimization matching module to solve the problem that the scale shape in different domains affects the detection accuracy;

[0007] Step S3: Design a dynamic semantic covariance enhancement module to interpolate or extrapolate data samples using semantic space to generate new data samples, thereby expanding the data;

[0008] Step S4: Perform knowledge distillation. The teacher model and the student model are trained in a collaborative manner. The teacher model uses distillation loss to guide the student model to promote its performance. The student model continuously optimizes parameter weights through the exponential moving average (EMA) strategy and gradually approaches the teacher model. Finally, the features of the teacher model are used for cross-domain small sample target detection.

[0009] In a preferred embodiment, step S1 specifically includes the following steps:

[0010] Step S11: Obtain a public small sample segmentation training set from the Internet and obtain relevant annotations of the training data; the model is trained from a base class dataset with a large number of annotated instances for each class, and then the model is migrated and fine-tuned to the case where each class in the target dataset has only a few samples; the base class dataset and the new class dataset come from different fields, the MSCOCO2017 dataset is set as the base class dataset, and the ArTaxOr, UODD, and DIOR datasets are used as new class datasets, and their fields are respectively biological, underwater, and aviation;

[0011] Step S12: After the data is loaded, different data enhancements are performed on the data of the teacher network and the student network; the teacher network adopts a weak enhancement method, i.e., random flipping, and the student network implements enhancement measures, including random color jittering, grayscale, Gaussian blurring, and random shearing, to simulate complex changes in real scenes and force the student network to learn more robust feature representations;

[0012] Step S13: Initialize the network weights and parameters of the image to be detected in the cross-domain small sample target detection dataset using a pre-trained ResNet50 feature extraction network.

[0013] In a preferred embodiment, step S2 specifically includes the following steps:

[0014] Step S21: Define A to represent the detection box, B to represent the real box, Area(·) to represent the area of ​​the corresponding area, and A∩B to represent the intersection area of ​​the detection box and the real box. First, obtain the width w and height h of each detection box, as well as the width w′ and height h′ of the real box, and calculate their areas Area(A) and Area(B). The specific calculation is as follows:

[0015] Area(A)=w×h, Area(B)=w′×h′

[0016] Step S22: Calculate the width difference s between the center point of the detection frame and the real frame c w and height difference s c h, and the distance sigma between them; the calculation formula is as follows:

[0017]

[0018] Among them, x′ min , x′ max are the minimum and maximum horizontal coordinates of the real box, respectively, min 、x max are the minimum and maximum horizontal coordinates of the detection box respectively; similarly, y′ min , y′ max are the minimum and maximum ordinates of the real box, y min ,y max are the minimum and maximum ordinates of the detection box respectively;

[0019] Step S23: according to the angle threshold threshold and the calculation formula of the center point angle α, the angle loss angle_cost is calculated; the calculation formula of the center point angle is as follows, and the angle loss calculation formula is specifically as follows:

[0020]

[0021] Among them, σ represents the scale of the anchor box;

[0022] Step S24: Calculate the convex hull width cw and convex hull height ch of the detection frame and the real frame, as well as the center point distance sigma, and then calculate the distance loss distance_cost and shape loss shape_cost; the convex hull width cw refers to the maximum distance between the detection frame and the real frame in the horizontal direction, that is, the maximum overlapping length of the detection frame and the real frame in the horizontal direction; the convex hull height ch refers to the maximum distance between the detection frame and the real frame in the vertical direction, that is, the maximum overlapping height of the detection frame and the real frame in the vertical direction; the specific calculation formula is as follows:

[0023] CW=max(w,w′), ch=max(h,h′)

[0024]

[0025] distance_cost = 2-exp(γ·rho x )-exp(γ·rho y )

[0026]

[0027] Where w is the width of the detection box, h is the height of the detection box, w′ is the width of the real box, and h′ is the height of the real box. In the calculation of shape and distance loss, the rho value is introduced to measure the shape difference between the detection box and the real box. x Refers to the difference in width between the detection box and the real box; rho y Refers to the difference in height between the detection box and the real box; the calculation of the rho value involves the center point difference s c w and s c h is the ratio of the convex hull width cw and the convex hull height ch; ∈ is a very small positive number, which is used to prevent the calculation of rho from being divided by zero when the convex hull width cw and the convex hull height ch are zero; γ represents a weight parameter, which is used to adjust the relative importance of shape and distance losses; the larger the γ, the greater the influence of shape loss and distance loss on the dssom value; the smaller the γ, the smaller the influence of shape loss and distance loss;

[0028] Step S25: Calculate the area of ​​the intersection region according to the calculation formula of the intersection region. area , the specific calculation is as follows:

[0029] irtter area =(x max_inter -x min_inter )×(y max_inter -y min_inter )

[0030] Among them, x max_inter and x min_inter is the maximum and minimum horizontal coordinates of the intersection area; y max_inter and min_inter is the maximum and minimum ordinates of the intersection area;

[0031] Step S26: Finally, the area of ​​the intersection region is divided by the area of ​​the union region, and then half of the distance loss and shape loss are subtracted to obtain the final IoU loss value L dssom , the specific calculation formula is as follows:

[0032]

[0033] In a preferred embodiment, step S3 specifically includes the following steps:

[0034] Step S31: Design a covariance matrix estimation module to dynamically estimate the covariance matrix of different categories of features based on the input feature data; this module maintains the covariance matrix, mean matrix and sample quantity information of each category during the training process, and updates these statistical information based on new samples;

[0035] First, reshape the features into the shape of N×1×A to match the dimensions required by subsequent operations, where N and A are the number of samples and the number of features, respectively. Then, calculate the one-hot encoding of the label to determine the category membership of each sample. Next, calculate the mean vector ave of the feature data under each category. c,a , as the representative vector of the category features; ave c,a The specific calculation formula is as follows:

[0036]

[0037] Among them, c represents the category index, a represents the dimension of the feature vector, and features i , a is the reshaped feature value, i represents the sample index, onehot c,i is the one-hot category vector of sample i, Amount c , a represents the number of features of category c;

[0038] Step S32: Based on the above covariance matrix generation, implicit semantic data expansion is performed; first, the feature weight matrix M_ij is obtained from the network weights, which maps the features to the category space; the shape of the matrix is ​​N×C×A, where: N is the number of samples, C is the number of categories, and A is the number of features; the formula of the feature weight matrix M_ij is as follows:

[0039] M_ij=weight_m×ones(N,C,A)

[0040] Among them, weight_m is the weight parameter of the network, and ones(N, C, A) is a full-1 tensor with a shape of N×C×A;

[0041] Step S33: Obtain the target feature weight matrix M from the feature weight matrix according to the true label labels _ kj, used to calculate the pseudo-label of the sample; the shape of the target feature weight matrix is ​​the same as the feature weight matrix; the formula of the target feature weight matrix M_kj is as follows:

[0042] M_kj=gather(M_ij, 1, labels)

[0043] Where gather(·) means indexing according to a given dimension;

[0044] Step S34: Calculate the pseudo label isda_aug_y of the sample according to the feature weight matrix, the target feature weight matrix, and the covariance matrix; the pseudo label will be used to calculate the loss value, and the formula of the pseudo label isda_aug_y is as follows:

[0045] sigma2=ratio×(weight_m-M_kj) 2 ×CoVariance c,a

[0046] isda_aug_y=y+0.5×sigma2

[0047] Among them, ratio is the ratio parameter of loss, CoVariance c , a is the covariance matrix obtained from the covariance estimator, y is the original output of the model, sigma2 represents the adjustment value of the pseudo label of the sample, which adjusts the original output to a form more suitable for the target domain;

[0048] Step S35: Finally, the cross entropy loss function is used to calculate the loss value between the pseudo label and the true label; this loss value will be used as the training target of the model, and the loss value will be minimized by optimizing the network parameters; the loss value L aug The formula is as follows:

[0049] L aug =CE(isda_aug_y, labels)

[0050] Among them, CE represents the cross entropy loss function, and labels is the true label of the sample.

[0051] In a preferred embodiment, step S4 specifically includes the following steps:

[0052] Step S41: First, based on the classic framework of cross-domain target detection, it is divided into two stages, namely the training stage and the testing stage; in the training stage, the base class data is used to train the detector to improve the detection performance; then, before fine-tuning the new class, the initialized detector parameters are copied to the student model; the student model needs to go through a "running-in" stage before formal distillation; in this stage, the model uses the standard detection supervision loss to make fine adjustments for a specific number of new class data to meet the needs of cross-domain small sample target detection; after the running-in is completed, the weights of the student model will be synchronized to the teacher model to prepare for subsequent distillation training; when distilling, the teacher model and the student model are trained in a collaborative manner;

[0053] Step S42: Use the knowledge distillation architecture to take the strongly enhanced image as the input of the student model and the weakly enhanced image as the input of the teacher model to provide reliable pseudo labels; in the supervision branch, calculate the supervised detection loss of the student model: the classification loss L cls and the localization loss L loc ; In the new class sample In the case of supervised detection loss L S The calculation formula is as follows:

[0054]

[0055] in, It represents the image value. It represents the true value of the label, s refers to the new class sample, i refers to the i-th new class sample, L cls represents the classification loss, L loc It represents the positioning loss;

[0056] Step S43: The teacher and student share the same architecture and are initialized with the same weights after the burn-in step; the image is processed independently by the student and the teacher; the teacher is used to generate thousands of box candidates for the weakly enhanced version and remove redundancy through non-maximum suppression (NMS); the student model uses the strongly enhanced version for prediction and obtains the distillation loss L by calculating the detection loss between the student prediction and the pseudo-label D ; L D The specific calculation formula is as follows:

[0057]

[0058] in, For the correct result in the case of distillation, Predictions generated for the distillation case;

[0059] Step S44: An exponential moving average (EMA) is used to separate the teacher and the student. The teacher is used to perform reasoning and evaluation on the test image. It is worth noting that no data augmentation operation is performed on the input image during the reasoning phase.

[0060] At this point, before knowledge distillation, the loss function calculation formula is as follows:

[0061] L=L S +L dssom +L aug

[0062] Among them, L represents the value of the loss function, L S It represents the supervised detection loss value, L dssom It represents the loss value of the cross-domain scale shape optimization matching module generated in step S2, L aug It represents the loss value of the dynamic semantic covariance enhancement module generated in step S3;

[0063] After the knowledge distillation calculation, the new loss function calculation formula is as follows:

[0064] L=L S +L D +L dssom +Laug

[0065] Among them, L D It represents the distillation loss;

[0066] Step S45: Test the trained teacher model on the validation set to obtain the final detection accuracy and obtain a training model suitable for the new class data set.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] 1. Different from small sample target detection, considering the domain disjointness between base class and new class, introducing small sample learning on the basis of cross-domain target detection can be more in line with actual application scenarios.

[0069] 2. By utilizing the teacher and student networks in knowledge distillation, the two develop synergistically, which can achieve excellent detection performance in different fields and with fewer samples, greatly improving the generalization ability of the model.

[0070] 3. The scale and shape differences between the detection box and the real box are taken into consideration. From a cross-domain perspective, the problem of scale and shape affecting detection accuracy in different fields is solved, and a cross-domain scale and shape matching module is designed.

[0071] 4. We further solved the problem of data scarcity in cross-domain small sample target detection and proposed an implicit semantic data expansion method, called the dynamic semantic covariance enhancement module, which uses the semantic space to interpolate or extrapolate data samples to generate new data samples, thereby expanding the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 The present invention is a flowchart of a method implementation of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0073] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0074] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0075] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0076] like Figure 1 As shown, the trial example of the present invention provides a cross-domain small sample target detection method based on scale matching and semantic enhancement, comprising the following steps:

[0077] Step S1: Use a cross-domain small sample target detection universal training set, and use strong and weak data enhancement methods on the data respectively, and send the data to the ResNet50 backbone network for feature training. Specifically, it includes the following steps:

[0078] Step S11: Obtain a public small sample segmentation training set from the Internet and obtain relevant annotations of the training data; the model is trained from a base class dataset with a large number of annotated instances for each class, and then the model is migrated and fine-tuned to the case where each class in the target dataset has only a few samples. The base class dataset and the new class dataset come from different fields. The MSCOCO2017 dataset is set as the base class dataset, and the ArTaxOr, UODD, and DIOR datasets are used as new class datasets, and their fields are respectively biological, underwater, and aviation;

[0079] Step S12: Since the knowledge distillation architecture is used, different data enhancements are performed on the teacher network and the student network after the data is loaded. The teacher network only uses a weak enhancement method, i.e., random flipping, to maintain its stability to the original data. The student network implements stronger enhancement measures, including random color jitter, grayscale, Gaussian blur, and random shearing, to simulate the complex changes in real scenes and force the student network to learn more robust feature representations;

[0080] Step S13: Initialize the network weights and parameters of the image to be detected in the cross-domain small sample target detection dataset using a pre-trained ResNet50 feature extraction network.

[0081] Step S2: Use the trained feature extraction network as the feature extraction module and introduce a cross-domain scale shape optimization matching module to solve the problem that the scale shape in different domains affects the detection accuracy. Specifically, the following steps are included:

[0082] Step S21: Define A to represent the detection box, B to represent the real box, Area(·) to represent the area of ​​the corresponding region, and A∩B to represent the intersection area of ​​the detection box and the real box. First, obtain the width w and height h of each detection box, as well as the width w′ and height h′ of the real box, and calculate their areas Area(A) and Area(B). The specific calculation is as follows:

[0083] Area(A)=w×h, Area(B)=w′×h′

[0084] Step S22: Calculate the width difference s between the center point of the detection frame and the real frame c w and height difference s c h, and the distance sigma between them. The calculation formula is as follows:

[0085]

[0086] Step S23: According to the angle threshold threshold and the calculation formula of the center point angle, the angle loss angle_cost is calculated. The calculation formula of the center point angle is as follows, and the angle loss calculation formula is specifically as follows:

[0087]

[0088] Step S24: Calculate the convex hull width cw and convex hull height ch of the detection frame and the real frame, as well as the center point distance sigma, and then calculate the distance loss distance_cost and shape loss shape_cost. The convex hull width cw refers to the maximum distance between the detection frame and the real frame in the horizontal direction, that is, the maximum overlapping length of the detection frame and the real frame in the horizontal direction; the convex hull height ch refers to the maximum distance between the detection frame and the real frame in the vertical direction, that is, the maximum overlapping height of the detection frame and the real frame in the vertical direction. The specific calculation formula is as follows:

[0089] CW=max(w,w′), ch=max(h,h′)

[0090]

[0091] distance_cost = 2-exp(γ·rho x )-exp(γ·rho y )

[0092]

[0093] Among them, the rho value is introduced in the calculation of shape and distance loss to measure the shape difference between the detection box and the real box. xRefers to the difference in width between the detection box and the real box; rho y Refers to the difference in height between the detection box and the real box; the calculation of the rho value involves the center point difference s c w and s c h is the ratio of the convex hull width cw and the convex hull height ch. ∈ is a very small positive number, which is used to prevent the calculation of rho from being divided by zero when the convex hull width cw and the convex hull height ch are zero. γ represents a weight parameter used to adjust the relative importance of shape and distance losses. The larger γ is, the greater the influence of shape loss and distance loss on the dssom value; the smaller γ is, the smaller the influence of shape loss and distance loss is;

[0094] Step S25: Calculate the area of ​​the intersection region according to the calculation formula of the intersection region. area , the specific calculation is as follows:

[0095] inter_area=(x max_inter -x min_inter )×(y max_inter -y min_inter )

[0096] Step S26: Finally, the area of ​​the intersection region is divided by the area of ​​the union region, and then half of the distance loss and shape loss are subtracted to obtain the final IoU loss value L dssom , the specific calculation formula is as follows:

[0097]

[0098] By comprehensively considering the differences in scale and shape, the matching degree between the detection box and the true box can be evaluated more accurately. Taking into account the differences in scale and shape between the detection box and the true box, it has better adaptability to the scale and shape changes of the target. In cross-domain scenarios, since data from different fields may have different scale and shape characteristics, L dssom It can more accurately measure the degree of match between the detection box and the true box.

[0099] Step S3: Design a dynamic semantic covariance enhancement module, use the semantic space to interpolate or extrapolate the data samples to generate new data samples, thereby expanding the data. Specifically, the following steps are included:

[0100] Step S31: In order to enhance the features generated by the model, it is necessary to calculate the data augmentation value based on the covariance matrix, model weights and sample labels. A covariance matrix estimation module is designed to dynamically estimate the covariance matrix of different categories of features based on the input feature data. This module maintains the covariance matrix, mean matrix and sample quantity information of each category during the training process, and updates these statistical information based on new samples;

[0101] First, reshape the features into the shape of N×1×A to match the dimensions required by subsequent operations, where N and A are the number of samples and the number of features, respectively. Then calculate the one-hot encoding of the label to determine the category membership of each sample. Next, calculate the mean vector ave of the feature data under each category c,a , as the representative vector of the category features. ave c,a The specific calculation formula is as follows:

[0102]

[0103] Among them, c represents the category index, a represents the dimension of the feature vector, and features i,a is the reshaped feature value, i represents the sample index, onehot c,i is the one-hot category vector of sample i, Amount c,a The number of features representing category c;

[0104] Step S32: Based on the above covariance matrix generation, data expansion of implicit semantics is performed. First, this section obtains the feature weight matrix M_ij from the network weights, which maps the features to the category space. The shape of this matrix is ​​N×C×A, where: N is the number of samples, C is the number of categories, and A is the number of features. The formula of the feature weight matrix M_ij is as follows:

[0105] M_ij=weight_m×ones(N,C,A)

[0106] Among them, weight_m is the weight parameter of the network, and ones(N, C, A) is a full-1 tensor with a shape of N×C×A;

[0107] Step S33: Obtain the target feature weight matrix M_kj from the feature weight matrix according to the real label labels, which is used to calculate the pseudo label of the sample. The shape of the target feature weight matrix is ​​the same as the feature weight matrix. The formula of the target feature weight matrix M_kj is as follows:

[0108] M_kj=gather(M_ij, 1, labels)

[0109] Where gather(·) means indexing according to a given dimension;

[0110] Step S34: Calculate the pseudo label isda_aug_y of the sample according to the feature weight matrix, the target feature weight matrix, and the covariance matrix. The pseudo label will be used to calculate the loss value. The formula of the pseudo label isda_aug_y is as follows:

[0111] sigma2=ratio×(weight_m-M_kj) 2 ×CoVariance c,a

[0112] isda_aug_y=y+0.5×sigma2

[0113] Among them, ratio is the ratio parameter of loss, CoVariance c,a is the covariance matrix obtained from the covariance estimator, y is the original output of the model, and sigma2 represents the adjustment value of the pseudo label of the sample, which adjusts the original output to a form more suitable for the target domain;

[0114] Step S35: Finally, the cross entropy loss function is used to calculate the loss value between the pseudo label and the true label. This loss value will be used as the training target of the model, and the loss value will be minimized by optimizing the network parameters. aug The formula is as follows:

[0115] L aug =CE(isda_aug_y, labels)

[0116] Among them, CE represents the cross entropy loss function, and labels is the true label of the sample.

[0117] Step S4: Perform knowledge distillation, and the teacher model and the student model are trained in a collaborative way. The teacher model uses distillation loss to guide the student model to promote its performance improvement; while the student model continuously optimizes the parameter weights through the exponential moving average (EMA) strategy, gradually moving closer to the teacher model. Finally, the features of the teacher model are used for cross-domain small sample target detection. Specifically, the following steps are included:

[0118] Step S41: First, based on the classic framework of cross-domain target detection, it is divided into two major stages, namely the training stage and the testing stage. In the training stage, the base class data is used to train the detector to improve the detection performance. Subsequently, before fine-tuning the new class, the initialized detector parameters are copied to the student model. The student model needs to go through a "running-in" stage before formal distillation. In this stage, the model uses standard detection supervision losses to make fine adjustments to a specific number of new class data to meet the needs of cross-domain small sample target detection. After the running-in is completed, the weights of the student model will be synchronized to the teacher model to prepare for subsequent distillation training. During distillation, the teacher model and the student model are trained in a collaborative manner;

[0119] Step S42: The knowledge distillation architecture is used to take the strongly enhanced image as the input of the student model and the weakly enhanced image as the input of the teacher model to provide reliable pseudo labels. The purpose of this is to increase the diversity of the student model, thereby improving the performance and generalization ability of the model. In the supervision branch, the supervised detection loss (classification loss L cls and the localization loss L loc ). In the new class sample In the case of supervised detection loss L S The calculation formula is as follows:

[0120]

[0121] in, It represents the image value. It represents the true value of the label, s refers to the new class sample, i refers to the i-th new class sample, L cls represents the classification loss, L loc It represents the positioning loss;

[0122] Step S43: The teacher and student share the same architecture and are initialized with the same weights after the burn-in step. The image is processed independently by the student and the teacher. The teacher is used to generate thousands of box candidates for the weakly enhanced version and remove redundancy through non-maximum suppression (NMS). Although most redundant boxes are removed, there may still be some non-foreground candidate boxes. Therefore, only candidates with foreground scores above the threshold are retained as pseudo boxes. The student model uses the strongly enhanced version for prediction and obtains the distillation loss L by calculating the detection loss between the student prediction and the pseudo label. D .L D The specific calculation formula is as follows:

[0123]

[0124] in, For the correct result in the case of distillation, Predictions generated for the distillation case;

[0125] Step S44: To alleviate the overfitting problem, an exponential moving average (EMA) is used to separate the teacher and the student. The teacher is used to perform reasoning and evaluation on the test image. It is worth noting that in the reasoning stage, no data augmentation operation is performed on the input image;

[0126] At this point, before knowledge distillation, the loss function calculation formula is as follows:

[0127] L=L S +L dssom +L aug

[0128] Among them, L represents the value of the loss function, L S It represents the supervised detection loss value, L dssom It represents the loss value of the cross-domain scale shape optimization matching module generated in step S2, L aug It represents the loss value of the dynamic semantic covariance enhancement module generated in step S3;

[0129] After the knowledge distillation calculation, the new loss function calculation formula is as follows:

[0130] L=L S +L D +L dssom +L aug

[0131] Among them, L D It represents the distillation loss;

[0132] Step S45: Test the trained teacher model on the validation set to obtain the final detection accuracy and obtain a training model suitable for the new class data set.

[0133] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0134] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0135] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0136] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0137] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

[0138] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive other forms of cross-domain small sample target detection methods based on scale matching and semantic enhancement under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of the present invention should be covered by this patent.

Claims

1. A cross-domain small sample target detection method based on scale matching and semantic enhancement, It is characterized in that The following steps are involved: Step S1: Use a cross-domain small sample target detection universal training set, and use strong and weak data enhancement methods on the data respectively, and send the data to the ResNet50 backbone network for feature training; Step S2: Using the trained feature extraction network as the feature extraction module, introducing a cross-domain scale shape optimization matching module to solve the problem that the scale shape in different domains affects the detection accuracy; Step S3: Design a dynamic semantic covariance enhancement module to interpolate or extrapolate data samples using semantic space to generate new data samples, thereby expanding the data; Step S4: Perform knowledge distillation, and the teacher model and the student model are trained in a collaborative manner; the teacher model uses distillation loss to guide the student model to promote its performance improvement; The student model continuously optimizes parameter weights through the exponential moving average (EMA) strategy, gradually approaching the teacher model; finally, the characteristics of the teacher model are used to perform cross-domain small sample target detection.

2. According to claim 1, a cross-domain small sample target detection method based on scale matching and semantic enhancement, It is characterized in that Step S1 specifically includes the following steps: Step S11: Obtain a public small sample segmentation training set from the Internet and obtain relevant annotations of the training data; the model is trained from a base class dataset with a large number of annotated instances for each class, and then the model is migrated and fine-tuned to the case where each class in the target dataset has only a few samples; the base class dataset and the new class dataset come from different fields, the MSCOCO2017 dataset is set as the base class dataset, and the ArTaxOr, UODD, and DIOR datasets are used as new class datasets, and their fields are respectively biological, underwater, and aviation; Step S12: After the data is loaded, different data enhancements are performed on the data of the teacher network and the student network; the teacher network adopts a weak enhancement method, i.e., random flipping, and the student network implements enhancement measures, including random color jittering, grayscale, Gaussian blurring, and random shearing, to simulate complex changes in real scenes and force the student network to learn more robust feature representations; Step S13: Initialize the network weights and parameters of the image to be detected in the cross-domain small sample target detection dataset using a pre-trained ResNet50 feature extraction network.

3. According to claim 1, a cross-domain small sample target detection method based on scale matching and semantic enhancement, It is characterized in that Step S2 specifically includes the following steps: Step S21: Define A to represent the detection box, B to represent the real box, Area(·) to represent the area of ​​the corresponding area, and A∩B to represent the intersection area of ​​the detection box and the real box. First, obtain the width w and height h of each detection box, as well as the width w′ and height h′ of the real box, and calculate their areas Area(A) and Area(B). The specific calculation is as follows: Area(A)=w×h, Area(B)=w′×h′ Step S22: Calculate the width difference s between the center point of the detection frame and the real frame c w and height difference s c h, and the distance sigma between them; the calculation formula is as follows: Among them, x′ min , x′ max are the minimum and maximum horizontal coordinates of the real box, respectively, min 、x max are the minimum and maximum horizontal coordinates of the detection box respectively; similarly, y′ min , y′ max are the minimum and maximum ordinates of the real box, y min ,y max are the minimum and maximum ordinates of the detection box respectively; Step S23: according to the angle threshold threshold and the calculation formula of the center point angle α, the angle loss angle_cost is calculated; the calculation formula of the center point angle is as follows, and the angle loss calculation formula is specifically as follows: Among them, σ represents the scale of the anchor box; Step S24: Calculate the convex hull width cw and convex hull height ch of the detection frame and the real frame, as well as the center point distance sigma, and then calculate the distance loss distance_cost and shape loss shape_cost; the convex hull width cw refers to the maximum distance between the detection frame and the real frame in the horizontal direction, that is, the maximum overlapping length of the detection frame and the real frame in the horizontal direction; the convex hull height ch refers to the maximum distance between the detection frame and the real frame in the vertical direction, that is, the maximum overlapping height of the detection frame and the real frame in the vertical direction; the specific calculation formula is as follows: CW=max(w,w′), ch=max(h,h′) distance_cost=2-exp(γ·rho x )-exp(γ·rho y ) Where w is the width of the detection box, h is the height of the detection box, w′ is the width of the real box, and h′ is the height of the real box. In the calculation of shape and distance loss, the rho value is introduced to measure the shape difference between the detection box and the real box. x Refers to the difference in width between the detection box and the real box; rho y Refers to the difference in height between the detection box and the real box; the calculation of the rho value involves the center point difference s c w and s c h is the ratio of the convex hull width cw and the convex hull height ch; ∈ is a very small positive number, which is used to prevent the calculation of rho from being divided by zero when the convex hull width cw and the convex hull height ch are zero; γ represents a weight parameter, which is used to adjust the relative importance of shape and distance losses; the larger the γ, the greater the influence of shape loss and distance loss on the dssom value; the smaller the γ, the smaller the influence of shape loss and distance loss; Step S25: Calculate the area of ​​the intersection region according to the calculation formula of the intersection region. area , the specific calculation is as follows: irtter area =(x max_inter -x min_inter )×(y max_inter -y min_inter ) Among them, x max_inter and x min_inter is the maximum and minimum horizontal coordinates of the intersection area; y max_inter and min_inter is the maximum and minimum ordinates of the intersection area; Step S26: Finally, the area of ​​the intersection region is divided by the area of ​​the union region, and then half of the distance loss and shape loss are subtracted to obtain the final IoU loss value L dssom , the specific calculation formula is as follows:

4. According to claim 1, a cross-domain small sample target detection method based on scale matching and semantic enhancement, It is characterized in that Step S3 specifically includes the following steps: Step S31: Design a covariance matrix estimation module to dynamically estimate the covariance matrix of different categories of features based on the input feature data; this module maintains the covariance matrix, mean matrix and sample quantity information of each category during the training process, and updates these statistical information based on new samples; First, reshape the features into the shape of N×1×A to match the dimensions required by subsequent operations, where N and A are the number of samples and the number of features, respectively. Then, calculate the one-hot encoding of the label to determine the category membership of each sample. Next, calculate the mean vector ave of the feature data under each category. c,a , as the representative vector of the category features; ave c , the specific calculation formula of a is as follows: Among them, c represents the category index, a represents the dimension of the feature vector, and features i,a is the reshaped feature value, i represents the sample index, onehot c,i is the one-hot category vector of sample i, Amount c,a The number of features representing category c; Step S32: Based on the above covariance matrix generation, implicit semantic data expansion is performed; first, the feature weight matrix M_ij is obtained from the network weights, which maps the features to the category space; the shape of the matrix is ​​N×C×A, where: N is the number of samples, C is the number of categories, and A is the number of features; the formula of the feature weight matrix M_ij is as follows: M_ij=weight_m×ones(N,C,A) Among them, weight_m is the weight parameter of the network, and ones(N, C, A) is a full-1 tensor with a shape of N×C×A; Step S33: Obtain the target feature weight matrix M from the feature weight matrix according to the true label labels _ kj, used to calculate the pseudo-label of the sample; the shape of the target feature weight matrix is ​​the same as the feature weight matrix; the target feature weight matrix M _ The formula for kj is as follows: M_kj=gather(M_ij, 1, labels) Where gather(·) means indexing according to a given dimension; Step S34: Calculate the pseudo label isda_aug_y of the sample according to the feature weight matrix, the target feature weight matrix, and the covariance matrix; the pseudo label will be used to calculate the loss value, and the formula of the pseudo label isda_aug_y is as follows: sigma2=ratio×(weight_m-M_kj) 2 ×CoVariance c,a isda_aug_y=y+0.5×sigma2 Among them, ratio is the ratio parameter of loss, CoVariance c , a is the covariance matrix obtained from the covariance estimator, y is the original output of the model, sigma2 represents the adjustment value of the pseudo label of the sample, which adjusts the original output to a form more suitable for the target domain; Step S35: Finally, the cross entropy loss function is used to calculate the loss value between the pseudo label and the true label; this loss value will be used as the training target of the model, and the loss value will be minimized by optimizing the network parameters; the loss value L aug The formula is as follows: L aug =CE(fish_aug_y, labels) Among them, CE represents the cross entropy loss function, and labels is the true label of the sample.

5. According to claim 1, a cross-domain small sample target detection method based on scale matching and semantic enhancement, It is characterized in that Step S4 specifically includes the following steps: Step S41: First, based on the classic framework of cross-domain target detection, it is divided into two stages, namely the training stage and the testing stage; in the training stage, the base class data is used to train the detector to improve the detection performance; then, before fine-tuning the new class, the initialized detector parameters are copied to the student model; the student model needs to go through a "running-in" stage before formal distillation; in this stage, the model uses the standard detection supervision loss to make fine adjustments for a specific number of new class data to meet the needs of cross-domain small sample target detection; after the running-in is completed, the weights of the student model will be synchronized to the teacher model to prepare for the subsequent distillation training; when distilling, the teacher model and the student model are trained in a collaborative manner; Step S42: Use the knowledge distillation architecture to take the strongly enhanced image as the input of the student model and the weakly enhanced image as the input of the teacher model to provide reliable pseudo labels; in the supervision branch, calculate the supervised detection loss of the student model: the classification loss L cls and the localization loss L loc ; In the new class sample In the case of supervised detection loss L S The calculation formula is as follows: in, It represents the image value. It represents the true value of the label, s refers to the new class sample, i refers to the i-th new class sample, L cls represents the classification loss, L loc It represents the positioning loss; Step S43: The teacher and student share the same architecture and are initialized with the same weights after the burn-in step; the image is processed independently by the student and the teacher; the teacher is used to generate thousands of box candidates for the weakly enhanced version and remove redundancy through non-maximum suppression (NMS); the student model uses the strongly enhanced version for prediction and obtains the distillation loss L by calculating the detection loss between the student prediction and the pseudo-label D ; L D The specific calculation formula is as follows: in, For the correct result in the case of distillation, Predictions generated for the distillation case; Step S44: An exponential moving average (EMA) is used to separate the teacher and the student. The teacher is used to perform reasoning and evaluation on the test image. It is worth noting that no data augmentation operation is performed on the input image during the reasoning phase. At this point, before knowledge distillation, the loss function calculation formula is as follows: L=L S +L dssom +L aug Among them, L represents the value of the loss function, L S It represents the supervised detection loss value, L dssom It represents the loss value of the cross-domain scale shape optimization matching module generated in step S2, L aug It represents the loss value of the dynamic semantic covariance enhancement module generated in step S3; After the knowledge distillation calculation, the new loss function calculation formula is as follows: L=L S +L D +L dssom +L aug Among them, L D It represents the distillation loss; Step S45: Test the trained teacher model on the validation set to obtain the final detection accuracy and obtain a training model suitable for the new class data set.

Citation Information

Patent Citations

  • Remote sensing image cross-domain small sample classification method based on pseudo label uncertainty perception

    CN117152503A

  • Knowledge distillation method, system, medium and equipment for cross-domain passive domain data

    CN118298279A

  • Object tracking

    WO2015074173A1

  • Training method and training apparatus for rotating-ship target detection model, and storage medium

    WO2023116631A1

Cited By

  • Domain adaptive target detection method and system based on adversarial double-teacher knowledge distillation

    CN121437856A

  • A Domain-Adaptive Target Detection Method and System Based on Adversarial Dual-Teacher Knowledge Distillation

    CN121437856B