A remote sensing weakly supervised fine-grained object detection and recognition method and device

Abstract

The present application relates to the technical field of computer vision and image processing, and discloses a remote sensing weakly supervised fine-grained target detection and recognition method and device, based on a general text prompt corresponding to a small amount of labeled remote sensing data samples and a large amount of unlabeled remote sensing data samples, by extracting the geometric features of the samples and the cross-modal features representing the visual text differences, combining the fine-grained class labels of the labeled remote sensing data samples, prior knowledge prototypes of various fine-grained classes are constructed; then, the prior knowledge prototypes of various fine-grained classes are used to construct fine-grained soft labels of the unlabeled remote sensing data samples, which are used as supervision signals to realize the training of the model, solve the cost consumption problem of fine-grained labeling, improve the label labeling efficiency of the unlabeled remote sensing data samples, and further improve the model training efficiency and target recognition efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, and in particular to a method and apparatus for fine-grained target detection and recognition using remote sensing with weak supervision. Background Technology

[0002] With the rapid development of computer vision and deep learning technologies, the demand for fine-grained recognition of targets of interest is increasing. However, fine-grained recognition relies on large-scale, high-quality, finely labeled data. In the field of remote sensing, obtaining such labels is extremely costly and requires specialized prior knowledge, resulting in poor scalability of dataset construction and severely limiting the generalization ability of models in open scenarios. Furthermore, training supervised solely by labeled information lacks exploration of fine-grained prior information.

[0003] To alleviate the reliance on massive amounts of labeled data, few-shot learning has been introduced. Existing few-shot learning methods typically rely on only a small number of labeled samples for model training or rapid adaptation. However, the core of few-shot learning is meta-learning, which requires a large amount of labeled base class data for meta-training and then rapid adaptation on new classes using a small number of samples. It cannot effectively utilize a large amount of unlabeled data.

[0004] However, few-shot learning typically relies on only a small number of labeled samples for model adaptation, failing to effectively utilize the large amount of readily available unlabeled data in remote sensing images, resulting in a waste of data resources. Furthermore, in real-world remote sensing scenarios, acquiring a large number of fine-grained base class annotations is also costly, limiting its application scenarios.

[0005] Therefore, how to reduce the reliance on finely labeled data, make full use of massive amounts of unlabeled data, and effectively mine fine-grained prior information of targets in order to improve the detection accuracy and efficiency of fine-grained targets in remote sensing images has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] Therefore, the technical problem to be solved by the present invention is to overcome the problem of low target detection and recognition efficiency caused by insufficient prior information and high cost of fine-grained annotation in the prior art.

[0007] To address the aforementioned technical problems, this invention provides a remote sensing weakly supervised fine-grained target detection and recognition method, comprising: Obtain labeled and unlabeled remote sensing data samples corresponding to any general text prompt; the labeled remote sensing data samples include target directed bounding boxes and fine-grained category labels, and the unlabeled remote sensing data samples include target directed bounding boxes; Based on the target's directed bounding box, obtain the geometric features and target region of interest of the labeled remote sensing data sample; Based on the text features of general text prompts and the visual spatial features of the target region of interest, cross-modal features of labeled remote sensing data samples are constructed. Based on the fine-grained category labels, geometric features, and cross-modal features corresponding to each labeled remote sensing data sample, we construct prior knowledge prototypes for various fine-grained categories. Based on the geometric features and cross-modal features of unlabeled remote sensing data samples, and the Euclidean distance between them and the prior knowledge prototypes of various fine-grained categories, the class distance vector of unlabeled remote sensing data samples is obtained. Based on the category distance vector of unlabeled remote sensing data samples, Euclidean distance is converted into similarity to obtain the similarity vector of unlabeled remote sensing data samples, and then normalized to obtain fine-grained soft labels for unlabeled remote sensing data samples. Unlabeled remote sensing data samples are input into the rotating target detection model to obtain the predicted bounding box and predicted class probability. Combined with the corresponding target oriented bounding box and fine-grained soft label, the bounding box regression loss and classification loss are constructed to train the rotating target detection model and obtain the trained rotating target detection model. The remote sensing image to be identified is input into the trained rotating target detection model to obtain fine-grained target detection and recognition results.

[0008] Preferably, the acquisition of the geometric features of the remote sensing data sample includes: Based on the four vertices of the target's directed bounding box, obtain the rotated box represented by the center point to get the target's length and width; Based on the target's directed bounding box, the target's region of interest is cropped, and a depth estimation map of the target's region of interest is constructed; Based on the depth values ​​at each pixel location in the depth estimation map of the target's region of interest, the estimated height of the target is obtained; The geometric features that make up a remote sensing data sample are based on the length, width, and estimated height of the target.

[0009] Preferably, the estimated height of the target is obtained based on the depth values ​​at each pixel location in the depth estimation map of the target's region of interest, including: Four rectangular regions of preset size are cropped at the four vertices of the target region of interest; Calculate the average height of the region of interest based on the average height of each rectangular region. Obtain depth estimation map The maximum value in Calculate the difference between its height and the average height of the target's region of interest to obtain the estimated height of the target. , represented as: ; in, The first part represents the region of interest of the target. The rectangular region at each vertex Represents a rectangular area The average height in .

[0010] Preferably, the acquisition of cross-modal features of remote sensing data samples includes: Perform word segmentation and encoding on general text prompts to obtain general text features; Visual encoding is performed on the target region of interest to obtain the target's spatial visual features. After global average pooling, the features are interpolated to the same dimension as the general text features to obtain matching spatial visual features. Calculate the cosine similarity between matching spatial visual features and general text features as cross-modal features of remote sensing data samples.

[0011] Preferably, the general text prompt is segmented and encoded to obtain general text features, including: Perform word segmentation on general text prompts to obtain word sequence; Learnable embedding layers are used to map word sequences to word vector sequences and position encoding is performed to obtain word order-based word vector sequences. A text encoder is used to encode word order-based word vector sequences to obtain semantic vectors, which serve as general text features.

[0012] Preferably, the cosine similarity between the matching spatial visual features and the general text features is calculated. , represented as: ; in, Represents matching spatial visual features, denoted as , This represents a linear interpolation operation. Indicates global average pooling. Representing spatial visual characteristics, Dimensions representing general textual features; Represents general text features. This represents the magnitude of the eigenvector.

[0013] Preferably, based on the fine-grained category labels, geometric features, and cross-modal features corresponding to each labeled remote sensing data sample, prior knowledge prototypes for various fine-grained categories are constructed, including: Calculate the mean and variance of the geometric features of all labeled remote sensing data samples with the same fine-grained category labels; Calculate the mean and variance of cross-modal features for all labeled remote sensing data samples with the same fine-grained category labels; Get all those with the first The mean and variance of the geometric features and cross-modal features of the labeled remote sensing data samples with fine-grained category labels constitute the first... Prior knowledge prototypes for fine-grained categories , represented as: ; in, and Representing all those with the first Geometric features of labeled remote sensing data samples with fine-grained category labels The mean and variance, and Representing all those with the first Cross-modal features of labeled remote sensing data samples with fine-grained category labels The mean and variance of.

[0014] Preferably, based on the geometric features and cross-modal features of unlabeled remote sensing data samples, and the Euclidean distance between them and prior knowledge prototypes of various fine-grained categories, the class distance vector of the unlabeled remote sensing data samples is obtained, including: Based on the Geometric features of an unlabeled remote sensing data sample Cross-modal features Calculate the first The first unlabeled remote sensing data sample and the first Euclidean distance of prior knowledge prototypes for fine-grained categories , represented as: ; Get the The Euclidean distances between each unlabeled remote sensing data sample and the prior knowledge prototypes of all fine-grained categories constitute the first... The class distance vector of an unlabeled remote sensing data sample , represented as: ; in, This indicates that the hyperparameter controls the weights. This represents the total number of categories in the fine-grained category. This indicates transpose.

[0015] Preferably, unlabeled remote sensing data samples are input into the rotating target detection model to obtain predicted bounding boxes and predicted class probabilities. These are then combined with the corresponding oriented target bounding boxes and fine-grained soft labels to construct bounding box regression loss and classification loss, including: Predicted bounding boxes based on unlabeled remote sensing data Directed bounding box with target Smoothing L1 function Construct bounding box regression loss , represented as: ; Predicting category probabilities based on unlabeled remote sensing data With fine-grained soft labels Cross-entropy loss function Construct classification loss , represented as: ; in, This indicates the preset temperature coefficient.

[0016] This embodiment provides an apparatus based on the remote sensing weakly supervised fine-grained target detection and recognition method as described above, comprising: The sample acquisition module is used to acquire labeled and unlabeled remote sensing data samples corresponding to any general text prompt; the labeled remote sensing data samples include target directed bounding boxes and fine-grained category labels, and the unlabeled remote sensing data samples include target directed bounding boxes. The prior knowledge construction module is used to obtain the geometric features and regions of interest of labeled remote sensing data samples based on the directed bounding boxes of the targets; to construct the cross-modal features of labeled remote sensing data samples based on the text features of general text prompts and the visual spatial features of the regions of interest of the targets; and to construct the prior knowledge prototypes of various fine-grained categories based on the fine-grained category labels, geometric features and cross-modal features corresponding to each labeled remote sensing data sample. The module for constructing fine-grained soft labels for unlabeled remote sensing data samples is used to obtain the category distance vector of unlabeled remote sensing data samples based on the geometric features and cross-modal features of the unlabeled remote sensing data samples and the Euclidean distance between them and the prior knowledge prototypes of various fine-grained categories. Based on the category distance vector of the unlabeled remote sensing data samples, the Euclidean distance is converted into similarity to obtain the similarity vector of the unlabeled remote sensing data samples, and then normalized to obtain the fine-grained soft labels of the unlabeled remote sensing data samples. The model training module is used to input unlabeled remote sensing data samples into the rotating target detection model, obtain the predicted bounding box and predicted class probability, combine the corresponding target oriented bounding box and fine-grained soft label, construct the bounding box regression loss and classification loss, train the rotating target detection model, and obtain the trained rotating target detection model. The recognition module is used to input the remote sensing image to be recognized into the trained rotating target detection model to obtain fine-grained target detection and recognition results.

[0017] Compared with the prior art, the above-described technical solution of the present invention has the following advantages: The remote sensing weakly supervised fine-grained target detection and recognition method of this invention is based on a small number of labeled remote sensing data samples and a large number of unlabeled remote sensing data samples corresponding to general text prompts. By extracting the geometric features of the samples and cross-modal features that characterize the differences between visual and textual data, and combining them with the fine-grained category labels of the labeled remote sensing data samples, it constructs prior knowledge prototypes for various fine-grained categories. Then, using the prior knowledge prototypes of various fine-grained categories, it constructs fine-grained soft labels for the unlabeled remote sensing data samples, and uses these as supervision signals to train the model. This solves the cost problem of fine-grained labeling, improves the labeling efficiency of unlabeled remote sensing data samples, and thus improves the model training efficiency and target recognition efficiency.

[0018] The training set of this invention does not require fine-grained annotation, only directed bounding boxes. By combining geometric features and cross-modal features, it uses a loss function for iterative updates, which greatly reduces the annotation cost while maintaining excellent recognition performance and lower training cost. Moreover, it only introduces a small number of parameter calculations during the preprocessing process, without increasing the computational burden during training and inference, thus improving the model training efficiency. Attached Figure Description

[0019] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein: Figure 1 This is a flowchart of the steps of the remote sensing weak supervision fine-grained target detection and recognition method of the present invention; Figure 2 This is a schematic diagram illustrating the principle of the remote sensing weak supervision fine-grained target detection and recognition method of the present invention; Figure 3 This is a schematic diagram showing the results of fine-grained classification based on the method of this invention. Figure 4 This is a depth estimation map obtained based on the method of this invention. Detailed Implementation

[0020] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0021] Reference Figure 1 The flowchart shown is a step-by-step diagram of the remote sensing weakly supervised fine-grained target detection and recognition method of the present invention. The present invention proposes a weakly supervised paradigm for fine-grained target detection tasks in images. By extracting geometric features and visual text difference features, fine-grained soft labels are obtained as supervision signals to solve the cost problem of fine-grained labeling. The specific steps include S101 to S108.

[0022] S101: Obtain labeled and unlabeled remote sensing data samples corresponding to any general text prompt; the labeled remote sensing data samples include target directed bounding boxes and fine-grained category labels, and the unlabeled remote sensing data samples include target directed bounding boxes.

[0023] Specifically, in this embodiment, labeled remote sensing data samples are used as the validation and test sets, and unlabeled remote sensing data samples are used as the training set. The ratio of the training set, validation set, and test set is 5:2:3.

[0024] S102: Based on the target's directed bounding box, obtain the geometric features and target region of interest of the labeled remote sensing data sample.

[0025] This embodiment utilizes a feature extractor to obtain the target region of interest and its features based on the target's directed bounding box.

[0026] This embodiment includes the following steps when extracting geometric features from labeled remote sensing data samples: Based on the target directed bounding box The four vertices are transformed to obtain the result based on the center point. Representing the rotating frame , with the length of the rotating frame ,Width To obtain the target length With width ; Based on the target's directed bounding box, the region of interest of the target is cropped. And using depth estimation model Construct a depth estimation map of the target region of interest. ; Based on the depth values ​​at each pixel location in the depth estimation map of the target's region of interest, the estimated height of the target is obtained. ; Based on the goal of long ,Width Compared with estimated height Geometric features that make up remote sensing data samples ; in, Indicates the first target on the directed bounding box. The coordinates of the vertices, Indicates the rotation angle of the bounding box; depth estimation model It is a computer vision model based on CNN, commonly used models include the Depth Anything model.

[0027] Specifically, obtaining the estimated altitude of the target includes: Four rectangular regions of preset size are cropped at the four vertices of the target region of interest; Calculate the average height of the region of interest based on the average height of each rectangular region. Obtain depth estimation map The maximum value in Calculate the difference between its height and the average height of the target's region of interest to obtain the estimated height of the target. , represented as: ; in, The first part represents the region of interest of the target. The rectangular region at each vertex Represents a rectangular area The average height in .

[0028] In this embodiment, the target's height estimate is calculated using the maximum height estimate and the approximate height of the ground surrounding the target. Depth estimation map. Each pixel location stores the depth value of that point, taking the maximum value from the depth estimation map. .right Four rectangular areas were cut out from the four corners of the region to approximate the height of the ground around the target. Assuming The coordinates of the top left corner of the region are , width is Gao Wei The width and height of the angular region are defined as follows: , Therefore, the four rectangular regions of preset size defined in this embodiment are represented as follows: Top left corner area: ; Top right corner area: ; Bottom left corner area: ; Bottom right corner area: .

[0029] S103: Based on the text features of general text prompts and the visual spatial features of the target region of interest, construct cross-modal features for labeled remote sensing data samples, including: S103-1: Perform word segmentation and encoding on general text prompts to obtain general text features; S103-2: Visually encode the target region of interest to obtain the target's spatial visual features, and then perform global average pooling and interpolate to the same dimension as the general text features to obtain matching spatial visual features; S103-3: Calculate the cosine similarity between the matching spatial visual features and the general text features. As a cross-modal feature of remote sensing data samples, it is represented as: ; in, To represent matching spatial visual features, for cross-modal matching, the spatial visual features of the target are globally averaged and pooled, then interpolated to the same dimension as the general text features, represented as follows: , This represents a linear interpolation operation. Indicates global average pooling. Representing spatial visual characteristics, Dimensions representing general textual features; Represents general text features. This represents the magnitude of the eigenvector.

[0030] In this embodiment, the general text prompt is segmented to obtain a word sequence; a learnable embedding layer is used to map the word sequence into a word vector sequence, and positional encoding is performed to obtain a word vector sequence based on word order; a text encoder is used to encode the word vector sequence based on word order to obtain a semantic vector, which serves as the general text feature. , represented as: ;in, General text prompts Word segmentation is performed to obtain a sequence of word elements. This sequence is then mapped to a sequence of word vectors through a learnable embedding layer, and positional encoding is added to preserve word order information. As a text encoder, using clip as the text encoder, it encodes natural language descriptions into dense semantic vectors with a dimension of 512.

[0031] In this embodiment, a visual encoder is used to extract the spatial visual feature map of the target in the input region of interest. , represented as: ;in, A visual encoder is a standard convolutional neural network backbone used to extract regions of interest from the input target. Spatial visual features are extracted, and an ORCNN two-stage object detection model is used as the detection model, with the target region of interest input. Features of head output It has 7×7×256 dimensions.

[0032] S104: Based on the fine-grained category labels, geometric features, and cross-modal features corresponding to each labeled remote sensing data sample, construct prior knowledge prototypes for various fine-grained categories, including: S104-1: Calculate the mean and variance of the geometric features of all labeled remote sensing data samples with the same fine-grained category labels; S104-2: Calculate the mean and variance of cross-modal features for all labeled remote sensing data samples with the same fine-grained category labels; S104-3: Obtain all those with the first The mean and variance of the geometric features and cross-modal features of the labeled remote sensing data samples with fine-grained category labels constitute the first... Prior knowledge prototypes for fine-grained categories , represented as: ; in, and Representing all those with the first Geometric features of labeled remote sensing data samples with fine-grained category labels The mean and variance, and Representing all those with the first Cross-modal features of labeled remote sensing data samples with fine-grained category labels The mean and variance of.

[0033] In this embodiment, the feature extraction methods used for labeled and unlabeled remote sensing data samples are the same when acquiring their geometric features and cross-modal features.

[0034] S105: Based on the geometric features and cross-modal features of unlabeled remote sensing data samples, and the Euclidean distance between them and prior knowledge prototypes of various fine-grained categories, obtain the category distance vector of unlabeled remote sensing data samples, including: Based on the Geometric features of an unlabeled remote sensing data sample Cross-modal features Calculate the first The first unlabeled remote sensing data sample and the first Euclidean distance of prior knowledge prototypes for fine-grained categories , represented as: ; Get the The Euclidean distances between each unlabeled remote sensing data sample and the prior knowledge prototypes of all fine-grained categories constitute the first... The class distance vector of an unlabeled remote sensing data sample , represented as: ; in, This represents the hyperparameter control weight, which is set to 1.2 in this embodiment; This represents the total number of categories in the fine-grained category. This indicates transpose.

[0035] S106: Based on the category distance vector of unlabeled remote sensing data samples, convert the Euclidean distance into similarity, obtain the similarity vector of unlabeled remote sensing data samples, and normalize it to obtain fine-grained soft labels for unlabeled remote sensing data samples, including: S106-1: Based on the inverse relationship between similarity and distance, the first... The first unlabeled remote sensing data sample and the first Euclidean distance of prior knowledge prototypes for fine-grained categories Convert to the first The first unlabeled remote sensing data sample and the first Similarity of prior knowledge prototypes for fine-grained categories , represented as: ; S106-2: Obtain the... The similarity between each unlabeled remote sensing data sample and the prior knowledge prototypes of all fine-grained categories constitutes the first... Similarity vectors of unlabeled remote sensing data samples , represented as: ; S106-3: Normalize the similarity vector to obtain the first... Fine-grained soft labels for unlabeled remote sensing data samples , represented as: ; .

[0036] S107: Input unlabeled remote sensing data samples into the rotating target detection model to obtain the predicted bounding box and predicted class probability. Combine the corresponding directed bounding box of the target and fine-grained soft label to construct the bounding box regression loss and classification loss. Train the rotating target detection model to obtain the trained rotating target detection model.

[0037] In this embodiment, the model is based on a rotated object detection framework. The training set does not require fine-grained annotations, only directed bounding boxes. Fine-grained soft labels are obtained through extracted geometric features and cross-modal features representing visual-text differences. As a supervisory signal, the image and annotations are input into the detection model, which generates directed bounding box predictions. and category prediction probability The model parameters are updated through gradient backpropagation by constraining the bounding box prediction branch and the fine-grained category prediction branch respectively using a loss function, thus achieving end-to-end training.

[0038] Specifically, predicted bounding boxes based on label-free remote sensing data Directed bounding box with target Smoothing L1 function Construct bounding box regression loss , represented as: ; Predicting category probabilities based on unlabeled remote sensing data With fine-grained soft labels Cross-entropy loss function Construct classification loss , represented as: ; The total loss function is expressed as: ; in, This represents the preset temperature coefficient, which is 0.6 in this embodiment.

[0039] S108: Input the remote sensing image to be identified into the trained rotating target detection model to obtain fine-grained target detection and recognition results.

[0040] In this embodiment, to fully verify the effectiveness of the proposed method in fine-grained object classification tasks, a fully supervised comparative experiment was conducted with the baseline model on the same dataset. All experiments were performed on an environment equipped with a single NVIDIA RTX4090 24-GB GPU, implemented using PyTorch 1.13.0 and the MMDetection / MMRotate toolbox; the model was trained using the SGD (momentum 0.9, initial learning rate 0.0025) optimizer, employing a 12-round training strategy on the FAIR1M-v2.0 aircraft dataset. Parameter settings were as follows: N=11 .

[0041] Reference Figure 2 The diagram illustrates the principle of the remote sensing weakly supervised fine-grained target detection and recognition method of the present invention. First, ORCNN is used as the baseline model, and Clip is used as the text encoder. The directed bounding boxes of the target are used to obtain the Region of Interest (RoI) and its geometric characteristics. The RoIhead of ORCNN is used to obtain visual features and calculate the differences between the visual features and the text features of Clip. Fine-grained prototypes for each category are constructed, and soft labels for the training samples are calculated. These soft labels are then combined with the directed bounding boxes as supervision signals and input to the ORCNN model to train the model.

[0042] Reference Figure 3The diagram shows the results of fine-grained classification based on the method of this invention, displaying the true labels (ground values) and model predictions for three sets of test samples. As can be seen from the comparison, the model provided in this embodiment is completely consistent with the ground values ​​in predicting the target fine-grained category.

[0043] Reference Figure 4 As shown, this is a depth estimation map obtained based on the method of the present invention. The image shows a sample image. According to the depth estimation map of the target's bounding box clipping region, the depth value of the aircraft part is significantly higher than that of other parts, and the outline of the aircraft can be clearly depicted.

[0044] To further assess the performance of weakly supervised fine-grained detection, mean precision was analyzed. For evaluation, precision, recall, and mean precision (mAP) were used as metrics, expressed as follows: Precision is defined as: ; Recall rate is defined as: ; Mean precision is defined as the average precision of a model at different recall levels. Here, TP, FP, FN, and TN represent true positives, false positives, false negatives, and true negatives, respectively. By comparing these metrics, the recognition accuracy of the model across different categories and the overall sample can be comprehensively evaluated.

[0045] To investigate the contribution of each module to the model performance, ablation experiments were designed on the FAIR1M-v2.0 aircraft dataset. Using ORCNN as the baseline, geometric features, similarity, and loss functions were progressively integrated, with mean accuracy (mAP) used as the evaluation metric. Table 1 shows the ablation experiments performed on the baseline model combined with the method proposed in this application for target recognition.

[0046] Table 1 Ablation Experiment Table

[0047] This invention also provides an apparatus for a remote sensing weakly supervised fine-grained target detection and recognition method, the specific apparatus including: The sample acquisition module is used to acquire labeled and unlabeled remote sensing data samples corresponding to any general text prompt; the labeled remote sensing data samples include target directed bounding boxes and fine-grained category labels, and the unlabeled remote sensing data samples include target directed bounding boxes. The prior knowledge construction module is used to obtain the geometric features and regions of interest of labeled remote sensing data samples based on the directed bounding boxes of the targets; to construct the cross-modal features of labeled remote sensing data samples based on the text features of general text prompts and the visual spatial features of the regions of interest of the targets; and to construct the prior knowledge prototypes of various fine-grained categories based on the fine-grained category labels, geometric features and cross-modal features corresponding to each labeled remote sensing data sample. The module for constructing fine-grained soft labels for unlabeled remote sensing data samples is used to obtain the category distance vector of unlabeled remote sensing data samples based on the geometric features and cross-modal features of the samples and the Euclidean distance between them and the prior knowledge prototypes of various fine-grained categories. Based on the category distance vector of the unlabeled remote sensing data samples, the Euclidean distance is converted into similarity to obtain the similarity vector of the unlabeled remote sensing data samples, and then normalized to obtain the fine-grained soft labels of the unlabeled remote sensing data samples. The model training module is used to input unlabeled remote sensing data samples into the rotating target detection model, obtain the predicted bounding box and predicted class probability, combine the corresponding target oriented bounding box and fine-grained soft label, construct the bounding box regression loss and classification loss, train the rotating target detection model, and obtain the trained rotating target detection model. The recognition module is used to input the remote sensing image to be recognized into the trained rotating target detection model to obtain fine-grained target detection and recognition results.

[0048] The remote sensing weakly supervised fine-grained target detection and recognition device of this embodiment is used to implement the aforementioned remote sensing weakly supervised fine-grained target detection and recognition method. Therefore, the specific implementation of the remote sensing weakly supervised fine-grained target detection and recognition device can be found in the embodiment section of the remote sensing weakly supervised fine-grained target detection and recognition method above. For example, the sample acquisition module is used to implement step S101 in the aforementioned remote sensing weakly supervised fine-grained target detection and recognition method; the prior knowledge construction module is used to implement steps S102, S103, and S104 in the aforementioned remote sensing weakly supervised fine-grained target detection and recognition method; the unlabeled sample fine-grained soft label construction module is used to implement steps S105 and S106 in the aforementioned remote sensing weakly supervised fine-grained target detection and recognition method; the model training module is used to implement step S107 in the aforementioned remote sensing weakly supervised fine-grained target detection and recognition method; and the recognition module is used to implement step S108 in the aforementioned remote sensing weakly supervised fine-grained target detection and recognition method. Therefore, its specific implementation can be referred to the description of the corresponding embodiments, and will not be repeated here.

[0049] The remote sensing weakly supervised fine-grained target detection and recognition method of this invention is based on a small number of labeled remote sensing data samples corresponding to general text prompts and a large number of unlabeled remote sensing data samples. It extracts geometric features of the samples and cross-modal features representing visual-text differences, and combines these with fine-grained category labels of labeled remote sensing data samples to construct prior knowledge prototypes for various fine-grained categories. Then, using these prior knowledge prototypes, it constructs fine-grained soft labels for unlabeled remote sensing data samples, which are used as supervision signals to train the model. This solves the cost problem of fine-grained annotation, improves the labeling efficiency of unlabeled remote sensing data samples, and thus improves model training efficiency and target recognition efficiency. The training set of this invention does not require fine-grained annotation, only directed bounding boxes. By combining geometric features and cross-modal features, it uses a loss function for iterative updates, greatly reducing annotation costs while maintaining excellent recognition performance and finer training costs. Furthermore, it only introduces a small number of parameter calculations during preprocessing, without increasing the computational burden during training and inference, thus improving model training efficiency.

[0050] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0051] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0052] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0053] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0054] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A remote sensing weakly supervised fine-grained target detection and recognition method, characterized in that, include: Obtain labeled and unlabeled remote sensing data samples corresponding to any general text prompt; the labeled remote sensing data samples include target directed bounding boxes and fine-grained category labels, and the unlabeled remote sensing data samples include target directed bounding boxes; Based on the target's directed bounding box, obtain the geometric features and target region of interest of the labeled remote sensing data sample; Based on the text features of general text prompts and the visual spatial features of the target region of interest, cross-modal features of labeled remote sensing data samples are constructed. Based on the fine-grained category labels, geometric features, and cross-modal features corresponding to each labeled remote sensing data sample, we construct prior knowledge prototypes for various fine-grained categories. Based on the geometric features and cross-modal features of unlabeled remote sensing data samples, and the Euclidean distance between them and the prior knowledge prototypes of various fine-grained categories, the class distance vector of unlabeled remote sensing data samples is obtained. Based on the category distance vector of unlabeled remote sensing data samples, Euclidean distance is converted into similarity to obtain the similarity vector of unlabeled remote sensing data samples, and then normalized to obtain fine-grained soft labels for unlabeled remote sensing data samples. Unlabeled remote sensing data samples are input into the rotating target detection model to obtain the predicted bounding box and predicted class probability. Combined with the corresponding target oriented bounding box and fine-grained soft label, the bounding box regression loss and classification loss are constructed to train the rotating target detection model and obtain the trained rotating target detection model. The remote sensing image to be identified is input into the trained rotating target detection model to obtain fine-grained target detection and recognition results.

2. The remote sensing weakly supervised fine-grained target detection and identification method according to claim 1, characterized in that, The acquisition of geometric features of remote sensing data samples includes: Based on the four vertices of the target's directed bounding box, obtain the rotated box represented by the center point to get the target's length and width; Based on the target's directed bounding box, the target's region of interest is cropped, and a depth estimation map of the target's region of interest is constructed; Based on the depth values ​​at each pixel location in the depth estimation map of the target's region of interest, the estimated height of the target is obtained; The geometric features that make up a remote sensing data sample are based on the length, width, and estimated height of the target.

3. The remote sensing weakly supervised fine-grained target detection and identification method according to claim 2, characterized in that, Based on the depth values ​​at each pixel location in the depth estimation map of the target's region of interest, the estimated height of the target is obtained, including: Four rectangular regions of preset size are cropped at the four vertices of the target region of interest; Calculate the average height of the region of interest based on the average height of each rectangular region. Obtain depth estimation map The maximum value in Calculate the difference between its height and the average height of the target's region of interest to obtain the estimated height of the target. , is represented as: ； in, The first part represents the region of interest of the target. The rectangular region at each vertex Represents a rectangular area The average height in .

4. The remote sensing weakly supervised fine-grained target detection and identification method according to claim 1, characterized in that, The acquisition of cross-modal features of remote sensing data samples includes: Perform word segmentation and encoding on general text prompts to obtain general text features; Visual encoding is performed on the target region of interest to obtain the target's spatial visual features. After global average pooling, the features are interpolated to the same dimension as the general text features to obtain matching spatial visual features. Calculate the cosine similarity between matching spatial visual features and general text features as cross-modal features of remote sensing data samples.

5. The remote sensing weakly supervised fine-grained target detection and identification method according to claim 4, characterized in that, The general text prompts are segmented and encoded to obtain general text features, including: Perform word segmentation on general text prompts to obtain word sequence; Learnable embedding layers are used to map word sequences to word vector sequences and position encoding is performed to obtain word order-based word vector sequences. A text encoder is used to encode word order-based word vector sequences to obtain semantic vectors, which serve as general text features.

6. The remote sensing weakly supervised fine-grained target detection and identification method according to claim 4, characterized in that, Calculate the cosine similarity between the matching spatial visual features and the general text features. , is represented as: ； in, Represents matching spatial visual features, denoted as , This represents a linear interpolation operation. Indicates global average pooling. Representing spatial visual characteristics, Dimensions representing general textual features; Represents general text features. This represents the magnitude of the eigenvector.

7. The remote sensing weakly supervised fine-grained target detection and identification method according to claim 1, characterized in that, Based on the fine-grained category labels, geometric features, and cross-modal features corresponding to each labeled remote sensing data sample, prior knowledge prototypes for various fine-grained categories are constructed, including: Calculate the mean and variance of the geometric features of all labeled remote sensing data samples with the same fine-grained category labels; Calculate the mean and variance of cross-modal features for all labeled remote sensing data samples with the same fine-grained category labels; Get all those with the first The mean and variance of the geometric features and cross-modal features of the labeled remote sensing data samples with fine-grained category labels constitute the first... Prior knowledge prototypes for fine-grained categories , is represented as: ； in, and Representing all those with the first Geometric features of labeled remote sensing data samples with fine-grained category labels The mean and variance, and Representing all those with the first Cross-modal features of labeled remote sensing data samples with fine-grained category labels The mean and variance of.

8. The remote sensing weakly supervised fine-grained target detection and identification method according to claim 7, characterized in that, Based on the geometric features and cross-modal features of unlabeled remote sensing data samples, and the Euclidean distance between these features and prior knowledge prototypes of various fine-grained categories, the class distance vectors of the unlabeled remote sensing data samples are obtained, including: Based on the Geometric features of an unlabeled remote sensing data sample Cross-modal features Calculate the first The first unlabeled remote sensing data sample and the first Euclidean distance of prior knowledge prototypes for fine-grained categories , is represented as: ； Get the The Euclidean distances between each unlabeled remote sensing data sample and the prior knowledge prototypes of all fine-grained categories constitute the first... The class distance vector of an unlabeled remote sensing data sample , is represented as: ； in, This indicates that the hyperparameter controls the weights. This represents the total number of categories in the fine-grained category. This indicates transpose.

9. The remote sensing weakly supervised fine-grained target detection and identification method according to claim 1, characterized in that, Unlabeled remote sensing data samples are input into a rotating target detection model to obtain predicted bounding boxes and predicted class probabilities. These are then combined with the corresponding oriented bounding boxes and fine-grained soft labels to construct bounding box regression and classification losses, including: Predicted bounding boxes based on unlabeled remote sensing data Directed bounding box with target Smoothing L1 function Construct bounding box regression loss , is represented as: ； Predicting category probabilities based on unlabeled remote sensing data With fine-grained soft labels Cross-entropy loss function Construct classification loss , is represented as: ； in, This indicates the preset temperature coefficient.

10. An apparatus based on the remote sensing weakly supervised fine-grained target detection and recognition method as described in any one of claims 1 to 9, characterized in that, include: The sample acquisition module is used to acquire labeled and unlabeled remote sensing data samples corresponding to any general text prompt; the labeled remote sensing data samples include target directed bounding boxes and fine-grained category labels, and the unlabeled remote sensing data samples include target directed bounding boxes. The prior knowledge construction module is used to obtain the geometric features and target regions of interest of labeled remote sensing data samples based on the target's directed bounding box; and to construct cross-modal features of labeled remote sensing data samples based on the text features of general text prompts and the visual spatial features of the target regions of interest. Based on the fine-grained category labels, geometric features, and cross-modal features corresponding to each labeled remote sensing data sample, we construct prior knowledge prototypes for various fine-grained categories. The module for constructing fine-grained soft labels for unlabeled remote sensing data samples is used to obtain the category distance vector of unlabeled remote sensing data samples based on the geometric features and cross-modal features of the unlabeled remote sensing data samples and the Euclidean distance between them and the prior knowledge prototypes of various fine-grained categories. Based on the category distance vector of the unlabeled remote sensing data samples, the Euclidean distance is converted into similarity to obtain the similarity vector of the unlabeled remote sensing data samples, and then normalized to obtain the fine-grained soft labels of the unlabeled remote sensing data samples. The model training module is used to input unlabeled remote sensing data samples into the rotating target detection model, obtain the predicted bounding box and predicted class probability, combine the corresponding target oriented bounding box and fine-grained soft label, construct the bounding box regression loss and classification loss, train the rotating target detection model, and obtain the trained rotating target detection model. The recognition module is used to input the remote sensing image to be recognized into the trained rotating target detection model to obtain fine-grained target detection and recognition results.

Images