An open target detection method, apparatus, device, and storage medium

Abstract

This application discloses an open object detection method, apparatus, device, and storage medium. The method includes: extracting local regions of interest from training images using a region proposal network, aligning their features using a feature deformation adapter, generating image embedding features using an image encoder, and calculating a local instance-level distillation loss using the image embedding features and the adapted local region features; inputting the text description corresponding to the training images into a text encoder for processing to generate text embedding features, and calculating a cross-modal feature alignment loss using the image and text embedding features; updating network parameters by combining the local instance-level distillation loss and the cross-modal feature alignment loss to obtain an open object detection model; and performing object detection using the open object detection model. This application reduces the feature semantic differences caused by local region feature deformation, enhances the deformation feature alignment capability, and effectively improves the generalization ability of the open object detection model for detecting unknown categories.

Description

Technical Field

[0001] This application relates to the field of target detection technology, and in particular to an open target detection method, apparatus, device and storage medium. Background Technology

[0002] The core challenge of open-vocabulary object detection lies in identifying new objects outside the training set. Existing methods typically rely on the cross-modal image-text alignment capabilities of visual language pre-trained models such as CLIP. However, CLIP's image-text pair training mechanism has inherent limitations: it extracts features from the entire image through global semantic encoding, lacking the ability to perceive fine-grained features of local, fine-grained regions of the image. In open-vocabulary object detection tasks, the detection model needs to focus on the fine-grained features of local ROIs (Regions of Interest). However, spatial deformation during local ROI feature extraction can lead to significant semantic differences between local region features and the global image features pre-trained by the CLIP model, resulting in a decrease in the accuracy of local open object detection. Summary of the Invention

[0003] This application provides an open target detection method, apparatus, device, and storage medium to improve the spatial deformation problem during local ROI feature extraction, which leads to significant semantic differences between local region features and global image features extracted by the CLIP model, thereby causing a decrease in the accuracy of local open target detection.

[0004] In view of this, the first aspect of this application provides an open target detection method, comprising:

[0005] Construct a target network, which includes a region proposal network, a feature deformation adapter, an image encoder, and a text encoder;

[0006] Local regions of interest are extracted from training images using a region proposal network, and the local regions of interest are feature-aligned using a feature deformation adapter to obtain adapted local region features.

[0007] The adapted local region features are input into the image encoder for processing to generate image embedding features. The local instance-level distillation loss is calculated using the image embedding features and the adapted local region features.

[0008] The text descriptions corresponding to the training images are input into a text encoder for processing to generate text embedding features. Cross-modal feature alignment loss is calculated using the image embedding features and the text embedding features.

[0009] The network parameters of the target network are updated by combining the local instance-level distillation loss and the cross-modal feature alignment loss to obtain the trained open target detection model;

[0010] The open object detection model is used to perform object detection on the image to be detected.

[0011] Optionally, there are multiple feature deformation adapters, each feature deformation adapter including a feature adapter and a deformation adapter. Each feature adapter consists of a pooling layer, an activation layer, and a fully connected layer, and all feature adapters share the pooling layer and the activation layer.

[0012] The feature adapter is used to perform feature mapping on the local region of interest and to perform residual connection with the local region of interest to obtain local region features;

[0013] The deformation adapter is used to select the corresponding deformation adaptation weights based on the discrete interval where the aspect ratio of the local region features is located, and then weight the local region features to obtain the adapted local region features.

[0014] Optionally, the method further includes:

[0015] The image encoder is used to extract global features from the training image to obtain the first global feature.

[0016] Average pooling and feature aggregation are performed on the adapted local region features corresponding to the training image to obtain the second global features;

[0017] Calculate the cosine similarity between the first global feature and the second global feature to obtain the global feature training loss;

[0018] Accordingly, the network parameters of the target network are updated by combining the local instance-level distillation loss and the cross-modal feature alignment loss to obtain a trained open target detection model, including:

[0019] The network parameters of the target network are updated by combining the local instance-level distillation loss, the cross-modal feature alignment loss, and the global feature training loss to obtain a trained open target detection model.

[0020] Optionally, the extraction of local regions of interest from the training image via a region proposal network includes:

[0021] The training images are processed by a region proposal network to generate candidate boxes with confidence scores;

[0022] Candidate boxes with an intersection-union ratio (IU) lower than the IU threshold and a confidence score higher than the confidence threshold are selected from the candidate boxes to obtain new category candidate boxes; the region corresponding to the new category candidate box is the region of interest.

[0023] Optionally, the formula for calculating the local instance-level distillation loss is as follows:

[0024]

[0025] In the formula, N represents the number of candidate boxes for the new class, and M represents the number of ground truth boxes for the base class. The local region features of the i-th local region of interest after deformation adaptation; The text embedding features describing the j-th text; Text embedding features generated from regions of interest based on the basic category. Text embedding features Features of the adapted local region Cosine similarity between them.

[0026] Optionally, calculating the cross-modal feature alignment loss using the image embedding features and the text embedding features includes:

[0027] The text embedding features of the text description and the corresponding image embedding features are used as positive sample pairs, and the text embedding features of the text description and the non-corresponding image embedding features are used as negative sample pairs. The cosine similarity between the text embedding features and the image embedding features is calculated to obtain the cross-modal feature alignment loss.

[0028] Optionally, the formula for calculating the cross-modal feature alignment loss is:

[0029]

[0030] In the formula, Let L be the set of known basic categories, and L be the number of known basic categories; A collection of text descriptions; The text embedding features describing the j-th text; Basic Category Corresponding text embedding features; Text embedding features generated from regions of interest based on the basic category; Text description generated for a local region of interest i of an unknown category; These are the weight parameters.

[0031] A second aspect of this application provides an open target detection device, comprising:

[0032] A network construction unit is used to construct a target network, which includes a region proposal network, a feature deformation adapter, an image encoder, and a text encoder;

[0033] The feature adaptation unit is used to extract local regions of interest from the training image through a region proposal network, and to perform feature alignment on the local regions of interest through a feature deformation adapter to obtain adapted local region features.

[0034] The first loss calculation unit is used to input the adapted local region features into the image encoder for processing, generate image embedding features, and calculate the local instance-level distillation loss through the image embedding features and the adapted local region features.

[0035] The second loss calculation unit is used to input the text description corresponding to the training image into the text encoder for processing, generate text embedding features, and calculate cross-modal feature alignment loss through the image embedding features and the text embedding features.

[0036] The parameter update unit is used to update the network parameters of the target network by combining the local instance-level distillation loss and the cross-modal feature alignment loss, so as to obtain the trained open target detection model.

[0037] The target detection unit is used to perform target detection on the image to be detected using the open target detection model.

[0038] A third aspect of this application provides an electronic device, the device including a processor and a memory;

[0039] The memory is used to store program code and transmit the program code to the processor;

[0040] The processor is configured to execute any one of the open target detection methods described in the first aspect according to the instructions in the program code.

[0041] A fourth aspect of this application provides a computer-readable storage medium for storing program code that, when executed by a processor, implements the open target detection method described in any of the first aspects.

[0042] As can be seen from the above technical solutions, this application has the following advantages:

[0043] The open object detection method provided in this application reduces the semantic differences in features caused by local feature deformation by designing a feature deformation adapter, thereby enhancing the alignment capability of deformed features. Furthermore, it proposes a granular hierarchical distillation framework to fully utilize the multi-level semantic features of images and construct cross-level knowledge transfer links, effectively improving the generalization ability of the open object detection model for detecting unknown categories. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A schematic flowchart of an open target detection method provided in this application embodiment;

[0046] Figure 2 A schematic diagram of an open target detection device provided in an embodiment of this application;

[0047] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0048] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0049] For easier understanding, please refer to Figure 1 This application provides an open target detection method, including:

[0050] Step 110: Construct the target network;

[0051] The target network in this embodiment includes a region proposal network, a feature deformation adapter, and a CLIP model. The CLIP model includes an image encoder and a text encoder. The image encoder can be a classic convolutional network (such as ResNet-50) or a visual Transformer (ViT), and the text encoder can use a Transformer network. The CLIP model in this application can be a pre-trained CLIP model.

[0052] The system comprises multiple feature deformation adapters, including feature adapters and deformation adapters. Each feature adapter consists of a pooling layer, an activation layer, and a fully connected layer. All feature adapters share the pooling and activation layers. The feature adapters are used to perform feature mapping on local regions of interest and to establish residual connections with these local regions of interest to obtain local region features. The deformation adapters are used to select corresponding deformation adaptation weights based on the discrete intervals where the aspect ratio of the local region features lies to weight the local region features, thereby obtaining adapted local region features.

[0053] Step 120: Extract local regions of interest from the training image through a region proposal network, and perform feature alignment on the local regions of interest through a feature deformation adapter to obtain adapted local region features.

[0054] Open-vocabulary object detection only requires training data for basic categories and a pre-trained CLIP model to train an object detector capable of detecting new categories. Its core process consists of two parts: object localization and region classification. Given that the basic training data only covers a limited number of categories and lacks prior information on new categories, simply relying on the pre-trained CLIP model for simple distillation is insufficient to effectively capture the rich semantic features of new categories.

[0055] To address the aforementioned issues, this application employs a region proposal network to process the original training images, generating candidate boxes with confidence scores. From these candidate boxes, boxes with an Intersection over Union (IoU) lower than the IoU threshold with the ground truth boxes of the base class are selected. (This indicates that it has low overlap with the base category and may belong to a new category), and the confidence score is higher than the confidence threshold. Candidate boxes (which can be further filtered to include background areas) are identified as potential new category candidate boxes, and the regions corresponding to these new category candidate boxes are designated as regions of local interest. It can be 0.2. It can be 0.7.

[0056] It should be noted that the base class ground truth bounding boxes are the target ground truth bounding boxes labeled in the training images for known base classes, including the target's location coordinates and corresponding class labels; the base class ground truth bounding boxes are essentially the manually labeled results of base classes, and serve as the benchmark for training object detection models and selecting new classes, and their acquisition depends on high-quality dataset labeling.

[0057] Because the candidate bounding boxes of the generated regions of interest (ROIs) have varying shapes, simply obtaining local region features using ROIAlign will result in ineffective alignment with the image embedding features extracted by the CLIP model due to feature deformation. This application's embodiment generates K feature adapters for each candidate bounding box of a ROI. Each feature adapter consists of a pooling layer, an activation layer R, and a fully connected layer FC. For the i-th local region of interest r i By weighting the residual connection with the identity mapping, the corresponding local region feature F can be obtained. i Taking the feature of the k-th local region of the i-th local region of interest as an example:

[0058]

[0059] In the formula, For weight parameters, Indicates the local region of interest r i The output features after pooling processing Indicates activation layer pair The processed output features Indicates a fully connected layer The output features after processing.

[0060] For the i-th local region of interest, K local region features can be output. In this process, all local region features are calculated using shared pooling and activation layers, while differentiated local region features are generated through different fully connected layers.

[0061] Based on the K local region features output above, deformation adaptation weights need to be assigned to each local region feature. This allows each deformation adapter to process only local features within a specific range. Specifically, it is based on the aspect ratio of the local features. The corresponding division is into K discrete intervals. When K=10, the discrete intervals are as follows: ,according to D K Weight the range and select a specific deformation adapter, such as the current one. In the discrete interval In the middle, then the deformation adaptation weight Then in Setting the position to 1 and setting the positions of all other weights to 0 is equivalent to selecting only the deformation adapter that processes this range to perform feature alignment on the local region features. The final deformation adapter recommended for the i-th local region of interest is... × This solution achieves targeted adaptation to local features of different shapes through structured adapter design and aspect ratio interval division, effectively solving the problem of cross-modal alignment failure caused by feature deformation.

[0062] Step 130: Input the adapted local region features into the image encoder for processing to generate image embedding features. Calculate the local instance-level distillation loss using the image embedding features and the adapted local region features.

[0063] Local region features weighted by a targeted instance deformation adapter are input into the image encoder for processing to generate corresponding image embedding features. Local instance-level distillation loss is then calculated using the image embedding features and the adapted local region features.

[0064] During training, image embedding features are generated from the local regions of interest (ROIs) of all training images read in the current batch using an image encoder, resulting in an image embedding feature queue. For any given ROI, instance-level distillation training is performed on the adapted ROI features using the image embedding feature queue, effectively classifying potential new target classes. Specifically, leveraging the cross-modal image-text alignment capability of the pre-trained CLIP model, the cosine similarity between the image embedding feature queue extracted by the image encoder and each adapted ROI feature is calculated. If the target in the current ROI is a base class, its similarity to the corresponding base class text is significantly higher. If the target in the current ROI is an unknown class, the CLIP model's powerful feature extraction capability can be used to extract features from the ROI, and the text class obtained by the CLIP model is used as the text description label for the potential new target. This method retains the classification accuracy of the base class and utilizes the open vocabulary representation capability of the CLIP model to provide an interpretable label generation path for semantic modeling of unknown classes, effectively improving the CLIP model's generalization ability for novel targets.

[0065] Specifically, for local regions of interest (ROIs) of basic categories: based on predefined text labels for basic categories (e.g., "car", "airplane"), corresponding text descriptions (e.g., category names) are predefined. A text encoder processes these predefined text descriptions to generate text embedding features. Then, the cosine similarity between the image embedding features of this ROI and all predefined text embedding features is calculated, and the text category corresponding to the highest similarity is taken as the prediction result. If the similarity is higher than a similarity threshold, it is determined to be a basic category, and supervised training is performed using the real labels.

[0066] For regions of interest (ROIs) of unknown categories: Based on open-vocabulary semantic matching, assuming the ROI is "drone" (unknown category), firstly, image embedding features of the ROI are extracted using an image encoder; secondly, the similarity between the image embedding features and the text embedding features of the base categories (such as "car" and "airplane") is calculated, and these similarities are all below a similarity threshold; then, the closest text embedding feature, such as "small aircraft," is matched in the CLIP model's text library by calculating cosine similarity, and this is used as a pseudo-label for the target in the ROI; the CLIP model learns the features of "drone" through this pseudo-label, and the accuracy of the text label can be further fine-tuned based on more ROIs in the future.

[0067] Step 140: Input the text description corresponding to the training image into the text encoder for processing to generate text embedding features. Calculate the cross-modal feature alignment loss using the image embedding features and the text embedding features.

[0068] The text descriptions corresponding to the training images are input into a text encoder for processing. The text descriptions are projected into the same dimensional space as the image embedding features to generate text embedding features. By using the text embedding features of the training images and their corresponding image embedding features as positive sample pairs, and the text embedding features of the training images and their non-corresponding image embedding features as negative sample pairs, the distance between the text embedding features and image embedding features is calculated using cosine similarity, resulting in the cross-modal feature alignment loss. The training objective is to maximize the cosine similarity between the text and image embeddings of each positive sample pair and minimize the cosine similarity between the text and image embeddings of each negative sample pair.

[0069] Step 150: Combine local instance-level distillation loss and cross-modal feature alignment loss to update the network parameters of the target network, and obtain the trained open target detection model;

[0070] In one embodiment, the local instance-level distillation loss and the cross-modal feature alignment loss can be weighted and summed to obtain the total loss. The network parameters of the target network are then updated using this total loss until the target network converges (e.g., reaches the maximum number of training iterations or the training error is lower than a preset error threshold), thus obtaining a trained open target detection model.

[0071] In another embodiment, in order to obtain global image context features of multi-class targets and complex environments, in addition to feature distillation through local regions of interest, comparative distillation of global image features is also performed. An image encoder extracts global features from the entire training image to obtain the first global feature; a trainable detector performs average pooling on several adapted local region features and aggregates the pooled features to obtain the second global feature; then, feature-level distillation is performed on the first global feature extracted by the image encoder and the second global feature extracted by the detector, and the global feature training loss is calculated by designing cosine similarity to achieve cross-model alignment of global context information.

[0072] Finally, the local instance-level distillation loss, cross-modal feature alignment loss, and global feature training loss are weighted and summed to obtain the total loss. The network parameters of the target network are updated using this total loss to obtain the trained open target detection model.

[0073] The formula for calculating the total loss in this embodiment is:

[0074]

[0075] In the formula, , , To balance the hyperparameters of each loss term, they are usually tuned using a validation set. For local instance-level distillation losses, The training loss is for global features. This is the cross-modal feature alignment loss.

[0076] The formula for calculating the local instance-level distillation loss is as follows:

[0077]

[0078]

[0079] In the formula, The number of potential new class target candidate boxes, The set of candidate boxes generated by the region proposal network. M is the set of true bounding boxes for the basic categories, and M is the number of true bounding boxes for the basic categories. Candidate boxes With bounding box The intersection and union ratio, Candidate boxes The confidence score, This is the intersection-union ratio threshold, used to filter candidate boxes with low overlap with the base category; This is the confidence threshold used to filter background areas; The local region features of the i-th local region of interest after deformation adaptation; Assign deformation adaptation weights to the k-th deformation adapter for the i-th local region of interest; The output feature of the k-th feature adapter; For the i-th local region of interest; FC k This is the fully connected layer for the k-th feature adapter; Text embedding features generated from regions of interest based on the basic category; The text embedding features describing the j-th text; Text embedding features Features of the adapted local region Cosine similarity between them.

[0080] The formula for calculating the global feature training loss is:

[0081]

[0082] In the formula, The first global feature for training image I; The features obtained by average pooling are the adapted local region features of the i-th local region of interest in training image I. To calculate the features corresponding to training image I after average pooling The second global feature after aggregation.

[0083] The formula for calculating cross-modal feature alignment loss is:

[0084]

[0085] In the formula, Let L be the set of known basic categories, and L be the number of known basic categories; A collection of text descriptions (including descriptions of the base categories and the generated new categories); The text embedding features describing the j-th text; Basic Category Corresponding text embedding features; Text description generated for a local region of interest i of an unknown category; These are the weight parameters.

[0086] Step 160: Perform target detection on the image to be detected using an open target detection model.

[0087] The image to be detected is input into an open object detection model. Local regions of interest (ROIs) are extracted through the region proposal network in the open object detection model. The ROIs are then aligned with features by a feature deformation adapter to obtain adapted local region features. The adapted local region features are then processed by an image encoder to obtain image embedding features. The similarity between the image embedding features and the text embedding features in the text library is calculated. The text category corresponding to the text embedding feature with the highest similarity is taken as the target category in the ROI, thus obtaining the object detection result.

[0088] This application employs a two-layer mechanism combining local instance distillation with global feature alignment, and leverages the cross-modal semantic representation capabilities of the CLIP model. While preserving the accuracy of basic class recognition, it provides a semantic similarity-based label generation path for unknown targets, effectively solving the zero-shot generalization problem in novel target classification. It is particularly suitable for target detection tasks in multi-class and complex scenarios.

[0089] This application designs an instance deformation adapter to reduce the semantic differences in features caused by local ROI feature deformation and enhance the alignment capability of deformation features. Furthermore, it proposes a granular hierarchical distillation framework to fully utilize multi-level semantic features of images, construct cross-level knowledge transfer links, and effectively improve the generalization ability of open object detection models for detecting unknown categories.

[0090] Please refer to Figure 2 This application also provides an open target detection device, including:

[0091] The network construction unit 210 is used to construct a target network, which includes a region proposal network, a feature deformation adapter, an image encoder, and a text encoder.

[0092] The feature adaptation unit 220 is used to extract local regions of interest from the training image through the region proposal network, and to perform feature alignment on the local regions of interest through the feature deformation adapter to obtain adapted local region features.

[0093] The first loss calculation unit 230 is used to input the adapted local region features into the image encoder for processing, generate image embedding features, and calculate the local instance-level distillation loss through the image embedding features and the adapted local region features.

[0094] The second loss calculation unit 240 is used to input the text description corresponding to the training image into the text encoder for processing, generate text embedding features, and calculate cross-modal feature alignment loss through image embedding features and text embedding features.

[0095] The parameter update unit 250 is used to update the network parameters of the target network by combining the local instance-level distillation loss and the cross-modal feature alignment loss, so as to obtain the trained open target detection model.

[0096] The target detection unit 260 is used to perform target detection on the image to be detected using an open target detection model.

[0097] As a further improvement, there are multiple feature deformation adapters. Each feature deformation adapter includes a feature adapter and a deformation adapter. Each feature adapter consists of a pooling layer, an activation layer, and a fully connected layer. All feature adapters share the pooling layer and the activation layer.

[0098] The feature adapter is used to perform feature mapping on the local region of interest and to perform residual connection with the local region of interest to obtain local region features;

[0099] The deformation adapter is used to select the corresponding deformation adaptation weights based on the discrete interval where the aspect ratio of the local region features is located, and then weight the local region features to obtain the adapted local region features.

[0100] As a further improvement, the device also includes a third loss calculation unit for extracting global features from the training image through an image encoder to obtain a first global feature;

[0101] The adapted local region features corresponding to the training image are subjected to average pooling and feature aggregation to obtain the second global features;

[0102] Calculate the cosine similarity between the first global feature and the second global feature to obtain the global feature training loss;

[0103] Correspondingly, the parameter update unit is specifically used to update the network parameters of the target network by combining the local instance-level distillation loss, cross-modal feature alignment loss, and global feature training loss, so as to obtain the trained open target detection model.

[0104] This application designs an instance deformation adapter to reduce the semantic differences in features caused by local ROI feature deformation and enhance the alignment capability of deformation features. Furthermore, it proposes a granular hierarchical distillation framework to fully utilize multi-level semantic features of images, construct cross-level knowledge transfer links, and effectively improve the generalization ability of open object detection models for detecting unknown categories.

[0105] Please refer to Figure 3 This application also provides an electronic device, which includes a processor 310 and a memory 320;

[0106] The memory 320 is used to store program code and transfer the program code to the processor 310;

[0107] The processor 310 is used to execute the open target detection method in the foregoing method embodiments according to the instructions in the program code.

[0108] This application also provides a computer-readable storage medium for storing program code, which, when executed by a processor, implements the open target detection method in the aforementioned method embodiments.

[0109] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0110] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.

[0111] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0115] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for executing all or part of the steps of the methods described in the various embodiments of this application through a computer device (which may be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0116] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An open target detection method, characterized in that, include: Construct a target network, which includes a region proposal network, a feature deformation adapter, an image encoder, and a text encoder; Local regions of interest are extracted from training images using a region proposal network, and the local regions of interest are feature-aligned using a feature deformation adapter to obtain adapted local region features. The adapted local region features are input into the image encoder for processing to generate image embedding features. The local instance-level distillation loss is calculated using the image embedding features and the adapted local region features. The text descriptions corresponding to the training images are input into a text encoder for processing to generate text embedding features. Cross-modal feature alignment loss is calculated using the image embedding features and the text embedding features. The network parameters of the target network are updated by combining the local instance-level distillation loss and the cross-modal feature alignment loss to obtain the trained open target detection model; The open target detection model is used to perform target detection on the image to be detected. The number of feature deformation adapters is multiple. Each feature deformation adapter includes a feature adapter and a deformation adapter. Each feature adapter consists of a pooling layer, an activation layer, and a fully connected layer. All feature adapters share the pooling layer and the activation layer. The feature adapter is used to perform feature mapping on the local region of interest and to perform residual connection with the local region of interest to obtain local region features; The deformation adapter is used to select the corresponding deformation adaptation weights based on the discrete interval where the aspect ratio of the local region features is located, and then weight the local region features to obtain the adapted local region features.

2. The open target detection method according to claim 1, characterized in that, The method further includes: The image encoder is used to extract global features from the training image to obtain the first global feature. Average pooling and feature aggregation are performed on the adapted local region features corresponding to the training image to obtain the second global features; Calculate the cosine similarity between the first global feature and the second global feature to obtain the global feature training loss; Accordingly, the network parameters of the target network are updated by combining the local instance-level distillation loss and the cross-modal feature alignment loss to obtain a trained open target detection model, including: The network parameters of the target network are updated by combining the local instance-level distillation loss, the cross-modal feature alignment loss, and the global feature training loss to obtain a trained open target detection model.

3. The open target detection method according to claim 1, characterized in that, The extraction of local regions of interest from training images via a region proposal network includes: The training images are processed by a region proposal network to generate candidate boxes with confidence scores; Candidate boxes with an intersection-union ratio (IU) lower than the IU threshold and a confidence score higher than the confidence threshold are selected from the candidate boxes to obtain new category candidate boxes; the region corresponding to the new category candidate box is the region of interest.

4. The open target detection method according to claim 3, characterized in that, The formula for calculating the local instance-level distillation loss is as follows: In the formula, N represents the number of candidate boxes for the new class, and M represents the number of ground truth boxes for the base class. The local region features of the i-th local region of interest after deformation adaptation; The text embedding features describing the j-th text; Text embedding features generated from regions of interest based on the basic category. Text embedding features Features of the adapted local region Cosine similarity between them.

5. The open target detection method according to claim 1, characterized in that, The calculation of cross-modal feature alignment loss using the image embedding features and the text embedding features includes: The text embedding features of the text description and the corresponding image embedding features are used as positive sample pairs, and the text embedding features of the text description and the non-corresponding image embedding features are used as negative sample pairs. The cosine similarity between the text embedding features and the image embedding features is calculated to obtain the cross-modal feature alignment loss.

6. The open target detection method according to claim 1, characterized in that, The formula for calculating the cross-modal feature alignment loss is as follows: In the formula, Let L be the set of known basic categories, and L be the number of known basic categories; A collection of text descriptions; The text embedding features describing the j-th text; Basic Category Corresponding text embedding features; Text embedding features generated from regions of interest based on the basic category; Text description generated for a local region of interest i of an unknown category; These are the weight parameters.

7. An open target detection device, characterized in that, include: A network construction unit is used to construct a target network, which includes a region proposal network, a feature deformation adapter, an image encoder, and a text encoder; The feature adaptation unit is used to extract local regions of interest from the training image through a region proposal network, and to perform feature alignment on the local regions of interest through a feature deformation adapter to obtain adapted local region features. The first loss calculation unit is used to input the adapted local region features into the image encoder for processing, generate image embedding features, and calculate the local instance-level distillation loss through the image embedding features and the adapted local region features. The second loss calculation unit is used to input the text description corresponding to the training image into the text encoder for processing, generate text embedding features, and calculate cross-modal feature alignment loss through the image embedding features and the text embedding features. The parameter update unit is used to update the network parameters of the target network by combining the local instance-level distillation loss and the cross-modal feature alignment loss, so as to obtain the trained open target detection model. The target detection unit is used to perform target detection on the image to be detected using the open target detection model. The number of feature deformation adapters is multiple. Each feature deformation adapter includes a feature adapter and a deformation adapter. Each feature adapter consists of a pooling layer, an activation layer, and a fully connected layer. All feature adapters share the pooling layer and the activation layer. The feature adapter is used to perform feature mapping on the local region of interest and to perform residual connection with the local region of interest to obtain local region features; The deformation adapter is used to select the corresponding deformation adaptation weights based on the discrete interval where the aspect ratio of the local region features is located, and then weight the local region features to obtain the adapted local region features.

8. An electronic device, characterized in that, The device includes a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the open target detection method according to any one of claims 1-6 according to the instructions in the program code.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code, which, when executed by a processor, implements the open target detection method according to any one of claims 1-6.

Images