Few-sample object detection method based on semantic enhancement feature generation and prediction optimization

Abstract

This invention discloses a few-shot object detection method based on semantically enhanced feature generation and prediction optimization. A few-shot object detection model is constructed, comprising a query branch, support branch, class-independent aggregation module, detection head, and detection result optimization module. A feature generator is then built to train the few-shot object detection model. The constructed feature generator integrates semantic and visual information, allowing the generator to strengthen category-centric representations through cross-modal constraints, thereby ensuring increased data diversity while clearly defining the boundaries between different categories. Furthermore, the prediction optimization module disclosed in this invention accurately filters out potential false alarms caused by bounding box offsets, ensuring that only the most reliable detection results are retained. The method proposed in this invention achieves superior performance compared to other methods.

Description

Technical Field

[0001] This application belongs to the field of few-sample object detection technology, and in particular relates to a few-sample object detection method based on semantic enhancement feature generation and prediction optimization. Background Technology

[0002] The Few Sample Object Detection (FSOD) task aims to simulate the human ability to identify objects with only a few observations, so that the model no longer relies on a large amount of labeled data.

[0003] Most state-of-the-art models are based on the two-stage Faster-R-CNN framework, but these methods fall far short of satisfactory classifier performance due to the lack of diversity in training data. To address this issue, researchers have proposed various data augmentation methods. For example, CNPB is an image-level sample augmentation method that inserts new class targets against different backgrounds. Other methods take a more flexible approach, augmenting at the feature level. Specifically, Hallucination proposes generating diverse training data for new classes by mimicking variations in the base class. DELO proposes generating features in zero-shot learning using a conditional variational autoencoder. Norm-VAE found that the lack of crop-related diversity led to performance degradation and then proposed generating crop-related features to improve the robustness of the classifier.

[0004] Despite these advancements, over-reliance on visual features of base classes can blur the generative boundary between novel and base classes. Finding a balance between generating features that are diverse enough to train a robust classifier and accurate enough to avoid misclassification remains a challenging problem.

[0005] Furthermore, existing methods focus on improving the performance of the detection head, but lack attention to the post-processing of the prediction results. Summary of the Invention

[0006] The purpose of this application is to provide a few-sample object detection method based on semantically enhanced feature generation and prediction optimization to overcome the problems mentioned in the background art. In this application, this method is also referred to as SE-FewDet.

[0007] To achieve the above objectives, the technical solution of this application is as follows:

[0008] A few-shot object detection method based on semantically enhanced feature generation and prediction optimization includes:

[0009] A few-shot target detection model is constructed, which includes a query branch, a support branch, a class-independent aggregation module, a detection head, and a detection result optimization module. A feature generator is then constructed to train the few-shot target detection model.

[0010] The image to be detected in the detection task is input into the trained small sample object detection model, and the features of the region of interest are extracted through the query branch.

[0011] The sample images of the detection task are input into the trained small sample object detection model, and the category features are extracted through the support branches;

[0012] The region of interest features and category features are aggregated by the class-independent aggregation module to obtain aggregated features. The aggregated features are then passed through the detection head to obtain preliminary detection results.

[0013] Input the preliminary detection results and a list of all category labels into the detection result optimization module to filter the negative prediction box and obtain the final detection results.

[0014] Furthermore, the feature generator includes a visual feature encoder, a visual latent space mapping layer, a semantic feature encoder, a semantic latent space mapping layer, and a latent space feature decoder, and the feature generator performs the following operations:

[0015] The region of interest features are input into a visual feature encoder to obtain a visual latent space representation.

[0016] The visual latent space representation is input into the visual latent space mapping layer to obtain its mean vector and variance vector in the visual latent space;

[0017] The category labels are input into the CLIP model to extract category semantic features, and then input into the semantic feature encoder to obtain the semantic latent space representation;

[0018] The semantic latent space representation is input into the semantic latent space mapping layer to obtain its mean vector and variance vector in the semantic latent space;

[0019] The mean and variance vectors in the visual latent space and the semantic latent space are fused to obtain a more robust category distribution;

[0020] The latent space vector is sampled from the category distribution and then input into the latent space feature decoder to obtain the reconstructed features.

[0021] Furthermore, the step of constructing the feature generator to train the few-shot target detection model includes:

[0022] The first stage involves freezing the parameters of the feature generator and training a small sample object detection model with the feature generator added.

[0023] The second stage involves freezing all parameters except for the feature generator and training the feature generator.

[0024] In the third stage, all parameters except for the feature generator and the detection head are frozen, and a few-sample object detection model with the feature generator is trained.

[0025] Furthermore, the loss function used to train the feature generator is:

[0026] L = L mse +β*L KL +L recons ;

[0027]

[0028]

[0029] L recons =‖f roi -f recons ||;

[0030] Where L is the total loss function, L mse For the squared error loss, L KL For relative entropy loss, L recons For reconstruction loss, β is the weighting coefficient, and μ is the weighting coefficient. i Let σ be the mean value in the visual latent space. i Let μ be the variance vector in the visual latent space. i ′ is the mean in the semantic latent space, σ i ′ is the variance vector in the semantic latent space. The mean of the category distribution. Let f be the variance vector of the class distribution. roi For the features of the region of interest, f recons To reconstruct the features, i represents the i-th dimension, and d l For latent space dimensions.

[0031] Furthermore, the fusion of the mean and variance vectors in the visual latent space and the semantic latent space to obtain a more robust category distribution is expressed by the following formula:

[0032]

[0033] Where α is the weighting coefficient, μ is the mean in the visual latent space, σ is the variance vector in the visual latent space, μ′ is the mean in the semantic latent space, and σ′ is the variance vector in the semantic latent space. The mean of the category distribution. This is the variance vector of the class distribution.

[0034] Furthermore, the step of inputting the preliminary detection results into the detection result optimization module to filter the negative prediction boxes and obtain the final detection results includes:

[0035] The corresponding target region is obtained from the image to be detected based on the predicted bounding box in the preliminary detection results;

[0036] Image features of the target region are obtained using the image encoder in the CLIP model;

[0037] The semantic features of all category labels are obtained using the text encoder in the CLIP model;

[0038] Calculate the cosine similarity matrix of image features and semantic features, and sum the similarity values ​​by taking a preset number of similarity values ​​from largest to smallest to obtain a sum value;

[0039] The ratio of the similarity value corresponding to the category label in the preliminary detection result to the sum value is calculated as the confidence level of the category label in the preliminary detection result, and the result with the higher confidence level is retained as the final detection result.

[0040] This application addresses the challenge of insufficient data diversity in small-sample object detection by proposing an innovative feature generation method. The SE-FewDet model introduces a novel structure called the Feature Generator SemVAE, which is designed to progressively integrate learned semantic information into the visual space. Therefore, the model can sample from a more robust class distribution and generate more diverse samples. SE-FewDet also includes a Precise Prediction Optimization (PPR) module for optimizing prediction results, which helps to further filter out false detection boxes, aiming to improve the reliability of the results. This module uses a simple yet effective post-processing method to address the large number of false positives in the results, significantly improving the reliability. Extensive experiments demonstrate that, in most cases, SE-FewDet outperforms several state-of-the-art methods. Attached Figure Description

[0041] Figure 1 This is a flowchart of the small sample target detection method of this application.

[0042] Figure 2 This is a structural diagram of the small sample target detection model in this application.

[0043] Figure 3 The overall model structure diagram for incorporating the feature generator in this application is shown.

[0044] Figure 4 This is a schematic diagram of the detection result optimization module in an embodiment of this application. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0046] One embodiment of this application, such as Figure 1 As shown, a few-shot object detection method based on semantically enhanced feature generation and prediction optimization is proposed, including:

[0047] Step S1: Construct a few-shot target detection model, which includes a query branch, a support branch, a class-independent aggregation module, a detection head, and a detection result optimization module, and construct a feature generator to train the few-shot target detection model.

[0048] The small-sample object detection model constructed in this application, such as Figure 2 As shown, it includes a query branch, a support branch, a class-agnostic aggregation module, a detection head, and a prediction result optimization module (PPR).

[0049] The query branch includes a backbone network, a candidate box extraction module, and a region of interest (ROI) alignment module. The backbone network uses ResNet-101 to extract feature maps; the candidate box extraction module (RPN) extracts candidate boxes; and the region of interest alignment module (RoI Align) extracts features from the regions of interest. The backbone network, RPN, and ROI Align in the query branch are all relatively mature technologies and will not be elaborated upon here.

[0050] The support branch includes a backbone network and a support feature extractor. The backbone network uses ResNet-101 to extract feature maps, while the support feature extractor uses a convolutional network to extract categorical features.

[0051] The class-independent aggregation module is used to fuse the features extracted from two branches, generating a weight vector and performing a dot product with the features from the query branch, thereby improving feature quality. The class-independent aggregation module is also a relatively mature technology in this field, and will not be elaborated upon further below.

[0052] The detection head contains a classification layer and a regression layer, both implemented as fully connected layers (with a dimension of 2048). The classification layer predicts the category label of the input feature, the regression layer predicts the location offset of the candidate box to which the input feature belongs, and the detection head outputs the detection result.

[0053] The Prediction Result Optimization (PPR) module is used to filter negative prediction boxes to obtain better detection results.

[0054] It should be noted that, except for the prediction result optimization module (PPR), the other parts of the small-sample target detection model in this application are relatively mature technologies in the field, and will not be elaborated upon here. The prediction result optimization module (PPR) will be described in detail in subsequent steps.

[0055] To better train the aforementioned few-shot object detection model, this application constructs a feature generator (SemVAE) to participate in the model's training process, thereby improving the detection performance of the trained model. The network model incorporating the SemVAE feature generator is as follows: Figure 3 As shown, the feature generator is added to the query branch to generate reconstructed features.

[0056] The feature generator SemVAE constructed in this application includes a visual feature encoder E and a visual latent space mapping layer (with parameters of...). ), semantic feature encoder SemE, semantic latent space mapping layer (parameters are...) The latent space feature decoder is Dec. The input dimensions of encoders E and SemE are 2048 and 512, respectively. Both encoders consist of three fully connected layers, while decoder Dec has two fully connected layers, each with 2048 hidden units. The latent space dimension d... h Set to 2048, and the latent space dimension d l Set to 512. The activation functions used for the hidden and output layers are LeakyReLU and ReLU, respectively.

[0057] In this embodiment, the feature generator SemVAE performs the following operations:

[0058] The features of the region of interest f roi The input is fed into the visual feature encoder to obtain the visual latent space representation h. roi ;

[0059] Representing the visual latent space as h roi The mean vector μ and variance vector σ in the visual latent space are obtained by inputting the visual latent space mapping layer;

[0060] The category labels are input into the CLIP model to extract category semantic features f. sem Then, it is input into the semantic feature encoder to obtain the semantic latent space representation h. sem ;

[0061] Represent the semantic latent space h sem The input semantic latent space mapping layer yields its mean vector μ′ and variance vector σ′ in the semantic latent space;

[0062] By fusing the mean μ and variance vector σ in the visual latent space with the mean μ′ and variance vector σ′ in the semantic latent space, a more robust category distribution is obtained.

[0063] From category distribution The latent space vector z is obtained by sampling and then input into the latent space feature decoder to obtain the reconstructed feature f. recons .

[0064] CLIP, short for Contrastive Language-Image Pre-Training, uses a text encoder to encode text. In this application, the category labels corresponding to the image are input into the CLIP model to extract category semantic features f. sem The following will not be elaborated further.

[0065] With the addition of a feature generator, the entire model training process includes:

[0066] The first stage involves freezing the parameters of the feature generator and training a small sample object detection model with the feature generator added.

[0067] The second stage involves freezing all parameters except for the feature generator and training the feature generator.

[0068] In the third stage, all parameters except for the feature generator and the detection head are frozen, and a few-sample object detection model with the feature generator is trained.

[0069] Specifically, the first stage is the basic training stage, which uses the base class dataset Cbase to train the entire model's detection capability against the base class. During this stage, the parameters of the SemVAE feature generator module are frozen. The second stage is to train SemVAE separately using Cbase. During this stage, all parameters except SemVAE need to be frozen. The third stage is the fine-tuning stage, which freezes all parameters except SemVAE and the detection head. The model is then fine-tuned using the new class dataset CNovel and the base class dataset Cbase. During this stage, the trained SemVAE is used to generate additional training data for the new class and is used together with the original data for training the detection head in subsequent models.

[0070] In the separate training phase of the feature generator SemVAE, for the image Img containing the target object O, its true class label is... SemVAE's input can be mainly divided into two branches: visual and semantic.

[0071] The input to the visual branch is the region of interest feature f. roi Region of interest features f roi The features output by ROI Align are the filtered target positive sample features, f roiThis is then encoded by the visual feature encoder E and mapped to d. h The hidden space is then mapped to the weights of the layer through the visual latent space. and bias Weight and bias The mean μ and variance σ of the visual latent space can be obtained mathematically and described as follows:

[0072]

[0073] The input to the semantic branch is the semantic feature f sem Semantic features f sem The text encoder of the CLIP model measures the category labels. This is obtained through encoding. sem This is then encoded by the semantic feature encoder SemE and mapped to d. h The semantic latent space is then mapped to the weights of the semantic latent space layer. and bias Weight and bias The mean μ′ and variance σ′ of the semantic latent space can be obtained through mathematical formulas as follows:

[0074]

[0075] To balance the visual latent space and the semantic latent space derived from semantic information mapping, a variable weight α is introduced, starting from 1 and decreasing by 0.1 every 1000 iterations until it reaches 0.5. This weight is used to balance the interference to the latent space feature decoder during the fusion of the visual and semantic latent spaces. This ensures that semantic information is gradually injected into the latent representation during training. middle:

[0076]

[0077] The latent space feature decoder Dec is used to fuse semantic and visual information to generate reconstructed features:

[0078] f recons =Dec(z);

[0079] in and Let represent the distribution of new categories in the latent space that integrates semantic and visual information. ε is a noise vector sampled from a standard normal distribution.

[0080] In one specific embodiment, during the training of the feature generator SemVAE, the parameters of SemVAE are updated by minimizing the loss L:

[0081] L = L mse +β*LKL+L recons ;

[0082] in:

[0083]

[0084] This embodiment calculates the mean and squared variance error loss L between two multivariate distributions. mse Aligning these distributions enhances category-specific information and helps to more clearly distinguish distributions corresponding to different categories in the latent space.

[0085] L KL L is the relative entropy between the posterior and prior distributions of SemVAE (chosen as the standard normal distribution), β is a weighting coefficient that balances latent space regularization and the quality of samples generated during training, used to constrain the class distribution boundary to make it tend towards a normal distribution. KL The relative entropy loss is calculated as follows:

[0086]

[0087] Reconstruction loss L recons Used to optimize the output of the latent space feature decoder, forcing the model to recover the details of the input data:

[0088] L recons =‖f roi -f recons ||;

[0089] Where f recons This represents the reconstructed features output by the latent space feature decoder Dec.

[0090] Where L is the total loss function, L mse For the squared error loss, L KL For relative entropy loss, L recons For reconstruction loss, β is the weighting coefficient, and μ is the weighting coefficient. i Let σ be the mean value in the visual latent space. i Let μ be the variance vector in the visual latent space. i ′ is the mean in the semantic latent space, σ i ′ is the variance vector in the semantic latent space. The mean of the category distribution. Let f be the variance vector of the class distribution. roi For the features of the region of interest, f recons To reconstruct the features, i represents the i-th dimension, and d l For latent space dimensions.

[0091] In this embodiment, during the training process of the feature generator, alignment of the visual latent space and semantic latent space is achieved, and the generated features are guided. For visual latent space distributions (μ, σ) and semantic latent space distributions (μ′, σ′) of the same category, the mean squared error is used to calculate the losses between μ and μ′, and between σ and σ′, respectively, thereby aligning the visual latent space and semantic latent space. Simultaneously, the reconstructed feature f is calculated. recons and f roi The mean squared error between the two is used to constrain the generated features, making them closer to the features of real samples.

[0092] It should be noted that the training of the entire model is a relatively mature technology in this field, and will not be elaborated on here.

[0093] Step S2: Input the image to be detected into the trained small sample object detection model, and extract the region of interest features through the query branch.

[0094] For a detection task with m categories and k sample images for each category, to detect the category of the image to be detected, the image to be detected is first input into a trained small sample object detection model, and the region of interest features are extracted through the query branch.

[0095] Specifically, in the query branch, the image to be detected is processed by the backbone network, RPN, and ROI Align to extract the region of interest features f. roi .

[0096] Step S3: Input the sample images of the detection task into the trained small sample object detection model, and extract the category features through the support branches.

[0097] For a detection task with m categories and k sample images for each category, the m*k samples are input into the support branch, and feature maps are obtained through the backbone network and the support feature extractor to obtain the category feature list F.

[0098] Step S4: Aggregate the region of interest features and category features through the class-independent aggregation module to obtain aggregated features. Pass the aggregated features through the detection head to obtain preliminary detection results.

[0099] The class-independent aggregation module performs a feature f for each region of interest. roi A feature is randomly selected from the list of category features F and aggregated. The aggregated feature is then input into the detection head.

[0100] The aggregated features are input into the detection head to obtain a label list l and a bounding box list b, which is the preliminary detection result (l, b).

[0101] Step S5: Input the preliminary detection results into the negative prediction box of the detection result optimization module to obtain the final detection results.

[0102] This step inputs the preliminary detection results (l, b) into the detection result optimization module, filters out the negative sample detection boxes, and obtains the final detection results.

[0103] Simply improving detector performance cannot reduce the proportion of negative predicted boxes in the results. These negative predicted boxes usually deviate from the true values ​​and may mislead the classifier. Therefore, this embodiment designs an accurate detection result optimization module (PPR) to filter these negative predicted boxes.

[0104] like Figure 4 As shown, given an image Img to be detected, the detection head outputs a label list l and a bounding box list b, R n It is a target region image cropped from Img based on b[n], where n is the index of b.

[0105] Calculate the similarity matrix between the image features of each target region image and the text features of each category label:

[0106] Sim = Softmax(Cos(ImgE(R) n ),TextE(C j )));

[0107] This embodiment uses the CLIP model image encoder (ImgE) to extract R. n Image features F R Using the Text Encoder (TextE) in the CLIP model, obtain the semantic feature list F of all labels in the category label list C. C Then, calculate the image features F. R Each category label C j The cosine similarity (Cos()) between semantic features is obtained by passing the Softmax function.

[0108] Then, the confidence score of each category label l[n] in the preliminary detection result label list l is calculated. n :

[0109]

[0110] Where C is a list containing m categories (Cbase∪Cnovel), and max() represents taking the first ten values. Sim[n][l[n]] represents the similarity value s corresponding to each category label l[n] in the preliminary detection result label list l. S represents the sum of similarity values, ordered from highest to lowest, with a predetermined number of similarity values. The confidence score is obtained by calculating the ratio of S to sum. i .

[0111] This embodiment calculates the confidence score of the nth predicted label l[n] by summing the first ten similarity values ​​in descending order. n Finally, a threshold is manually set for filtering, retaining predictions with scores exceeding the threshold. The threshold, determined experimentally, is 0.1. Then, the confidence score of each prediction is recalculated, and the predictions with high confidence scores are retained as the final output detection results.

[0112] On the PASCAL VOC dataset, as shown in Table 1, the technical solution adopted in this application has significant advantages compared with other methods. In 15 settings (each of the five different classes in the PASCAL VOC dataset is designated as a Novel class, thus dividing the dataset into different Novel sets (1, 2, 3), the performance of the model under different sample sizes (1, 2, 3, 5, 10-shots) was tested in each Novel set. Therefore, out of 3*5=15 settings, 14 settings achieved optimal performance, while 1 setting achieved the second-best result. Notably, the performance of the method in this application is significantly improved under different Novel Set 1 shooting conditions, with an improvement ranging from 3.1% to 6.9%, indicating that the method is robust under different shot sizes. Furthermore, it significantly narrows the gap between K=1 and K=10 (9.7% vs 5.9%). This demonstrates that the method in this application is very effective in improving the generalization defects caused by data scarcity.

[0113] Furthermore, in Novel Set 2 and Novel Set 3, the method of this application achieved average growth of 3.82% and 2.06%, respectively. This demonstrates that the application is not biased towards a specific category set and shows its generalization potential in a wider range of practical applications.

[0114] On the MS COCO dataset, considering the advantages of meta-learning in FSOD, such as rapid adaptation to new categories and improved model robustness, the meta-learning-based architecture VFA was chosen as the benchmark. Although VFA is not dominant on the MS COCO dataset, our proposed method significantly improves performance here. As shown in Table 2, our proposed method achieves an average improvement of 3% in the K=10 and K=30 settings. It ranks first in the 10-shot setting and second in the 30-shot setting, even approaching the performance of state-of-the-art non-meta-learning methods. These experimental data demonstrate that our proposed method can effectively alleviate the problem of insufficient model generalization performance caused by data scarcity and can effectively filter out negative samples in the results, improving the reliability of the model's prediction results.

[0115] Table 1

[0116]

[0117]

[0118] Table 1 shows the performance comparison between the proposed method and other methods on the PASCAL VOC dataset. Different Novel Sets represent different base class-new class partitions, and k-shots represent k labeled training samples for each new class. The above data represents the prediction accuracy nAP-50 of the new class when the IOU threshold is 0.5.

[0119] Table 2

[0120]

[0121] As shown in Table 2, existing methods on the MS COCO dataset are divided into two categories: meta-learning-based methods and non-meta-learning-based methods. The average prediction accuracy (nAP) for the new class is also presented when k = 10 and 30.

[0122] The embodiments described above are merely illustrative of implementation methods of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A few-sample target detection method based on semantically enhanced feature generation and prediction optimization, characterized in that, The few-shot object detection method based on semantically enhanced feature generation and prediction optimization includes: A few-shot target detection model is constructed, which includes a query branch, a support branch, a class-independent aggregation module, a detection head, and a detection result optimization module. A feature generator is then constructed to train the few-shot target detection model. The image to be detected in the detection task is input into the trained small sample object detection model, and the features of the region of interest are extracted through the query branch. The sample images of the detection task are input into the trained small sample object detection model, and the category features are extracted through the support branches; The region of interest features and category features are aggregated by the class-independent aggregation module to obtain aggregated features. The aggregated features are then passed through the detection head to obtain preliminary detection results. Input the preliminary detection results and the list of all category labels into the detection result optimization module to filter the negative prediction box and obtain the final detection results; The feature generator includes a visual feature encoder, a visual latent space mapping layer, a semantic feature encoder, a semantic latent space mapping layer, and a latent space feature decoder. The feature generator performs the following operations: The region of interest features are input into a visual feature encoder to obtain a visual latent space representation. The visual latent space representation is input into the visual latent space mapping layer to obtain its mean vector and variance vector in the visual latent space; The category labels are input into the CLIP model to extract category semantic features, and then input into the semantic feature encoder to obtain the semantic latent space representation; The semantic latent space representation is input into the semantic latent space mapping layer to obtain its mean vector and variance vector in the semantic latent space; The mean and variance vectors in the visual latent space and the semantic latent space are fused to obtain a more robust category distribution; The latent space vector is sampled from the category distribution and then input into the latent space feature decoder to obtain the reconstructed features.

2. The few-sample target detection method based on semantic enhancement feature generation and prediction optimization according to claim 1, characterized in that, The construction of the feature generator to train the few-sample object detection model includes: The first stage involves freezing the parameters of the feature generator and training a small sample object detection model with the feature generator added. The second stage involves freezing all parameters except for the feature generator and training the feature generator. In the third stage, all parameters except for the feature generator and the detection head are frozen, and a few-sample object detection model with the feature generator is trained.

3. The few-sample target detection method based on semantic enhancement feature generation and prediction optimization according to claim 2, characterized in that, The loss function used to train the feature generator is: ； ； ； ； in, For the total loss function, For the squared error loss, For relative entropy loss, To reconstruct the loss, These are weighting coefficients. The mean value in the visual latent space. Let be the variance vector in the visual latent space. The mean value in the semantic latent space. Let be the variance vector in the semantic latent space. The mean of the category distribution. Let Variance be the variance vector of the class distribution. For the features of the region of interest, To reconstruct the features, i represents the i-th dimension. For latent space dimensions.

4. The few-sample target detection method based on semantic enhancement feature generation and prediction optimization according to claim 1, characterized in that, The method of fusing the mean and variance vectors in the visual latent space and the semantic latent space to obtain a more robust category distribution is expressed by the following formula: ； in, These are the weighting coefficients. The mean value in the visual latent space. Let be the variance vector in the visual latent space. The mean value in the semantic latent space. Let be the variance vector in the semantic latent space. The mean of the category distribution. This is the variance vector of the class distribution.

5. The method for semantically enhanced feature generation and prediction optimization according to claim 1, characterized in that, The step of inputting the preliminary detection results into the detection result optimization module to filter the negative prediction box and obtain the final detection results includes: The corresponding target region is obtained from the image to be detected based on the predicted bounding box in the preliminary detection results; Image features of the target region are obtained using the image encoder in the CLIP model; The semantic features of all category labels are obtained using the text encoder in the CLIP model; Calculate the cosine similarity matrix of image features and semantic features, and sum the similarity values ​​by taking a preset number of similarity values ​​from largest to smallest to obtain a sum value; The ratio of the similarity value corresponding to the category label in the preliminary detection result to the sum value is calculated as the confidence level of the category label in the preliminary detection result, and the result with the higher confidence level is retained as the final detection result.

Images