Real-time open vocabulary video target detection method based on efficient time sequence aggregation and knowledge migration

Abstract

The invention discloses a real-time open vocabulary video target detection method based on efficient time sequence aggregation and knowledge migration, and relates to a computer vision technology. The problems of poor real-time performance, weak generalization ability of novel categories, insufficient stability of cross-frame detection and the like in the prior art are solved. Firstly, a sparse sampling strategy is adopted to select key frames to reduce calculation overhead; secondly, an enhanced supervision set is constructed, and a basic category real label and a novel category pseudo label generated by a teacher model are fused; extracting multi-scale anchor-level features through a backbone network, and aggregating time sequence features through an anchor-level memory attention module; and finally, reasoning by adopting a label space expert mixed strategy, and optimizing the model in combination with a multi-task loss. Compared with the prior art, the method has the advantages that excellent performance-efficiency balance is achieved, the method is outstanding in performance on data sets such as LV-VIS and BURST, the reasoning speed is increased by 70-120 times compared with an existing method, meanwhile, the robustness to novel categories and specific video challenges is enhanced, and the method is suitable for real-time application scenes such as intelligent monitoring and automatic driving.

Description

Technical Field

[0001] This invention relates to computer vision technology, specifically to a real-time open-vocabulary video object detection method based on efficient temporal aggregation and knowledge transfer. Background Technology

[0002] Traditional video object detection (VOD) methods are typically trained and evaluated on closed class sets, making it difficult to perceive novel concept classes in real-world scenes and prone to performance degradation in cross-domain deployments. Furthermore, video scenes exhibit significant temporal characteristics, presenting challenges such as motion blur, occlusion, object entry / exit from the frame, and abnormal poses. These problems are further amplified in open-vocabulary scenarios, limiting their application in open-world environments.

[0003] In recent years, open-vocabulary object detection methods have achieved the detection of objects of any category by introducing visual language models. However, when directly applied to video data, they ignore the rich temporal dependencies in videos, leading to a decline in detection performance when facing video-specific challenges such as motion blur, occlusion, and abnormal poses. Existing open-vocabulary video object detection methods, such as OV-VOD, although employing post-processing strategies for temporal modeling, rely on two-stage detectors, resulting in high inference latency and deployment costs. Furthermore, lightweight models are prone to catastrophic forgetting during cross-domain fine-tuning, weakening their generalization ability to novel categories. Summary of the Invention

[0004] The purpose of this invention is to address the problems of low inference efficiency, insufficient temporal modeling, and catastrophic forgetting in video object detection in open-vocabulary scenarios. It provides a real-time open-vocabulary video object detection method based on efficient temporal aggregation and knowledge transfer, solving the problems of instability in cross-frame detection of new concept categories, catastrophic forgetting in cross-domain adaptation of small models, and high inference overhead and difficulty in real-time deployment in existing methods. Through an end-to-end temporal modeling module and a knowledge distillation strategy, the detection stability for novel categories is significantly improved while maintaining high inference efficiency.

[0005] To achieve the above-mentioned objectives, the present invention provides the following technical solutions.

[0006] A real-time open-vocabulary video object detection method based on efficient temporal aggregation and knowledge transfer includes the following steps:

[0007] 1) Keyframe sparse sampling: Given a video dataset, uniformly sample keyframe sequences from each video in the video dataset;

[0008] 2) Text feature extraction and pseudo-label generation: The labels of the keyframe sequence are input into the text encoder through the text template to extract text features. At the same time, the keyframe sequence is input into the teacher model. The teacher model with frozen parameters is used to generate novel category pseudo-labels offline on the video training set. The pseudo-labels are fused with the real labels of the basic categories to construct an enhanced supervision set.

[0009] 3) Backbone network feature extraction: Input keyframes into the backbone network, extract and fuse multi-scale features through convolutional layers and path aggregation networks to generate anchor-level feature embeddings;

[0010] 4) Anchor-level memory attention modeling: various anchor point optimizations are performed on the anchor-level feature embedding, representative anchor point features are selected and stored in the memory bank, and historical frame features are aggregated through the memory attention mechanism to output temporal fusion features;

[0011] 5) Label Space Expert Hybrid Strategy: Input the temporal fusion features into the decoupled base category expert and novel category expert, and fuse the probability distribution of the two experts through label space perception gating to output the final category distribution;

[0012] 6) Joint model optimization: Calculate the loss function with the final class distribution and the enhanced supervision set, and update the gradients, including detection, bounding box regression and segmentation losses.

[0013] In step 1), the specific steps of the keyframe sparse sampling can be as follows: given an open vocabulary video object detection dataset (such as LV-VIS), which contains several video sequences, each video consisting of consecutive frames; use a sparse sampling strategy to select keyframes to reduce computational overhead; specifically, uniformly sample keyframe sequences from each video for subsequent processing to ensure that the model can focus on significant change regions in the video.

[0014] In step 2), the specific steps of text feature extraction and pseudo-label generation can be as follows: The labels of the video frame sequence obtained in step 1) are input into a text encoder through a text template (e.g., "an image containing {category name}") to extract text features; the text template is designed as a natural language description to enhance semantic alignment; simultaneously, a teacher model with strong open vocabulary awareness (e.g., CLIP-VIS) is used to generate novel category pseudo-labels offline on the video training set; the teacher model parameters are frozen, and pseudo-labels are generated through a single forward propagation; this process is represented as:

[0015]

[0016] in, Indicates a pseudo tag. This represents categories that are not visible during training. τ is the confidence score for predicting category c, and τ is a threshold hyperparameter used to filter low-quality predictions (default value is 0.5). It is used to filter low-quality predictions and ensure high fidelity of pseudo-labels. When τ is too small, noise will be introduced. When τ is too large, the number of pseudo-labels is insufficient, which will affect the supervision effect. Through experiments, τ=0.5 can achieve the best balance between the quality and quantity of pseudo-labels.

[0017] In step 3), the specific steps for backbone network feature extraction can be as follows: input the sampled video frames from step 1) into the backbone network (such as YOLOv8 or YOLO11) to extract multi-scale anchor-level feature embeddings; the backbone network is designed based on a real-time open vocabulary detector and includes feature extraction, regression head, and segmentation head; specifically, the backbone network first extracts spatial features through convolutional layers, and then uses a path aggregation network (PAN) to fuse multi-scale information to generate anchor-level feature embeddings. Where A is the number of anchor points and d is the feature dimension; this process uses pre-trained weight initialization to accelerate convergence and supports end-to-end training.

[0018] In step 4), the specific steps of the anchor-level memory attention modeling can be as follows: input the multi-scale features obtained in step 3) into the anchor-level memory attention module; this module enhances keyframe representation through temporal modeling. First, it applies a diverse anchor optimization (DAR) strategy to reduce background noise; specifically, it embeds anchor-level features... Perform non-maximum suppression (NMS) and TopK scoring, and select the top K (usually K=100) most representative anchor features: ;where K A (typically K=100); subsequently, the memory bank stores the optimized features of the keyframes. As a temporal memory; the memory attention mechanism aggregates historical frame features (assuming N supporting frames, the features are...). This is calculated through cross-attention; the process is represented as:

[0019]

[0020] In step 5), the specific steps of the label space expert fusion strategy can be as follows: input the temporal fusion features output in step 4) into the label space expert fusion strategy; this strategy decouples the base category experts (Experts) base ) and novelty category experts novel The basic category experts consist of frozen parameter copies of the network classification layer; novel category experts are fine-tuned and updated using video-specific data (such as LV-VIS); firstly, Aligning with the pre-extracted text embedding T (obtained by processing the category names through a text encoder), compute the full-class probability distribution for each expert: It is the probability distribution of all categories from the basic category experts. This is the probability distribution of all categories from the new category experts; subsequently, the probability distribution is fused through label space-aware gating, and the final category distribution is output; this process is represented as:

[0021]

[0022] in This is the gating factor (default setting is 0.7), which controls the confidence level of each expert in their area of ​​expertise; when the category belongs to the base category (C... b When the category belongs to the novel category (C), the experts in the basic category have higher weights; when the category belongs to the novel category (C), the experts in the basic category have higher weights. n When it comes to novel categories, experts have higher weighting.

[0023] In step 6), the specific steps of the joint optimization of the model can be as follows: combine the prediction results obtained in step 5) with the enhanced supervision set. Calculate the loss function and jointly optimize the model parameters; the loss function includes classification loss. Bounding box regression loss and segmentation loss Jointly optimize model parameters; the loss function is defined as: Specifically, the classification loss uses focal loss to handle class imbalance, the bounding box regression loss uses GIoU (Generalized Intersection over Union) loss to improve localization accuracy, and the segmentation loss generates instance masks based on binary cross-entropy. During training, the AdamW optimizer is used, with an initial learning rate of... The batch size was 32, the momentum was 0.9, and the weight decay was 0.025. The training period was 30 rounds, and the learning rate was scheduled using a cosine annealing strategy. Mosaic enhancement was disabled in the 29th round to accelerate convergence.

[0024] Compared with the prior art, the present invention has the following outstanding technical effects and advantages:

[0025] This invention proposes a real-time open vocabulary video object detection method based on YOVO. First, it employs a pseudo-label knowledge distillation strategy to generate high-quality pseudo-labels using a teacher model, enhancing the model's supervision of new categories. Second, it introduces an anchor-level memory attention module to achieve end-to-end temporal modeling, mitigating cross-frame detection instability. Furthermore, it designs a label space expert hybrid strategy, reducing overfitting during fine-tuning through expert decoupling and gating mechanisms. Extensive comparative experiments were conducted on the LV-VIS, BURST, and TAO datasets, demonstrating a significant performance improvement. Specifically, on the LV-VIS dataset, the APn index is improved by 6.1% compared to OV-VOD, while inference speed is increased by 70-120 times. Attached Figure Description

[0026] Figure 1 This is an overall framework diagram of the present invention.

[0027] Figure 2 This is a schematic diagram of an anchor-level memory attention strategy.

[0028] Figure 3 The image shows the prediction results of this invention in a motion-induced blurred scene.

[0029] Figure 4 The figure shows the prediction results of this invention under abnormal posture fields caused by motion. Detailed Implementation

[0030] The method of the present invention will be described in detail below with reference to the accompanying drawings and embodiments. This embodiment is implemented under the premise of the technical solution of the present invention, and provides implementation methods and specific operation processes. However, the protection scope of the present invention is not limited to the following embodiments.

[0031] Figure 1 This paper presents the overall framework diagram of the invention, fully demonstrating the entire process of real-time open vocabulary video object detection based on efficient temporal aggregation and knowledge transfer, as well as the interaction relationships between each module: The input keyframes are processed in two ways. One way is fed into the teacher model in the pseudo-label knowledge distillation (PLKD) mechanism, where pseudo-labels of novel categories are generated through scoring and selection. Simultaneously, combined with knowledge transfer and text prompts, the category labels are input into the text encoder through a text template to extract text features. The other way is fed into the YOLO backbone network to extract multi-scale features and further generate anchor-level features E. Subsequently, the anchor-level features E are input into the anchor-level memory attention mechanism (ALMA). This module calls the past frame features stored in the memory bank, selects effective supporting anchors, and filters duplicate and abnormal anchors through the Diverse Anchor Optimization (DAR) strategy. Then, the memory attention mechanism aggregates temporal information and outputs temporal fusion features. Finally, temporal fusion features The text features output by the PLKD mechanism are fed into the Label Space Expert Hybrid (LS-MoE) module. This module decouples the new category expert from the base category expert and fuses the probability distributions of the two experts through Label Space Aware Gating (LSAG) to obtain the final category distribution. This distribution is then input into the mask header and bounding box header, and finally outputs the segmentation mask and bounding box detection results of the target.

[0032] The real-time open vocabulary video target detection method based on efficient temporal aggregation and knowledge transfer described in this embodiment of the invention specifically includes the following steps:

[0033] A. Given an open vocabulary video object detection dataset (such as LV-VIS) containing several video sequences, each video consisting of consecutive frames; this invention uses a sparse sampling strategy to select keyframes, reducing computational overhead; specifically, keyframe sequences are uniformly sampled from each video for subsequent processing, ensuring that the model can focus on significant change regions in the video.

[0034] B. Input the labels of the video frame sequences obtained in step A into a text encoder (e.g., MobileCLIP-B) using a text template (e.g., "an image containing {category name}") to extract text features; the text template is designed as a natural language description to enhance semantic alignment; simultaneously, a teacher model with strong open-vocabulary awareness (e.g., CLIP-VIS) is used to generate novel category pseudo-labels offline on the video training set, such as... Figure 1 As shown; the teacher model parameters are frozen, and pseudo-labels are generated through a single forward propagation; this process is represented as:

[0035]

[0036] in Indicates a pseudo tag. This represents categories that are not visible during training. τ is the confidence score for predicted category c, and τ is a threshold hyperparameter used to filter low-quality predictions (default value is 0.5). This filter ensures high fidelity for pseudo-labels. If τ is too small, noise will be introduced; if τ is too large, the number of pseudo-labels will be insufficient, affecting the supervision effect. Experiments have shown that τ=0.5 achieves the best balance between the quality and quantity of pseudo-labels. The pseudo-labels and the true labels of the base categories... Integration to form a strengthened oversight system This serves as an additional supervisory signal for the student model; the strategy enhances the model's generalization ability to novel categories through consistency constraints while avoiding the introduction of additional inference overhead; the enhanced supervision set covers bounding boxes, categories, and segmentation masks, providing comprehensive supervision for multi-task learning.

[0037] C. For example Figure 1As shown, the video frames sampled in step A are input into a backbone network (such as YOLOv8 or YOLO11) to extract multi-scale anchor-level feature embeddings. The backbone network is designed based on a real-time open vocabulary detector and includes feature extraction, regression head, and segmentation head. Specifically, the backbone network first extracts spatial features through convolutional layers, and then uses a path aggregation network (PAN) to fuse multi-scale information to generate anchor-level feature embeddings. Where A is the number of anchor points and d is the feature dimension; this process uses pre-trained weight initialization to accelerate convergence and supports end-to-end training; multi-scale features capture information from low-level details (such as edges and textures) to high-level semantics (such as object shape and category), providing a foundation for subsequent temporal modeling; the regression head is responsible for predicting bounding box offsets, the segmentation head generates instance masks, and the object embedding head outputs feature vectors for alignment with text features; the backbone network supports multiple scale variants (such as YOVO-v8-S, M, L) to adapt to different computational resource constraints.

[0038] D. Input the multi-scale features obtained in step C into the anchor-level memory attention module. This module enhances keyframe representation through temporal modeling, specifically as follows: Figure 2 As shown; firstly, a Diverse Anchor Optimization (DAR) strategy is applied, using TopK-NMS to filter unique target representations and reduce background noise; specifically, the anchor-level feature embedding is performed... Perform non-maximum suppression (NMS) and TopK scoring, and select the top K (usually K=100) most representative anchor features:

[0039]

[0040] Where K A. Ensure diverse and non-redundant foreground semantics; the DAR strategy significantly reduces the computational overhead of the ALMA module by balancing positive and negative samples; subsequently, the memory bank stores the optimized features of the keyframes. As a temporal memory, it serves as a reference for subsequent frames; the memory attention mechanism aggregates features from historical frames (assuming N supporting frames, the features are...). This is calculated through cross-attention; the process is represented as:

[0041]

[0042] Among them W Q W K W V These are learnable linear projection parameters. To prevent gradient vanishing, use a scaling factor; Mapping back to the original feature space; this module is integrated end-to-end, requires no post-processing, and significantly improves cross-frame detection stability, making it particularly suitable for handling video-specific challenges such as occlusion and motion blur.

[0043] E. such as Figure 1 As shown, the temporal fusion features output in step D are input into the label space expert fusion strategy; this strategy decouples the base category expert (Expert) base ) and novelty category experts novel The basic category experts consist of frozen parameter copies of the network's classification layers, preserving domain-general knowledge acquired through large-scale pre-training; novel category experts are fine-tuned and updated using video-specific data (such as LV-VIS) to enhance generalization ability to new categories; firstly, Aligning with the pre-extracted text embedding T (obtained by processing the category names through a text encoder), compute the full-class probability distribution for each expert:

[0044]

[0045]

[0046] in, It is the probability distribution of all categories from the basic category experts. This is the probability distribution of all categories from the new category experts; both features and text embeddings are L2 normalized to enhance numerical stability; subsequently, the probability distribution is fused through label space-aware gating to output the final category distribution; this process is represented as:

[0047]

[0048] in, This is the gating factor (default setting is 0.7), which controls the confidence level of each expert in their area of ​​expertise; when the category belongs to the base category (C... b When the category belongs to the novel category (C), the experts in the basic category have higher weights; when the category belongs to the novel category (C), the experts in the basic category have higher weights. n When the novel category is selected, the expert weight is higher; this strategy introduces only minimal parameter overhead (because the parameters are shared before expert decoupling), effectively mitigating the overfitting and catastrophic forgetting problems during the fine-tuning process.

[0049] F. Compare the prediction results obtained in step E with the enhanced supervision set. Calculate the loss function and jointly optimize the model parameters; the loss function includes classification loss. Bounding box regression loss and segmentation loss Jointly optimize model parameters; the loss function is defined as: Specifically, the classification loss uses focal loss to handle class imbalance, the bounding box regression loss uses GIoU (Generalized Intersection over Union) loss to improve localization accuracy, and the segmentation loss generates instance masks based on binary cross-entropy. During training, the AdamW optimizer is used, with an initial learning rate of... The batch size was 32, momentum was 0.9, and weight decay was 0.025. The training cycle was 30 rounds, and the learning rate was scheduled using a cosine annealing strategy. Mosaic enhancement was disabled in the 29th round to accelerate convergence. The model was trained on a single NVIDIA RTX 4090 GPU for about 9 hours, which is 14 times less training cost than OV-VOD. By jointly optimizing the basic and novel classes, the model improves zero-shot generalization ability while maintaining real-time performance.

[0050] Figure 3 , Figure 4 A comparison chart of the prediction results of this invention and the YOLOE method is provided. The prediction results of the YOLOE method are shown at the top, and the prediction results of this invention are shown at the bottom. It can be observed from the prediction results chart that in some motion-induced blurred scenes and abnormal pose scenes, this invention can still accurately detect and segment the target, showing stronger robustness and generalization ability.

[0051] Table 1 shows the comparative experimental results of this invention on the LV-VIS dataset.

[0052] Table 1

[0053]

[0054] As shown in Table 1, YOVO achieves a superior performance-efficiency trade-off across all parameter scales; for example, YOVO-v8-L achieves an AP (Average Performance) in detection tasks. n 18.9% and AP b 17.4%, compared to OV-VOD in AP n It improves performance by 6.1% and increases inference speed by 70 times; at the same time, YOVO-v8-S achieves similar performance with only 1 / 3 of the parameters of OV-VOD, and improves speed by 120 times.

[0055] Table 2 shows the comparative experimental results of this invention on the BURST and TAO datasets.

[0056] Table 2

[0057]

[0058] In a zero-sample setting, YOVO-v8-L achieved AP on Burst.n 10.9%, reaching AP on TAO n The method achieves 11.1% improvement, significantly outperforming existing methods and demonstrating its strong cross-dataset generalization ability. It can be seen that the method of this invention achieves superior performance on three datasets. This method is efficient, lightweight, and suitable for real-time application scenarios.

[0059] Experimental results show that the present invention outperforms existing state-of-the-art methods on multiple public datasets, providing new directions and inspirations for research and application in this field.

[0060] The above embodiments are merely preferred embodiments of the present invention and should not be considered as limiting the scope of the present invention. All equivalent variations and improvements made within the scope of the present invention should still fall within the patent coverage of the present invention.

Claims

1. A real-time open-vocabulary video target detection method based on efficient temporal aggregation and knowledge transfer, characterized in that... Includes the following steps: 1) Keyframe sparse sampling: Given a video dataset, uniformly sample keyframe sequences from each video in the video dataset; 2) Text feature extraction and pseudo-label generation: The labels of the keyframe sequence are input into the text encoder through the text template to extract text features. At the same time, the keyframe sequence is input into the teacher model. The teacher model with frozen parameters is used to generate novel category pseudo-labels offline on the video training set. The pseudo-labels are fused with the real labels of the basic categories to construct an enhanced supervision set. 3) Backbone network feature extraction: Input keyframes into the backbone network, extract and fuse multi-scale features through convolutional layers and path aggregation networks to generate anchor-level feature embeddings; 4) Anchor-level memory attention modeling: various anchor point optimizations are performed on the anchor-level feature embedding, representative anchor point features are selected and stored in the memory bank, and historical frame features are aggregated through the memory attention mechanism to output temporal fusion features; 5) Label Space Expert Hybrid Strategy: Input the temporal fusion features into the decoupled base category expert and novel category expert, and fuse the probability distribution of the two experts through label space perception gating to output the final category distribution; 6) Joint model optimization: Calculate the loss function with the final class distribution and the enhanced supervision set, and update the gradients, including detection, bounding box regression and segmentation losses.

2. The real-time open vocabulary video target detection method based on efficient temporal aggregation and knowledge transfer as described in claim 1, characterized in that... In step 1), the specific steps of the keyframe sparse sampling are as follows: given an open vocabulary video object detection dataset containing several video sequences, each video consisting of consecutive frames; use a sparse sampling strategy to select keyframes to reduce computational overhead; specifically, uniformly sample keyframe sequences from each video for subsequent processing to ensure that the model can focus on significant change regions in the video.

3. The real-time open vocabulary video target detection method based on efficient temporal aggregation and knowledge transfer as described in claim 1, characterized in that... In step 2), the specific steps for text feature extraction and pseudo-label generation are as follows: the labels of the video frame sequence obtained in step 1) are input into the text encoder through a text template to extract text features; the text template is designed as a natural language description to enhance semantic alignment; simultaneously, a teacher model with strong open vocabulary awareness is used to generate novel category pseudo-labels offline on the video training set; the teacher model parameters are frozen, and pseudo-labels are generated through a single forward propagation; this process is represented as: in, Indicates a pseudo tag. This represents categories that are not visible during training. τ is the confidence score for predicting category c, and τ is a threshold hyperparameter used to filter low-quality predictions and ensure high fidelity of pseudo-labels.

4. The real-time open vocabulary video target detection method based on efficient temporal aggregation and knowledge transfer as described in claim 1, characterized in that... In step 3), the specific steps for backbone network feature extraction are as follows: the sampled video frames from step 1) are input into the backbone network to extract multi-scale anchor-level feature embeddings; the backbone network is designed based on a real-time open vocabulary detector and includes feature extraction, regression head, and segmentation head; specifically, the backbone network first extracts spatial features through convolutional layers, and then uses a path aggregation network to fuse multi-scale information to generate anchor-level feature embeddings. , where A is the number of anchor points and d is the feature dimension; This process uses pre-trained weight initialization to accelerate convergence and supports end-to-end training.

5. The real-time open vocabulary video target detection method based on efficient temporal aggregation and knowledge transfer as described in claim 1, characterized in that... In step 4), the specific steps of the anchor-level memory attention modeling are as follows: the multi-scale features obtained in step 3) are input into the anchor-level memory attention module; this module enhances keyframe representation through temporal modeling. First, it applies various anchor optimization strategies to reduce background noise; specifically, it embeds anchor-level features. Non-maximum suppression and TopK scoring are performed, and the top K most representative anchor features are selected: ;where K A; Subsequently, the memory bank stores optimized features of keyframes. As a form of chronological memory; The memory attention mechanism aggregates historical frame features. Assuming there are N supporting frames, the features are... This is calculated through cross-attention; the process is expressed as: 。 6. The real-time open vocabulary video target detection method based on efficient temporal aggregation and knowledge transfer as described in claim 1, characterized in that... In step 5), the specific steps of the label space expert fusion strategy are as follows: input the temporal fusion features output in step 4) into the label space expert fusion strategy; this strategy decouples the base category expert. base and novel category expert novel The basic category experts consist of frozen parameter copies of the network classification layer; novel category experts are fine-tuned and updated using video-specific data; firstly, Align with the pre-extracted text embedding T, and compute the full-class probability distribution for each expert: It is the probability distribution of all categories from the basic category experts. This is the probability distribution of all categories from the new category experts; subsequently, the probability distribution is fused through label space-aware gating, and the final category distribution is output; this process is represented as: in This is the gating factor (default setting is 0.7), which controls the confidence level of each expert in their area of ​​expertise; when the category belongs to the basic category C... b At that time, experts in the basic category have higher weight; When the category belongs to the novel category C n At that time, experts in novel categories have higher weighting.

7. The real-time open vocabulary video target detection method based on efficient temporal aggregation and knowledge transfer as described in claim 1, characterized in that... In step 6), the specific steps of the joint optimization of the model are as follows: the prediction results obtained in step 5) are combined with the enhanced supervision set. Calculate the loss function and jointly optimize the model parameters; the loss function includes classification loss. Bounding box regression loss and segmentation loss Jointly optimize model parameters; the loss function is defined as: . in, The classification loss uses focus loss to handle class imbalance, the bounding box regression loss uses GIoU loss to improve localization accuracy, and the segmentation loss is based on binary cross-entropy to generate instance masks; during training, the AdamW optimizer is used.

Images