Underwater passive cross-domain target detection method based on visual and language model guidance
By introducing a visual and language-enhanced pseudo-label generation module and a graph contrast learning module into the passive cross-domain target detection model, the problems of insufficient pseudo-label quality and source domain bias are solved, thereby improving the accuracy of underwater target detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-25
- Publication Date
- 2026-07-14
Smart Images
Figure CN122391788A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of underwater robot visual perception and target detection technology, and in particular to an underwater passive cross-domain target detection method guided by visual and language models. Background Technology
[0002] With the increasing demand for marine resource development and marine environmental monitoring, underwater target detection technology has significant application value in underwater robot navigation, underwater target recognition, marine ecological monitoring, and marine safety assurance. In recent years, with the development of deep learning technology, target detection methods based on convolutional neural networks or Transformer architectures have made significant progress in underwater target detection tasks.
[0003] In practical applications of underwater target detection, significant differences in data distribution often exist between different aquatic environments due to factors such as water scattering, light attenuation, and variations in imaging equipment. When the training data (source domain) and test data (target domain) exhibit distributional differences, model performance deteriorates significantly. While it's possible to collect and label target domain images to obtain a target domain dataset and retrain a model biased by the target domain data distribution, labeling data is extremely labor-intensive and time-consuming. To address this, researchers have proposed an unsupervised cross-domain target detection model. Unsupervised means that the target domain data lacks supervised labels; this model only requires labeled source domain images and unlabeled target domain images for training.
[0004] However, due to privacy and security principles, as well as practical factors such as storage and sharing limitations, training data from the source domain is often unavailable, making model training even more difficult. To address these issues, researchers have developed unsupervised passive cross-domain object detection (or passive cross-domain object detection) based on unsupervised cross-domain object detection models. "Passive" means that source domain data cannot be used; only pre-trained models trained on source domain data can be used. Cross-domain object detection refers to using a pre-trained model from the source domain and unlabeled image data from the target domain, supplemented by domain adaptation methods, for subsequent unsupervised model training.
[0005] Existing passive cross-domain object detection models cannot access any labeled source domain data. Therefore, pseudo-label self-training based on a teacher-student framework has become almost the only feasible technical approach. This approach consists of a teacher model and a student model, both pre-trained models initialized with source domain data. The teacher model receives a target domain image enhanced by weak image enhancement techniques such as random horizontal flipping, performs detection on this image, and uses the detection result as a pseudo-label to guide the student model's training. The student model receives a target domain image enhanced by strong image enhancement techniques such as color dithering, grayscale transformation, and Gaussian blur, and obtains a prediction result for this image. Subsequently, the student model calculates the detection loss by combining the pseudo-label supervision signal provided by the teacher model and its own prediction results, calculates the gradient of each layer's parameters using the backpropagation algorithm, and finally updates the model's parameters through an optimizer. The teacher model is updated using the exponential moving average (EMA) of the student model's parameters. This encourages the teacher model to continuously generate higher-quality pseudo-labels to guide the student model's training, thereby improving the detection performance of both the student and teacher models. The exponential moving average can be represented by formula (1):
[0006] (1)
[0007] in, Represents the smoothing coefficient. and Let T represent the parameters of the teacher and student models, respectively, and T represent the iteration step.
[0008] Since the research on this passive cross-domain target detection method is still in its early stages, it has the following shortcomings:
[0009] First, the pseudo-labels generated by existing methods are only produced by the teacher model, and the initialization parameters of the teacher model are obtained by pre-training with source domain data. It naturally carries the bias of the source domain data distribution. Therefore, it is difficult to guarantee the quality of pseudo-labels generated by the teacher model alone.
[0010] Secondly, existing methods update the teacher model parameters synchronously using an exponential moving average after each update of the student model parameters. When the student model is affected by low-quality pseudo-labels generated by the teacher model and makes incorrect updates, the update error will accumulate synchronously in the teacher model, causing the training to deteriorate or even collapse.
[0011] Third, this pseudo-label self-training method only focuses on the supervision information at the target instance level in the image, using the instance features such as the category and bounding box of a single target to construct supervision constraints, while ignoring the high-level structural information such as the inherent spatial position association, relative layout relationship and spatial interaction mode between the same or different targets in the same image.
[0012] Therefore, how to improve the cross-domain detection performance of underwater targets, at least by improving the quality of pseudo-labels in current passive cross-domain target detection and alleviating source domain bias, has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0013] This invention provides an underwater passive cross-domain target detection method based on vision and language models, which at least solves the problem of low underwater target cross-domain detection performance caused by the limited and biased sources of pseudo-labels in related technologies.
[0014] As one aspect of the present invention, a method for underwater passive cross-domain target detection based on vision and language models is provided, comprising:
[0015] A source domain dataset is constructed based on publicly available and labeled datasets for underwater target detection, and a target domain dataset is constructed based on unlabeled data from actual underwater application scenarios. The target domain dataset includes at least the data of underwater targets to be detected.
[0016] Construct a baseline model based on the Transformer architecture, and train the baseline model to obtain pre-trained weights based on the source domain dataset;
[0017] The teacher and student models are constructed based on the baseline model containing pre-trained weights to obtain the teacher-student model framework.
[0018] Under the framework of the teacher-student model, a pseudo-label generation module with visual and language enhancement is constructed to obtain an initial model for underwater passive cross-domain target detection guided by visual and language models. The pseudo-label generation module with visual and language enhancement is used to perform semantic completion and localization correction on the pseudo-labels generated by the teacher model based on the target domain dataset according to the cross-modal semantic prior of the visual and language model.
[0019] The initial underwater passive cross-domain target detection model guided by vision and language models is iteratively trained based on the target domain dataset to obtain an underwater passive cross-domain target detection model guided by vision and language models. The underwater passive cross-domain target detection model guided by vision and language models can detect underwater target data and obtain underwater target data detection results.
[0020] Furthermore, within the aforementioned teacher-student model framework, a visual and language-enhanced pseudo-label generation module is constructed, including:
[0021] Initial pseudo-labels are generated based on the teacher model;
[0022] Generate semantically enhanced pseudo-labels based on a visual and language teacher model;
[0023] The initial pseudo-labels and the semantically enhanced pseudo-labels are fused to obtain a visual and linguistic enhancement pseudo-label module for supervising student model training.
[0024] Further, generating initial pseudo-labels based on the teacher model includes:
[0025] The target domain dataset is input into the teacher model to obtain candidate box prediction results;
[0026] For each candidate box prediction result, the maximum class confidence score and the class corresponding to that maximum class confidence score are selected.
[0027] Based on the category adaptive thresholding strategy of Gaussian mixture model, a set of historical predictions is collected for each target category in the target domain dataset;
[0028] The historical prediction set for each target category is modeled using a two-component Gaussian mixture model to obtain the fitting results of the two-component Gaussian mixture model.
[0029] Based on the fitting results of the two-component Gaussian mixture model, the category threshold is estimated from the high-confidence component, and the initial pseudo-label set of the teacher model after threshold screening is obtained.
[0030] Furthermore, semantically enhanced pseudo-labels are generated based on the visual and language teacher model, including:
[0031] For each target category in the target domain dataset, a corresponding text prompt word is constructed, and a target category text prompt word set is obtained;
[0032] The target domain dataset and the target category text prompt word set are input together into the visual and language teacher model to obtain the visual and language candidate set;
[0033] Based on target category matching and confidence threshold, the visual and language candidate sets are filtered to obtain semantically enhanced pseudo-labels.
[0034] Further, a pseudo-label module for visual and language enhancement used to supervise student model training is obtained by fusing the initial pseudo-labels with the semantically enhanced pseudo-labels, including:
[0035] The candidate set is obtained by merging the pseudo-label sets of the initial pseudo-label and the semantically enhanced pseudo-label.
[0036] The confidence scores and category information in the candidate set are fused to obtain fused confidence scores and fused category information;
[0037] Based on the candidate set, the fused confidence level, and the fused category information, a pseudo-label module for visual and language enhancement used to supervise student model training is obtained.
[0038] Furthermore, the underwater passive cross-domain target detection method guided by vision and language models also includes:
[0039] A weight adaptive update module is constructed within the teacher-student model framework. This module is used to adaptively select the parameter update strategy for the teacher model.
[0040] Furthermore, a weight adaptive update module is constructed within the aforementioned teacher-student model framework, including:
[0041] The target domain dataset is input into the student model to obtain intermediate prediction results, and an entropy-based uncertainty is constructed based on the intermediate prediction results.
[0042] The initial uncertainty of the samples in the target domain dataset obtained in the teacher model is stored in a buffer queue, wherein the number of uncertainties in the buffer queue is not greater than a preset number threshold.
[0043] The initial uncertainty in the buffer queue is evaluated based on the parameters after the student model is trained to obtain the updated uncertainty;
[0044] The parameters of the teacher model are adaptively updated based on the difference between the updated uncertainty and the initial uncertainty.
[0045] Furthermore, the underwater passive cross-domain target detection method guided by vision and language models also includes:
[0046] A graph comparison learning module is constructed within the framework of the teacher-student model. The graph comparison learning module is used to construct the instance space structural relationship between the teacher model and the student model.
[0047] Furthermore, a graph contrastive learning module is constructed within the aforementioned teacher-student model framework, including:
[0048] Instance relationship modeling and graph convolution modeling are performed based on the instance features of the teacher model and student model respectively to obtain the graph structure of the teacher model and the graph structure of the student model.
[0049] Construct contrastive learning losses based on the graph structure of the teacher model and the graph structure of the student model, respectively.
[0050] Distillation constraints are introduced into the graph structures of both the teacher model and the student model to obtain a graph comparison learning module.
[0051] Further, the initial underwater passive cross-domain target detection model guided by vision and language models is iteratively trained based on the target domain dataset, including:
[0052] The target domain dataset is subjected to a first enhancement preprocessing and a second enhancement preprocessing respectively to obtain a first enhancement preprocessed dataset and a second enhancement preprocessed dataset, wherein the enhancement magnitude of the first enhancement preprocessing is smaller than the enhancement magnitude of the second enhancement preprocessing.
[0053] The first enhanced preprocessed dataset is input into the teacher model and the pseudo-label generation module respectively to obtain the pseudo-label set of the teacher model, and the second enhanced preprocessed dataset is input into the student model to obtain the prediction result of the student model;
[0054] The training iteration loss is calculated based on the supervision signal of the pseudo-label set of the teacher model to predict the results of the student model.
[0055] The initial underwater passive cross-domain target detection model guided by the vision and language models is iteratively trained based on the training iteration loss.
[0056] The present invention provides an underwater passive cross-domain target detection method guided by vision and language models. This method selects a suitable publicly available and labeled dataset for underwater target detection as the source domain dataset, collects unlabeled data from actual underwater application scenarios to form the target domain dataset, and uses this to build a cross-domain experimental scenario. A baseline model based on the Transformer architecture is constructed, and the baseline model is trained using the source domain dataset to obtain pre-trained weights. A teacher model and a student model are then constructed based on the baseline model containing the pre-trained weights to obtain a teacher-student model framework. A pseudo-label generation module with visual and language enhancements is built on the teacher and student models to obtain an underwater passive cross-domain target detection model guided by vision and language models. Finally, the initial underwater passive cross-domain target detection model guided by vision and language models is iteratively trained using the target domain dataset to obtain the final underwater passive cross-domain target detection model guided by vision and language models. This underwater passive cross-domain target detection method, guided by visual and language models, introduces a visual and language-enhanced pseudo-label generation module into the initial model of underwater passive cross-domain target detection. This visual and language-enhanced pseudo-label generation module can generate higher-quality pseudo-labels. As the visual base model contains rich, unbiased semantic features, it can effectively alleviate the source domain bias problem of pseudo-labels in the teacher model, thereby improving the accuracy of underwater passive cross-domain target detection. Attached Figure Description
[0057] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the following detailed description to explain the invention, but do not constitute a limitation thereof.
[0058] Figure 1 The flowchart shows the underwater passive cross-domain target detection method based on vision and language models provided by this invention.
[0059] Figure 2 The flowchart for constructing a pseudo-tag generation module with visual and language enhancement provided by the present invention is shown.
[0060] Figure 3 The flowchart for generating the initial pseudo-label provided by this invention.
[0061] Figure 4 The flowchart for generating semantically enhanced pseudo-tags provided by this invention.
[0062] Figure 5 This is a flowchart illustrating the fusion of initial pseudo-tags and semantically enhanced pseudo-tags provided by the present invention.
[0063] Figure 6 The flowchart shows the workflow of the weight adaptive update module provided by this invention.
[0064] Figure 7 This is a flowchart illustrating the iterative training of an initial model for underwater passive cross-domain target detection guided by vision and language models, as provided in this invention.
[0065] Figure 8 This is a schematic diagram of the architecture of the underwater passive cross-domain target detection model based on vision and language models provided by the present invention. Detailed Implementation
[0066] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0067] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0068] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of the invention described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0069] This embodiment provides an underwater passive cross-domain target detection method guided by vision and language models. Figure 1 This is a flowchart of an underwater passive cross-domain target detection method based on vision and language models according to an embodiment of the present invention, such as... Figure 1 As shown, it includes:
[0070] S100. Construct a source domain dataset based on publicly available and labeled datasets for underwater target detection, and construct a target domain dataset based on unlabeled data from actual underwater application scenarios. The target domain dataset includes at least the data of underwater targets to be detected.
[0071] In this embodiment of the invention, a suitable publicly available and labeled dataset for underwater target detection is selected as the source domain dataset, and data (unlabeled) from actual underwater application scenarios are collected to form the target domain dataset, and a cross-domain experimental scenario is built based on this.
[0072] Specifically, a labeled dataset with the same target detection category but a different data distribution than that of actual underwater application scenarios is selected as the source domain dataset, and images from actual underwater application scenarios are collected to form the target domain dataset. , Let s and t represent the source domain dataset and the target domain dataset, respectively, where s and t represent the source domain and the target domain, respectively. This represents the image data of the i-th sample in the source domain. This represents the image data of the j-th sample in the target domain. This represents the bounding box and category information labeled in the i-th sample of the source domain. , representing the center coordinates, width, and height of the bounding box, respectively. Indicates the corresponding category label, , These represent the total number of samples in the source domain dataset and the target domain dataset, respectively. Represents four-dimensional real space, They are all real numbers, together forming a four-dimensional vector.
[0073] The source domain dataset constructed in the above manner and target domain dataset Significant differences exist in data distribution (such as imaging environment, underwater lighting, noise interference, etc.), thus providing real and challenging cross-domain application scenarios for subsequent cross-domain migration methods.
[0074] S200. Construct a baseline model based on the Transformer architecture, and train the baseline model to obtain pre-trained weights based on the source domain dataset;
[0075] In this embodiment of the invention, a baseline model DeformabelDETR based on the Transformer architecture can be built, and the pre-trained weights can be obtained by training the baseline model using the source domain dataset selected in step S100.
[0076] Specifically, the Deformable DETR model consists of three parts: a backbone network, an encoder, and a decoder. First, the input batch X is processed by the ResNet-50 backbone network to extract a multi-scale convolutional feature set. Then, the features at each scale are flattened and stitched together in the spatial dimension, and the positional encoding and scale encoding are superimposed to obtain the serialized token features. ,in, Let F represent a three-dimensional real tensor, where B represents the batch size, N represents the number of tokens, and C represents the channel dimension. The encoder globally models F through multiple layers of self-attention modules and outputs pixel semantic memory Z. The decoder uses a set of fixed or learnable queries. As input, multi-scale information is adaptively sampled from Z through deformable cross-attention and target semantics are aggregated. Finally, a prediction set is generated by a feedforward network. During training, Hungarian matching is used to achieve a one-to-one assignment between predictions and true values, thereby determining positive and negative samples and calculating the corresponding classification and regression losses.
[0077] This invention employs a Deformable DETR model based on the Transformer architecture as the baseline model. This baseline model, by introducing a deformable attention mechanism, effectively alleviates the slow convergence speed and high computational complexity of the original DETR, while also enabling more flexible modeling of feature differences across different scene domains, thereby improving the model's detection performance in the target domain. Furthermore, compared to existing convolutional neural networks such as Faster R-CNN and YOLO, the baseline model based on Detection Transformer (DETR) in this invention introduces a self-attention mechanism, modeling the target detection task as an ensemble prediction problem. This allows for end-to-end target detection without the need for pre-defined anchor boxes and non-maximum suppression (NMS).
[0078] S300. Construct teacher and student models based on the baseline model containing pre-trained weights to obtain the teacher-student model framework.
[0079] In this embodiment of the invention, two copies of the baseline model containing the same pre-trained weights are made and set as the teacher model and the student model, respectively.
[0080] Specifically, after completing the structure of the Deformable DETR model, the baseline model is trained in a supervised manner using the constructed source domain dataset to obtain pre-trained weights with good detection capabilities. During the training process, the baseline model uses a target detection loss function based on set matching to calculate the loss of classification and regression between the predicted results and the ground truth labels. Its total detection loss can be expressed by formula (2):
[0081] (2)
[0082] in, This represents the optimal matching relationship between the prediction results obtained by the Hungarian matching algorithm and the actual annotations. Represents classification loss. This represents the bounding box regression loss, which is further derived from... The loss and the GIOU loss together constitute the total loss.
[0083] By iteratively optimizing the baseline model parameters using the aforementioned loss function, a Deformable DETR pre-trained model that converges on the source domain dataset is obtained. This pre-trained weight can provide an initialization foundation for subsequent cross-domain knowledge transfer and the construction of teacher and student models in the teacher-student framework.
[0084] S400. Under the framework of the teacher-student model, construct a pseudo-label generation module with visual and language enhancement to obtain an initial model for underwater passive cross-domain target detection guided by the visual and language model. The pseudo-label generation module with visual and language enhancement is used to perform semantic completion and localization correction on the pseudo-labels generated by the teacher model based on the target domain dataset according to the cross-modal semantic prior of the visual and language model.
[0085] It should be understood that in passive cross-domain object detection methods based on a teacher-student framework, the quality of the pseudo-labels generated by the teacher model directly determines whether the model optimization direction is correct. Because there are significant distribution differences between the source and target domain data, and the teacher model's initial parameters are trained using source domain data, it naturally carries the bias of the source domain data distribution. Therefore, the pseudo-labels it detects often involve a large number of false positives and false negatives. If iterative training and updates rely solely on the pseudo-labels generated by the teacher model, the aforementioned detection errors will be continuously amplified during the self-training process, leading to model error accumulation and performance degradation.
[0086] In this embodiment of the invention, to avoid the problem of inconsistent pseudo-label quality caused by pseudo-labels generated solely by the teacher model, this embodiment of the invention constructs a pseudo-label generation module with visual and language enhancement. This module introduces cross-modal semantic priors from visual and language models to perform semantic completion and localization correction on pseudo-labels, thereby improving the quality and reliability of pseudo-labels.
[0087] S500. The initial underwater passive cross-domain target detection model guided by vision and language models is iteratively trained according to the target domain dataset to obtain an underwater passive cross-domain target detection model guided by vision and language models. The underwater passive cross-domain target detection model guided by vision and language models can detect underwater target data to be detected and obtain underwater target data detection results.
[0088] In this embodiment of the invention, after collecting source and target domain datasets, constructing cross-domain scenarios, pre-training weights obtained from source domain training, building a teacher-student framework, and constructing a pseudo-label generation module with visual and language enhancement, an initial underwater passive cross-domain target detection model guided by a visual and language model is obtained. This initial model undergoes iterative training, continuously iterating and upgrading through multiple rounds of training to generate better cross-domain detection performance, ultimately resulting in a visual and language model-guided underwater passive cross-domain target detection model. This model can detect underwater target data and thus obtain underwater target data detection results.
[0089] Therefore, the underwater passive cross-domain target detection method based on vision and language models provided by this invention selects a suitable publicly available and labeled dataset for underwater target detection as the source domain dataset, collects data (unlabeled) from actual underwater application scenarios to form the target domain dataset, and uses this to build a cross-domain experimental scenario. A baseline model based on the Transformer architecture is constructed, and the baseline model is trained on the source domain dataset to obtain pre-training weights. A teacher model and a student model are constructed based on the baseline model containing the pre-training weights to obtain a teacher-student model framework. A pseudo-label generation module with vision and language enhancement is built on the teacher model and the student model to obtain an underwater passive cross-domain target detection model based on vision and language models. Finally, the initial underwater passive cross-domain target detection model based on vision and language models is iteratively trained on the target domain dataset to obtain the underwater passive cross-domain target detection model based on vision and language models. This underwater passive cross-domain target detection method, guided by visual and language models, introduces a visual and language-enhanced pseudo-label generation module into the initial model of underwater passive cross-domain target detection. This visual and language-enhanced pseudo-label generation module can generate higher-quality pseudo-labels. As the visual base model contains rich, unbiased semantic features, it can effectively alleviate the source domain bias problem of pseudo-labels in the teacher model, thereby improving the accuracy of underwater passive cross-domain target detection.
[0090] In embodiments of the present invention, such as Figure 2 As shown, a visual and language-enhanced pseudo-label generation module is constructed within the teacher-student model framework, including:
[0091] S410. Generate initial pseudo-labels based on the teacher model;
[0092] Specifically, such as Figure 3 As shown, the initial pseudo-labels are generated based on the teacher model, including:
[0093] S411. Input the target domain dataset into the teacher model to obtain candidate box prediction results;
[0094] In this embodiment of the invention, the target domain unlabeled samples The teacher model is input to obtain candidate box prediction results, wherein the expression for the candidate box prediction results is:
[0095] (3)
[0096] in, This represents the candidate bounding box prediction results for the target domain dataset. This represents the predicted bounding box for the i-th sample. This represents the class probability vector of the i-th sample. C represents the total number of samples in the target domain dataset, and C represents the number of categories.
[0097] S412. For each candidate box prediction result, select the maximum class confidence and the class corresponding to the maximum class confidence.
[0098] In this embodiment of the invention, the maximum category confidence score is selected for each candidate box. and its categories The selection method can be found in formula (4):
[0099] (4)
[0100] S413. Collect the historical prediction set for each target category in the target domain dataset according to the category adaptive threshold strategy of Gaussian mixture model;
[0101] It should be noted that traditional teacher models filter pseudo-labels based on a fixed confidence threshold. To avoid filtering bias caused by class imbalance and cross-domain shifts due to the fixed threshold, this invention adopts a class-adaptive threshold strategy based on a Gaussian Mixture Model (GMM). This strategy collects the historical prediction set for each class c. .
[0102] S414. Model the historical prediction set of each target category based on the two-component Gaussian mixture model to obtain the fitting result of the two-component Gaussian mixture model.
[0103] Specifically, the modeling process based on the two-component Gaussian mixture model can be represented by formula (5):
[0104] ,
[0105] in, Let represent the weight coefficient of the k-th Gaussian component of category c, used to characterize the proportion of this component in the confidence distribution of this category, and satisfying that the sum of the coefficients of all components is 1. This represents the Gaussian distribution function.
[0106] S415. Based on the fitting results of the two-component Gaussian mixture model, estimate the category threshold from the high-confidence component to obtain the initial pseudo-label set of the teacher model after threshold screening.
[0107] In this embodiment of the invention, the class threshold is estimated from the high-confidence component based on the fitting result of the two-component Gaussian mixture model. Thus, the initial pseudo-label set of the teacher model after threshold screening is obtained. This strategy can dynamically adjust the screening criteria of different categories during training, thereby obtaining a more stable initial pseudo-label set in the target domain. This set can be represented by formula (6):
[0108] (6)
[0109] in, This indicates the category index of the prediction box.
[0110] S420. Generate semantically enhanced pseudo-labels based on the visual and language teacher model;
[0111] It should be understood that, in order to obtain higher quality pseudo-labels and stronger cross-domain semantic constraints, this embodiment of the invention introduces a general vision-language model, Grounding DINO, trained on large-scale data, and denoted as... Furthermore, it is built into the teacher-student framework in parallel as an additional visual and language teacher model, which introduces category priors into the pseudo-label generation process through displayed text prompts.
[0112] Specifically, such as Figure 4 As shown, semantically enhanced pseudo-labels are generated based on the visual and language teacher model, including:
[0113] S421. For each target category in the target domain dataset, construct corresponding text prompt words and obtain a set of target category text prompt words;
[0114] In this embodiment of the invention, a text prompt word is constructed for each category c. And form a set of prompt words. ,in, A natural language description representing the c-th type of target.
[0115] S422. Input the target domain dataset and the target category text prompt word set into the visual and language teacher model to obtain the visual and language candidate set;
[0116] In this embodiment of the invention, the target domain image is... With text prompt word set By inputting the visual and language teacher models together, a visual-language candidate set is obtained, which can be represented by formula (7):
[0117] (7)
[0118] in, This represents the predicted bounding box for the j-th query vector. This represents the cross-modal matching response between the prediction box and the text prompt, where M represents the number of predictions.
[0119] Furthermore, in order to align the visual-language output to the target category space, The corresponding phrase will be parsed and further mapped to obtain the category index. If the phrase cannot match any target category, it will be recorded as an invalid label and the prediction box will be removed. The category index can be represented by formula (8):
[0120] (8)
[0121] Meanwhile, the confidence score of the visual and language teacher model is defined as the maximum activation of the cross-modal response, which can be expressed by formula (9):
[0122] (9)
[0123] in, This represents the normalized activation function.
[0124] S423. Based on target category matching and confidence threshold, the visual and language candidate sets are filtered to obtain semantically enhanced pseudo-labels.
[0125] In this embodiment of the invention, by combining target category matching and confidence threshold, a set of semantically enhanced pseudo-labels based on the visual and language teacher model is obtained. This set can be represented by formula (10):
[0126] (10)
[0127] in, Represents the legal bounding box space (such as coordinate range and minimum width and height constraints). This indicates that the target category can be correctly matched. This represents the confidence thresholds for visual and linguistic responses.
[0128] S430. The initial pseudo-labels and the semantically enhanced pseudo-labels are fused to obtain a visual and language enhancement pseudo-label module for supervising student model training.
[0129] In this embodiment of the invention, the pseudo-label set predicted by the teacher model... The set of pseudo-labels predicted by the visual and language teacher model They are complementary in terms of semantic feature information. The former usually has a more stable classification output, but may experience missed detections and localization errors in cross-domain scenarios; the latter relies on cross-modal semantic matching, which can often fill in missed detections and provide more reasonable candidate box positions. In order to take advantage of both at the same time, this embodiment of the invention adopts a weighted box fusion method to perform weighted fusion of the candidate boxes predicted by the two at the spatial location level, generating a final set of pseudo-labels with visual and language enhancement. It is used to supervise the training of student models.
[0130] Specifically, a visual and language enhancement pseudo-label module for supervising student model training is obtained by fusing the initial pseudo-labels with the semantically enhanced pseudo-labels, such as... Figure 5 As shown, it includes:
[0131] S431. Merge the pseudo-label sets of the initial pseudo-label and the semantically enhanced pseudo-label to obtain a candidate set;
[0132] In this embodiment of the invention, the pseudo-label sets generated by the two are merged into a candidate set. For any set of candidate boxes with high spatial overlap in the set... If its If the bounding boxes are not in the same fusion group, they are divided into the same fusion group. The bounding boxes within the fusion group are weighted and averaged according to their confidence levels to obtain the fused bounding boxes, which can be represented by formula (11):
[0133] (11)
[0134] S432. The confidence scores and category information in the candidate set are fused to obtain the fused confidence scores and fused category information;
[0135] In this embodiment of the invention, the confidence and category information of the fused bounding box are obtained based on the confidence and category information of the first two candidate boxes. The specific definition can be expressed by formula (12):
[0136] (12)
[0137] S433. Obtain a pseudo-label module for visual and language enhancement used to supervise student model training based on the candidate set, the fused confidence level, and the fused category information.
[0138] Finally, the set of pseudo-labels for visual-language augmentation used to supervise student model training is obtained, which can be represented by formula (13):
[0139] (13)
[0140] Therefore, the three steps in the pseudo-tag generation module that integrates visual and language enhancement in this embodiment of the invention can effectively avoid the phenomena of missed detection, false detection and positioning deviation caused by a single pseudo-tag source, thereby effectively solving the problem of low pseudo-tag generation quality and achieving the goal of improving the accuracy of underwater passive cross-domain target detection.
[0141] In this embodiment of the invention, incorporating a visual and language teacher model can effectively improve the quality of pseudo-label generation. To avoid the inevitable issues of missed detections, false detections, and inaccurate localization in scenarios with large domain offsets, and to prevent the training errors generated by these low-quality pseudo-labels from continuously propagating back to the teacher model through the EMA in the traditional teacher-student framework, leading to error accumulation and ultimately model performance degradation or even collapse, this embodiment of the invention further includes:
[0142] A weight adaptive update module is constructed within the teacher-student model framework. This module is used to adaptively select the parameter update strategy for the teacher model.
[0143] Specifically, a weight adaptive update module is constructed within the aforementioned teacher-student model framework, combined with... Figure 6 As shown, it includes:
[0144] (1) Input the target domain dataset into the student model to obtain intermediate prediction results, and construct an entropy-based uncertainty based on the intermediate prediction results;
[0145] In passive cross-domain target detection, the uncertainty of detection is reflected in both category classification and bounding box regression. Therefore, embodiments of this invention design a prediction uncertainty index that simultaneously covers classification and localization.
[0146] First, the target domain image Inputting the student model yields intermediate prediction results, which are outputs for the classification branches. With regression branch output Where 6 represents the output of the 6th layer in the decoder, N represents the number of queries / candidates, C represents the number of categories, and D represents the regression dimension.
[0147] Secondly, the variance of the prediction results of different decoding layers of the input sample is calculated, and the global mean is taken as the uncertainty score. This score can effectively measure the volatility of the model prediction results. The uncertainty scores of classification and regression can be expressed by formulas (14) and (15):
[0148] (14)
[0149] (15)
[0150] in, This indicates that the variance of the candidate features is calculated.
[0151] Finally, the uncertainty scores of the classification and regression models are combined to obtain the final uncertainty score of the model, which can be expressed by formula (16):
[0152] (16)
[0153] The variance can be regarded as a proxy indicator related to entropy. When the outputs of different layers of the model are more consistent and more concentrated, the variance is smaller, which represents a more certain prediction. The smaller the uncertainty score, the larger the variance.
[0154] (2) Store the initial uncertainty of the samples in the target domain dataset obtained in the teacher model into a buffer queue, wherein the number of uncertainties in the buffer queue is not greater than a preset number threshold;
[0155] In this embodiment of the invention, the strategy maintains a short-term buffer of no more than 5 in length during model training to store the uncertainty scores of the most recent m≤5 samples and dynamically adjust the timing of teacher model parameter updates.
[0156] The i-th cached sample Input the teacher model, put it into a short-term buffer queue, and record the uncertainty at the time of its enqueue. This uncertainty can be expressed by formula (17):
[0157] (17)
[0158] in, This indicates the iteration step at which it joins the queue.
[0159] (3) Evaluate the initial uncertainty in the buffer queue based on the parameters after training the student model to obtain the updated uncertainty;
[0160] In this embodiment of the invention, after the student model has undergone several training steps, the cached sample is re-evaluated under its current parameters to obtain the uncertainty of the sample center. This new uncertainty can be expressed by formula (18):
[0161] (18)
[0162] (4) The parameters of the teacher model are adaptively updated based on the difference between the updated uncertainty and the initial uncertainty.
[0163] In this invention, the change in uncertainty of the sample is calculated based on the new and old uncertainties. If the student model is training and iterating towards a more stable prediction direction, then a large number of samples should satisfy [the desired outcome]. Therefore, this invention uses the majority criterion as the update trigger condition for the teacher model, which can be expressed by formula (19):
[0164] (19)
[0165] in, This is an indicator function, used to determine... If true, output 1; otherwise, output 0. A value of 0.5 indicates that the trigger condition is met only if the uncertainty of at least half of the cached samples decreases.
[0166] At the same time, the weight adaptive update strategy updates the gating variable according to the triggering condition. If the formula (19) is satisfied, the gate variable is 1; otherwise, it is 0.
[0167] Finally, based on the gated variable adaptive update teacher model, the fixed EMA update strategy is changed to a gated adaptive EMA update strategy, which can be expressed by formula (20):
[0168] (20)
[0169] It should be noted that simply freezing the teacher model parameter updates cannot fundamentally solve the problem of the student model continuously shifting due to noise interference. In order to prevent the student from failing to meet formula (19) to enter the training stable region for a long time, this invention introduces a re-initialization strategy. The specific implementation details of this strategy are as follows:
[0170] First, the student model parameters are divided into modules. ,in This indicates the modules that need to be retained, which include key representation modules such as backbone networks, encoders, and relational graph networks. For modules that need to be rolled back and reset.
[0171] Secondly, the present invention records the student model's stable state parameters initially recorded by the weight adaptive update module as follows: Then, partial reinitialization can be represented by formula (21):
[0172] ,(twenty one)
[0173] Finally, this strategy can improve training recovery capability by resetting modules that are more easily perturbed by noisy pseudo-labels (such as classification heads and regression heads) while preserving the learned robust feature representations of the target domain.
[0174] In this embodiment of the invention, to avoid the target detection model ignoring high-level structural information such as spatial location associations and relative layout relationships between the same or different targets, the underwater passive cross-domain target detection method guided by vision and language models further includes:
[0175] A graph comparison learning module is constructed within the framework of the teacher-student model. The graph comparison learning module is used to construct the instance space structural relationship between the teacher model and the student model.
[0176] Specifically, a graph contrast learning module is constructed within the aforementioned teacher-student model framework, including:
[0177] (1) Perform instance relationship modeling and graph convolution modeling based on the instance features of the teacher model and student model respectively to obtain the graph structure of the teacher model and the graph structure of the student model;
[0178] First, let the instance features output by the teacher model or student model at a certain decoding layer be... Where N represents the number of instances and d represents the feature dimension.
[0179] Secondly, in order to model the relationships between instance features, query features for that instance feature are generated through learnable linear transformations. Bond features .in, and This is the learnable parameter matrix for the corresponding feature.
[0180] Furthermore, the similarity of instance keys is defined based on the inner product of the query and the key. To obtain a stable graph structure, the similarity matrix is normalized and squared to construct a weighted adjacency matrix, which can be represented by formula (22):
[0181] ,(twenty two)
[0182] in, This represents the weight of the relationship between instance i and instance j.
[0183] Finally, a three-layer graph convolutional network is used to model the relationships between instance features. After the instance features are propagated through the three layers, a graph feature representation containing instance relationship information is obtained. Through the above modeling, the graph structures of the instance features output by the teacher and student models can be obtained respectively. This relationship modeling can be expressed by formula (23):
[0184] ,(twenty three)
[0185] in, , Represents the convolution weights of the l-th layer graph. This represents a non-linear activation function.
[0186] (2) Construct contrastive learning losses based on the graph structure of the teacher model and the graph structure of the student model respectively;
[0187] It should be noted that graph convolutional propagation alone is insufficient to constrain the feature distribution of instances at the spatial structure level. Therefore, this embodiment of the invention further introduces graph contrastive learning loss for constraint, the specific implementation details of which are as follows:
[0188] First, based on the similarity matrix between instances Normalize it and set a threshold to generate a positive sample mask, which can be represented by formula (24):
[0189] ,(twenty four)
[0190] in, This indicates whether instances i and j are considered a positive sample pair in the graph structure. This represents the normalized similarity matrix. This is the similarity threshold between instances.
[0191] Secondly, the instance feature vectors of the teacher or student model after graph convolution are passed through a non-linear projection head. Mapping to the contrastive learning embedding space and normalizing it yields a unit vector. .
[0192] Finally, a contrastive learning loss based on the form of InfoNCE is constructed. This loss encourages instances that are closely related in the graph structure to remain close in the embedding space, while widening the distance between weakly related instances, thereby improving the stability of the feature structure in an unsupervised manner. This loss can be expressed by Equation (25):
[0193] (25)
[0194] in, This represents the temperature coefficient, which can be set to 0.07 for scaling the similarity.
[0195] (3) Distillation constraints are introduced into both the graph structure of the teacher model and the graph structure of the student model to obtain the graph comparison learning module.
[0196] To prevent the destruction of original category information during graph convolutional inference, this invention introduces a semantic distillation constraint. The specific implementation of this constraint is as follows:
[0197] First, the instance features after graph convolution are input into the classification head to obtain predicted logits.
[0198] Secondly, the Kullback–Leibler divergence constraint ensures consistency with the original classification output. This constraint guarantees that the graph structure modeling process enhances relational information without reducing the original category semantic information. This constraint can be expressed by formula (26):
[0199] (26)
[0200] in, and These represent the classification logits for the student model and the teacher model, respectively.
[0201] Therefore, by constructing a graph contrast learning module, the embodiments of the present invention can extend the self-training process from traditional instance-level supervision to feature structure-level constraints.
[0202] In this embodiment of the invention, the initial model for underwater passive cross-domain target detection guided by vision and language models is iteratively trained based on the target domain dataset, such as... Figure 7 As shown, it includes:
[0203] S510. Perform a first enhancement preprocessing and a second enhancement preprocessing on the target domain dataset to obtain a first enhancement preprocessed dataset and a second enhancement preprocessed dataset, wherein the enhancement magnitude of the first enhancement preprocessing is smaller than the enhancement magnitude of the second enhancement preprocessing.
[0204] Specifically, the first enhancement preprocessing may include weak enhancement processing such as random horizontal flipping, and the second enhancement preprocessing may include strong enhancement processing such as color dithering, grayscale transformation and Gaussian blur.
[0205] S520. Input the first enhanced preprocessing dataset into the teacher model and the pseudo-label generation module respectively to obtain the pseudo-label set of the teacher model, and input the second enhanced preprocessing dataset into the student model to obtain the prediction result of the student model;
[0206] Specifically, the weakly enhanced target domain image is input into the teacher model and the visual and language teacher model (Grounding DINO), and the set of text prompt words is also input into the visual and language teacher model. Simultaneously, the strongly enhanced target domain image is input into the student model.
[0207] S530. Calculate the training iteration loss based on the supervision signal of the pseudo-label set of the teacher model for the prediction results of the student model;
[0208] Specifically, the pseudo-labels predicted by the teacher model are filtered using a thresholding strategy based on a two-component Gaussian mixture model to obtain the pseudo-label set. The pseudo-labels predicted by the visual-language teacher model are denoted as the pseudo-label set. Subsequently, a weighted box fusion strategy was used to merge the two into a set of pseudo-labels. Simultaneously, the results predicted by the student model are obtained based on the pseudo-label set. The supervisory signal is used to calculate the loss of the result. .
[0209] Furthermore, instance relationship graphs are constructed for both the teacher and student models, and graph structure relationships are modeled based on graph convolutional neural networks. The modeled instance feature vectors are then used to calculate the contrastive learning loss based on the InfoNCE form. Meanwhile, to preserve the category information of this feature, semantic distillation loss is introduced. After the above process, the total loss of the model training is obtained. The gradient of this loss at each layer of the model is calculated using the backpropagation algorithm, and then the parameters of the student model are updated using the AdamW optimizer. The total loss can be expressed by formula (27):
[0210] (27)
[0211] in, This represents the balance coefficient.
[0212] S540. The initial model for underwater passive cross-domain target detection guided by the vision and language model is iteratively trained according to the training iteration loss.
[0213] Finally, the parameters of the teacher model are adaptively updated using the gated EMA strategy of the adaptive weight update module. The updated teacher model iteratively generates higher-quality pseudo-labels. Subsequently, the model is continuously upgraded through multiple rounds of training to achieve better cross-domain detection performance. The overall framework of the model is as follows: Figure 8 As shown.
[0214] In summary, the underwater passive cross-domain target detection method based on visual and language models provided by this invention introduces a visual and language-enhanced pseudo-label generation module, a weight adaptive update module, and a graph contrastive learning module into the initial model of underwater passive cross-domain target detection based on visual and language models. The visual and language-enhanced pseudo-label generation module further introduces the visual and language foundational model Grounding DINO, which serves as a visual-language teacher model to provide new pseudo-label guidance. Fusing it with the pseudo-labels generated by the teacher model can produce higher-quality pseudo-labels. As a visual foundational model, it contains rich, unbiased semantic features, effectively alleviating the source domain bias problem of pseudo-labels from the teacher model. The weight adaptive update module dynamically selects the strategy for updating teacher parameters based on the change in uncertainty of the student model for cached samples. This strategy effectively alleviates the noise accumulation and training degradation problems caused by the fixed EMA update method. The graph contrastive learning module captures the spatial structural connections between instances through instance relationship graph construction and graph convolutional neural network construction, and uses graph contrastive learning loss and distillation loss to bring these relationships closer while maintaining consistency in category information. This passive underwater cross-domain target detection method, guided by vision and language models, has achieved good detection results in various complex aquatic environments. Experimental results show that the model of this invention has high average accuracy and good generalization ability in underwater cross-domain scenarios, and can significantly improve the accuracy and stability of target detection.
[0215] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of the present invention, and the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.
Claims
1. A passive underwater cross-domain target detection method guided by vision and language models, characterized in that, include: A source domain dataset is constructed based on publicly available and labeled datasets for underwater target detection, and a target domain dataset is constructed based on unlabeled data from actual underwater application scenarios. The target domain dataset includes at least the data of underwater targets to be detected. Construct a baseline model based on the Transformer architecture, and train the baseline model to obtain pre-trained weights based on the source domain dataset; The teacher and student models are constructed based on the baseline model containing pre-trained weights to obtain the teacher-student model framework. Under the framework of the teacher-student model, a pseudo-label generation module with visual and language enhancement is constructed to obtain an initial model for underwater passive cross-domain target detection guided by visual and language models. The pseudo-label generation module with visual and language enhancement is used to perform semantic completion and localization correction on the pseudo-labels generated by the teacher model based on the target domain dataset according to the cross-modal semantic prior of the visual and language model. The initial underwater passive cross-domain target detection model guided by vision and language models is iteratively trained based on the target domain dataset to obtain an underwater passive cross-domain target detection model guided by vision and language models. The underwater passive cross-domain target detection model guided by vision and language models can detect underwater target data and obtain underwater target data detection results.
2. The underwater passive cross-domain target detection method based on vision and language models as described in claim 1, characterized in that, Within the aforementioned teacher-student model framework, a pseudo-label generation module with visual and language enhancements is constructed, including: Initial pseudo-labels are generated based on the teacher model; Generate semantically enhanced pseudo-labels based on a visual and language teacher model; The initial pseudo-labels and the semantically enhanced pseudo-labels are fused to obtain a visual and linguistic enhancement pseudo-label module for supervising student model training.
3. The underwater passive cross-domain target detection method based on vision and language models as described in claim 2, characterized in that, Initial pseudo-labels are generated based on the teacher model, including: The target domain dataset is input into the teacher model to obtain candidate box prediction results; For each candidate box prediction result, the maximum class confidence score and the class corresponding to that maximum class confidence score are selected. Based on the category adaptive thresholding strategy of Gaussian mixture model, a set of historical predictions is collected for each target category in the target domain dataset; The historical prediction set for each target category is modeled using a two-component Gaussian mixture model to obtain the fitting results of the two-component Gaussian mixture model. Based on the fitting results of the two-component Gaussian mixture model, the category threshold is estimated from the high-confidence component, and the initial pseudo-label set of the teacher model after threshold screening is obtained.
4. The underwater passive cross-domain target detection method based on vision and language models as described in claim 2, characterized in that, Generate semantically enhanced pseudo-labels based on a visual and language teacher model, including: For each target category in the target domain dataset, a corresponding text prompt word is constructed, and a target category text prompt word set is obtained; The target domain dataset and the target category text prompt word set are input together into the visual and language teacher model to obtain the visual and language candidate set; Based on target category matching and confidence threshold, the visual and language candidate sets are filtered to obtain semantically enhanced pseudo-labels.
5. The underwater passive cross-domain target detection method based on vision and language models as described in claim 2, characterized in that, The initial pseudo-labels and the semantically enhanced pseudo-labels are fused to obtain a visual and language enhancement pseudo-label module for supervising student model training, including: The candidate set is obtained by merging the pseudo-label sets of the initial pseudo-label and the semantically enhanced pseudo-label. The confidence scores and category information in the candidate set are fused to obtain fused confidence scores and fused category information; Based on the candidate set, the fused confidence level, and the fused category information, a pseudo-label module for visual and language enhancement used to supervise student model training is obtained.
6. The underwater passive cross-domain target detection method based on vision and language model guidance according to any one of claims 1 to 5, characterized in that, The underwater passive cross-domain target detection method based on vision and language models also includes: A weight adaptive update module is constructed within the teacher-student model framework. This module is used to adaptively select the parameter update strategy for the teacher model.
7. The underwater passive cross-domain target detection method based on vision and language models as described in claim 6, characterized in that, Within the aforementioned teacher-student model framework, a weight adaptive update module is constructed, including: The target domain dataset is input into the student model to obtain intermediate prediction results, and an entropy-based uncertainty is constructed based on the intermediate prediction results. The initial uncertainty of the samples in the target domain dataset obtained in the teacher model is stored in a buffer queue, wherein the number of uncertainties in the buffer queue is not greater than a preset number threshold. The initial uncertainty in the buffer queue is evaluated based on the parameters after the student model is trained to obtain the updated uncertainty; The parameters of the teacher model are adaptively updated based on the difference between the updated uncertainty and the initial uncertainty.
8. The underwater passive cross-domain target detection method based on vision and language model guidance according to any one of claims 1 to 5, characterized in that, The underwater passive cross-domain target detection method based on vision and language models also includes: A graph comparison learning module is constructed within the framework of the teacher-student model. The graph comparison learning module is used to construct the instance space structural relationship between the teacher model and the student model.
9. The underwater passive cross-domain target detection method based on vision and language models as described in claim 8, characterized in that, A graph contrastive learning module is constructed within the aforementioned teacher-student model framework, including: Instance relationship modeling and graph convolution modeling are performed based on the instance features of the teacher model and student model respectively to obtain the graph structure of the teacher model and the graph structure of the student model. Construct contrastive learning losses based on the graph structure of the teacher model and the graph structure of the student model, respectively. Distillation constraints are introduced into the graph structures of both the teacher model and the student model to obtain a graph comparison learning module.
10. The underwater passive cross-domain target detection method based on vision and language model guidance according to any one of claims 1 to 5, characterized in that, The initial underwater passive cross-domain target detection model guided by vision and language models is iteratively trained based on the target domain dataset, including: The target domain dataset is subjected to a first enhancement preprocessing and a second enhancement preprocessing respectively to obtain a first enhancement preprocessed dataset and a second enhancement preprocessed dataset, wherein the enhancement magnitude of the first enhancement preprocessing is smaller than the enhancement magnitude of the second enhancement preprocessing. The first enhanced preprocessed dataset is input into the teacher model and the pseudo-label generation module respectively to obtain the pseudo-label set of the teacher model, and the second enhanced preprocessed dataset is input into the student model to obtain the prediction result of the student model; The training iteration loss is calculated based on the supervision signal of the pseudo-label set of the teacher model to predict the results of the student model. The initial underwater passive cross-domain target detection model guided by the vision and language models is iteratively trained based on the training iteration loss.